Forem: adaboese

AI Content is Not Short-Term Arbitrage

adaboese — Thu, 08 Feb 2024 14:34:45 +0000

Whether you like it or not, AI is going to change the way we create content.

Recently, there has been a flood of naysayers who claim that AI-generated content is short-term arbitrage. Like this article:

https://ahrefs.com/blog/ai-content-is-short-term-arbitrage/

I have a lot of respect for Rayan, but he is taking a too narrow view of the potential of AI-generated content.

Content was never the moat

First, let's address the Elephant in the room: your writing style is not a moat. Even with today's capabilities, given sufficient amount of direction, AI can mimic your writing style to a T. The moat has always been the data and unique insights. The content is just a way to get the data and insights to the users. This will remain true even when AI-generated content becomes the norm.

The real-question is whether AI-generated content can produce unique insights and data. The answer is yes, but not yet. We are getting there.

Short-term arbitrage

It is hard to talk about AI content without mentioning the recent "penalization" of websites that used AI-generated content. The "SEO heist" is one that got the most attention for both the rapid rise and fall of the website. So let's talk about it.

Every single instance of websites getting "penalized" for use of AI content happened in the context of outputting thousands of articles, getting a temporary boost, and then sinking. This comes as no surprise. Google publicly disclosed that they use multi-model evaluation algos to establish how to rank content. The first stage is the naive/cheap algo that basically gives any website the benefit of doubt. I suppose at the time they never expected anyone to be adding thousands of articles... well those thousands of articles are getting the benefit of doubt across many keywords. However, no surprise that once we enter verification stage (which includes broader spectrum of variables, including bounce rate, time on site, backlinks and whatnot) the websites tank. So... don't do that. Build organically over time.

Related, contrary to what everyone on Reddit will want you to think, no one at Google is manually penalizing your website just because your tweet goes viral talking about how you came to top using AI. And if they do, you will be notified of it through Google Search Console Manual Action report, but the reason won't be "because you've used AI".

Just based on common sense, Google wants only 3 things out of your-content:

factually accurate content
unique insights or new data points
recognized by users to be valuable [based on engagement, backlinks, etc]

Achieve these 3 things and it won't matter whether you use USD 100/hour copywriter, USD 15/hour copywriter, or AI.

The state of AI content generation

Most "AI content generators" are not yet where they would tick all 3 boxes, esp. if we compare to a good human copywriter. However, even if we are not there yet, we will be there very soon. Like, within 12 months at most. Not because of advancements in Large Language Models (LLMs) to be clear. I think LLMs are already there. What needs to catch up is orchestration of content generation.

LLMs are already very capable of text analyzes and even making logical conclusions based on provided data. They are good at narrative telling. They are good at spotting logical inconsistencies, etc.

What they are not good at is doing all of this in one-go. The statistical model just doesn't have the capability to think this far ahead or correct itself after output has been sent to users. This is why Retrieval-Augmented Generation (RAG) and multi-stage output generation is necessary to produce something that resembles human-level output.

Just to give a high-level idea, AIMD uses 20+ different APIs to:

research the topic
identify the most important data points about the topic
aggregate supporting data
validate the data

All of this work is done before even starting to write the outline of the article. These APIs include Google Search Console, "people also ask", keyword research tools, and many more.

I am intentionally being vague about the APIs used beyond the obvious ones, because I don't want to give away the secret sauce.

And then when it starts writing, it keeps fact checking everything that is written in the context of the greater article. And then when it is done, it goes through each section again looking for logical inconsistencies, style inconsistencies, etc.

The biggest downside of this approach is that:

it takes a long time to generate article
costs a lot more
a lot more unstable (many things tend to break when talking with so many services)

But all of 3 of those problems are solvable. So it is just a matter of time.

How to spot bad AI content generator?

I currated a list of AI content generators that are currently available. I have 25 on the list at the moment, and I keep a close eye on all of them.

Without even looking at the output, the biggest tell-tale sign of a bad AI content generator are:

real-time output - if the output is streamed
cheap price – if you are paying less than ~1 USD per article
fast output – if the output is generated in a few minutes

If either of these 3 are true, then you are likely just paying for a ChatGPT wrapper. You get what you pay for.

Not surprisingly, the content produced using zero-shot approach is going to include logical inconsistencies, style inconsistencies, and factual inaccuracies. It is not going to be able to reference its sources or make compelling arguments.

Just as a benchmark, I am working really hard to get AIMD to produce articles in under 10 minutes. And I am not there yet. The data queries alone take a good chunk of time. The first draft is usually reduced by good ~40-60% just for the sake of fact checking and logical consistency. And then it goes through few more iterations of fact checking and logical consistency checks.

To be clear, I am not making a claim that https://aimd.app is there yet. I believe AIMD is the most advanced AI content generator out there, but even then you would be shooting yourself in the foot if you were to use its output without human editorial. However, I am confident that we will be there within 12 months and that then the moat will be the unique data and perspectives that you can provide to the AI, and not the "packaging" of the content.

Using Vector Embeddings to Overengineer 404 pages

adaboese — Wed, 17 Jan 2024 15:55:30 +0000

After spending a significant amount of time working with vector embeddings, I've started to see more and more use cases for them for every day problems. One of the most interesting ones I've seen is using vector embeddings to find the page that the user was looking for when they hit a 404 page.

What is a vector embedding?

A vector embedding is a way to represent a word or phrase as a vector. This is useful because it allows us to do math on words and phrases. For example, we can find the word that is closest to another word by finding the word with the smallest distance between the two vectors.

Financial Times has a great interactive article that explains vector embeddings in more detail.

How can we use vector embeddings to find the page that the user was looking for?

We can use vector embeddings to find the page that the user was looking for by finding the page with the smallest distance between the vector of the page and the vector of the user's query. In the context of a 404 page, user's query is the URL that they were trying to access.

It is suprisingly simple:

we need to create a database of all the pages on our site
we need to create a vector embedding for each page URL
we need to create a vector embedding for the user's query
we need to find the page with the smallest distance between the vector of the page and the vector of the user's query

In case of AIMD, I am doing this all in-memory, but you could also do this in a database (e.g. Pinecone). It all depends on how much data you have and how much compute you have available.

Deciding on a vector embedding model

The first step is to decide on a vector embedding model. I am using Supabase/gte-small because it is small model and it outperforms OpenAI's text-embedding-ada-002 model.

I wrote this abstraction that creates a vector embedding for a given text:

import { pipeline } from '@xenova/transformers';

export const generateEmbedding = async (subject: string): Promise<number[]> => {
  const generateEmbedding = await pipeline(
    'feature-extraction',
    'Supabase/gte-small',
  );

  const result = await generateEmbedding(subject, {
    normalize: true,
    pooling: 'mean',
  });

  if (result.type === 'float32') {
    return Array.from(result.data) as number[];
  }

  throw new Error('Expected embedding type to be float32');
};

Creating a database of all the pages on our site

The next step is to create a database of all the pages on our site.

Let's assume that we have an array of all the pages on our site:

type SitemapEntry = {
  loc: string;
};

const staticPages: SitemapEntry[] = [
  {
    loc: 'https://aimd.app/',
  },
  {
    loc: 'https://aimd.app/blog',
  },
  {
    loc: 'https://aimd.app/blog/2024-01-15-top-seo-trends-for-2024-what-should-you-focus-on',
  },
  {
    loc: 'https://aimd.app/blog/2024-01-07-maximizing-article-visibility-understanding-and-applying-e-e-a-t-in-seo',
  },
  // ...
];

We can then create a database of all the pages on our site by creating a vector embedding for each page URL:

type Metadata = Record<string, string | number | boolean>;

type DatabaseEntry = {
  metadata: Metadata;
  vector: number[];
};

const entries: DatabaseEntry[] = [];

for (const page of staticPages) {
  const vector = await generateEmbedding(page.loc);

  entries.push({
    metadata: {
      url: page.loc,
    },
    vector,
  });
}

entries is now a database of all the pages on our site.

Finding the page with the smallest distance between the vector of the page and the vector of the user's query

The last step is to find the page with the smallest distance between the vector of the page and the vector of the user's query.

Let's assume that we have a user's query:

const query = 'https://aimd.app/blog/2024-01-17-using-vector-embeddings-to-overengineer-404-pages';

First, we need to create a vector embedding for the user's query:

const queryVector = await generateEmbedding(query);

Then, we need a way to calculate a distance between two vectors. For this, we can use cosine similarity:

import similarity from 'compute-cosine-similarity';

Finally, we can find the page with the smallest distance between the vector of the page and the vector of the user's query:

const closestEntry = entries.reduce((closestEntry, entry) => {
  const distance = similarity(queryVector, entry.vector);

  if (distance > closestEntry.distance) {
    return {
      distance,
      entry,
    };
  }

  return closestEntry;
}, {
  distance: -Infinity,
  entry: null,
});

closestEntry.entry is now the page that has the most similar URL to the page the user was similar.

The best part is that this does not even need to be the exact page that the user was looking for, e.g. in case the page was removed. It will be whichever page has the most similar URL to the page the user was looking for.

Using with Remix

Just to complete the example, here is how you would use this with Remix:

// app/routes/$.tsx

import { type MetaFunction } from '@remix-run/node';
import { Link, useLoaderData } from '@remix-run/react';
import { json, type LoaderFunctionArgs } from '@remix-run/server-runtime';
import { findNearestUrl } from '#app/services/sitemap.server';

export const meta: MetaFunction = () => {
  return [
    {
      title: '404',
    },
  ];
};

export const loader = async ({ request }: LoaderFunctionArgs) => {
  const nearestUrl = await findNearestUrl(request.url);

  return json({
    nearestUrl,
  });
};

const Route = () => {
  const data = useLoaderData<typeof loader>();

  return (
    <div>
      <h1>404</h1>
       <p>Were you looking for <Link to={data.nearestUrl.url}>{data.nearestUrl.url}</Link>?</p>
    </div>
  );
};

export default Route;

Now your 404 page will suggest the page that the user was most likely looking for.

Examples of this in the wild

Here is how that looks once deployed:

Even though all of these pages do not exist, they all produce a 404 page that links to the correct page.

In practice, this type of hint will be most useful for pages that were removed or renamed, e.g. I have accidentally introduced numerous 404s on this site by changing the dates of the posts.

Do we even need 404 pages?

This is a bit of a tangent, but I think it is worth mentioning that we might not even need 404 pages. Instead, we could just redirect the user to the page that they were looking for.

Realistically, the only reason we have 404 pages is because we don't know what the user was looking for. But if we can use vector embeddings to find the page that the user was looking for, then we can just redirect them to that page.

I will be experimenting with this on AIMD in the future.