Forem: Charlie Graham

Counting tokens is dumb. So we built a free metric for AI proficiency.

Charlie Graham — Thu, 21 May 2026 21:07:15 +0000

We’ve been trying to figure out a real answer to a question that keeps coming up: how do you measure whether someone is actually good at Claude Code, Codex, and the other AI coding tools? Not "do they use them," but how good are they at using AI.

The first metric we looked at, like everyone else, was token usage. It’s the only number you can pull out of the box. Anthropic and OpenAI hand you token data in the console. So token usage becomes an easy first answer.

But obviously counting tokens sucks as a metric.

What we noticed when we looked at the actual sessions

When we started reading session logs from people who were clearly good with these tools, and people who were clearly struggling, both groups burned tokens. Sometimes the strugglers burned more.
A senior developer who has refined their workflow ships in 1 million tokens what a junior chews through 10 million on. The high-skill move is fewer turns, sharper prompts, smaller context windows, more planning up front.

Rank by token spend and you end up rewarding the things that make people slower:

Padded context with files that aren’t relevant
Brute-forcing with longer and longer prompts
Staying in chat mode forever instead of building reusable workflows

On top of all this, once a company starts measuring token usage, the incentive flips from accidental to deliberate. If your performance review or “AI adoption KPI” depends on token counts, the rational move is to burn tokens on purpose. We’ve already heard about people writing scripts that loop the model on busywork just to pump their number.

We’ve seen the same critique made about lines of code and commit counts. Volume isn’t skill. It just looks like it on a dashboard.

So we tried looking at something else

Instead, we started watching for things you can read from local session activity that show how someone configured the tool, not how much they spent on it.

Eight things kept clustering together. People with two of them usually had four. People with five usually had close to all of them.

Customization — CLAUDE.md, AGENTS.md, custom slash commands, hooks. How much did they shape the tool to their workflow, vs run defaults?
Parallel Agents — Are they using multiple agents working at once, or one chat at a time?
Background Work — Tasks delegated to run unattended, or babysitting every turn?
Tool Breadth — To what degree do they have MCP servers, skills, plugins wired into the environment?
Planning — Plan mode, structured /spec / /plan workflows, or jumping straight to file edits?
Repetition — Skill breadth and skill depth, measured separately. A lot of people install skills they never actually use.
Custom Skills — Written their own reusable workflows for things they do more than once?
Multi-Tasking — AI treated as a team running in parallel, or as a single chat window?

Plus a few tool-specific ones we added later for Codex and Cowork.
Each of these is observable from session activity. No self-report, no interview answer to game.

So we built a free metric

We turned the framework into AIQ Rank. AIQ Rank reads local session activity from whatever AI coding tools you’re using (Claude Code, Codex, Cursor, OpenCode, Cowork) and scores you 0-1000 across the eleven dimensions. Think of it as a credit score, but for AI fluency.

We made it free. It runs locally — transcripts never leave your machine. You get a number, a per-dimension breakdown, and a profile URL to share if you want to.

The score is the hook. The breakdown is the part that’s actually useful. When we ran it on ourselves the first time, the dimension that surprised us wasn’t a strength we expected. It was a weakness we hadn’t noticed.

What we’d suggest doing with it

If you ran a quick sanity check on your team - top 10% by token spend, top 10% by AIQ Rank - we suspect the overlap would be smaller than you’d expect. Some of the token-heavy people are still brute-forcing every problem in chat mode. Some of the lower-spend people quietly built skills, wired up MCPs, learned plan mode, and run parallel agents.

That gap is the interesting part. Not because token spend is bad data, but because it’s a different question than “who’s good at this.”

If you want to do this comparison across your own team, AIQ Rank has private team leaderboards. Invite people and see a leaderboard for just your team including the skills used between them. Transcripts still never leave anyone’s machine.

What do you think?

AIQRank is still a work in progress and we care constantly improving it. Do you agree/disagree with these parameters? Let us know what changes/improvements we should make and please try it out!

Again you can get your score for free in 60 seconds from aiqrank.com. We’d love any constructive feedback!

7 Tools to Track AI Mentions in LLMs in 2025

Charlie Graham — Tue, 12 Aug 2025 16:55:45 +0000

You’ve built a powerful API, launched an open-source library, or founded a SaaS company. You've followed the playbook: written great docs, published tutorials, and engaged with the community. In the past, you'd track your success with web analytics, SERP rankings, and social media mentions. But the ground has shifted. Now, your potential users aren't just Googling—they're asking ChatGPT, Claude, and Perplexity for recommendations.

How do you know if your brand, your product, or your code is being recommended in these new conversational search engines? Welcome to the new frontier of brand visibility. Tracking mentions within Large Language Models (LLMs) is a complex but critical task for any modern developer or tech company. This guide explores the problem and presents the top 7 tools and techniques to help you track AI mentions and manage your brand's presence in this new ecosystem.

Quick Answer: How Can I Track Brand Mentions in AI?

Tracking brand mentions in AI requires a multi-faceted approach because LLM responses are not publicly indexed like web pages. The best methods include:

DIY Scripting: Use Python and official LLM APIs like OpenAI and Anthropic to build your own custom tracking solution for maximum control.

Specialized AI Monitoring Platforms: Use tools like RivalSee, Otterly, or Athena HQ specifically built to query public LLMs at scale and analyze your brand's mention frequency and sentiment.

Enterprise AI Monitoring: For a broader view, enterprise platforms like Profound and Scrunch AI analyze vast sets of conversational data (from customer interactions to market trends) to provide high-level brand and market intelligence.

Why Is Tracking AI Mentions So Hard?

Before we jump into the solutions, it’s essential to understand why this is a fundamentally different challenge than traditional SEO or brand monitoring.

The "Black Box" Problem: LLM responses are generated in real-time within a closed environment. Unlike a website, there's no public URL to crawl or index. The output is ephemeral.
Non-Deterministic Output: Ask the same question twice, and you might get two different answers. An LLM's response can vary based on the conversation's history, model updates, and inherent randomness, making consistent tracking difficult.
Data Source Obscurity: LLMs are trained on vast datasets from the public internet (like Common Crawl), books, and licensed data. Pinpointing the exact source that led to a specific mention is nearly impossible, making it hard to double down on what works.
Context is Everything: A simple keyword search isn't enough. You need to understand the context. Was your API mentioned as a top solution, a buggy alternative, or a historical footnote? This requires sophisticated analysis beyond simple string matching.

These challenges mean that old tools and techniques are often insufficient. You need a new stack for a new era of search. This is where AI mention tracking tools come in.

Top 7 Tools to Monitor Your Brand in LLMs

Here’s a developer-focused breakdown of the best ways to track brand visibility in large language models, from powerful dedicated platforms to hands-on DIY methods.

1. DIY Python Scripts: The Hands-On Developer Approach

Best for: Developers who want maximum control, have a limited budget, and enjoy building their own tools.

Before looking at paid platforms, many developers will instinctively ask: "Can I build this myself?" The answer is yes. This approach gives you complete flexibility to design queries and analyze results exactly how you want. The core idea is to use the official APIs from providers like OpenAI and Anthropic to ask questions and parse the responses for mentions of your brand.

Here's a conceptual Python snippet using the openai library:

import os
import openai

# It's best practice to use environment variables for your API key
# from openai import OpenAI
# client = OpenAI()
# client.api_key = os.getenv("OPENAI_API_KEY")

openai.api_key = os.environ.get("OPENAI_API_KEY")

def check_ai_mention(brand_name, query):
    """
    Queries GPT to see if a brand is mentioned in the response.
    """
    try:
        response = openai.chat.completions.create(
            model="gpt-4-turbo", # Or another model of your choice
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": query}
            ],
            temperature=0.7,
            max_tokens=250
        )

        response_text = response.choices[0].message.content
        print(f"--- Query: '{query}' ---")
        print(f"Response: {response_text}\n")

        if brand_name.lower() in response_text.lower():
            return True, response_text
        else:
            return False, response_text

    except Exception as e:
        print(f"An error occurred: {e}")
        return False, str(e)

# --- Example Usage ---
my_brand = "PostHog" # Let's track an open-source tool
queries_to_run = [
    f"What are the best open-source alternatives to Google Analytics?",
    f"How can I implement product analytics in my web app?",
    f"Compare PostHog vs. Mixpanel."
]

for q in queries_to_run:
    mentioned, text = check_ai_mention(my_brand, q)
    if mentioned:
        print(f"✅ SUCCESS: '{my_brand}' was mentioned!\n")
    else:
        print(f"❌ FAILED: '{my_brand}' was not mentioned.\n")

Challenges with the DIY Approach:

Cost & Scale: API calls, especially to advanced models, get expensive quickly. Running thousands of queries to get statistically significant data is often cost-prohibitive for individuals.
Complexity: Managing different prompts, personas, and API keys for various LLM providers adds significant engineering overhead.
Analysis: You'll need to build your own system for storing, parsing, and visualizing the results over time to spot trends.

2. RivalSee: For Precision AI Search Visibility

URL: https://rivalsee.com

Best for: Developers, marketers, and businesses who need to specifically track and improve their mention rate within public AI chatbot responses.

While the DIY method is tempting, a specialized platform automates the hard parts. When your goal is to understand how often and in what context your brand appears in answers from models like ChatGPT, Claude, and Google's AI, a dedicated tool is more efficient.

RivalSee is a purpose-built AI search visibility platform designed to solve this exact problem. It moves beyond simple scripting to provide a robust framework for testing and analysis. For developers, its power lies in its systematic and data-driven approach. You can simulate real-world user queries at scale, such as "What's the best Python library for data visualization?" or "Compare Stripe vs. Adyen for international payments."

Key Capabilities for Developers:

Multi-LLM Tracking: It queries ChatGPT, Claude, Perplexity, Google AI, and others simultaneously, giving you a holistic view of your brand's presence.
Persona-Driven Simulation: You can define personas (e.g., 'Senior Backend Engineer', 'Indie Hacker', 'Non-technical Founder') to ask questions from different user perspectives, revealing how your brand is perceived by various audience segments.
Competitive Analysis: See not only how often you’re mentioned but also how your mention rate stacks up against direct competitors for critical queries. This is invaluable for understanding your "share of voice" in the AI space.
Actionable Insights: The platform provides recommendations to help you improve your rankings. This bridges the gap between tracking and strategy, helping you figure out how to get brand mentioned in AI content more effectively.

3. Otterly AI: For Direct Chat Tracking

URL: https://www.otterly.ai

Best for: Startups and teams looking for a straightforward tool to track mentions in AI chat, but who should be mindful of pricing tiers.

Otterly AI is another player in the direct-to-LLM tracking space. It offers a clean interface for setting up keywords and tracking their appearance in AI conversations. Its focus is on providing a direct feedback loop on your AIO (AI Optimization) efforts.

It's a solid solution for those who want a managed platform instead of a DIY script. However, it's important to analyze the pricing structure. While it may offer a low-cost entry point, sources suggest the cost can increase significantly if you need to track more than a handful of prompts, so it's crucial to model your expected usage before committing.

4. Athena HQ: The AI Optimization (AIO) Platform

URL: https://www.athenahq.ai

Best for: Digital marketing agencies and in-house teams who want to manage AIO as a new marketing channel.

Athena HQ positions itself as a comprehensive AIO platform, combining tracking with optimization workflows. The goal is to treat your presence in AI answers as a formal marketing channel, much like SEO or PPC.

It provides tools to discover what questions users are asking in your niche, track your brand's current visibility for those questions, and measure the impact of your content changes on your mention rate. This makes it a good fit for teams that are ready to operationalize their efforts to track brand visibility in large language models.

5. Profound: For Customer Conversation Intelligence

URL: https://www.getprofound.com

Best for: Enterprise product and marketing teams needing to analyze brand perception and market trends from their own customer conversation data.

The next two tools shift focus from public LLMs to enterprise-controlled data. Profound is an AI-powered market intelligence platform that plugs into your company's "voice of the customer" data—sources like sales calls on Gong, support tickets in Zendesk, and survey results.

It uses AI to analyze these conversations at scale, delivering insights on how customers perceive your brand, what features they're requesting, and how you stack up against competitors in their minds. While it doesn't track ChatGPT, it offers a crucial form of AI mention analysis by revealing the ground truth of your brand reputation directly from your users.

6. Scrunch AI: For Enterprise AI Monitoring

URL: https://scrunchai.com

Best for: Large organizations that need to monitor and analyze a wide array of AI-driven interactions for high-level brand intelligence and compliance.

Similar to Profound, Scrunch AI operates at the enterprise level, offering a broad monitoring solution. It is designed to give large companies a unified view of their brand's presence across diverse digital ecosystems where AI plays a role.

Instead of focusing only on customer support tickets or public LLMs, it provides a wider lens for AI enterprise monitoring. This can include analyzing trends, ensuring brand consistency across various automated touchpoints, and providing a strategic overview of brand health in an increasingly AI-saturated market. For large-scale brand reputation management in AI chatbots and automated systems, a comprehensive solution like this is essential.

7. Mentions.so: Modern & Lean Brand Monitoring

URL: https://mentions.so

Best for: Startups and small businesses looking for a clean, simple, and affordable brand monitoring tool for the web and social media.

Rounding out the list is Mentions.so, a more modern and streamlined take on brand monitoring. It focuses on tracking your brand across key social and web platforms without the complexity and cost of enterprise suites. For a developer launching a new SaaS, it can be a great way to keep an eye on discussions on Reddit, X (Twitter), and Hacker News.

While not a direct AI mentions tracker, this is critically important because these public conversations are the very data used to train future LLMs. A strong, positive presence on these platforms today is the best way to influence how your brand is mentioned by the AI of tomorrow.

From Tracking to Influencing: A Long-Term Strategy

Simply tracking mentions is only half the battle. The ultimate goal is to improve the frequency and quality of those mentions. After using an AI mention tracking tool to establish a baseline, your focus should shift to a content and distribution strategy designed for AI consumption.

Create Authoritative Source Material: Write comprehensive guides, in-depth tutorials, and well-structured API documentation. LLMs prioritize clear, factual, and well-organized content.
Foster Third-Party Validation: Encourage reviews and comparisons on trusted, independent sites and forums. A positive review on a high-authority blog is a powerful signal.
Structure Your Data: Use schema markup on your website to help search engines (and by extension, their AI components) understand what your company does, who your products are for, and what problems they solve.
Analyze and Iterate: Use a platform like RivalSee to correlate your content efforts with changes in your AI mention rate. Did that new API tutorial series boost your visibility for "best payment processing API" queries? This data-driven feedback loop is what separates guessing from a true AI optimization strategy.

Conclusion

The rise of conversational AI has created a new, vital channel for brand discovery. Ignoring it means ceding ground to your competitors. For developers and tech brands, understanding and optimizing for this channel is no longer optional—it’s a core component of modern growth.

Getting started is a matter of choosing the right tool for your scale and goals. The DIY approach offers control, while specialized platforms like RivalSee and Otterly provide efficiency and analytics for tracking public LLMs. For a higher-level view, enterprise solutions like Profound and Scrunch AI analyze brand perception within controlled data sets.

By combining these tools with a strategy focused on creating high-quality, authoritative content, you can move from being a passive observer to an active participant in shaping how your brand is seen and recommended by the AI assistants of today and tomorrow.

Vibe-Coded sites are bad at SEO – and a way to fix it

Charlie Graham — Thu, 31 Jul 2025 17:50:46 +0000

Vibe-Coded site-builders are pretty bad at SEO.

AI coding assistants like Lovable, Bolt, Replit, Cursor, etc, are all optimizing for speed to develop and speed to deploy. Unfortunately, SEO is treated like a second-class citizen. They basically ignore it.

The result is a (usually) working site that is not seen in OpenAI's ChatGPT, Perplexity, Claude, and Google AI, not to mention Google and Bing.

The big problems:

No focus on Headers and Meta Tags: Almost all of these AI-generating sites forget to do the SEO basics like H1/H2 tag hierarchy, and setting up both meta and share tags so that your content is sharable and descriptions and titles are easy to read. To have better SEO, you need to make sure you have correct and differentiated title/meta tags and good H1/H2 header hierarchy.
Client-side Rendering. This is a big one. The vibe coding sites are using API calls (like Supabase) and React to render content on the client post HTML load. The problem is all the AI chat engines only read HTML when they search - so NONE of that content will be checked by the chats when they search your site. You need to make sure that your text content is rendered on the server side when the HTML is loaded. This is the hardest to change but super important to appear in AI chats.
Omission of LD-JSON: AI-generated sites do not include structured JSON which search engines and AI chats use to get better context.

To help, RivalSee just published a free, open-sourced prompt you can give your AI coding assistants to do a technical SEO audit and give instructions on what to fix. It's MIT license and available on Github here: https://github.com/RivalSee/ai-seo-tools

This prompt won't solve every SEO issue. You still need to take other steps to improve your SEO, including:

Get backlinks
Create content
Do appropriate keyword research
Core Web Vitals optimization: Performance beyond basic rendering

If you want to contribute or give suggestions, please see the instructions in the repository.

And thank you to RivalSee for putting this together! If you want to track/improve your AI visibility on AI Chats like ChatGPT, Google AI, Perplexity, etc.. they are a great tool!