Forem: Cyris ⭐️ JavaScript

Making Your Documentation AI-Friendly: A Practical Guide

Cyris ⭐️ JavaScript — Fri, 11 Apr 2025 21:57:26 +0000

Hey there! 👋 I've been working with AI and web scraping for a while now, and I want to share some practical tips on how to make your documentation more AI-friendly. Instead of trying to block all bots (which never really works), let's focus on creating structured, reliable channels for beneficial AI while keeping the bad actors out.

📅 The LLM Knowledge Gap

Here's the thing about Large Language Models - they're only as good as their training data. Most LLMs have a knowledge cutoff date, meaning they don't know anything that happened after their last training session.

This creates a real problem when users ask about:

New features released after the cutoff
Breaking changes in recent updates
Security patches and bug fixes
Current best practices

That's why it's crucial to give LLMs direct access to your latest documentation. When an LLM can pull in your current docs on demand, it can provide accurate, up-to-date information instead of relying on potentially outdated training data.

🤔 Why AI Can't Read Your Docs

Before we dive into solutions, let's look at the most common roadblocks I've seen that prevent AI from properly accessing documentation:

Client-Side Rendering: Some AI crawlers can't execute JavaScript, so if your content is rendered client-side, it's invisible to them
Complex Navigation: JavaScript-based routing and hash navigation break traditional crawling
Inconsistent Structure: Varying HTML layouts and tags make it hard for AI to understand content relationships
Authentication Walls: Login requirements block automated access completely
Rate Limiting: Aggressive rate limits prevent thorough documentation crawling
Anti-Bot Measures: CAPTCHAs and similar tools block all automated access, good or bad
Performance Issues: Slow loading times and timeouts interrupt the crawling process

The good news? All of these problems have solutions, and that's what we're going to cover next. Let's make your docs AI-friendly!

🚀 Roll Out the Welcome Mat for Good AI

Instead of making bots guess their way through your site, let's make it crystal clear who's welcome. Here's how:

Allowlisting: Use robots.txt to explicitly allow trusted bots. Set up your WAF or firewall to recognize these good actors by their user agents and IP ranges.
Double-Check: Don't just trust the user agent string. Verify bots by:
- Checking their IP ownership
- Using verified bot lists (like Cloudflare's)
- Looking out for new verification standards
Clear Signals: Use robots.txt to publicly declare which bots you're cool with, even if you're enforcing rules elsewhere.

🤖 Common AI Crawlers You'll Encounter

First things first - you need to know who's knocking at your door. While robots.txt isn't perfect, it's still the standard way to signal your intentions. Here's a quick reference of the main players:

Bot	User-Agent	What They Do	Allow Example	Block Example
OpenAI (ChatGPT)	GPTBot	Web crawling for model improvement	`User-agent: GPTBot<br>Allow: /`	`User-agent: GPTBot<br>Disallow: /`
OpenAI (Plugin)	ChatGPT-User	Plugin actions	`User-agent: ChatGPT-User<br>Allow: /`	`User-agent: ChatGPT-User<br>Disallow: /`
Google AI	Google-Extended	Gemini, Vertex AI	`User-agent: Google-Extended<br>Allow: /`	`User-agent: Google-Extended<br>Disallow: /`
Anthropic	anthropic-ai	General crawling	`User-agent: anthropic-ai<br>Allow: /`	`User-agent: anthropic-ai<br>Disallow: /`
Perplexity	PerplexityBot	Web crawling	`User-agent: PerplexityBot<br>Allow: /`	`User-agent: PerplexityBot<br>Disallow: /`
Common Crawl	CCBot	Public web data	`User-agent: CCBot<br>Allow: /`	`User-agent: CCBot<br>Disallow: /`
Facebook	FacebookBot	Platform needs	`User-agent: FacebookBot<br>Allow: /`	`User-agent: FacebookBot<br>Disallow: /`

Note: This list changes often, and remember - robots.txt is more of a suggestion than a rule.

📦 Give AI Direct Access to Your Data

The best way to ensure AI gets your docs right? Give them direct access to structured data. Here's how to do it:

1. Use llms.txt

The llms.txt standard is a game-changer for making your site AI-friendly. It's like a robots.txt for LLMs, but way more powerful. Here's why you should use it:

Standardized Format: A simple markdown file that tells LLMs exactly how to use your site
Easy to Implement: Just add a /llms.txt file to your root directory
Human & Machine Readable: Works for both humans and AI
Context-Aware: Helps LLMs understand your site's structure and purpose
Markdown Support: Automatically serves markdown versions of your pages (just add .md to URLs)

Here's a quick example of what your llms.txt might look like:

# Your Project Name

> A brief description of what your project does and why it's awesome

## Documentation

- [Getting Started](https://your-site.com/getting-started.html.md): Quick start guide
- [API Reference](https://your-site.com/api.html.md): Complete API documentation

## Optional

- [Contributing Guide](https://your-site.com/contributing.html.md): How to contribute

The best part? It's super simple to implement and works with existing tools. Check out the llms.txt GitHub repo for more details and examples.

2. Build an API and MCP server

Create a public API specifically for your documentation. This gives you:

Clean, structured data in JSON/Markdown
More reliable than HTML scraping
Better control over access
Easier for AI devs to find and use

But here's the cool part - you can take it a step further by setting up an MCP (Model Context Protocol) server. This lets AI models request documentation directly from your source, ensuring they always get the latest version. Here's how it works:

// Example MCP server setup
const express = require('express');
const app = express();

// MCP endpoint for documentation
app.get('/mcp/docs', async (req, res) => {
  const { version, section } = req.query;

  // Fetch latest docs from your source
  const docs = await fetchLatestDocs(version, section);

  // Return in a format AI can easily consume
  res.json({
    content: docs,
    metadata: {
      lastUpdated: new Date(),
      version: version,
      source: 'your-docs-repo'
    }
  });
});

// Start the server
app.listen(3000, () => {
  console.log('MCP server running on port 3000');
});

The benefits of using an MCP server:

Always Fresh: AI gets the latest docs directly from your source
Version Control: Specify which version of docs you want
Structured Access: Clean, consistent data format
Analytics: Track which parts of your docs are most useful to AI

You can even combine this with your llms.txt file to point AI to your MCP endpoints:

# Your Project Docs

> Latest documentation available via MCP

## API Endpoints

- [MCP Documentation Server](https://your-site.com/mcp/docs): Get the latest docs
- [MCP Search](https://your-site.com/mcp/search): Search across all versions

This approach gives you the best of both worlds - structured access through an API and real-time updates through MCP. The AI can request exactly what it needs, when it needs it, and you maintain complete control over the content.

Just look at how all these companies and users building 3rd party tools are leveraging MCP servers. MCP Server Directory

3. Provide Direct Markdown Access

Sometimes the simplest solution is the best. You can make your documentation directly available as markdown files that AI can easily consume. Here's how:

Single File Approach: Create a comprehensive docs.md that links to all your documentation
Downloadable Archive: Offer a .zip of all your docs in markdown format
GitHub/GitLab: Host your docs in a public repository (many LLMs can read these directly)

Here's a quick example of a well-structured markdown index:

# Project Documentation

## Core Features
- [Getting Started](getting-started.md)
- [API Reference](api.md)
- [Configuration](config.md)

## Advanced Topics
- [Architecture](architecture.md)
- [Performance Tuning](performance.md)
- [Security](security.md)

However, there are some challenges to watch out for:

Version Control: Markdown files can get out of sync with your main docs
Context Limits: Some docs might exceed LLM context windows (e.g., GPT-4's 128k limit)
Formatting Issues: Complex markdown features might not render correctly in all LLMs
Link Management: Relative links might break when files are moved or downloaded

To mitigate these issues:

Use a docs-as-code approach with automated sync
Split large docs into smaller, focused files
Keep markdown simple and well-structured
Use absolute URLs for external links
Include a last-updated timestamp in each file

🏗️ Structure Your Content for Machines

Make your docs easy for machines to understand:

Use Semantic HTML: <article>, <section>, <nav> - not just endless <div>s
Add Schema.org: Give machines explicit semantic info
Keep It Clean: Remove ads, repeated nav links, and cookie banners from the main content

🔧 Fix Technical Roadblocks

Make sure bots can actually get to your content:

Rendering: If you're heavy on JavaScript, use SSR or Dynamic Rendering
Navigation: Use standard <a href> tags and provide sitemaps
Code Examples: Make them accessible text with clear language tags
Versioning: Make it clear in URLs and metadata
Speed: Optimize for quick loading

🛡️ Anti-Scraping vs. AI: What Works?

Here's the lowdown on different techniques and how they stack up against AI scrapers:

Technique	How It Works	Against Basic Bots	Against AI	What to Do
robots.txt	Voluntary rules	Moderate	Low	Use Allow for good bots; combine with other methods
Meta Tags	Page-level instructions	N/A	N/A	Use sparingly; doesn't stop scraping
CAPTCHA	Human verification	High	Moderate	Avoid on public docs; allowlist good bots
Rate Limiting	IP-based limits	Moderate	Low	Higher limits for verified bots
UA Filtering	Block by user agent	Low-Moderate	Very Low	Mainly for allowing good bots
JS Rendering	Client-side content	High	Moderate	Use SSR for bots; provide APIs
Complex Nav	Non-standard links	Moderate-High	Low-Moderate	Stick to standard links
Inconsistent HTML	Varying structure	Low	Low-Moderate	Use consistent templates
Login Walls	Authentication	Very High	High	Keep public access; API for trusted AI
Fingerprinting	Browser ID	High	Moderate-High	Part of layered defense
Behavior Analysis	Pattern detection	High	High	Tune carefully
Bot Management	Multi-signal detection	Very High	High-Very High	Configure allowlisting

Remember: The goal isn't to block all bots - it's to welcome the good ones while keeping the bad ones out. With these strategies, you can make your documentation both AI-friendly and secure.

🛠️ Tools That Bypass Restrictions

While we've focused on making your docs AI-friendly, it's worth noting that some companies are building tools to bypass these restrictions. For example, Firecrawl offers a service that can:

Handle JavaScript-heavy sites
Bypass rate limiting
Parse dynamic content
Convert web pages to LLM-ready data

However, these services come with a subscription cost and can be blocked by more sophisticated anti-bot measures. That's why implementing the solutions we've discussed is still the best long-term approach - it's more reliable, cost-effective, and maintains control over how your documentation is accessed.

Thanks for reading! If you found this guide helpful give it a ⭐️, and feel free to share it with others. If you notice any inaccuracies, please do let me know. I welcome all feedback. Hit me up on X/Twitter.

P.S. Don't forget to check out the MCP Server Directory to see how others are implementing these strategies in the wild.

Reverse engineering a private API with MITM Proxy

Cyris ⭐️ JavaScript — Thu, 20 Aug 2020 09:30:02 +0000

Disclaimer: This is for educational purposes, do not abuse this method.

Developers can often struggle to find the data they need for their personal projects due to certain services locking down their API's.

So I wanted to show you a way you can get the data you need.

If you prefer video, I've got you covered.

What is MITM Proxy and how does it work?

MITM stands for Man-in-the-Middle and mitmproxy is exactly that. It allows you to send data through a proxy that acts as a man-in-the-middle so you can read the request and responses being sent to and from the server.

Before we begin

We'll be installing this on a Macbook and capturing the data sent from an iPhone. However mitmproxy supports multiple platforms and devices. You can find out more in their documentation.

Installation

Head on over to mitmproxy.org and follow their installation instructions. You should be able to open mitmproxy from the terminal with:

$ mitmweb

mitmweb is the GUI version that should open up in a new tab once it's running.

With mitmweb running we can configure our device

You are going to need the local IP address of the computer running mitmproxy. You can Google how to find this on your computer.

On your device follow these instructions:
(Detailed tutorial in video above)

Enable the proxy setting on your devices connection sections.
Point the IP address to the local IP address of your computer
Set the port number to 8080
Authentication is not needed
Open your mobile browser (iPhone must be Safari)
Go to mitm.it
Click to download the certificate for your device type
On iPhone you then need to visit the setting page. You will be prompted about the recently downloaded certificate. Confirm the installation of the certificate there.

Bypassing iPhone's extra security

Go to Settings -> General -> About -> Certificate Trust Settings
Enable full trust of root certificates.

Setup is done, on to the fun stuff

Now you should be able to open up apps on your device and you will start seeing traffic popup in mitmweb

In our example we are capturing the data being sent to Thingivese from the mobile app. If you look closely on the right hand side you will find the authentication header. In this header you will find the token being used to validate you as a user.

This is the token we can now use to query the API ourselves in our own personal application.

Now try query the API yourself.

Here we are pulling the latest listings using Axios and sending the bearer token along with it.

const http = require("http");
const Axios = require("axios");

http
    .createServer(function (req, res) {

        const url = "https://api.thingiverse.com/featured?page=1&per_page=10&return=complete";
        const token = "5a2d072366d7039776e4c35c5f32efaf";

        Axios.get(url, {
                headers: {
                    'Authorization': `token ${token}`
                }
            })
            .then((result) => {
                res.setHeader('Content-Type', 'application/json');
                res.end(JSON.stringify(result.data))
            })
            .catch((error) => {
                console.error(error)
            })
    })
    .listen(3000); //the server object listens on port 3000

Worth noting

This may will not work for all applications as they may use 'Certificate Pinning' to which you'll need an Android or Jailbroken iOS device.

You can read more about Certificate Pinning in the docs: Certificate Pinning

Thanks for reading, for a more in-depth tutorial watch the video above. Follow me @Sudo_Overflow on Twitter for more.

Let me know below if you have any other tips or tricks.

DIY: Generating dynamic images on the fly for Email Marketing

Cyris ⭐️ JavaScript — Fri, 22 May 2020 06:25:47 +0000

I was recently tasked to find a lightweight method to generate dynamic images on the fly for Email Campaigns. Sure we could use third-party solutions to do just that but for a fee. These are great services but being a developer I wanted to see if I could build my own that fits my needs.

But with a catch

A few rules I had set forward for myself.

It needs to be lightweight
No headless browsers
No screenshot tools
No saving and serving images
Needs to be fast

Why do we need this?

Email development has come a long way in terms of what is possible but how email is coded still lacks far behind traditional web development. Email still uses <table> for layout although <div> is becoming more popular each day.

Security concerns prevent us from using scripts such as JavaScript in email, not to mention it would immediately be spammed. People and companies are still using software like Outlook 2010 so cross-platform support for certain elements and layouts is about as messy as those $1 DVD bins.

In general, Email is largely static, boring, and struggles to capture your target market's attention. So what can we do about this?

Introducing dynamically generated imagery

One thing that works on 99% of email clients is images. So we focus our attention on improving that. Using dynamic images allows us to personalize the email to the reader with great custom fonts, designs and even custom animated GIF's to grab the reader's attention.

The above example is personalizing the header of the email with the recipients first name in a custom font on a background image.

Let's get building

In summary we are going to build a simple Express server with NodeJS. This uses the node-canvas module to draw exactly what we want before exporting the canvas as a PNG.

Initialise a project and install the dependencies

$ npm init
$ npm install express --save
$ npm install canvas --save

Create server.js and require the dependencies needed

Don't forget to register your font files.

const express = require('express')
const { registerFont, createCanvas, loadImage } = require('canvas')
const app = express()
const port = 3000

// We need to register our font file to be used in canvas
registerFont('./fonts/Sign-Painter-Regular.ttf', { family: 'signpainter' })

app.get('/', (req, res) => {
    res.send('Hello World!')
})

app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`))

If all is working you should be able to start your application with node server and visit your hello world at http://localhost:3000

Create a custom GET route for the image

This should pull in the query parameters to be used in the canvas. In our case, we just want the name so all we have to do is:

app.get('/header', (req, res) => {

    // Grab first name from query
    let firstname = decodeURI(req.query.name)

    // Custom canvas added here
    // ...
    // ...
})

Add your canvas logic inside the route

From our design we know that the only personalization is going to be the first name. So the rest of it can be a "background-image" so to speak.

app.get('/header', (req, res) => {

    // Grab first name from query
    let firstname = decodeURI(req.query.name) + "!";

    // Define the canvas
    const width = 600 // width of the image
    const height = 474 // height of the image
    const canvas = createCanvas(width, height)
    const context = canvas.getContext('2d')

    // Define the font style
    context.textAlign = 'center'
    context.textBaseline = 'top'
    context.fillStyle = '#FFFFFF'
    context.font = "80px 'signpainter' bold";

    // Load and draw the background image first
    loadImage('./images/background.jpg').then(image => {

        // Draw the background
        context.drawImage(image, 0, 0, 600, 474)

        // Draw the text
        context.fillText(firstname, 300, 150)

        // Convert the Canvas to a buffer
        const buffer = canvas.toBuffer('image/png')

        // Set and send the response as a PNG
        res.set({ 'Content-Type': 'image/png' });
        res.send(buffer)
    })
})
})

After drawing the background image and text, we are then converting the canvas into a buffer and sending the response back to the client as a PNG image. This allows the client to load the dynamic image on their side.

Time to run this.

Start your app with node server and visit the new route you created at http://localhost:3000/header?name=@Sudo_Overflow

And there you have it

You can now merge the first name into your email's <img> tag as
<img src="http://localhost:3000/header?name={{FirstName}}"> and have it automatically generated for you.

A special thank you to @flaviocopes for the idea of using Canvas. You can check out his article here.

The full project can be found on my Github

If you know of ways to improve this or have feedback, let me know in the comments or Twitter at @sudo_overflow

Use Twitter to find developer friends near you

Cyris ⭐️ JavaScript — Thu, 21 May 2020 05:48:34 +0000

Being a developer can be tough, especially doing it all alone. That's why I always recommend surrounding yourself with like-minded 'Developer Friends' that support and motivate you.

Learn from each other, challenge each other, and work together to achieve the goals you set out. You'll not only learn about team dynamics but communication and collaboration. These are all important skills you'll need when working in a team as a developer.

So if you're looking to connect with like-minded individuals in your area then this trick is for you.

I'm intrigued, go on.

Twitter has a really cool feature that not many people know about. You can search for tweets using geolocation data.

A lot of tweets are automatically tagged with geolocation data such a latitude, longitude, and radius (skip ahead if you want to disable this). Using Twitter's own search or API we can query for tweets using these fields:

// The breakdown
geocode:latitude,longitude,radius
// Actual data
geocode:-36.8484597,174.7633315,10km 
// Actual data with search terms included
geocode:40.7127753,-74.0059728,5mi "Coding" OR "Programming"

Let's give it a try

Here we are searching for anyone tweeting related developer terms within a 5-mile radius of Facebook's Headquarters in California.

Now make it even easier

You're not going to want to keep Googling the coordinates and building your custom search strings each and every time. So I built a quick tool that does it all for you.

Developer Friends will accept any address, location you can throw at it and it will automatically generate the coordinates and build your custom search query for you. Include the predefined search terms with the tickboxes or even add your own in the text box.

After hitting search you'll be taken to Twitter with your own custom search query and everyone listed should be within the location and radius you specified. Now go make some developer friends. 👍

...

I don't want to appear in these searches

No worries, you can disable geotagging on your tweets within Twitter's settings. You can read more about it here

I hope you'll be able to find, follow, and make plenty of developer friends. If you know any other cool tricks, let me know in the comments or on Twitter @sudo_overflow