Forem: Andrew Judd

One API Call Changed Everything

Andrew Judd — Mon, 25 May 2026 13:36:50 +0000

Sunday morning. I'm about ready to type everything by hand and call it a weekend.

But I want to try one more thing. Instead of OCR to extract characters and then code to figure out what those characters mean - what if I just send the image to a vision model and ask what the document says?

The Code

import openai
import base64
import json
from pathlib import Path

def extract_document_vision(image_path):
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode("utf-8")

    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": """Extract handwritten documents into structured 
                JSON. Identify:
                - title: the document title or heading
                - items: array of {quantity, unit, item} for any 
                  listed items with measurements
                - instructions: array of step strings for any 
                  procedural content
                - notes: any additional annotations or side notes

                If something is crossed out, ignore it.
                If you can't read something clearly, make your 
                best interpretation and add a "uncertain": true 
                flag to that field."""
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{image_data}"
                        }
                    }
                ]
            }
        ],
        response_format={"type": "json_object"}
    )

    message = response.choices[0].message
    if message.content is None:
        raise ValueError(f"No content returned. Finish reason: {response.choices[0].finish_reason}. Refusal: {message.refusal}")
    return json.loads(message.content)

if __name__ == "__main__":
    for image_file in sorted(Path("images").glob("*")):
        print(f"\n--- {image_file.name} ---")
        try:
            result = extract_document_vision(image_file)
            print(json.dumps(result, indent=2))
        except ValueError as e:
            print(f"Skipped: {e}")

No pre-processing. No regex. No parser. One API call and a prompt in English.

Compare that to yesterday.

What Came Back

The document that Textract turned into "2 1/4 c fleur, 1 tso bokrig sado":

{
  "title": "Chocolate Chip Cookies",
  "items": [
    {"quantity": "2 1/4", "unit": "cups", "item": "flour"},
    {"quantity": "1", "unit": "tsp", "item": "baking soda"},
    {"quantity": "1", "unit": "tsp", "item": "salt"},
    {"quantity": "1", "unit": "cup", "item": "butter", "notes": "2 sticks"}
  ],
  "instructions": [
    "Preheat oven to 375°F",
    "Combine flour, baking soda and salt in small bowl",
    "Beat butter, granulated sugar, brown sugar and vanilla extract in large mixer bowl until creamy",
    "Add eggs, beating well",
    "Gradually beat in flour mixture",
    "Stir in chocolate chips",
    "Drop rounded tablespoon of dough onto ungreased baking sheets",
    "Bake for 9 to 11 minutes or until golden brown"
  ],
  "notes": null
}

First try. "c" became "cups." "tsp" stayed "tsp" because that's already standard. It caught "(2 sticks)" as a note on the butter and put it in the right field.

Why

OCR asks "what characters are in this image?" Hard problem when the characters are messy handwriting.

The vision model asks "what does this document say?" Sounds like the same question. It's not.

Think about how you read someone's handwriting. You don't decode each letter and build words from shapes. You look at the whole thing and between context and layout and your knowledge of language, you just know. Even when individual letters are a mess.

That's what's happening here. The model isn't a better letter-recognizer. It's skipping that problem entirely.

The Stuff That Broke OCR

The crossed-out line that killed my parser? Vision model saw the strikethrough, ignored it, read the correction. No code for that. Just worked.

Marginal notes Textract mixed into the main text? Identified as supplementary. Put in the "notes" field.

Abbreviations Tesseract turned into garbage? Interpreted from context.

The layout I spent 200 lines of regex on? Figured out on its own. Titles in "title." Items in "items." Steps in "instructions."

Three Approaches, Same Document

Tesseract:

Chocohite Ch p Cookes
2 114 cps flcar
1 tso bokrg sado
l tsp sit
1 c (2 stcks) btter

Textract (after all the pre-processing and parsing):

Title: Chocokite Chtp Cookes (confidence: 0.67)
Items:
  - 2 1/4 c fleur
  - 1 tso bokrig sado  
  - l tsp slt
  - 1 c (2 stcks) btter
[MANUAL REVIEW REQUIRED - 4 items below confidence threshold]

Vision API:

{
  "title": "Chocolate Chip Cookies",
  "items": [
    {"quantity": "2 1/4", "unit": "cups", "item": "flour"},
    {"quantity": "1", "unit": "tsp", "item": "baking soda"},
    {"quantity": "1", "unit": "tsp", "item": "salt"},
    {"quantity": "1", "unit": "cup", "item": "butter", "notes": "2 sticks"}
  ]
}

Metric	Tesseract	Textract	Vision API
Character accuracy	30-40%	40-60%	95%+
Structure accuracy	N/A	~30%	~90%
Manual review needed	~90%	~70%	~5-10%
Pre-processing	Yes	Yes (6 params)	None
Lines of code	~50	300+	~30
Dev time	~4 hours	~40 hours	~2 hours

The vision model's mistakes are small. A "3" that might be an "8." An abbreviation it flagged as uncertain. Stuff you catch in seconds. Not garbled output you have to retype.

By Sunday afternoon, everything is processed. The thing I spent all of Saturday failing to do took a couple of hours once I changed the approach.

When the Cleanup Code Becomes the Project

Andrew Judd — Fri, 22 May 2026 12:47:20 +0000

Tesseract can't do handwriting. Time to spend money.

AWS Textract. Cloud service, built-in handwriting detection, pay per page. If I'm paying for it, the output should at least be usable.

Textract

import boto3
from pathlib import Path

def extract_document(image_path):
    client = boto3.client('textract')

    with open(image_path, 'rb') as f:
        image_bytes = f.read()

    response = client.detect_document_text(
        Document={'Bytes': image_bytes}
    )

    lines = []
    for block in response['Blocks']:
        if block['BlockType'] == 'LINE':
            lines.append({
                'text': block['Text'],
                'confidence': block['Confidence']
            })

    return lines

if __name__ == "__main__":
    for image_file in sorted(Path("images").glob("*")):
        print(f"\n--- {image_file.name} ---")
        for line in extract_document(image_file):
            print(f"  [{line['confidence']:5.1f}%] {line['text']}")

Confidence scores are a nice touch. Accuracy is better - maybe 40-60% on a good document.

But "better" isn't "good enough." "2 1/4 cups flour" comes back as "2 1/4 c fleur." "1 tsp baking soda" becomes "1 tso bokrig sado."

The Real Problem

Even when it gets the words right, Textract doesn't know what any of it means. Flat text. Lines in reading order. My documents have titles, ingredient lists, instruction paragraphs, notes in margins. Textract sees none of that. Just characters on a page.

So now I'm writing parsers.

import boto3
import json
import re
from pathlib import Path

def extract_document(image_path):
    client = boto3.client('textract')

    with open(image_path, 'rb') as f:
        image_bytes = f.read()

    response = client.detect_document_text(
        Document={'Bytes': image_bytes}
    )

    lines = []
    for block in response['Blocks']:
        if block['BlockType'] == 'LINE':
            lines.append({
                'text': block['Text'],
                'confidence': block['Confidence']
            })

    return lines

def parse_structured_data(raw_lines):
    title = None
    items = []
    instructions = []

    quantity_pattern = r'^(\d+[\s/]*\d*)\s*(cups?|tbsp?|tsp|oz|lbs?|g|ml|c)\s+(.+)'

    for line in raw_lines:
        text = line['text'].strip()
        match = re.match(quantity_pattern, text, re.IGNORECASE)

        if match:
            items.append({
                'quantity': match.group(1),
                'unit': match.group(2),
                'item': match.group(3)
            })
        elif not title:
            title = text
        else:
            instructions.append(text)

    return {'title': title, 'items': items, 'instructions': instructions}

if __name__ == "__main__":
    for image_file in sorted(Path("images").glob("*")):
        print(f"\n--- {image_file.name} ---")
        raw_lines = extract_document(image_file)
        result = parse_structured_data(raw_lines)
        print(json.dumps(result, indent=2))

Works on 30% of the documents. The other 70% break at least one assumption. Title not on line one. Quantities written backwards. Abbreviations I've never seen. Crossed-out text mixed into the content. Multi-line entries split apart.

Every new document, a new edge case. Every new edge case, another if, another regex.

Saturday Night

Here's where I'm at:

Pre-processing with 6 configurable parameters
200+ lines of regex and heuristics
70% of documents still need a human
Accuracy I'm being generous calling 30%

The parser is now more work than just typing things by hand.

And every time I fix one document's output, three others break. The heuristics are fragile. Interconnected. Basically untestable because no two documents look alike.

One document has a crossed-out line. Original text scratched out, correction written above. Any person glances at it and reads the correction. Half a second.

Textract returns both lines. Jumbled. My parser doesn't know what a strikethrough is. Teaching it would mean analyzing the spatial layout of ink strokes. That's not a text problem anymore. That's a computer vision problem.

I'm a full day in. The system I'm building reads worse than I do, and the code to make it slightly less bad is growing faster than the documents it's supposed to process.

The OCR Rabbit Hole

Andrew Judd — Thu, 21 May 2026 15:44:33 +0000

Long weekend. Pile of handwritten documents on the desk. They need to become structured data - searchable fields in an app, not just scanned images.

OCR's been around forever. This should take a couple of hours.

The Documents

They're handwritten. Different people, different paper, different decades. Faded ink on some. Crossed-out words on others. Notes crammed into margins where there wasn't really room.

Any person can read these in about five seconds. Remember that.

Tesseract

Saturday morning. Tesseract is free, it's open source, I can run it on my own machine. Good enough to start.

import pytesseract
from PIL import Image
from pathlib import Path

def extract_document(image_path):
    img = Image.open(image_path)
    text = pytesseract.image_to_string(img)
    return text

if __name__ == "__main__":
    for image_file in sorted(Path("images").glob("*")):
        print(f"\n--- {image_file.name} ---")
        print(extract_document(image_file))

Printed text? Fine. Handwriting? "2 1/4 cups flour" comes back as "2 114 cps flcar."

I try different config options. Page segmentation modes. OCR engine modes. Character allowlists. Each one helps somewhere and breaks something else.

Pre-Processing

Maybe the images are the problem.

import pytesseract
from PIL import Image, ImageFilter, ImageEnhance, ImageOps
import numpy as np
from pathlib import Path

def preprocess_document(image_path):
    img = Image.open(image_path)
    img = img.convert('L')
    img = ImageOps.autocontrast(img, cutoff=2)
    img = img.filter(ImageFilter.SHARPEN)
    img = ImageEnhance.Contrast(img).enhance(1.8)

    img_array = np.array(img)
    threshold = np.mean(img_array)
    img_array = ((img_array > threshold) * 255).astype(np.uint8)

    return Image.fromarray(img_array)

if __name__ == "__main__":
    for image_file in sorted(Path("images").glob("*")):
        print(f"\n--- {image_file.name} ---")
        img = preprocess_document(image_file)
        print(pytesseract.image_to_string(img))

Contrast boosting helps the faded ones. Hurts the clear ones. Binarization destroys the subtle parts of the handwriting. Sharpening makes some letters readable and others bleed together.

Every fix is a trade-off. I'm tuning parameters per document. That's not automation.

30-40% accuracy on the handwriting. Saturday morning gone.

Tesseract is great at printed text. That's not what I have.

Mistakes Happen

Andrew Judd — Tue, 19 Oct 2021 13:15:05 +0000

Software development is a mostly “behind the scenes” operation. This is great because we don’t need to be in the limelight to do our job, nor to be good at it. But, this also opens up the door for other possibilities as well.

We have a lot of different tools at our disposal for the code that we write, and with those tools, we also have a responsibility to use for good. And guess what, we are also human. So, we make mistakes!

When doing anything in life, mistakes are inevitable. They are made in every occupation, it’s how we react to them that matters. Unfortunately I’ve seen countless times people trying to “protect themselves” by covering up the mistake. We have the tools available to us to do some of this. But I have to ask why is it necessary? Just because we are sitting behind a screen and can slave away for a few extra hours to remedy it, doesn’t make it right. Sure, admitting that you made a mistake is a bit of a slap to the ego, it should be! It further proves that you are not infallible. On the flip side, if you don’t admit it to yourself or just try to cover it up, and you are able to repair it it may end up actually inflating your ego. Telling yourself, “See, I messed up and was able to power through and fix it, I’m an amazing developer.” This however, is missing the point. It’s a learning experience for you.

The bigger issue that I find when people try to cover up mistakes is the fact that, it gets the problem solved today, but:

What happens if you make bigger mistakes when covering up this one?
What happens if you don’t even realize you are missing one piece of the puzzle? (i.e. if you are restoring data in a database, and return data to 2 out of the 3 required tables)

Problem solving is part of our job, and these issues could just be looked at as another problem to solve, which is fair however, not favourable.

What happens if you make a mistake that you can’t cover up? What happens if one of your co-workers find out that you covered something else up? That’s when things start to go downhill fast. Your reputation has been tarnished. The worst part about it is the fact people know you do this, so sometimes the unasked question (or asked question) is “what else did you cover up?”. This of course will instill doubt within your co-workers. They also will lose faith in you, and at this point it’s difficult to rebuild this trust. You built part of the working relationship on lies, and betrayed your fellow developers, how can they trust you?

When you do make a mistake that you are not a fan of, instead of covering it up, do the following:

Get a good understanding of the mistake that was made, so you are able to explain it
Come up with possible solutions on how to resolve it
Mention to your team that you made this mistake, and have come up with a few options and ask colleagues for their opinions
Work with your colleagues to resolve the issue
Come up with an action plan to prevent this from happening in the future

Yes, doing this may add a scratch on your reputation but it will also help to add a feather into your cap. You have shown your co-workers a vulnerable side, and that you are fallible. Because of that, they will, over time, trust you more. They will know you tell the truth and aren’t trying to save face. You may also become one of the people that they go to when they inevitably make a mistake.

Mistakes suck, but are inevitable as humans. It’s how we choose to react and grow from them which shows a level of maturity and integrity that you have. The initial ego hit sucks, but the trust that you develop as a team member will help to reduce the overall burn from this mistake.