Forem: Wu Haotian

Garbage collecting OpenCode's 2.4 GB session database

Wu Haotian — Wed, 15 Apr 2026 14:11:21 +0000

I've been using OpenCode from time to time for about a month, mainly working on a single side project in a tiny VPS. The disk free space is getting less and less so I decided to do some cleanup, then this caught my eye:

$ ls -lh ~/.local/share/opencode/opencode.db
-rw-r--r-- 1 ubuntu ubuntu 2.4G Apr 15 03:25 opencode.db

One month, non-heavy usage, single project, 2.4 GB of session database. That seems crazy.

What's eating the space

I asked an agent to give me more background and it turns out, OpenCode stores the full thinking & reasoning output for every turn in a single SQLite database. In my case, most of my 2.4 GB database is reasoning tokens and tool results, only less than 1% is actual conversation text. I'm not sure what's the real value of keeping them all, especially in a single SQLite database (credit to the strong SQLite), and I'm surprised to see OpenCode has no built-in cleanup for this. So this will just keep growing forever, until it becomes a black hole larger than node_modules.

For reference, Claude Code stores 86 MB for the same period. Codex uses 95 MB.

Let's garbage collect OpenCode session database

So I ( actually claude code ) built ocgc (OpenCode Garbage Collector).

uv tool install ocgc

ocgc status tells you where the space goes:

ocgc analyze shows your heaviest sessions and how fast the database is growing:

For the actual garbage collect part, ocgc purge:

ocgc purge --subagents --older-than 7d

809 sessions, 1.8 GB. Gone. Then ocgc vacuum to actually reclaim the disk space, from 2.3 GB down to 438 MB.

After cleaning up the database, I also noticed OpenCode's snapshot/ and session_diff/ directories can take up quite some space too, so ocgc cleans those as well.

Go check your ~/.local/share/opencode/opencode.db. Disk space doesn't grow on trees.

Making OpenClaw remember what it's doing after compaction

Wu Haotian — Mon, 06 Apr 2026 15:25:46 +0000

OpenClaw is great and I've been using it across a dozen Discord channels: travel planning, coding projects, daily briefings, each in its own thread, concurrently. During the journey, I found OpenClaw could have amnesia: lost track of key guidance and restrictions, or forgot the task it was working on, oftentimes, especially as conversations got long.

Why this happens

Although AI looks like magic and works like magic, under the hood it still has its boundaries, and in this case, its context window. The mighty AI can only process a certain amount of information, and if your threads go beyond that, OpenClaw uses compaction to summarize previous conversations, thus reducing the amount of information AI needs to process.
The thing is, compaction is not (and cannot be) lossless. Your critical guidance, rules you must follow, tasks in progress, all could be lost during the compaction, and voilà, your agent has amnesia.

The fix

Instead of putting everything inside your conversation only, I created openclaw-pawpad to give each individual OpenClaw session a freeform note file and a structured task list to read and update. This information is persisted on disk and does not get impacted by compaction.

How? The plugin registers a hook that reads the files and injects their content into the system prompt on every agent turn. The injected context looks like this:

<pawpad>
Your persistent session state — survives context compaction.
Update via pawpad_tasks and pawpad_note tools.

<tasks description="Session tasks (2/4 done)">
- [x] (high) Set up Docker build + Nginx reverse proxy
- [x] (high) Fix TLS certificate error (switched to distroless)
- [ ] (low) Write deployment docs
</tasks>
<notes description="Session notes">
- Runtime image: gcr.io/distroless/static-debian12 (not Alpine — TLS issues)
- LLM backend: localhost:4141, model gpt-5-mini
- Nginx: proxy_cache keys_zone needs restart (not reload) to clear
- This is an open source project — no local-only dependencies allowed
</notes>
</pawpad>

The agent gets tools to manage the task list and freeform notes. Writes are atomic so you don't end up with half-written files.

Setup

It's easy to set up:

openclaw plugins install openclaw-pawpad

No configuration needed. Restart OpenClaw and it works.

Does it help?

I've been playing with it for a few weeks. Before pawpad, my agent would get amnesia after ~15 minutes of heavy conversation. Now the tasks and notes persist across compactions and the agent never forgets.

But since I'm the one creating this plugin, my experience could just be placebo and your mileage may vary. So feel free to test and let me know your thoughts!

Make (my) git fetch & pull 100x faster on monorepos

Wu Haotian — Sat, 06 Sep 2025 00:00:00 +0000

I work on a massive git monorepo where hundreds of developers push changes daily.
The repo is huge, and most commits happen on branches I don’t even care about. Unsurprisingly, Git doesn’t perform well at this scale.

Why `git fetch` is slow by default

By default, running git fetch downloads everything from the remote, even branches you’ll never touch.
Instead, if you only fetch the branch you’re working on, e.g. git fetch origin main, Git only pulls objects related to main, skipping everything else.

On my machine, this simple change made fetches about 100x faster, just by downloading less.

But what about `git pull`?

Here’s the problem:
git pull (which is really just git fetch + git rebase/merge) will always fetch all branches by default. That means even if you only care about your current branch, git pull stays slow.

The fix? Run git pull origin main.
This fetches only what you need and makes pulls significantly faster.

Making it less tedious

Of course, typing git pull origin main every single time is painful.
My solution is to create a Git alias that automatically pulls only the current branch (credit to this Stack Overflow post):

# Add this to your ~/.gitconfig
[alias]
    p = !git pull origin $(git branch --show-current)

Now, instead of git pull, just run git p, and enjoy faster pulls without thinking about it.

Takeaway

If you’re stuck with a giant monorepo, you don’t have to suffer every time you run git pull.
By narrowing fetches to just the branch you’re on and automating it with an alias, you can get massive speedups with almost no downside.

Decorator type gymnastics in Python

Wu Haotian — Fri, 28 May 2021 00:00:00 +0000

You need to type hints your decorator

Say you have a simple decorator for adding logging before calling a function.

import logging

def add_log(f):
    def wrapper(*args, **kwargs):
        logging.info("Called f!")
        return f(*args, **kwargs)
    return wrapper


@add_log
def two_sum(a, b):
    return a + b

One day you decide to add type hints for this module -- it's easy to add type hints for two_sum

def two_sum(a: int, b: int) -> int:
    return a + b

But you need to add type hints for your decorator (add_log in this case) too, or you'll get Anyed wrapped function. mypy's reveal_type can be used for verifying this.

import logging

def add_log(f):
    def wrapper(*args, **kwargs): # type: ignore
        logging.info("Called f!")
        return f(*args, **kwargs)
    return wrapper


def two_sum(a: int, b: int) -> int:
    return a + b

reveal_type(two_sum)  # Revealed type is 'def (a: builtins.int, b: builtins.int) -> builtins.int'
reveal_type(add_log(two_sum))  # Revealed type is 'Any'

Simple type hints for simple decorators

Let's adding type hints for this simple decorator. For a simple decorator which doesn't modify the functions' arguments and return (like add_log above ), TypeVar should do the job pretty well.

from typing import TypeVar, Callable, cast
import logging

TCallable = TypeVar("TCallable", bound=Callable)

def add_log(f: TCallable) -> TCallable:
    def wrapper(*args, **kwargs): # type: ignore
        logging.info("Called f!")
        return f(*args, **kwargs)
    return cast(TCallable, wrapper)


@add_log
def two_sum(a: int, b: int) -> int:
    return a + b


reveal_type(two_sum)  # Revealed type is 'def (a: builtins.int, b: builtins.int) -> builtins.int'
reveal_type(add_log(two_sum))  # Revealed type is 'def (a: builtins.int, b: builtins.int) -> builtins.int'

Hard type hints for hard decorators

But, what if you want to modify the arguments and/or return value? Well, there's no easy way to type arguments, but at least you can type return value correctly

from typing import TypeVar, Callable, Awaitable, cast
R = TypeVar('R')
def sync_to_async(f: Callable[..., R]) -> Callable[..., Awaitable[R]]:
    async def wrapper(*args, **kwargs): # type: ignore
        return f(*args, **kwargs)
    return cast(Callable[..., Awaitable[R]], wrapper)

@sync_to_async
def two_sum(a: int, b: int) -> int:
    return a + b

reveal_type(two_sum)  # Revealed type is 'def (*Any, **Any) -> typing.Awaitable[builtins.int*]'
reveal_type(two_sum(2, "3"))  # Revealed type is 'typing.Awaitable[builtins.int*]

The dark way for typing arguments & return values

You can do some type gymnastics.. by generating numbers of TypeVars and using overload.

from typing import TypeVar, Callable, Awaitable, overload

A = TypeVar('A')
B = TypeVar('B')
C = TypeVar('C')
D = TypeVar('D')
E = TypeVar('E')
RV = TypeVar('RV')
@overload
def sync_to_async(f: Callable[[A], RV]) -> Callable[[A], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B], RV]) -> Callable[[A, B], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B, C], RV]) -> Callable[[A, B, C], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B, C, D], RV]) -> Callable[[A, B, C, D], Awaitable[RV]]:
    ...
@overload
def sync_to_async(f: Callable[[A, B, C, D, E], RV]) -> Callable[[A, B, C, D, E], Awaitable[RV]]:
    ...
def sync_to_async(f):
    async def wrapper(*args, **kwargs):
        return f(*args, **kwargs)
    return wrapper

@sync_to_async
def two_sum(a: int, b: int) -> int:
    return a + b

@sync_to_async
def do_log(content: str) -> None:
    print(content)

reveal_type(two_sum)  # Revealed type is 'def (builtins.int*, builtins.int*) -> typing.Awaitable[builtins.int*]'
reveal_type(do_log)  # Revealed type is 'def (builtins.str*) -> typing.Awaitable[None]'

If you insist to go this way, here's the code snippet I used for generating code above:

gymnastics = 5

for i in range(gymnastics):
    char = chr(i + 65)
    print(f"{char} = TypeVar('{char}')")

print("RV = TypeVar('RV')")

for i in range(gymnastics):
    chars = [chr(n + 65) for n in range(i + 1)]
    args = ", ".join(chars)
    print(f"""@overload
def sync_to_async(f: Callable[[{args}], RV]) -> Callable[[{args}], Awaitable[RV]]:
    ...""")

The Future: PEP-612

PEP-612 defines ParamSpec and Concatenate. They can make type hinting decorators pretty easy:

from typing import Concatenate, ParamSpec

P = ParamSpec('P')
R = TypeVar('R')

def with_context(f: Callable[Concatenate[Context, P], R]) -> Callable[P, R]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> R:
        return f(context, *args, **kwargs)
    return inner

@with_context
def request(context: Context) -> int:
    return 42

The sad thing is PEP-612 is not widely supported, as of now mypy does not fully support it.

Fin

May the type be with you.

Don't forget `py.typed` for your typed Python package

Wu Haotian — Wed, 17 Feb 2021 00:00:00 +0000

Intro

pixelmatch-py, a Python library I maintained for comparing images, received a Pull Request to add type hints about 10 months ago. After merging this PR & releasing a new version, I thought the library's users will magically get their type working.

Until today, I was reading cryptography's Changelog and a line got my attention:

Added a py.typed file so that mypy will know to use our type annotations.

After reading PEP-561 and mypy documentation I'm sure that I didn't publish the package right: I should include a py.typed file, or the type checker won't use the type hints provided by the package.

Adding `py.typed`

It's faily simple to include this file: just touch a py.typed file in your package directory and include it in your distribution.

I'm using poetry, so I added

packages = [
  {include = "pixelmatch"},
  {include = "pixelmatch/py.typed"},
]

under the [tool.poetry] section of pyproject.toml.

If you're using setup.py, you can add package_data to setup call:

setup(
    package_data={"pixelmatch": ["py.typed"]},
)

Release a new version for your package then type informations from your packages should works.

Fin

If you're a Python package maintainer, be sure to include py.typed file for your typed package!

Using Celery with RabbitMQ's Lazy Queue

Wu Haotian — Wed, 15 Apr 2020 01:57:15 +0000

Celery & RabbitMQ basics

Skip this part if you know what Celery & RabbitMQ is.

Celery is a task queue system in Python. Celery can help you run something in the background, schedule cronjobs and distribute workloads across multiple servers.

RabbitMQ is a message broker. A message broker is a program to help you send messages. Celery needs to use a message broker to send & receive messages, and they highly recommend using RabbitMQ.

What is Lazy Queue?

When you publish a message into a RabbitMQ's regular queue, the message is stored into the memory so they can be delivered to consumers pretty fast. RabbitMQ will try to keep as much as messages it can until it hit a memory threshold. When it happens, RabbitMQ will save messages to disk("page out"), which will block the queue and prevent publishers from publishing any new messages.

As you may notice, if you have a lot of messages to publish, a regular queue will

use a lot of memory, depending on your message size and number
sometimes stop receiving new messages ( when paging out ), make the performance less predictive.

A lazy queue will try to save the message to disk as early as possible, therefore minimize the memory usage ( use constant space in most cases ) with the cost of throughput time, disk I/O and CPU usage.

Should I use Lazy Queue?

I'd always use lazy queues unless there are some really high-performance requirements. Lazy queues use significantly less memory and can hold a lot more messages than regular queues. CloudAMQP recommends to enable lazy queues to get predictable performance..

But before you do, I highly recommend you to carefully read the Caveats and Limitations section of lazy queues' documentation, and do some benchmarks yourself.

How to Celery with RabbitMQ's Lazy Queue

Specify queue arguments in Celery

Say you have a celery app

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_routes = ([
    ('account.tasks.*', {
        'queue': 'account'
    }),
    ('server.tasks.*', {
        'queue': 'server'
    })
])

You can specify queue_arguments to make queues lazy.

from celery import Celery
from kombu import Queue

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_queues = [
    Queue('account', queue_arguments={'x-queue-mode': 'lazy'}),
    Queue('server', queue_arguments={'x-queue-mode': 'lazy'}),
]

app.conf.task_routes = ([
    ('account.tasks.*', {
        'queue': 'account'
    }),
    ('server.tasks.*', {
        'queue': 'server'
    })
])

Or, if you haven't specified any task routes

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

In this case, Celery will route them into a default queue called celery by default. You need to change the default queue into lazy mode.

from celery import Celery
from kombu import Queue

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_queues = [
    Queue('celery', queue_arguments={'x-queue-mode': 'lazy'}),
]


@app.task
def add(x, y):
    return x + y

If the queue is already created, you may encounter a PRECONDITION_FAILED error saying the queue was declared with inequivalent arg 'x-queue-mode'. You should consider deleting the queue ( WILL LOSE ALL MESSAGES INSIDE IT ) or changing queue mode via policy.

Changing Queue Mode using RabbitMQ's policy

You can also change queues' mode using RabbitMQ's policy. This allows you to change an existing queue's mode without removing & recreating it. You can set a policy in shell using rabbitmqctl:

rabbitmqctl set_policy Lazy "^celery$" '{"queue-mode":"default"}' --apply-to queues

The above command changes the queue celery into lazy mode.

I prefer to specify queue arguments in Celery since it can be version controlled and don't require extra operations on the production server.

Enjoy.

Originally published at https://blog.whtsky.me/tech/2020/using-celery-with-rabbitmqs-lazy-queue/

Fixing Django's "populate() isn't reentrant" by printing traceback

Wu Haotian — Sun, 12 Apr 2020 16:05:00 +0000

Note: This article is based on Django 2.2.11

I was maintaining a huge Django app. Everything looked fine until I tried to run python manage.py -h:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "venv/lib/python3.6/site-packages/django/apps/registry.py", line 83, in populate
    raise RuntimeError("populate() isn't reentrant")
RuntimeError: populate() isn't reentrant

Hmmm, okay.

The exception was from execute_from_command_line in manage.py, which is generated by Django thus in great chance not the root cause of the exception.
I googled populate() isn't reentrant but the results weren't helpful enough to me.

Let's dig into Django's source code then. Followed the traceback from exception we can locate the populate function

# An RLock prevents other threads from entering this section. The
# compare and set operation below is atomic.
if self.loading:
    # Prevent reentrant calls to avoid running AppConfig.ready()
    # methods twice.
    raise RuntimeError("populate() isn't reentrant")
self.loading = True

Looks like the populate got called twice. Since execute_from_command_line is expected, let's record the traceback from the other call and print it out.

Modify venv/lib/python3.6/site-packages/django/apps/registry.py:

# An RLock prevents other threads from entering this section. The
# compare and set operation below is atomic.
if self.loading:
    # Prevent reentrant calls to avoid running AppConfig.ready()
    # methods twice.
    raise RuntimeError("populate() isn't reentrant.\n" + "\n".join(self.prev_traceback))
self.loading = True
import traceback
self.prev_traceback = traceback.format_stack()

Rerun python manage.py -h, and we can see the nice traceback.

  File "server/__init__.py", line 1, in <module>
    from .celery import app as celery_app

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "server/celery.py", line 11, in <module>
    django.setup()

  File "venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)

  File "venv/lib/python3.6/site-packages/django/apps/registry.py", line 85, in populate
    self.prev_traceback = traceback.format_stack()

Looks like someone called django.setup() in our celery module.

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'balisong_server.settings')
django.setup()

app = Celery('balisong')

After talking with the code author and looking into celery's document, I'm definite that the django.setup() here should be removed.
And removing this solves the problem.

Lessons learned: when in doubt, just print traceback.

Originally published at https://blog.whtsky.me/tech/2020/populate_isnt_reentrant/