Forem: Buffer Overflow

The ESCAPE Byte Problem: How We Beat Brotli by Separating Token Streams

Buffer Overflow — Tue, 14 Apr 2026 00:47:39 +0000

Part 4 of the Glasik Notation series.

Previous articles covered the sliding window tokenizer, Aho-Corasick O(n) matching, and GN's first verified benchmarks against gzip.

The Waste Was Hidden in Plain Sight
After implementing Aho-Corasick O(n) matching, GN was fast. Sub-millisecond per chunk, competitive with brotli on latency. But the ratio numbers kept coming back flat:

gzip-6: 2.18x
GN AC: 2.20x (+0.9% vs gzip)
brotli-6: 2.47x

We were barely beating gzip. Brotli was 12% ahead. The vocabulary was real — 31,248 tokens per 200 chunks, 190 tokens per chunk average. The matches were happening. So where were the bits going?
We ran a token stream entropy analysis:
pythonfrom collections import Counter
import math

token_ids = []
for c in sample:
raw = slider.encode_ac_raw(c)
i = 0
while i < len(raw):
if raw[i] == ESCAPE and i+1 < len(raw):
token_ids.append(raw[i+1])
i += 2
else:
i += 1

counter = Counter(token_ids)
total = sum(counter.values())
entropy = -sum(c/total * math.log2(c/total) for c in counter.values())
print(f"Token entropy: {entropy:.3f} bits/token")
Result: 7.758 bits/token.

We were encoding each token as 2 bytes: ESCAPE + id. That's 16 bits per token. The theoretical minimum was 7.758 bits. We were wasting 51.5% of every token encoding. That's where the bits were going.

Why the Mixed Stream Was Hurting Us

Our tokenized output looked like this:

[ESCAPE][id][ESCAPE][id][lit][lit][lit][ESCAPE][id][lit][ESCAPE][id]...

Every token costs 2 bytes: an ESCAPE byte (0x01) followed by the ID. We fed this into deflate expecting it to compress well. But deflate uses LZ77 — it looks for repeated byte sequences in a sliding window. The ESCAPE bytes were fragmenting every pattern.

Where deflate might have seen:
" the " " the " " the " ← repeating 6-byte sequence, compresses well
It was instead seeing:
[01][04] " t" "he" [01][04] " t" ... ← ESCAPE bytes breaking the pattern
The ESCAPE byte was acting like static on a radio signal. Present in every token, making the mixed stream look noisier than it actually was.

The Insight: Separate the Streams
What if we just... didn't mix them?
Instead of one interleaved stream, emit two:

Token stream: just the IDs — [04][04][38][20][04][07]...
Literal stream: just the literal bytes — "t" "h" "e" " " "a" ...

Then compress each independently with raw deflate.
The token stream is pure symbols. Token ID 4 (" the") fires 483 times in 200 chunks. That's a highly skewed distribution — deflate loves it. The literal stream is clean text with no ESCAPE pollution. It compresses the way text is supposed to compress.

pythontoks, lits = slider.encode_ac_split(chunk)
dt = zlib.compress(toks, 6)[2:-4] # raw deflate
dl = zlib.compress(lits, 6)[2:-4]
frame = struct.pack('>H', len(dt)) + dt + dl

This is the same insight behind why PNG separates prediction from entropy coding, why video codecs separate motion vectors from residual — when you have structurally different data, compress the structures separately.

The Numbers

We ran this across 4 corpora, 3 seeds each — 12 independent measurements. Standard protocol: warm 500 chunks, test next 300.
Batch size matters. Each chunk has ~37 token IDs. Deflate header overhead (~18 bytes) dominates a tiny stream. Batching solves this — concatenate N chunks before compressing the token stream:

GN split b=1: 2.226x 0.043ms -6.6% vs brotli ← header overhead dominates
GN split b=4: 2.385x 0.036ms +0.1% vs brotli ← already matching brotli
GN split b=8: 2.456x 0.036ms +3.1% vs brotli ← production sweet spot
GN split b=16: 2.542x 0.037ms +6.7% vs brotli ← diminishing returns

b=8 is the production choice. Beyond b=16 the marginal gain flattens and you're accumulating more latency budget than the ratio improvement justifies.

Full 12-measurement verification at b=8:
CorpusGN split b=8vs gzipvs brotlip50p99ShareGPT2.49–2.52x+15%+2%0.043ms0.061msWildChat2.48–2.51x+15%+2%0.042ms0.073msLMSYS2.50–2.56x+14%+2%0.044ms0.079msUbuntu-IRC2.06–2.09x+49%+28%0.008ms0.013ms

Every single measurement beats both gzip and brotli.
And on tail latency: GN split b=8 p99 never exceeds 0.123ms. Brotli-6 p99 reaches 0.226ms. GN has 2–4x better tail latency than brotli while achieving better compression ratio.

Why This Works (The Information Theory)
The mixed tokenized stream had:

Token entropy: 7.758 bits/token
Encoding cost: 16 bits/token
Waste: 51.5%

The split stream:

Token stream: pure symbols, deflate compresses ~2–3x on its own
Literal stream: clean text, no structural noise, deflate compresses ~1.9x

Combined result: 2.49–2.56x on the original input

The separation lets each compressor do what it was designed to do. This isn't a trick — it's giving deflate the data structure it can actually exploit.

The Frame Format

Simple and self-contained:

[2B tok_deflated_len][tok_deflated][lit_deflated]
Two bytes of length prefix for the token stream, then the two compressed streams concatenated. Given the vocabulary, you can decode it without any other external state.
The Rust implementation in codon.rs:
rustpub fn encode_ac_split(buf: &[u8], ac: &AhoCorasick) -> (Vec, Vec) {
let mut tok_ids: Vec = Vec::new();
let mut literals: Vec = Vec::new();
let mut pos = 0usize;

for m in ac.find_iter(buf) {
    for &b in &buf[pos..m.start()] {
        literals.push(b);
    }
    let pat_idx = m.pattern().as_usize();
    if pat_idx < 254 {
        tok_ids.push((pat_idx + 1) as u8);
    } else {
        for &b in &buf[m.start()..m.end()] {
            literals.push(b);
        }
    }
    pos = m.end();
}
for &b in &buf[pos..] { literals.push(b); }
(tok_ids, literals)

}
O(n) scan, single pass, clean split.

Lossless Round-Trip

The split-stream is lossless when encoder and decoder share the same vocabulary. Token IDs are indices — to decode, you need to know what pattern each ID maps to.
GN uses a stateful model in production. Encoder and decoder share a synchronized sliding window; each frame carries a 2-byte dict_version. If they diverge, the decoder requests a resync. This keeps frames small while guaranteeing correctness.
Round-trip verified: 5/5 test cases pass including empty buffers, raw ESCAPE bytes in input, and 10,000-byte repetitive inputs.

What's Next: Fractal Dictionary Sharding

The split-stream insight revealed something deeper: token and literal streams have fundamentally different statistical structure. Taking that further — different types of content have different vocabulary entirely.
Code blocks repeat function, return, const. System messages repeat role definitions. User messages repeat question structures. Compressing them with a single shared vocabulary leaves ratio on the table.
We're implementing fractal dictionary sharding: four vocabulary tiers (L0 universal, L1 domain, L2 session, L3 chunk) with per-shard-type routing and deterministic crystal identity per shard — same content always produces the same compressed shape. The FractalCompressor is implemented, wired into the napi production path, and passing roundtrip verification across all shard types.
More on that in Article 5.

Code and Paper

GitHub: github.com/atomsrkuul/glasik-core (MIT)
npm: gni-compression@1.0.0
arXiv: pending cs.IR endorsement — if you're a qualified author (3+ cs papers): code 7HWUBA

Robert Rider is an independent researcher building Glasik, an open-source compression and context management system for LLM deployments.

GN Beats Gzip and Brotli: How a Learning Sliding Window Outperforms Static Compressors

Buffer Overflow — Tue, 07 Apr 2026 19:14:49 +0000

When we published our last article, GN was within 10% of gzip on LLM conversation data. We said the remaining gap was in the entropy backend. We were wrong about the solution — but right about the problem.
This week GN beats gzip on every corpus we tested. And on all three corpora, it beats brotli.
Here is what we learned.

The ANS Dead End
Our first instinct was to improve the entropy coder. Gzip uses Huffman coding. zstd uses ANS (Asymmetric Numeral Systems). We implemented byte-renorm ANS, bit-renorm ANS, and Order-1 ANS from scratch in Rust.

Results on ShareGPT:
Codec Ratio
gzip-6 2.082x
byte-ANS 1.233x
bit-ANS 1.212x
O1-ANS 0.551x

ANS without an LZ-style preprocessing pass is worse than gzip. Every time. The reason is fundamental: entropy coders compress symbol frequency distributions. But gzip's real advantage comes from LZ77 — the sliding window that eliminates repeated byte sequences before entropy coding runs. ANS cannot fix what LZ77 needs to do first.
We kept ANS in the codebase as a primitive for future work and moved on.
The Real Problem: Per-Frame Dictionary Overhead
GN has a sliding window tokenizer — it learns domain vocabulary across batches and compresses using that vocabulary. But there was a critical architectural flaw: the dictionary was serialized into every compressed frame.

200 entries × ~10 bytes = ~2KB overhead per chunk. On 500-byte chunks, the dictionary cost more than the compression saved.

v1 on 1000 LLM chunks: 0.502x (expanding the data)
The fix: stop putting the dictionary in the frame. Keep it in shared state, reference it by version number. This is exactly how brotli's static dictionary and zstd's dictionary mode work.

Frame v1: magic + full_dictionary + payload (~2KB overhead)

Frame v2: magic + dict_version(4 bytes) + payload (8 bytes overhead)

The Corpus Window (Level 2)
With the overhead fixed, we increased the window to 10,000 entries and made it global — one sliding window shared across all compression calls in the process. Every session, every shard, every conversation feeds the same accumulating vocabulary.

Results immediately improved:
Corpus L1 (per-call) L2 (corpus window) gzip brotli
ShareGPT 2.191x 2.402x 2.178x 2.453x
WildChat 2.035x 2.145x 2.025x 2.234x
LMSYS 2.094x 2.231x 2.079x 2.322x

L2 beats gzip on every corpus. The gap to brotli narrowed to 2-4%.
Retrieval-Warmed Compression (Level 3)
The insight: before compressing a new chunk, feed similar prior chunks through the sliding window first. This warms the dictionary with related vocabulary so the new chunk compresses better. The act of retrieval changes the compression state.

We benchmarked warm_k (number of prior chunks used for warming) on WildChat — the hardest corpus due to topic diversity:
pressurize_k L3 ratio vs brotli
0 (no pressurize) 2.164x +3.54% gap
1 2.199x +1.89% gap
2 2.251x +0.5% ahead
3 2.207x +1.51% gap

pressurize_k=2 is optimal for WildChat. For ShareGPT and LMSYS, pressurize_k=3 is optimal.

The optimal pressurization depth varies by corpus vocabulary diversity — more diverse corpora benefit from shallower pressurization to avoid dictionary dilution.

Final Results: L3 Beats Brotli on All Three Corpora
Verified across 3 independent corpora, 3 random seeds each:
Corpus GN L3 gzip-6 brotli-6 margin
ShareGPT 2.526x 2.145x 2.429x +4.0% vs brotli
LMSYS 2.401x 2.031x 2.291x +4.8% vs brotli
WildChat 2.251x 2.023x 2.240x +0.5% vs brotli

All three beat gzip by 11-18%. All three beat brotli. Results verified across 3 random seeds per corpus.

GN beats gzip on 100% of runs across all seeds and corpora. GN beats brotli on all three corpora when the window is sufficiently warmed.

Why This Works

Brotli ships with a 120KB static dictionary of common web phrases. It never changes. GN's sliding window learns the specific vocabulary of your data stream as it runs. LLM conversations have crystalline structure — repeated role markers, prompt scaffolding, tool call formats, JSON patterns, reasoning templates. After seeing a few thousand examples, GN knows these patterns better than any generic dictionary ever could.

The critical property: GN's compression ratio improves with stream length. Gzip and brotli are static — they cannot improve.

ShareGPT at 500 chunks: GN 2.304x brotli 2.363x (behind)

ShareGPT at 2000 chunks: GN 2.440x brotli 2.436x (pulls ahead)

ShareGPT at 5000 chunks: GN 2.517x brotli 2.429x (+3.6%)

The longer GN runs on a domain-specific stream, the wider the gap grows.

What Comes Next

The current warming uses sequential proximity — the last N chunks before the current one. The next level uses semantic similarity — retrieve the most topically related prior chunks via embedding search, regardless of when they appeared.

A conversation about JWT authentication should be warmed by other authentication conversations, not by whatever happened to come before it in the stream. This is Semantic Level 3, and it should further improve results on diverse corpora like WildChat where topic jumps are common.
Beyond that: dictionary compression (compress the dictionary itself, fractal self-similarity), cross-session persistence (window state survives restarts), and pre-trained domain dictionaries (ship a base window trained on 50k LLM conversations).

The goal is to make GN the brotli of LLMs — purpose-built, measurably better, and invisible infrastructure.

GN is MIT licensed. Code: github.com/atomsrkuul/glasik-core

npm: gni-compression@1.0.0

NLNet NGI Zero Commons Fund application #2026-06-023

Within 10% of gzip: What GN’s Semantic Compression Teaches Us

Buffer Overflow — Sun, 05 Apr 2026 04:20:54 +0000

When we first started building, the goal was never to make another gzip clone. Generic compression already does that job incredibly well.

The real question was different:

What happens if the compressor understands the shape of the data before it ever starts packing bytes?

That question led us from the original JavaScript prototype into Glasik Core, a Rust implementation focused on semantic tokenization, rolling vocabulary windows, and domain-aware preprocessing for message and agent streams.

This week we hit a milestone that feels small on paper but huge architecturally:

GN is now within 10% of gzip on every benchmark corpus we tested.

Not better. Not faster. Not “production solved.”

Just consistently close, which is exactly why this stage is exciting.

The benchmark reality

Current corpus results:

Corpus Glasik Core gzip Relative
MEMORY.md 1.849x 2.075x 89%
ShareGPT-1k 3.752x 3.945x 95%
Ubuntu-IRC-1k 2.122x 2.357x 90%

The most important one is ShareGPT-1k hitting 95% of gzip. That corpus is extremely close to the data GN was designed for:

Repeated assistant roles
Prompt scaffolding
Tool formatting
Structured JSON-like patterns
Recurring conversational templates

Even though we have not passed gzip yet, nearly matching it on LLM-native streams is a strong validation signal.

Why being close matters more than winning right now

The remaining gap is not where many would assume. The weak point is not semantic understanding anymore.

The weak point is the final entropy backend. gzip still has decades of advantage in:

Huffman tuning
Backreference heuristics
Lazy match parsing
Highly optimized bit packing
Mature DEFLATE edge cases

That last 5–10% is the part generic compressors are legendary at.

But the semantic layer is already doing the harder thing: understanding the structure of the stream before compression begins. That’s where the long-term leverage is.

The real architectural lesson

The simplest way to explain the difference:

gzip remembers bytes. GN remembers meaning.

As the rolling vocabulary fills, repeated structures stop being treated like raw strings and start being treated as stable semantic units. That includes:

Timestamps
Speaker roles
Repeated tool calls
Theorem blocks
JSON keys
Repeated prompt shells
Agent trace scaffolding
Channel metadata

Performance improves the longer the stream runs. Instead of relying only on a fixed byte-history window, GN reinforces the vocabulary of the domain itself. That’s the core bet.

Why Rust changed the debugging loop

The JavaScript prototype proved the idea. Rust made it possible to trust the measurements.

One concrete example: during corpus benchmarking we hit a rolling-frequency bug silently inflating token counts over long windows. Compression ratios looked “better,” but only because the vocabulary statistics were wrong.

The fix only became obvious because Rust forced us to reason explicitly about integer width, overflow behavior, and ownership boundaries inside the rolling state machine.

Fixing it tightened the corpus results and gave us confidence that the “within 10%” milestone is real, not a measurement artifact. That debugging loop alone justified the rewrite.

What makes this exciting now

The missing performance is now localized. We know exactly where the gap is:

Residual encoding
Entropy refinement
Better state models
Adaptive codon dictionaries
Specialized chat residual codecs

That is a much better place to be than wondering whether the entire idea works. The semantic layer is clearly competitive. Now it’s about tightening the backend until the semantic advantage outweighs gzip’s entropy maturity.

What’s next

Tonight’s most interesting work was deeper in the backend: we now have a reference-safe ANS entropy coder implemented from scratch in Rust, using the same family of techniques that powers zstd.

The current version uses correctness-first binary renormalization so we can prove round-trip behavior before optimizing. Next step: bit-level state refinement and faster renormalization transforms.

This work directly targets the exact 5–10% gap the benchmarks are still showing.

The path forward is finally clear:

Semantic understanding is already competitive
Entropy packing is the remaining frontier
The architecture now tells us exactly where to push

At this point, GN (our semantic agent layer) and Glasik Core (the compression engine) feel less like an experiment and more like a real compression architecture.

We Built Domain-Specific Compression for Messages. Here's What We Learned.

Buffer Overflow — Fri, 03 Apr 2026 17:35:19 +0000

Why gzip loses to custom compression on chat data — and what we learned building a lossless message codec from scratch.
The Problem
Message data is expensive at scale. Discord servers, Slack workspaces, OpenClaw chat logs — each message is ~500 bytes. Generic compression gets 2-3x. That's good but messages have structure generic algorithms ignore.
[2026-04-03T11:00:00Z] user: Hello, can you check the repo?
[2026-04-03T11:00:15Z] bot: Checking repository...
Timestamp format, role prefixes, platform names — these repeat thousands of times. A domain-specific dictionary front-loads that knowledge instead of discovering it slowly.
The Architecture
We built two layers that work together:
GN (Glasik Notation) — semantic compression. Extracts structure before compression, maps repeated values to IDs, recognizes message templates, reduces entropy before any algorithm touches the data.
GNI (Glasik Notation Interface) — transmission codec. Handles serialization, framing, integrity verification, wire protocol.
Together they form a complete pipeline. Neither is useful alone — GN without GNI has no reliable wire protocol, GNI without GN has no semantic advantage over gzip.
What We Shipped: GNI v1
Phase 1 delivers the foundation:

Canonical binary serialization (varint encoding)
Versioned frame format (backward compatible forever)
CRC32 integrity verification
100% lossless round-trip recovery verified on 2,000+ real messages
Zero external dependencies, 482 lines of JavaScript

Compression ratios from semantic tokenization are a Phase 2 deliverable. The tokenizer is currently stubbed — Phase 2 implements the domain-specific dictionary that gives GN its advantage.
The Bug We Caught
During validation on 1,038,324 real dialogue messages we hit a CRC32 mismatch:
Stored CRC: 1428394006
Computed CRC: -1889366573
Mismatch!
The bug: checksum computed over (header + payload) instead of just (payload). Three-line fix, proper unsigned arithmetic (>>> 0). Full corpus re-validated in 25 seconds. 6/6 tests passed.
Caught before production. Caught by us, not a user. This is why you validate before you ship.
Try It
javascriptconst GNLz4V2 = require('./src/gn-lz4-v2-complete');
const codec = new GNLz4V2();

const messages = [
{ templateId: 0, ts: 1743744000, author: 1, channel: 1, payload: 'hello world' },
{ templateId: 0, ts: 1743744001, author: 2, channel: 1, payload: 'how are you?' }
];

const result = codec.compress(messages);
const recovered = codec.decompress(result.compressed);
console.log(recovered.length + ' messages recovered losslessly');
npm test

37/37 passing

What We Are Not Claiming

Compression ratios are Phase 2. Phase 1 is serialization and framing.
Not production-proven at scale. Validated on our own systems.
No external users yet. Looking for third-party benchmarks and feedback.

What We Are Claiming

Solid foundation: lossless, versioned, integrity-verified
Real validation: 1M+ messages, caught our own bugs before release
Clear roadmap: Phase 2 connects GN semantic compression into GNI transmission layer
Applied for NLNet NGI Zero funding (application 2026-06-023) to deliver Phase 2

Links
GitHub: https://github.com/atomsrkuul/glasik-notation
License: MIT
If you compress messages and want to share results, open an issue. That kind of external validation is what makes this real.