Forem: Gideon Towolawi

The Compiler Maze

Gideon Towolawi — Sun, 24 May 2026 18:57:16 +0000

Why Compiler is a very breathtaking project for devs and single dev

I just got into the parser stage 2 days ago and the design was flowing like a breeze. Since my compiler is focused as a system programming compiler for my custom language, I was at the type checking (local type checking) phase.

This is where I created a util.hpp file in which I wrote a namespace containing polymorphic code as it is a global type system my compiler will use to verify type at compile time.

My compiler makes sure it does everything at compile time as it really benefits — creating a custom language means you're definitely tired of the scope of the current language compiler, so that's why we're here!

Type System

The compiler tries to be ambitious like Zig and more than it, as it supports variable width types from 1 to 65535 for both unsigned and signed. I won't lie, I learned the formula for signed and unsigned, but like I said, the compiler scope got in my way: how do I store 2^65535 - 1 for unsigned value range? As we famous graphics devs and codecs know, u8 is from 0 to 255. Wait, it's unrelated.

My Ambitions

I'm a rising video codec engineer but no traditional DCT, motion vector in my vision. I did try to invent a proprietary upscaler called GVEST (Geometry Vector Expansion Spatial Transform). It's better than bilinear by a few tens of dB in edge construction without blurring, and same dB on a noisy image. If you want to see the images, I will try to upload the graph and simulation results which AI did run Python code to generate, in another Non-Compiler Series of ours. So to cut the story short, I knew entropy coding and video codecs need not just runtime precision but memory precision. And don't get me wrong — Gemini AI helped me fix the formula, as fitting the above range results from the above formula is not my path and I didn't see it coming, and I don't want another hour of burnout. I want to add theoretical bit-size memory allocation to the value of that variable bits to memory, meaning if a u8 (which is 1 byte) and the value was 5 which doesn't take the whole 8 bits or byte, why waste memory? But it's non-trivial as unpacking them will be a big fight, but I'm still going that route.

What's still breathtaking

Won't lie — I'm very ambitious. I've got so much type system, DSA to optimize at compile time. I asked AI why no language optimizes tree-based DSA as I knew little about them. All my life from JS to C++ wasn't more than structs and arrays (with vector in C++'s case). I did learn DSA in my days as a JS dev, but C++ is different in everything: containers abstraction, STL, iterators, the .data() + index, end or the .begin(), .end() — it's everywhere in C++, and beginners and experienced devs definitely still go to the docs once in a while. And honestly, the docs aren't great too.

I went through serious burnout today as I want my language, even if C++-like, to have ownership and lifetime and memory validation, if you've seen my ambition above. It will also be friendly and clear with error messages, and I'm already proud of my lexer stage error messages on my string and char warnings and recovery on unterminated literals. It's really a breathtaking journey.

Closing it all up

If you liked the architecture and insights you've seen or haven't, I welcome you in the comments section, and if you want to subscribe to my Newsletter — though it's in early stage — become my subscriber on Substack: Substack

I'm happy building the project as I'm not looking at where I am but where it's going to be. I'm breaking the LLVM oath solo, and you can too. If you follow up, everything counts for me.

Why These and That in Every System

Gideon Towolawi — Sun, 24 May 2026 13:20:17 +0000

We all know about DSA—Data Structures and Algorithms. But that's not all there is to programming, no matter the field. I emphasize this especially in software engineering: DSA is almost never the fix on its own.

Today marks over a week and a half on my lexer since the beginning of May 2026, when I decided to build a compiler. I've gone through different choices, code patterns, rewrites. But you know what? I didn't read a single compiler book. No Compiler Dragon Book—the bible of the field. No PDF, no 2,000 pages just to experience someone else's experience into yours.

This article isn't saying you shouldn't read books. But if you're a systems engineer like me—and my experience is pretty low, roughly 2.5 years in C++ and years in programming—we know tools aren't always the answer. In fact, DSA isn't either. This isn't about toy projects. We're talking about real production-grade projects where every performance detail and line of code must be understood perfectly, not just written pretty.

While writing the lexer in C++, coming from Rust after ditching C++ for a while, I hated my comeback. Rust's ownership model and the compiler fighting me on every borrow? I ran from it. Went back to C++ where debugging a struct—which was my token object—meant just printing fields and moving on. No lifetime elision wars, no fighting the borrow checker because my token held a reference to source text that might outlive the lexer stream. In C++, I own my memory, I leak it, I fix it. The cost is visible immediately. In Rust, the cost was hidden behind compiler errors that were technically correct but solved someone else's problem, not mine. I didn't need zero-cost abstractions. I needed to see my token struct in a debugger without wrestling with the language first.

Those questions turned out to be simple. Deceptively so.

I noticed that while having only the basics of compilers—its stages, just the prior experience (in my case: Lexer → AST → IR → Codegen)—if I had actually asked the same questions as the first person who attempted it, about its architecture, what it solves, and the coding reality... because as we all know, design doesn't map well to code. In the worst case, if we try to find the ratio, we can use only the most abstract and subtle design—not overly architected, but rearchitected through code. This is a classic system design phase, but I realized it through grit.

So the problem, which you've probably guessed, is we want to solve two problems. First: no manual assembly. Second: high-level code and its own problem—which is, how do they get back to their root?

The best approach—not the only approach, but the best—was to use what we all call a lexer today: a stream-broken, tokenized, labeled source code.

But why? Why not parse straight from the character stream? Why add a whole stage just to label things?

Because the problem isn't "how do I read code." The problem is how do I transform high-level intent into machine execution without writing machine code by hand. And that problem has two parts: the human writes symbols, the machine needs instructions. The gap between them is where every cost hides.

So you ask the two questions. First: what do I actually need to know about this source? Not "what data structure should I use"—that's the DSA trap. You don't reach for a hash map because hash maps are fast. You reach for it when your problem is "I need to check if I've seen this identifier before in O(1)." The problem first. The structure second.

Second: where does the cost arrive? Design won't show you. Design is clean boxes and arrows. Code reveals it. When I wrote my lexer, I didn't hit the cost in the diagram. I hit it when I realized a recursive descent parser trying to backtrack over raw character streams was burning CPU on re-lexing the same identifier five times. The cost wasn't visible in the "Lexer → Parser" box. It was visible in the profiler, in the branching, in the cache misses from string comparisons.

That's when the engineering choice crystallized. The lexer isn't there because compilers "should have one." It's there because tokenization is the point where you pay the string cost once, then never again. You transform the variable-length, unpredictable, cache-unfriendly character soup into fixed-size, predictable, cache-friendly labels. That's not a fancy design pattern. That's solving the first sub-problem: how do I make the rest of the pipeline fast enough to be usable?

Now, why can't the parser live in the lexer? Why not just build the AST while tokenizing?

Because the lexer solves a linear problem. It walks left-to-right, one pass, no memory of nesting depth. The parser solves a non-linear problem. Take a complex recursive function—nested lambdas, match arms, closures capturing environments. The lexer sees this:

FN  IDENT  LPAREN  IDENT  COLON  IDENT  RPAREN  LBRACE  MATCH  IDENT  LBRACE...

Flat. Stateless. A conveyor belt of labels. It has no stack to track "this brace closes the lambda, not the match arm." It cannot, by design, handle recursion because recursion requires a tree, and trees require a builder that remembers where it is in the structure.

The parser sees the same tokens, but at depth 3, inside a match arm, inside a closure, knowing exactly which brace closes which scope. The lexer sees RBRACE and thinks "end of something." The parser knows "end of match arm, inside lambda, inside function." That's the difference between a label and a structure.

Could you force the lexer to track a stack? Could you make it "smart"? Sure. Now you've merged two problems into one stage. And when that recursive function nests ten levels deep, your lexer isn't tokenizing anymore—it's predicting, it's branching, it's holding state that grows with input complexity. The cost arrives in CPU exhaustion, in stack overflows, in the maze of entry and exit points where you can't tell if you consumed the full path or just the happy path. Burnout in code, not in design.

And here's what they don't tell you: the burnout hits the engineer before it hits the machine. The junior who reads three compiler books before writing a line of code. The team that designs for 1M users at 100 users. The developer who builds a distributed system because "microservices are best practice" when a monolith would have shipped in a week. The cost arrives in the human first—in the paralysis of premature abstraction, in the exhaustion of solving problems you don't have yet. That's why these stages exist. Not because a book prescribed them. Because that problem needed that solution.

The AST exists because some problems are inherently non-linear, and pretending they're linear doesn't make them linear—it makes them expensive. The parser is where you accept that cost upfront, where you build the explicit tree, where you make recursion manageable by giving it a structure that matches its nature. The AST isn't standard because every language has different non-linear truths. SIRL's match arms with explicit types need different nodes than C's switch statements. The AST shape is dictated by what your language actually does, not by what some book says it should look like.

Same for the IR. Same for the lexer. We know the stages—Lexer → Parser → AST → IR → Codegen—but the implementations diverge because the problems diverge. A JIT compiler skips the AST for hot paths because its problem is latency, not optimization. An embedded compiler uses a different IR because its problem is register pressure, not vectorization.

This is the grit. The design phase gives you the illusion that you've solved it. The code phase reveals you haven't. Design doesn't map to code. It maps to intention. Code maps to reality. And reality is where the cost lives—in the branching, in the memory layout, in the cache lines, in the non-linear paths that design documents politely ignore.

Books give you answers to questions you don't have yet. The problem is yours; the answer should be too. Not from blind DSA application. Not from following a chapter. From asking what you actually need to know, and having the guts to let the code show you where the cost lives.

So the lexer stays dumb. The parser stays recursive. The AST stays language-specific. Not because it's elegant. Because each stage solves exactly one problem, pays exactly one cost, and exposes exactly one interface to the next stage. That's systems engineering. Not DSA for DSA's sake. Problem first. Cost second. Structure last.

And when you finally lower to IR and codegen to assembly, you've answered both original problems. No manual ASM. High-level code returned to its roots. But the path there wasn't found in a book. It was found by asking what the problem actually is, where the cost actually arrives, and having the grit to let code—not design, not someone else's experience—tell you the truth.

Two questions. Everything else is just typing.

The Compiler: Heart and Tools of All Software

Gideon Towolawi — Mon, 18 May 2026 16:36:00 +0000

The Compiler: Heart and Tools of All Software

Every program you have ever run — your operating system, your browser, the app that woke you up this morning, the firmware in your coffee machine — was once just text. Human-readable text. Ideas typed by someone who understood a problem well enough to describe its solution.

But computers do not read ideas. They read instructions. Binary. Electrical signals that mean nothing without precise interpretation.

The bridge between human intention and machine execution is the compiler. It is the most consequential piece of software ever invented. Without it, computer science as we know it does not exist.

What Computer Science Would Be Without Compilers

Imagine a world where every programmer writes raw machine code. Not assembly — actual binary. Opcodes and operands encoded by hand. Every program is a miracle of patience, and every bug is a nightmare of hexadecimal archaeology.

In this world:

Software development is artisanal, not industrial. A single application takes years.
Portability is a myth. Every CPU architecture requires rewriting everything from scratch.
Abstraction dies. There are no functions, no types, no modules — just raw memory and jumps.
Security is impossible. Human minds cannot track the state of thousands of registers and memory locations simultaneously.

Computer science without compilers is not computer science. It is digital craftsmanship at the limit of human endurance. The compiler is what lets us think in concepts instead of circuits.

The Compiler as a Pipeline of Principles

A compiler is not a single program. It is a pipeline of transformations, each stage reducing complexity and increasing structure. The quality of a compiler depends entirely on the principles baked into each stage.

Most people know the classical stages:

Lexer — characters → tokens
Parser — tokens → syntax tree
Semantic Analysis — syntax tree → validated intermediate representation
Optimization — IR → faster IR
Code Generation — IR → machine code

But this description misses the point. The stages are not just mechanical steps. They are guardians of meaning.

Stage 1: The Lexer — Dumb by Design

The lexer is where principles begin. Its job is simple: convert a stream of characters into a stream of tokens. int, x, =, 42, ;.

A bad lexer tries to be smart. It merges = = into ==. It strips whitespace because "it doesn't matter." It reconstructs strings and throws away the original quotes.

A principled lexer stays dumb. It emits raw tokens with precise spatial information — where each token starts, where it ends, what line, what column. It does not interpret. It does not merge. It does not discard.

Why? Because semantics belong to the parser. The lexer cannot know whether :: is a scope resolution operator or two separate colons in a ternary expression. It cannot know whether whitespace inside a string literal is significant or decorative. By staying dumb, the lexer preserves all information for downstream stages to make informed decisions.

The token structure I use reflects this:

struct Token {
  TokenType type;      // what kind of token
  std::string lexeme;  // the raw text
  size_t line;         // visual line for errors
  size_t column;       // visual column for errors
  size_t span_to;      // exclusive byte offset in source
};

span_to is the critical field. It lets the parser reconstruct multi-token operators. It lets the formatter preserve original spacing. It lets the LSP highlight exact ranges. The lexer does not use this information — it merely records it, faithfully and without interpretation.

This is the first principle: reduce at the right stage, never earlier.

Why Principles Matter More Than Performance

It is tempting to optimize the lexer. Merge tokens early. Strip separators. Compress the token stream. These optimizations feel productive.

They are traps.

Every piece of information discarded in the lexer is a piece of information that cannot be recovered in the parser, the semantic analyzer, or the code generator. A stripped space cannot be restored for formatting. A merged == cannot be split back if the parser needs to report "unexpected token = after =". An interpreted string literal loses the original escape sequences.

The cost of a "smart" lexer is permanent information loss. The cost of a dumb lexer is a slightly larger token stream — trivial to optimize later, impossible to reconstruct if deleted early.

This principle extends through every compiler stage:

Parser: Validate syntax strictly, but do not constant-fold yet
Semantic Graph: Resolve types and ownership, but do not lower to machine concepts yet
IR: Represent semantics faithfully, optimize only when correctness is provable
Backend: Generate code for the target, but never modify semantic truth

Each stage has one job. Each stage does that job completely. No stage does another stage's work prematurely.

Building Correct by Construction

The compiler is not just a tool. It is a proof system. It proves that your program means what you think it means, that it will not leak memory, that it will not access invalid lifetimes, that it will execute deterministically across architectures.

This is not about being clever. It is about being correct by construction.

What Comes Next

Over the next weeks, I will document each stage of compiler construction in detail:

Why the lexer stays dumb and what that enables
How the semantic graph builds structure from raw tokens
What compile-time invariants mean for systems programming
How to translate semantics into machine resources without losing correctness

If you are building compilers, thinking about language design, or simply curious about how software becomes real, subscribe to the newsletter. I share what I learn, what I get wrong, and how to avoid the traps I fall into.

The compiler is the heart of software. Understanding it is understanding how we turn thought into action.

Building a systems language that writes like C++ and proves safety like Rust, without the mental overhead. Join the newsletter for weekly deep-dives on compiler architecture, language design, and systems programming.

I'm Building a Multi-Target Compiler Backend from Scratch — No LLVM, No Crutches

Gideon Towolawi — Sun, 17 May 2026 20:49:04 +0000

Hi there, i'm Gideon. Roughly 1.5 years of writing C++ from the ground up — ray tracers, video codecs, and now a compiler. No frameworks. No LLVM. Just me, the hardware manuals, and a lot of wrong turns.

This post starts a series where I document the build in real time. I'm currently in the parser stage. By the end, I want a compiler that emits x86-64 and SPIR-V from a C++-like language, with SIMD vectorization and security-hardened codegen baked in.

What I'm Actually Building

Not a programming language. A compiler backend toolkit — the part that turns intermediate representation into fast machine code across multiple targets.

The pipeline:

Source → Parser → AST → SSMOL (HIR) → MREL (LIR) → x86-64 / SPIR-V / ARM64 / RISC-V / WASM

MREL is my target-agnostic low-level IR. It knows about virtual registers, stack slots, and machine operations — but not physical register names. The backend handles that per-target.

Why Not Just Use LLVM?

LLVM is 4 million lines of code. It solves everyone's problem and no one's perfectly. I need:

Fine-grained control over SIMD width selection per target
Constant-time crypto primitive emission with secret register annotations
Security obfuscation passes (control flow flattening, opaque predicates)
A codebase I fully understand and can license

Building from scratch is slower. But I own every decision.

Where I Am Right Now

Parser stage. Hand-written recursive descent. C++-like syntax with:

Functions, structs, basic types
Ownership semantics (borrowed from my Rust phase, simplified)
Explicit SIMD types (v128, v256, v512)

The parser emits an AST that gets lowered to SSMOL — my high-level IR that knows about types, ownership, and semantics.

What's Next

SSMOL → MREL lowering (types to sizes, structs to offsets, control flow to basic blocks)
MREL → x86-64 backend (register allocation, instruction selection, ELF emission)
One working program: compile, link, run main() that returns 42

Then SPIR-V compute kernels. Then the rest.

What I'll Write About

Each stage, when I hit it. The problems that took me three days to solve. The specs I wrote to keep myself honest. The wrong assumptions that cost me a week.

Not polished tutorials. Build logs from someone actually building.

Follow This Series If

You work in systems, compilers, or graphics
You're curious what "building from scratch" actually looks like
You want to see if I crash or ship

Substack: (https://ayndlr.substack.com)

Closing

This is post 1 of however many it takes. Next post: parsing expressions with operator precedence and why I gave up on Pratt parsing.

Follow for the crash or the ship. Either way, it's real.