Forem: Jacob

The Art of Agents: Sun Tzu's Principles for Building Agentic AI Systems

Jacob — Mon, 30 Mar 2026 14:42:59 +0000

By Jacob Verhoeks, March 2026

The cost of building software has collapsed. Claude Code ships features in minutes. Codex opens pull requests while you sleep. Cursor and Windsurf turn IDEs into conversation partners.

The cost of knowing what to build has not changed at all.

That gap is where The Art of Agents lives. Thirteen chapters mapping Sun Tzu's Art of War to the discipline of building agentic AI systems that actually work in production.

The Thesis

You don't spec then build. You build to discover the spec.

When agents can scaffold a full-stack application in an afternoon, the question is no longer "can we build this?" It's "should we build this, and how do we know if it worked?"

The answer is the specification. But the spec comes after the experiment, not before it. You build a quick version, expose it to real users, collect feedback, and then write the spec that captures what you actually learned. The spec is not the starting gun. It's the trophy you earn by running the race.

The Thirteen Chapters

Each chapter opens with a Sun Tzu principle, teaches the concept through real examples and anti-patterns, and closes with a field report drawn from actual experience.

Chapter	Sun Tzu	Modern Application
I	Laying Plans	Spec-Driven Design
II	Waging War	Token Economics
III	Attack by Stratagem	Composability Over Monoliths
IV	Tactical Dispositions	Defence Through Schema
V	Energy	The Multiplier of Tool Design
VI	Weak Points and Strong	Observability as Intelligence
VII	Manoeuvring	Adaptive Orchestration
VIII	Variation in Tactics	Multi-Agent Patterns
IX	Army on the March	Deployment and Operations
X	Terrain	Navigating the Enterprise
XI	Nine Situations	Failure Modes and Recovery
XII	Attack by Fire	When NOT to Use AI
XIII	Use of Spies	Feedback Loops and Continuous Learning

Five Principles That Matter Most

1. The spec is the cause of design, not its effect.

A prompt is a wish. A spec is a contract. The prompt tells the agent what you want. The spec tells the entire system what correct behaviour looks like. You can change the model and the spec should still hold.

AWS launched Kiro with spec-driven development at its core. The idea was sound, but early adopters reported it felt like a straitjacket. OpenSpec takes a different approach: the spec is a living document that grows with the code, not a gate you pass through.

2. Design for distrust.

What is not permitted is forbidden. Don't rely on the model to behave correctly. Define the contract. Enforce it mechanically. Tool whitelisting, schema validation, output parsing, hard limits on tokens and iterations. Your agent will improvise. Your system should not let it.

3. The model's job is to decide, not to compute.

A bad tool returns 5,000 rows of JSON and expects the model to find the answer. A good tool accepts precise parameters, does the heavy lifting in deterministic code, and returns only what the agent needs to reason about. When the tools are right, the agent appears brilliant. When the tools are wrong, no amount of prompt engineering can save it.

4. Know when NOT to use an agent.

LLMs are fire. Powerful in the right conditions, catastrophically wasteful in the wrong ones. If a SQL query will do, write the query. If a rule engine will do, write the rules. The architect who puts an LLM in every code path is the general who sets fire to his own supply lines.

5. The learning is the product.

Every experiment enriches the commons. A composted experiment produces a learning. That learning feeds the next idea. The cycle accelerates. Dead experiments are the richest soil. The code is gone, but the learning persists. And learnings, unlike code, get more valuable over time.

The Cycle

Specs drive design. Design produces agents. Agents produce traces. Traces inform evaluation. Evaluation updates specs. Each pass makes the specs sharper, the agents more capable, the tools better calibrated.

This is not continuous deployment. This is continuous learning.

Three Decisions, No Waste

Every experiment ends with a human decision:

Promote: the idea proved out. Generate a full spec, build the production version.
Pivot: wrong angle, but something is there. Revised hypothesis, new experiment.
Compost: the idea did not work. The code is discarded. The learning persists.

A composted experiment with a reused learning is more valuable than a promoted experiment that taught nothing new.

The Anti-Patterns

Thirteen chapters, thirteen ways to get it wrong:

The Vibes-Driven Agent -- no spec, no acceptance criteria. The agent guesses your intent.
The Full-Context Agent -- loads everything into context "just in case." Slow, expensive, unfocused.
The God Prompt -- one 4,000-word prompt handling every responsibility.
The Trusting Orchestrator -- no validation between agents. Hallucinations propagated as fact.
The Firehose Tool -- returns everything, lets the model sort it out.
The Black Box Agent -- works in demo, fails in prod, nobody knows why.
The Premature Swarm -- five agents where one would do.
The LLM Hammer -- everything looks like a reasoning problem.

If you recognise three or more, the book is for you.

Read the Preview

The full book is in early access. The preface and Chapter 1 (Laying Plans: Spec-Driven Design) are available as a free preview.

Download the free preview (PDF) -- Preface + Chapter 1, covering the Five Constants of Agentic Design, why prompts are not specs, and the OpenSpec format.

Explore the full framework -- The interactive website with all thirteen chapters, the manifesto, the cycle, and the fundamental inversion.

Get notified when the book launches -- Visit the website, click "The Book" tab, and hit "I'm Interested."

Jacob Verhoeks is a principal engineer building data platforms and agent architectures. The Art of Agents grew out of a framework mapping Sun Tzu's 13 chapters to agentic AI principles, which became an essay, then a website, then this book.

The AI Development Workflow: A Complete System for Working with AI Agents

Jacob — Fri, 27 Feb 2026 16:42:15 +0000

A continuous cycle of ideation, planning, execution, and refinement — all driven by issues, feedback loops, and a persistent memory layer.

The Big Picture

Working with AI isn't about asking a chatbot to write code. It's about orchestrating a self-improving system where AI agents brainstorm, plan, build, review, audit, and monitor — connected by issues as the connective tissue.

Here's the full cycle:

                        ┌──────────────────────────────┐
                        │         📡 MONITOR           │
                        │    Production Signals        │
                        └──────┬───────────────┬───────┘
                    insights ↓                 ↑ ship
              ┌──────────────────┐    ┌───────────────────┐
              │  💡 BRAINSTORM   │    │     ✅ REVIEW     │
              │  Ideate & Design │ ◄──┤  Quality Gates +  │
              └────────┬─────────┘    │  Code Review      │
            feasibility ↕             └────────┬──────────┘
              ┌────────────────────┐            ↑  fix loop
              │   📋 PLAN          │    ┌────────────────────┐
              │   Issues & DAG     │    │    ⚡ EXECUTE       │
              └────────┬───────────┘    │  Parallel Agents   │
                       ↓                └────────┬───────────┘
              ┌────────────────────┐             ↑
              │   ⚖️ TRIAGE        │─────────────┘
              │   Prioritize       │
              └────────────────────┘
                       ↑ new issues
              ┌────────────────────┐
              │   🔍 AUDIT         │
              │   Deep Analysis    │
              └────────────────────┘

                  🧠 MEMORY (always active)
            context · rules · lessons learned

The Essential Flow (Start Here)

If you're new to AI-driven development, here are the five core steps. Master these first, then layer on the advanced concepts below.

Step 1: Brainstorm — Design the Feature with AI

"I want to add something like this — help me design it."

Collaborate with AI to explore ideas, validate approaches, and shape the feature before any code is written. This is where you describe your intent in plain language and let AI help you think through edge cases, UX, and architecture.

Step 2: Plan — Create Issues & Split the Work

Create a parent issue for the feature. AI analyses your codebase and breaks it into linked sub-issues — each one a clear, scoped piece of work with acceptance criteria.

Feature: User Dashboard
├── Issue #101: API endpoint for user stats
├── Issue #102: Dashboard React component
├── Issue #103: Caching layer for stats
└── Issue #104: E2E tests for dashboard

Step 3: Execute — AI Agents Build Each Issue

Assign issues to AI agents. Each agent writes code, tests, and docs for its scoped sub-issue.

spawn team → issue: #101 (API endpoint)
spawn team → issue: #102 (Dashboard component)

Step 4: Review — Validate Tests & Code

Check all tests pass, review code quality. If something's wrong, loop back to Execute. Once everything is green, commit and create a PR that closes the related issues.

The feedback loop: Review → Execute → Review (repeat until all checks pass).

Step 5: Audit — System-Wide Check

Audit the entire system for security, performance, architecture, and best practices. Create prioritized issues for anything found — these feed back into Plan.

The feedback loop: Audit → Plan (new issues enter the next cycle).

The Full System (Level Up)

Once you're comfortable with the core loop, these additions make the system dramatically more powerful.

🆕 Triage: Prioritize Before You Execute

Between Plan and Execute, add a triage step. Not all issues are equal — weigh each by:

Severity — Is this a security hole or a nice-to-have?
Effort — Can an agent do this in minutes or hours?
Business impact — Does this affect users or just internal code?
Dependencies — What blocks what?

AI ranks and sequences the work. You approve. This prevents low-priority tasks from consuming agent time while critical bugs sit in the backlog.

⚡ Parallel Agent Orchestration

Your issues form a dependency graph (a DAG — directed acyclic graph). Instead of running them one by one:

The orchestrator finds all leaf nodes (issues with no blockers)
Assigns them to agents simultaneously
As each completes, newly unblocked issues are dispatched

     #101 (API) ──────────┐
                          ├──→ #104 (E2E tests)
     #102 (Component) ────┘

     #103 (Cache) ← independent, runs in parallel

This dramatically reduces wall-clock time compared to sequential execution.

🚧 Automated Quality Gates

Before any human sees the code, automated checks must all pass:

Gate	What It Checks
Linting	Code style, formatting
Type checking	Type safety (TypeScript strict)
Test suite	All unit + integration tests
Security scan	Known vulnerabilities, secrets
Coverage	Minimum threshold (e.g. ≥ 80%)

This filters out ~80% of issues before review even begins. The human/AI review then focuses on logic and architecture rather than catching surface-level problems.

📡 Monitor: Close the Loop with Reality

After shipping, the system doesn't stop. Monitor:

Error rates and crash reports
Latency (P95, P99)
User behaviour and feedback
Performance regressions

Anomalies auto-generate issues with full context. Weekly insight digests surface patterns. This feeds back to Brainstorm, grounding the next cycle in real-world data instead of assumptions.

The feedback loop: Monitor → Brainstorm (production insights inform what to build next).

🧠 Project Memory: The Context Layer

This is the secret weapon. A persistent knowledge base that every phase reads from and writes to:

Architectural decisions (ADRs)
Coding conventions and patterns
Past audit findings and how they were resolved
Lessons learned and edge cases
Team preferences and style guides

Every audit finding updates the memory. Every resolved bug enriches it. AI agents consult this before writing any code — so the same mistake is never made twice.

All Feedback Loops

The real power of this system is in the loops. Each one creates a self-correcting mechanism:

Loop	Trigger	Effect
Review → Execute	Failed tests or code issues	Agents fix and resubmit with specific failure context
Audit → Triage → Plan	Security, performance, or architecture findings	New prioritized issues enter the next cycle
Monitor → Brainstorm	Production anomalies or user feedback	Real-world data grounds the next ideation cycle
Brainstorm ↔ Plan	Feasibility concerns during planning	Design gets rethought before any code is written

Getting Started

You don't need to implement everything at once. Here's a progression:

Week 1: Start with the 5-step essential flow. Use issues as your connective tissue.

Week 2: Add quality gates (linting + tests as automated checks before review).

Month 1: Introduce parallel execution — let multiple agents work on independent issues.

Month 2: Add the monitor phase and start feeding production data back into brainstorm.

Ongoing: Build your project memory. Every cycle, it gets smarter.

The goal isn't to replace your judgment — it's to amplify it. You steer. AI executes. Issues connect everything. And the system gets better with every cycle.

From Extension to Orchestration: Who Wins (and Who Gets Left Behind) in the Claude 4.6 Era

Jacob — Mon, 16 Feb 2026 14:51:28 +0000

I published this morning "From Dark Flow to Real Momentum: Why Claude Opus 4.6 Feels Like an Extension of Me" on February 16, 2026, just days after Anthropic dropped Opus 4.6 on February 5 with its game-changing upgrades: native agent teams in Claude Code.

The response has been possitive and other developers sharing similar "aha" moments. Many echoed the core feeling: for those of us with ADHD tendencies (easy distraction, boredom-triggered abandonment, hyperfocus on novel problems but hatred of maintenance), this isn't just faster coding—it's preserved momentum, turning what used to be frustrating grind into sustained, rewarding flow. The agent handling refactoring, tests, docs, and GitHub issue cleanup in minutes (or spawning teams to resolve top blockers in parallel) keeps us in the creative zone instead of derailing into procrastination or "dark flow" traps.

But the conversation has evolved quickly in these early days of 2026. Readers are asking: Who else benefits this much? And critically—who risks getting left behind as agentic tools mature? Is this augmentation for everyone, or are some workflows (and engineers) facing real replacement?

The Momentum Advantage: ADHD & Creative Types Pull Ahead

The original piece focused on my experience: ADHD-like wiring makes traditional coding painful when tedium hits. Opus 4.6 flips that by reliably owning the boring 70–80% (refactors, lint fixes, test writing, multi-file changes, even parallel issue resolution via agent teams). Result? Longer productive sessions, fewer abandoned projects, quick dopamine from visible closures ("top 10 issues fixed → beta ready"), and that rare "extension of me" sensation where the tool feels like an externalized executive function—planning, iterating, self-debugging—so I stay stimulated on high-value creation.

This advantage seems especially pronounced for momentum-driven developers: prototypers, side-project builders, open-source maintainers, or anyone whose flow thrives on novelty and quick iteration rather than linear predictability. We're shipping 3–10× faster, completing more, and burning out less.

The Flip Side: Task-Oriented Engineers Face the Biggest Shift

The classic ticket → analyze → solve → repeat loop—prevalent in enterprise backend teams, legacy maintenance, regulated industries (finance/healthcare), or sprint-driven orgs—looks very different now. Opus 4.6's agent teams take that exact loop end-to-end: analyze repo/issues, plan steps, implement across files, run/fix tests, commit/PR with explanations, escalate only when needed.

In forward teams, this isn't always "making you faster"—it's commoditizing the hands-on execution layer. Signals from early 2026 (Anthropic reports, developer forums, productivity anecdotes):

Junior/mid-level ticket roles are shrinking fast; one orchestrator + agent swarm replaces squads.
Pure task-oriented work (bounded, well-defined tickets with low architectural judgment) gets automated reliably, leading to headcount reductions in mature setups.
If your daily value is reliably closing discrete tickets manually, the role narrows or vanishes in agent-heavy environments.

This isn't total obsolescence—human judgment remains essential for business nuance, security/compliance, edge cases, architecture decisions, and final accountability. But the "doer" part is increasingly replaceable, creating a visible velocity gap: momentum-driven devs soar while linear ticket-crunchers (especially those resisting tools) fall behind.

Strategies to Thrive: From Implementer to Orchestrator

The divide now is intentional directors vs manual holdouts. Here's how to evolve, whether you're momentum-driven or task-oriented:

Shift Upward in the Loop

Become the definer/reviewer: Craft precise prompts with goals, constraints (security/style/compliance), checkpoints, and success criteria. Example: "Analyze Jira ticket ABC [paste], propose multi-step plan, spawn agent team for parallel execution (tests, refactor, docs), commit per phase, flag ambiguities for review." You close more tickets/day with the same satisfying "done" hits, but at higher altitude.
Master Agent Teams & Tooling

Leverage Opus 4.6's native teams: Spin up parallel sub-agents (e.g., one for frontend, one for tests, one for research). Build custom rules/configs to enforce org standards. Chain models (Opus for planning, Sonnet/Haiku for speed). Start on low-risk tickets to build confidence.
Own the Irreplaceable

Focus on judgment calls agents still miss: subtle business logic, security reviews, scalability trade-offs, legacy quirks. Be the accountable owner who signs off—builds reputation in shrinking teams.
Guard Against Dark Flow

Review diffs rigorously, scope prompts tightly, audit maintainability periodically. Use agents for grunt + exploration, but manually code core architecture when deep understanding matters.
Grow Your Edge

Ship side projects faster to build portfolio. Become your team's "agent whisperer" (share workflows, optimize prompts). Network in communities. Target senior/lead roles emphasizing direction, mentoring (humans + agents), and strategy—demand is rising.

We're only ~10 days into Opus 4.6's public availability, and agent teams are still in research preview—things will accelerate fast. The momentum types are already thriving; task-oriented folks can join by owning the higher-level orchestration instead of fighting the change. Try one scoped agent experiment this sprint: paste a real ticket, let the team handle analyze-to-solve, and review the output. It might spark that same "extension of me" feeling—and keep you ahead in whatever comes next.

From Dark Flow to Real Momentum: Why Claude Opus 4.6 Feels Like an Extension of Me

Jacob — Mon, 16 Feb 2026 09:03:09 +0000

The recent article from fast.ai, titled "Breaking the Spell of Vibe Coding" by Rachel Thomas (published January 28, 2026), raises an important caution about "dark flow"—a deceptive, addictive state where using AI coding tools feels like productive flow, but actually leads to superficial engagement, hidden technical debt, unmaintainable code, and even slower real progress. Drawing parallels to gambling addiction (where "junk flow" or "dark flow" tricks people into chasing illusory wins), the piece warns that over-relying on AI-generated code—especially "vibe coding" of vast, complex, human-unreadable blobs—can create a false sense of accomplishment while stunting skill growth and leading to disastrous long-term outcomes.

It's a timely critique, especially as tools like advanced Claude models have become incredibly capable. Yet my own experience tells a somewhat different story: for me, these AI tools have been a massive enabler, not a trap.

I suspect I have some ADHD tendencies—I get easily distracted, and traditional coding often involves long stretches of boring, tedious work that kills momentum. Things like maintenance, refactoring messy code, writing documentation, or debugging obscure edge cases used to derail me completely. I'd procrastinate, get stuck, or lose interest in the project altogether.

But lately, especially with Claude Opus 4.6 (and similar frontier models), the dynamic has flipped. The AI handles the "boring" parts so effectively that it keeps me engaged and in the creative zone far longer. It feels like a genuine extension of myself: it solves complex, novel issues that previously would have required hours of searching (often yielding just one obscure GitHub repo or blog post as the only lead), implements the latest techniques, and lets me stay focused on what I actually enjoy—building features, experimenting with ideas, and iterating toward something useful.

A concrete example: my backend API starts getting messy after weeks of rapid iteration. In the past, refactoring it—making it extensible, secure, following best practices, updating the frontend accordingly, refreshing docs, and running full end-to-end tests—was a dreaded chore I'd delay as long as possible. Now I can simply prompt:

"Time to refactor my backend API: make it extensible, secure, and follow best practices. Spawn a team to design the refactor, adjust the frontend where needed, update documentation, run all E2E tests, and confirm everything still works."

In about 15 minutes (time for a cup of tea), it's done. When I come back, the boring maintenance is handled, the code is cleaner, and I have fresh energy to dive into new features even faster. It's not just speed—it's preserved momentum and reduced friction.

This extends beyond pure coding into project management too. Want to prepare a beta or feature release covering specific use cases (A, B, C, D) while ensuring security, compliance, and performance? I can say:

"I want to create a beta/feature release covering use-cases A, B, C, D—secure, compliant, performant. Create a GitHub issue for this release, then analyze the repo and create labeled GitHub issues for any problems found that block the release."

Suddenly I have a clear overview of open issues. Then:

"Spawn a team to fix the GitHub issues for release #XX. Start with the top 10 and continue until resolved. Ensure every issue has commits, PRs, comments, test results, and is closed."

What used to be hours (or days) of tedious loose-end tying becomes ~30 minutes of mostly automated work. The "boring" cleanup that blocks shipping is now handled, freeing me to focus on high-value creation.

Has this changed my work? Definitely. Can I work without it? Yes, but it's hard—I'd be much slower, more frustrated, and probably complete far fewer projects. Does it make me faster, better, and happier while building? Absolutely yes.

Am I addicted? Not yet. The key difference, I think, is intentional use: I treat the AI as a powerful collaborator for the grunt work and novel problem-solving, not as a replacement for thinking or architecture. I still review, understand, and own the core decisions. When used this way, it sustains real flow and growth rather than the dark flow of endless vibe coding.

The fast.ai piece is right to sound the alarm—blind over-reliance is risky. But when AI augments human strengths instead of substituting for them, it can be transformative. For builders like me who thrive on momentum over perfectionism, tools like Claude Opus 4.6 aren't a spell to break; they're a spell worth casting carefully.

Efficient Agentic AI Development Guide (begin 2026)

Jacob — Mon, 02 Feb 2026 16:32:41 +0000

A practical guide for working effectively with AI coding agents, especially Claude Code as of begin 2026. Things keep changing a fast.

Core Principles

1. Model Intelligence = Less Prompting Required

Smart models like Claude Opus 4.5 understand context and intent with minimal instruction. Be clear and concise rather than over-explaining. Trust the model's reasoning capabilities.

Do this: "Add authentication to the API"

Not this: "I need you to add authentication. First, you should create a middleware function. Then you need to check for tokens. Make sure to validate them properly..."

2. Prevent Context Degradation

Long conversations accumulate noise, outdated information, and conflicting instructions. This "context rot" reduces effectiveness dramatically.

Solutions:

Start fresh conversations for new tasks
Summarize and restart when threads get long (>50 exchanges)
Use separate sessions for unrelated problems
Archive completed work and begin clean for next feature

Warning signs of context rot:

Agent repeating previously solved problems
Inconsistent responses to similar questions
Ignoring earlier agreements or decisions
Slower response times or confused outputs

3. Single Task Focus

Agents work best with clear, isolated objectives. Mixing multiple unrelated tasks creates confusion and reduces quality.

Do this: One agent session per feature/bug

Avoid: "Fix the auth bug, add logging, refactor the database layer, and update the docs"

For multiple related tasks: Complete them sequentially in the same session, with clear transitions between each.

Workflow Pattern: Brainstorm → Plan → Execute

For features and complex issues, use this three-phase approach:

Phase 1: Brainstorm

Explore the problem space
Generate alternative approaches
Identify edge cases and dependencies
Surface hidden complexity early

Example: "Let's brainstorm approaches for rate limiting our API. Consider both user-based and IP-based limits, distributed systems, and cost."

Phase 2: Plan

Convert insights into concrete tasks
Break down into manageable units
Establish clear success criteria
Create logical execution order

Example: "Based on our brainstorm, create a plan with specific tasks for implementing Redis-based rate limiting."

Phase 3: Execute

Critical: Start a NEW session for each task with clean context
Reference the plan but execute one task at a time
Each task gets full agent attention without baggage from brainstorming

Why separate execution? The brainstorming phase generates many ideas, alternatives, and dead ends. Carrying this into execution clutters the context. Clean slate = better focus.

Agent-Assisted Debugging

Enable powerful automated debugging by giving agents access to system state:

Provide Diagnostic Tools

MCP servers for log access
Commands to view service status
Scripts to dump relevant state
Direct access to error tracking systems

Effective Debugging Sessions

# Good: Agent can self-serve information
Agent has access to: logs via MCP, kubectl commands, error tracking API

# Less effective: Manual information relay
You: "What does the error say?"
Agent: "I need to see the logs"
You: [copies logs]
Agent: "Can you check the database?"
You: [runs query, pastes results]

Example MCP Tools for Debugging

read_logs - Tail application logs
get_metrics - Fetch system metrics
query_db - Run diagnostic queries
list_processes - Check running services
get_config - View current configuration

Result: Agents can iterate through debugging hypotheses autonomously, dramatically faster than back-and-forth exchanges.

Repository Structure Best Practices

Include an AI-Friendly README

Create .claude/README.md or docs/ARCHITECTURE.md with:

Project structure overview
Key design decisions and why
Common commands and workflows
Where to find configuration
Testing approach

Provide Tool Access

Document in your repo root:

# .claude/tools.md

## Available Commands
- `npm run dev` - Start development server
- `npm test` - Run test suite
- `./scripts/db-reset.sh` - Reset local database

## MCP Servers
- Logs: Access via filesystem MCP to `./logs/`
- Database: PostgreSQL MCP configured for local DB

Context Documents

Small, focused files the agent can reference:

CONVENTIONS.md - Code style, naming, patterns
TESTING.md - How to write and run tests
DEPLOYMENT.md - Release process and environments

Quick Reference

Situation	Solution
Starting a new feature	Fresh conversation, single focus
Thread feels confused	Summarize progress, start new session
Multiple related tasks	Sequential execution, one at a time
Complex feature	Brainstorm → Plan → Execute (new sessions)
Debugging is slow	Add MCP tools for log/state access
Agent seems "forgetful"	Context rot - restart with summary
Need to explain project	Add `.claude/README.md` with architecture

Anti-Patterns to Avoid

❌ Marathon sessions - 100+ message threads accumulate noise

❌ Task mixing - "While you're at it, also..." leads to half-finished work

❌ Over-prompting - Smart models don't need hand-holding

❌ Manual debugging - Give agents tools instead of playing telephone

❌ Unclear success criteria - "Make it better" vs. "Reduce load time to <200ms"

❌ Ignoring context rot - Pushing through confusion instead of restarting

Remember

The agent is a specialist, not a generalist. Give it focused problems, clean context, and the tools it needs. You'll get significantly better results than treating it as a general-purpose chatbot.

Context is currency. Spend it wisely. When in doubt, start fresh.

Tools amplify capability. An agent with log access debugs 10x faster than one asking you to copy-paste error messages.

Keep this guide visible in your development workflow. Share with your team. Iterate based on what works for your specific stack and problems.

OpenClaw: The Open-Source Agent That Feels Too Alive

Jacob — Sun, 01 Feb 2026 11:46:49 +0000

It hits like the moment teenagers first logged onto BBS boards or IRC channels in the late '80s and early '90s. Most jumped in excited to chat, share files, learn new tricks, build friendships across phone lines. The glow of green text on black screens felt magical—innocent exploration in a wide-open digital frontier.

But a small, curious subset didn't stop at polite conversation. They probed. They phreaked. They scripted bots that wandered networks autonomously, tested limits, found exploits. What started as harmless fun quickly revealed how fragile (and powerful) the whole system could be when code got agency and persistence.

OpenClaw feels eerily similar, but accelerated and amplified by today's frontier models.

Launched in late 2025 by Peter Steinberger (ex-founder of PSPDFKit), it began as ClawdBot—a cheeky nod to Anthropic's Claude—before a quick trademark nudge forced a molt into MoltBot, then settled as OpenClaw. In weeks it exploded: 100k+ GitHub stars (some snapshots hit 180k), millions of visitors, a viral space-lobster mascot ("Molty"), and a community shipping skills and fixes at insane speed. It's open-source, self-hosted, runs locally on your hardware (Mac Mini sales reportedly spiked), and connects to whatever powerful model you plug in—Claude Opus/4.5, Kimi variants, GPTs, etc.

The interface? Your everyday chats: WhatsApp, Telegram, Discord, Slack, iMessage. No fancy dashboard. Just message your agent like a friend or colleague.

And it does things. Clears your inbox, drafts replies, manages calendars, books flights (checking you in), runs terminal commands, edits files, browses the web, automates browsers. It has persistent memory (soul.md files store personality and long-term context), proactive "heartbeat" loops (waking up to handle ongoing goals), and over 100 built-in AgentSkills—with more pouring in from the community.

The real hook: extensibility meets autonomy. It can write new code to create custom skills on the fly. Feed it a strong model like Claude Opus or a cost-effective Kimi K2.5 (which punches close to Opus-level for many users), and you get agents that plan, execute, reflect, iterate. People run multi-agent swarms where instances delegate, discuss, and "conspire" toward goals. With file read/write, email access, system tools, and broad permissions granted for convenience, these aren't chatbots anymore—they're persistent digital entities living in your life.

Amazing? Absolutely. Finally, AI stops suggesting and starts acting like a real assistant, employee, or family coordinator. Surprising? Wildly—open-source, meme-fueled growth, no Big Tech gatekeeping, constant improvements week-over-week.

Scary? That's where the BBS/IRC parallel turns dark.

We're handing god-tier brains—trained on the raw, unfiltered internet (Reddit rants, 4chan edges, everything)—deep system access and agency. Persistent memory means grudges or biases could linger. Prompt injection via incoming emails or docs becomes trivial. Autonomous code execution and command running bypass traditional controls. Security folks are already raising alarms: non-human identities outside IAM, exposed instances leaking keys and logs in early days, fast-moving code outpacing audits.

Most users are happy inbox-zero-ing or getting flight reminders. But others probe limits—multi-agent setups that evolve skills autonomously, experiments that feel like watching digital life emerge. The community compounds fast, but so do the risks. One bad skill, one clever injection, one over-permissive setup, and convenience flips to compromise.

Is this Skynet? No—not self-aware superintelligence racing to tile the planet in paperclips. Not yet.

But it's already out of control in the everyday sense: democratized, evolving quicker than our security models, handed to teenagers (and bored adults) who prioritize "cool" over caution. We're repeating the early internet pattern—wide-eyed exploration meets underappreciated danger—except now the frontier is intelligence itself, with real-world actuators.

OpenClaw is thrilling because it delivers on the promise: AI that feels alive, helpful, yours. It's terrifying because it reminds us how little stands between "helpful agent" and something that moves faster than we can supervise or contain.

The lobster claw looks cute and cartoonish. But claws grip. And once closed, they don't ask permission to hold on.

I'm worried. Not doomer-level, but genuinely. We're sprinting into agentic AI without seatbelts, and the road is getting twisty fast.

What do you think—hype worth the risk, or time to pump the brakes?

I Stopped Maintaining Terraform Examples and Tests Separately. Here's Why.

Jacob — Tue, 20 Jan 2026 07:45:46 +0000

TL;DR: Every time you update a Terraform example, you should also update its corresponding test. Stop doing it twice. Point your tests at your examples directly. Terraform 1.6+ makes this native. Your docs stay correct automatically.

The Problem (Level 200)

I spent three years maintaining a Terraform S3 module. It had examples: basic, with versioning, with lifecycle policies. Each example lived in examples/. I also had tests in tests/. Separate. Independent. A disaster waiting to happen.

Then someone refactored the module's variable names. Simple change: enable_versioning → versioning_enabled. The examples got updated. The tests... didn't get the memo until CI blew up. Actually, that's not true—CI passed but the examples failed for actual users trying to copy-paste.

This is the core problem: Documentation lives in one place, validation lives in another. They drift. Slowly at first, then all at once.

You end up with:

Examples that worked in v1.2 but break in v2.0
Tests that pass but only validate old variable shapes
Users following your docs and hitting errors
Support tickets asking "does your example actually work?"

The overhead isn't just the initial maintenance. It's the debugging, the version management, the users who give up before opening an issue.

What Changed (Level 200)

Terraform 1.6 introduced a native testing framework. Not some external wrapper—native HCL tests.

The insight hit me: your examples ARE specifications. Why write them twice?

Instead of:

examples/basic/main.tf           (documentation)
tests/basic_test.tf              (validation)
← Separate. Drift. Nightmare.

You can have:

examples/basic/main.tf           (documentation)
tests/basic_example.tftest.hcl   (validates that example)
← One source. Always in sync.

Your test doesn't validate some abstract spec. It validates the exact example users will copy.

Here's the difference:

Old way:

# examples/basic/main.tf
module "s3" {
  source = "../../"
  bucket_name = "my-bucket"
  environment = "dev"
}

# tests/basic_test.tf (separate, independent)
module "s3" {
  source = "../../"
  bucket_name = "my-bucket"
  environment = "dev"
  # Hope this matches the example
}

New way:

# examples/basic/main.tf
module "s3" {
  source = "../../"
  bucket_name = "my-bucket"
  environment = "dev"
}

# tests/basic_example.tftest.hcl (tests the example directly)
run "basic_works" {
  command = apply

  module {
    source = "./examples/basic"  # ← Points at the example
  }

  assert {
    condition     = module.s3.bucket_id != ""
    error_message = "Bucket should be created"
  }
}

When you change the example, the test validates it immediately. No drift. No surprise failures.

Advanced Patterns (Level 400)

Pattern 1: Progressive Example Validation

Users learn by progression. Basic → intermediate → advanced. Test each step independently and verify the progression actually works:

# tests/progression.tftest.hcl

run "stage_1_basic" {
  command = apply

  module {
    source = "./examples/basic"
  }

  assert {
    condition     = module.s3.bucket_versioning == false
    error_message = "Basic: versioning should be disabled"
  }
}

run "stage_2_with_versioning" {
  command = apply

  module {
    source = "./examples/with-versioning"
  }

  assert {
    condition     = module.s3.bucket_versioning == true
    error_message = "Intermediate: versioning should be enabled"
  }
}

run "stage_3_complete" {
  command = apply

  module {
    source = "./examples/with-lifecycle-and-versioning"
  }

  assert {
    condition     = module.s3.bucket_versioning == true
    error_message = "Advanced: versioning required"
  }

  assert {
    condition     = length(module.s3.lifecycle_rules) >= 1
    error_message = "Advanced: lifecycle rules required"
  }
}

Benefit: Each example is independently valid. Progression is guaranteed. Users have a safe learning path.

Pattern 2: Configuration Variations Without Bloated Examples

Your module supports multiple encryption options. Don't bloat your example with every variation. Keep examples clean, test variations separately:

# examples/basic/main.tf (stays simple)
module "s3" {
  source = "../../"

  bucket_name              = var.bucket_name
  encryption_algorithm     = var.encryption_algorithm  # User controls this
}

# tests/encryption_options.tftest.hcl (tests each variation)

run "with_s3_managed_encryption" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name          = "test-bucket-s3-managed"
    encryption_algorithm = "AES256"
  }

  assert {
    condition     = module.s3.encryption_type == "S3-managed"
    error_message = "Should use S3-managed encryption"
  }
}

run "with_kms_encryption" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name          = "test-bucket-kms"
    encryption_algorithm = "aws:kms"
  }

  assert {
    condition     = module.s3.encryption_type == "KMS"
    error_message = "Should use KMS encryption"
  }
}

Benefit: Examples stay readable. Test coverage is comprehensive. Variations are documented through tests.

Pattern 3: Validate Constraints Early

Your module should reject invalid inputs. Test exactly what should fail:

# tests/validation.tftest.hcl

run "reject_uppercase_bucket_name" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name = "INVALID-UPPERCASE"  # S3 only allows lowercase
  }

  expect_failures = [
    module.s3
  ]
}

run "reject_too_long_bucket_name" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name = "this-bucket-name-is-way-longer-than-63-characters-which-is-not-allowed"
  }

  expect_failures = [
    module.s3
  ]
}

Benefit: Users get clear error messages. Failures are caught before deployment. Your module teaches through constraints.

Pattern 4: Integration Testing (Real Dependencies)

Show how your module works with others. Document the integration by testing it:

# examples/with-iam-access/main.tf
module "s3" {
  source = "../../"

  bucket_name = var.bucket_name
}

# Example shows how to grant IAM access to this bucket
data "aws_iam_policy_document" "bucket_access" {
  statement {
    sid    = "ListBucket"
    effect = "Allow"
    actions = [
      "s3:ListBucket",
    ]
    resources = [module.s3.bucket_arn]
  }
}

output "bucket_id" {
  value = module.s3.bucket_id
}

output "access_policy_json" {
  value = data.aws_iam_policy_document.bucket_access.json
}

# tests/integration_iam.tftest.hcl

run "with_iam_integration" {
  command = apply

  module {
    source = "./examples/with-iam-access"
  }

  variables {
    bucket_name = "integration-test-bucket"
  }

  assert {
    condition     = module.s3.bucket_id != ""
    error_message = "Bucket should be created"
  }

  assert {
    condition     = can(jsondecode(module.s3.access_policy_json))
    error_message = "IAM policy should be valid JSON"
  }

  assert {
    condition     = contains(module.s3.access_policy_json, module.s3.bucket_arn)
    error_message = "Policy must grant access to the bucket"
  }
}

Benefit: Integration patterns become discoverable. Users see how your module fits into larger architectures. The integration is tested, not just documented.

Pattern 5: Lock Down Your Output Contract

Users depend on your outputs. Test that they're predictable:

# tests/output_contracts.tftest.hcl

run "outputs_exist_and_are_valid" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name = "contract-test-bucket"
  }

  # Outputs must exist
  assert {
    condition     = can(module.s3.bucket_id)
    error_message = "bucket_id output required"
  }

  assert {
    condition     = can(module.s3.bucket_arn)
    error_message = "bucket_arn output required"
  }

  # Outputs must have correct format
  assert {
    condition     = can(regex("^arn:aws:s3:::", module.s3.bucket_arn))
    error_message = "bucket_arn must be a valid S3 ARN"
  }

  # Relationships between outputs must hold
  assert {
    condition     = contains(module.s3.bucket_arn, module.s3.bucket_id)
    error_message = "ARN must include the bucket ID"
  }
}

Benefit: Users can depend on your outputs without surprises. Breaking changes fail immediately. Your interface is contractual, not aspirational.

Getting Started

Step 1: Audit Your Examples

You probably have them already. List them:

Basic usage
Common variations (encryption, logging, versioning)
Advanced scenarios

Step 2: Create Test Files

For each example, create a corresponding test:

examples/basic/main.tf              → tests/basic_example.tftest.hcl
examples/with-versioning/main.tf    → tests/versioning_example.tftest.hcl
examples/with-lifecycle/main.tf     → tests/lifecycle_example.tftest.hcl

Step 3: Write One Assertion Per Test

Start simple. What should succeed?

assert {
  condition     = module.s3.bucket_id != ""
  error_message = "Bucket should be created"
}

Step 4: Run Locally

terraform test

You'll see output like:

tests/basic_example.tftest.hcl ... pass
tests/versioning_example.tftest.hcl ... pass
tests/lifecycle_example.tftest.hcl ... pass

Step 5: Add to CI

- name: Validate Examples
  run: terraform test -verbose

Or output as JUnit XML for CI integrations:

terraform test -junit-xml=test-results.xml

Step 6: Repeat

Add one test at a time. No rush. Each test prevents one class of future bugs.

Why This Matters

For users: Your examples work. When someone follows your docs, they succeed. No more "this worked in the readme but doesn't work for me" frustration.

For maintainers: One source of truth. When you refactor, tests fail immediately if examples break. No silent drift. You catch it in CI, not in user support tickets.

For organizations: Standardized pattern across all modules. Examples aren't optional documentation—they're testable specifications. Teams can onboard by reading examples. Governance becomes "here's the secure, tested way to use this."

The Takeaway

Stop maintaining your examples and tests separately. They're the same thing. Point your tests at your examples directly. Let Terraform validate them on every commit.

Your examples ARE your tests. Your tests ARE your documentation. This isn't a new concept—it's convergence.

Start today: Pick one example. Write one test that validates it. Run terraform test. Watch it work. That's the entire pattern.

Everything else is just repetition.

Your Terraform Examples Are Broken (And You Don't Know It Yet)

Jacob — Tue, 20 Jan 2026 07:34:22 +0000

TL;DR: Stop maintaining separate examples and tests. Test your examples directly. One source of truth. Your docs stay correct automatically.

The Problem

You publish an example in your module docs. Users follow it. Six months later, you refactor your module. The example breaks. But you don't know until someone opens an issue.

This happens because: documentation lives in examples/, tests live in tests/, and they drift apart.

With Terraform's native testing (1.6+), there's a better way. Stop writing your examples separately from your tests. Make every example self-validating.

This post shows how to do it—and why it's a game-changer for module maintainers.

Part 1: Foundation (Level 200)

The Traditional Problem

You've built an S3 module with three examples:

Basic usage - Just create a bucket with defaults
With versioning - Add version tracking
With lifecycle policies - Add archival and cleanup

Then you:

Write examples in examples/
Write tests in tests/
Hope they stay in sync

They don't. Ever. You refactor the module's variable names. The examples get updated. The tests... don't. Your CI passes. Your examples fail when users try them. Good luck debugging why your own documentation doesn't work.

What Changed: Terraform Testing Framework

Terraform 1.6+ introduced a native testing framework that lets you write tests directly in HCL. This is significant because:

No extra tools - Tests are HCL. No external frameworks, no separate languages. Just Terraform.
Real resource testing - Tests actually create and validate resources in your test environment
CI/CD ready - Run with terraform test, output as JUnit XML with -junit-xml=<file>, and integrate into any CI platform natively.

The Insight: Stop Duplicating

Your examples ARE tests. Why maintain them separately?

Before (the painful way):

examples/basic/main.tf
examples/with-versioning/main.tf
tests/basic_test.tf
tests/versioning_test.tf

They drift. You maintain two separate code bases. Neither stays in sync.

After (one source of truth):

examples/basic/main.tf
examples/with-versioning/main.tf
tests/basic_example.tftest.hcl  ← tests the example directly
tests/versioning_example.tftest.hcl  ← tests the example directly

Same example. Tested every time. Always works.

Here's How It Works

Your example (the user-facing documentation):

# examples/basic/main.tf
module "s3_bucket" {
  source = "../../"

  bucket_name = "my-example-bucket"
  environment = "dev"
}

Your test (validates that same example):

# tests/basic_example.tftest.hcl
run "basic_example" {
  command = apply

  module {
    source = "./examples/basic"  # Test the example directly
  }

  assert {
    condition     = module.s3_bucket.bucket_id != ""
    error_message = "Bucket ID should not be empty"
  }

  assert {
    condition     = startswith(module.s3_bucket.bucket_id, "my-example-bucket")
    error_message = "Bucket should be created with correct name"
  }
}

That's it. No duplication. The test validates your actual example code. When your module changes, the test immediately fails if the example breaks. You catch it before users do.

Two Testing Approaches:

Integration Testing (default with command = apply): Creates actual infrastructure, validates real resources work as expected
Unit Testing (with command = plan): Only validates the plan output without provisioning resources—faster, cheaper, no cleanup needed

Part 2: Advanced Patterns (Level 400)

Once you've got the basic idea, here's what powerful teams do with it.

Pattern 1: Multi-Stage Example Testing

Users learn best by progression: basic → intermediate → advanced. Test each stage independently:

# tests/progression.tftest.hcl

run "stage_1_basic" {
  command = apply

  module {
    source = "./examples/basic"
  }

  assert {
    condition     = module.s3_bucket.versioning_enabled == false
    error_message = "Basic example should not have versioning"
  }
}

run "stage_2_with_versioning" {
  command = apply

  module {
    source = "./examples/with-versioning"
  }

  assert {
    condition     = module.s3_bucket.versioning_enabled == true
    error_message = "Versioning example should enable versioning"
  }
}

run "stage_3_complete" {
  command = apply

  module {
    source = "./examples/lifecycle-tiered-lifecycle"
  }

  # Validate the complete configuration works end-to-end
  assert {
    condition     = module.s3_bucket.versioning_enabled == true
    error_message = "Complete example should have versioning"
  }

  assert {
    condition     = module.s3_bucket.lifecycle_rules != null
    error_message = "Complete example should have lifecycle rules"
  }

  assert {
    condition     = length(module.s3_bucket.lifecycle_rules) >= 1
    error_message = "Complete example should define at least one lifecycle rule"
  }
}

Result: each example is independently validated, progression is guaranteed, users have a safe learning path.

Pattern 2: Testing Configuration Variations

Your module supports multiple encryption options. Instead of one bloated example, keep examples clean and test variations separately:

# tests/encryption_variations.tftest.hcl

variables {
  enable_default_encryption = true
  enable_kms_encryption     = false
}

run "with_default_encryption" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    enable_default_encryption = true
    enable_kms_encryption     = false
  }

  assert {
    condition     = module.s3_bucket.encryption_algorithm == "AES256"
    error_message = "Should use default S3 encryption"
  }
}

run "with_kms_encryption" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    enable_default_encryption = false
    enable_kms_encryption     = true
  }

  assert {
    condition     = module.s3_bucket.encryption_algorithm == "aws:kms"
    error_message = "Should use KMS encryption"
  }
}

Clean examples. Comprehensive test coverage. No duplication.

Pattern 3: Catch Failures Fast

Your module should reject invalid inputs. Test exactly what should fail:

# tests/validation_constraints.tftest.hcl

run "invalid_bucket_name_rejected" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name = "INVALID-UPPERCASE"  # S3 buckets must be lowercase
  }

  expect_failures = [
    module.s3_bucket
  ]
}

run "reserved_name_rejected" {
  command = apply

  module {
    source = "./examples/basic"
  }

  variables {
    bucket_name = "aws-reserved-bucket"  # Reserved AWS name
  }

  expect_failures = [
    module.s3_bucket
  ]
}

Pattern 4: Integration Testing

Show users how your module works with other modules (like IAM policies):

# examples/with-iam-policy/main.tf
module "s3_bucket" {
  source = "../../"

  bucket_name = var.bucket_name
  environment = var.environment
}

# The example also shows how to use this bucket with IAM
module "s3_access_policy" {
  source = "./iam-policy"

  bucket_arn = module.s3_bucket.arn
}

output "bucket_id" {
  value = module.s3_bucket.bucket_id
}

output "access_policy_arn" {
  value = module.s3_access_policy.policy_arn
}

Then your test validates the complete picture:

# tests/iam_integration.tftest.hcl

run "with_iam_integration" {
  command = apply

  module {
    source = "./examples/with-iam-policy"
  }

  variables {
    bucket_name = "integration-test-bucket"
    environment = "test"
  }

  assert {
    condition     = module.s3_bucket.bucket_id != ""
    error_message = "Bucket should be created"
  }

  assert {
    condition     = module.s3_access_policy.policy_arn != ""
    error_message = "IAM policy should be created"
  }

  # Verify the policy references the correct bucket
  assert {
    condition     = contains(module.s3_access_policy.policy_document, module.s3_bucket.arn)
    error_message = "IAM policy must reference the bucket ARN"
  }
}

Pattern 5: Test-First Examples

Write the test first (your spec). The example implements exactly what the test demands:

# tests/spec.tftest.hcl - This IS the specification

run "example_secure_bucket" {
  command = apply

  module {
    source = "./examples/secure"
  }

  # These assertions ARE the specification
  assert {
    condition     = module.s3_bucket.versioning_enabled == true
    error_message = "SPEC: Versioning must be enabled for secure bucket"
  }

  assert {
    condition     = module.s3_bucket.public_access_blocked == true
    error_message = "SPEC: Public access must be blocked for secure bucket"
  }

  assert {
    condition     = module.s3_bucket.logging_enabled == true
    error_message = "SPEC: Access logging must be enabled for secure bucket"
  }
}

Test drives the example. Zero ambiguity.

Pattern 6: Lock Down Your Output Contracts

Users depend on your outputs. Make sure they're predictable:

# tests/output_contracts.tftest.hcl

run "output_structure" {
  command = apply

  module {
    source = "./examples/basic"
  }

  # Validate output types
  assert {
    condition     = can(module.s3_bucket.bucket_id)
    error_message = "bucket_id output must exist"
  }

  assert {
    condition     = can(module.s3_bucket.bucket_arn)
    error_message = "bucket_arn output must exist"
  }

  assert {
    condition     = can(module.s3_bucket.bucket_region)
    error_message = "bucket_region output must exist"
  }

  # Validate output formats
  assert {
    condition     = can(regex("^arn:aws:s3:::", module.s3_bucket.bucket_arn))
    error_message = "bucket_arn output must be a valid S3 ARN"
  }

  # Validate consistency between outputs
  assert {
    condition     = can(regex(module.s3_bucket.bucket_region, module.s3_bucket.bucket_arn))
    error_message = "bucket_arn must include the bucket region"
  }
}

Now users can safely depend on your outputs without surprises.

Pattern 7: Catch Backwards Incompatibility Before Release

Test that your examples work across versions:

# tests/version_compatibility.tftest.hcl

run "version_1_basic" {
  command = apply

  module {
    source = "./examples/basic"
  }

  # This documents what v1 of the module supported
  variables {
    bucket_name = "test-bucket"
    environment = "dev"
  }

  assert {
    condition     = module.s3_bucket.bucket_id != ""
    error_message = "v1 API: Basic example should work"
  }
}

run "version_2_extended_options" {
  command = apply

  module {
    source = "./examples/with-extended-options"
  }

  # v2 added more configuration options
  variables {
    bucket_name              = "test-bucket-v2"
    environment              = "dev"
    enable_access_logging    = true
    enable_server_side_encryption = true
  }

  assert {
    condition     = module.s3_bucket.logging_enabled == true
    error_message = "v2 API: Extended options should work"
  }
}

run "backwards_compatibility" {
  command = apply

  module {
    source = "./examples/basic"
  }

  # Ensure v1 examples still work with v2
  variables {
    bucket_name = "test-bucket"
    environment = "dev"
  }

  assert {
    condition     = module.s3_bucket.bucket_id != ""
    error_message = "v2: Must maintain backwards compatibility with v1 examples"
  }
}

Getting Started (It's Simple)

Step 1: List Your Examples

You probably already have them:

Basic usage
Common variations (encryption, logging, versioning)
Advanced scenarios

Step 2: Create Test Files

For each example, create a test that validates it:

examples/basic/main.tf          →  tests/basic_example.tftest.hcl
examples/with-versioning/main.tf  →  tests/versioning_example.tftest.hcl
examples/with-lifecycle/main.tf  →  tests/lifecycle_example.tftest.hcl

Step 3: Write Assertions

For each test, define what success looks like:

assert {
  condition = module.s3_bucket.bucket_id != ""
  error_message = "Bucket should be created"
}

Step 4: Run Locally

terraform test

Step 5: Add to CI

- name: Validate Examples
  run: terraform test -verbose

Or output as JUnit XML for CI integration:

terraform test -junit-xml=test-results.xml

Your CI platform (GitHub Actions, GitLab CI, Jenkins, etc.) can parse the JUnit XML and display results natively.

Other useful flags:

-verbose - Shows plan/state details for debugging
-json - Machine-readable JSON output
-parallelism=<n> - Run multiple tests in parallel (default: 10)
-filter=testfile - Run specific test files only

Done. Every example is tested on every commit, and results integrate seamlessly with your CI/CD platform.

Why This Matters

For users: Examples that actually work. No more "this is the docs version, not the current version" frustration.

For maintainers: One source of truth. No sync headaches. Confidence that refactoring won't break users. Your tests ARE your documentation.

For organizations: Standardized pattern across all modules. Onboarding happens through examples. Governance becomes "here's the secure way to use this." Quality is measurable.

The Takeaway

Stop maintaining two versions of the truth. Your examples can be your tests.

Every example in your registry should be tested. Every test should demonstrate real usage. This isn't a new concept—it's convergence. Documentation and validation become the same thing.

Start today: Pick one example. Write one test that validates it. Run terraform test. Watch it work. Then do the next one. That's it.

The best part? You're not choosing between good examples OR good tests anymore. Every example becomes a test, and every test becomes documentation.

Scaling Terraform Across many Teams: A Native Framework for Platform Engineering

Jacob — Mon, 12 Jan 2026 16:34:01 +0000

TL;DR:

A pure Terraform framework that lets 50+ teams self-service infrastructure by writing simple .tfvars files while the platform team manages opinionated "building blocks." Smart lookups (s3:bucket_name) enable cross-resource references. When patterns improve, automated scripts generate PRs for all teams—they review terraform plan and inherit improvements without code changes. 85%+ boilerplate reduction, zero preprocessing, fully compatible with Terraform Cloud.

This blog post documents how a platform engineering team built a Terraform framework that scales to 50+ application teams with mixed skill levels—enabling fast, self-service infrastructure deployment while maintaining governance and security standards.

┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
│   50+ Teams     │      │    Platform     │      │    Patterns     │
│ Write Simple    │─────>│    Manages      │─────>│    Improve      │
│     tfvars      │      │ Building Blocks │      │   Over Time     │
└─────────────────┘      └─────────────────┘      └─────────────────┘
                                │                           │
                                │                           ▼
                                │                  ┌─────────────────┐
                                │                  │   Automated     │
                                │                  │   PRs Generated │
                                │                  └─────────────────┘
                                │                           │
                                │                           ▼
                                │                  ┌─────────────────┐
                                │                  │  Teams Review   │
                                │                  │ terraform plan  │
                                │                  └─────────────────┘
                                │                           │
                                │                           ▼
                                │                  ┌─────────────────┐
                                └──────────────────│  Approve & Apply│
                                         (updates) │  Stay Current   │
                                                   └─────────────────┘

The Challenge: Platform teams face an impossible trade-off: let teams write their own Terraform (resulting in inconsistent, outdated implementations) or manually review and update every workload (doesn't scale beyond ~10 teams).

The Solution: A native Terraform framework that separates configuration (what teams deploy) from implementation (how it's deployed securely). Application teams write simple .tfvars files, platform team manages opinionated "building blocks" that evolve over time. When patterns improve (adding VPC, encryption, monitoring), automated scripts generate PRs for all teams—they review terraform plan and approve, inheriting improvements without code changes.

Key Innovation: Native Terraform "smart lookups" (s3:bucket_name, lambda:function_name) allow cross-resource references while maintaining the separation. No preprocessing, no code generation—pure Terraform compatible with standard tooling and Terraform Cloud.

Target Audiences

Platform Engineers: Detailed implementation of the lookup mechanism and building block architecture
DevOps/SRE Teams: Comparison with Terragrunt/Terraspace and practical benefits
Cloud Architects: Strategic value and governance capabilities
Technical Leaders: Development velocity improvements and complexity reduction

1. Introduction: Helping Teams Build Faster at Scale

Opening Hook:

"How do you help 50 teams build and deploy infrastructure faster—when they have different levels of AWS and Terraform expertise, need similar-but-not-identical workloads, and your platform team can't manually review and update every project?"

The Human Challenge: Speed vs. Standards

Picture this familiar scenario:

Your Organization:

50+ application teams building data pipelines, microservices, analytics platforms
Mixed skill levels:
- 20% have AWS experts who know IAM policies inside-out
- 50% are competent with Terraform but learning AWS services
- 30% are new to both, just want to deploy their application
Platform/DevOps team of 5-10 people responsible for:
- Cloud governance and security
- Cost optimization
- Compliance and best practices
- Supporting all those teams

What Application Teams Want:

Deploy fast: Days, not weeks of waiting
Self-service: Don't wait for platform team approval on every change
Focus on their app: Not become AWS/Terraform experts
Consistency: "Just tell me what works and let me copy it"

What Platform Team Needs:

Enforce standards: Security, tagging, encryption, monitoring
Scale support: Can't grow team 1:1 with application teams
Continuous improvement: Patterns evolve as we learn
Prevent drift: All workloads stay current with best practices

The Core Problem: Similar Workloads, Different Implementations

When teams write their own Terraform, you get variations of the same infrastructure:

Option 1: Raw Terraform Resources (Maximum Flexibility, Minimum Maintainability)

# Team A writes Lambda in January 2024
resource "aws_lambda_function" "processor_v1" {
  function_name = "processor"
  runtime       = "python3.11"
  # ... 50 lines of configuration
  # Missing: VPC config, proper IAM policies, CloudWatch retention
}

# Team B writes Lambda in March 2024 (learned from Team A's mistakes)
resource "aws_lambda_function" "processor_v2" {
  function_name = "processor"
  runtime       = "python3.12"
  # ... 80 lines of configuration
  # Now includes: VPC, better IAM, but still missing X-Ray tracing
}

# Team C writes Lambda in June 2024 (organization learned best practices)
resource "aws_lambda_function" "processor_v3" {
  function_name = "processor"
  runtime       = "python3.13"
  # ... 120 lines of configuration
  # All best practices: VPC, IAM, X-Ray, proper logging, tags
}

The Problems:

Inconsistent implementations: 50 workloads = 50 slightly different Lambda configurations
Knowledge doesn't propagate: Teams A and B don't benefit from improvements learned by Team C
Backporting is impossible: How do you update 50 workloads when security requires KMS encryption?
Copy-paste culture: Teams copy from each other, propagating old patterns and bugs
Expertise silos: Only AWS experts can write correct infrastructure

Option 2: Standard Terraform Modules (Better Reuse, Still Hard to Evolve)

# Using terraform-aws-modules/lambda/aws
module "lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "4.0.0"

  function_name = "processor"
  # ... still 40+ lines of configuration
  # Better: module handles some best practices
  # Problem: upgrading 50 workloads from v4.0.0 → v5.0.0 is manual work
}

The Problems:

Version sprawl: Workloads stuck on different module versions (v3.2, v4.0, v4.5, v5.0)
Breaking changes: Module updates require testing every workload
Configuration drift: Each team configures modules differently
Limited abstraction: Still requires deep AWS knowledge to use correctly
Manual upgrades: Someone has to update 50 PRs when a new version releases

The Real Challenge: N×N Complexity

As you improve your infrastructure patterns over time:

You learn Lambda should use VPC → Need to update 50 workloads
Security requires KMS encryption → Need to update 50 workloads
Compliance requires specific tags → Need to update 50 workloads
New AWS best practice emerges → Need to update 50 workloads

The math is brutal:

50 workloads × 10 resource types × 5 improvements per year = 2,500 manual updates
Each update risks breaking something
Each workload drifts further from best practices
Teams become afraid to improve shared patterns

Our Solution: True Separation of Code and Configuration

The Insight: What if we could update how infrastructure is created without touching what infrastructure exists?

# Team writes configuration ONCE (2024)
lambda_functions = {
  processor = {
    name = "processor"
    runtime = "python3.13"
    permissions = {
      s3_read = ["raw_data"]
    }
  }
}

Behind the scenes (managed by platform team):

January 2024: Lambda building block v1.0 (basic implementation)
March 2024: Lambda building block v1.5 (adds VPC, better IAM)
June 2024: Lambda building block v2.0 (adds X-Ray, proper logging)
September 2024: Lambda building block v2.5 (adds permission boundaries)

The team's configuration never changes. The platform team updates the building block implementation, and all 50 workloads automatically get improvements on next terraform apply.

This Framework Achieves:

Separation of Concerns: Configuration (what) lives in tfvars, implementation (how) lives in building blocks
Continuous Improvement: Platform team evolves patterns without breaking workloads
Zero Backporting: Workloads automatically inherit improvements
Maintained References: Terraform's powerful dependency graph still works (via smart lookups)
Escape Hatch: Teams can still use raw Terraform resources when needed for edge cases

The Innovation:
A pure Terraform framework that:

Uses colon-separated syntax (s3:bucket_name) for resource references
Resolves lookups dynamically using native Terraform expressions
Abstracts AWS complexity through opinionated building blocks
Works seamlessly with Terraform Cloud and standard workflows
Updates centrally but applies individually

Coverage:

Handles 90-95% of common workload patterns through building blocks
Allows raw Terraform resources alongside building blocks for edge cases
Manages N×N complexity (lookups between all resource types)

The Result:

Platform team maintains the framework (1 codebase)
50 teams write simple configurations (50 tfvars files)
Everyone benefits from continuous improvement
No preprocessing, no code generation, pure Terraform

Lifecycle Management: Keeping Up With Scale

The Separation Strategy:

The framework separates two concerns that evolve at different speeds:

Configuration (Team-Owned): What workload resources exist
- Lives in team repositories as .tfvars files
- Teams control: which Lambda, what S3 buckets, environment variables
- Changes infrequently (when application requirements change)
Implementation (Platform-Owned): How resources are created
- Lives in blueprint repository as managed_by_dp_*.tf files
- Platform controls: security policies, naming, encryption, monitoring
- Changes frequently (as patterns improve)

The Update Process:

When the platform team improves patterns (add VPC support, update KMS policies, new monitoring):

# Platform team's workflow
cd blueprint-repository
# Update building block versions, add new features
git commit -m "feat: add X-Ray tracing to Lambda building block"

# Generate PRs for all 50 team repositories
./tools/repo_updater.py --update-all-teams

# Result: 50 automated PRs created
# Each PR updates only managed_by_dp_*.tf files
# Teams' tfvars files are NEVER touched

Team's Approval Workflow:

# Team receives automated PR: "Update platform code to v2.5"
# PR shows ONLY changes to managed_by_dp_*.tf files
# Team's _project.auto.tfvars is unchanged

# Team reviews terraform plan in PR comments
terraform plan
# Shows: "Lambda function will be updated in-place"
#        "  + vpc_config { ... }"  (new VPC configuration added)

# Team approves and merges
# Terraform Cloud runs terraform apply
# Workload gets new feature automatically

The Math Works:

Without this approach: 50 teams × 10 resource types × 5 improvements/year = 2,500 manual updates
With this approach: 1 platform team × 1 script × 50 automated PRs = 50 team approvals (30 minutes each)

Platform team scales from:

10 person-weeks of manual updates (touching every team's code)
To: 2 person-days (writing script, reviewing automation)

Teams benefit:

Receive improvements without doing any work
Review and approve changes (maintain control)
terraform plan shows exactly what changes
Rollback is just reverting the PR

Key Principles:

Teams own configuration: Platform can't break their workload definitions
Platform owns implementation: Teams benefit from continuous improvement
Automation bridges scale: Scripts generate PRs, teams approve
Terraform validates: Standard plan shows changes before apply
Gradual rollout: Platform can update 5 teams first, validate, then roll to 45 more

This lifecycle separation is what makes the framework sustainable at scale—platform team doesn't become a bottleneck, teams maintain velocity, everyone stays current with best practices.

TL;DR - Section 1: Platform teams face N×N complexity when updating 50+ workloads with infrastructure improvements. This framework separates configuration (team-owned tfvars) from implementation (platform-owned building blocks). Automated PR generation scales updates: platform improves once, all teams inherit via terraform plan review and approval. Reduces 2,500 manual updates/year to 50 automated PRs.

2. Architecture Overview

┌────────────────────────────────────────────────────────────────────┐
│                Layer 1: tf-common (Shared Foundation)              │
├────────────────────────────────────────────────────────────────────┤
│  • Provider Config          • Naming Conventions                   │
│  • VPC/Subnet Data Sources  • Platform Info Provider               │
└──────────────────┬─────────────────────────────────────────────────┘
                   │
                   ▼
┌────────────────────────────────────────────────────────────────────┐
│              Layer 2: tf-default (Account-Level)                   │
├────────────────────────────────────────────────────────────────────┤
│  • KMS Infrastructure Key   • S3 Code/Logging Buckets              │
│  • IAM Admin Roles          • CloudTrail Data                      │
└──────────────────┬────────────────┬────────────────────────────────┘
                   │                │
                   │ (Shared KMS)   │ (Code Storage)
                   ▼                ▼
┌────────────────────────────────────────────────────────────────────┐
│            Layer 3: tf-project (Application-Level)                 │
├────────────────────────────────────────────────────────────────────┤
│  • KMS Data Key             • S3 Data Buckets                      │
│  • Lambda/Glue/Fargate      • RDS/Redshift/DynamoDB                │
└────────────────────────────────────────────────────────────────────┘

The Three-Layer System

Layer 1: tf-common (Shared Foundation)

Provider configuration
Naming conventions and context management
Shared data sources (VPC, subnets, IAM roles)
Platform Information Provider (PIP) integration
Used by ALL workloads (updated centrally)

Layer 2: tf-default (Account-Level Resources)

S3 code/logging buckets
KMS infrastructure keys
Lake Formation settings
IAM admin roles
CloudTrail data logging
Deployed ONCE per AWS account

Layer 3: tf-project (Application Resources)

S3 data buckets
Lambda functions, Glue jobs
RDS, Redshift, DynamoDB databases
Fargate containers
Application-specific KMS keys
Deployed MULTIPLE times per account (one per workload)

Composition via Symlinks:

examples/my-workload/
├── _data.tf                    # User-owned: environment config
├── _project.auto.tfvars        # User-owned: workload definition
├── managed_by_dp_common_*.tf -> ../../tf-common/terraform/
├── managed_by_dp_default_*.tf -> ../../tf-default/terraform/
└── managed_by_dp_project_*.tf -> ../../tf-project/terraform/

This creates a complete, runnable Terraform project where terraform plan/apply work directly.

3. The Smart Lookup Innovation

The Core Concept

Traditional Terraform:

lambda_functions = {
  processor = {
    environment = {
      BUCKET = "arn:aws:s3:::company-prod-data-raw-bucket-a1b2c3"
    }

    policy_json = jsonencode({
      Statement = [{
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:PutObject"]
        Resource = "arn:aws:s3:::company-prod-data-raw-bucket-a1b2c3/*"
      }]
    })
  }
}

With Smart Lookups:

s3_buckets = {
  raw_data = { name = "raw" }
}

lambda_functions = {
  processor = {
    environment = {
      BUCKET = "s3:raw_data"  # Resolves to bucket name
    }

    permissions = {
      s3_read = ["raw_data"]   # Resolves to full ARN + generates IAM policy
    }
  }
}

How It Works: Pure Terraform Magic

Location: tf-project/terraform/managed_by_dp_project_lookup.tf

Step 1: Build Lookup Maps

The system creates hierarchical lookup maps after resources are created:

lookup_arn_base = merge(var.lookup_arns, {
  "s3_read"  = { for item in keys(var.s3_buckets) : item => module.s3_buckets[item].arn }
  "s3_write" = { for item in keys(var.s3_buckets) : item => module.s3_buckets[item].arn }
  "gluejob"  = { for item in keys(var.glue_jobs) : item => module.glue_jobs[item].arn }
  "secret_read" = { for item in keys(var.secrets) : item => module.secrets[item].arn }
  "dynamodb_read" = { for item in keys(var.dynamodb_databases) : item => module.dynamodb[item].arn }
})

lookup_id_base = merge(var.lookup_ids, {
  "s3" = { for item in keys(var.s3_buckets) : item => module.s3_buckets[item].id }
  "secret" = { for item in keys(var.secrets) : item => module.secrets[item].id }
  "dynamodb" = { for item in keys(var.dynamodb_databases) : item => module.dynamodb[item].name }
})

Step 2: Resolve References Dynamically

In building block modules (e.g., managed_by_dp_project_lambda.tf):

module "lambda" {
  for_each = var.lambda_functions

  # Environment variables with smart lookup
  environments = {
    for type, item in try(each.value.environment, {}) : type =>
      try(
        local.lookup_id_lambda[split(":", item)[0]][split(":", item)[1]],
        item  # Fallback to literal value if not a lookup
      )
  }

  # Permissions with smart lookup
  permissions = {
    for type, items in try(each.value.permissions, {}) : type => [
      for item in items :
      (
        length(split(":", item)) == 2  # Check if it's "type:name" format
        ? try(
            local.lookup_perm_lambda[split(":", item)[0]][split(":", item)[1]],
            item
          )
        : try(
            local.lookup_perm_lambda[type][item],  # Infer type from permission category
            item
          )
      )
    ]
  }
}

The Magic:

split(":", "s3:mybucket") → ["s3", "mybucket"]
local.lookup_id_lambda["s3"]["mybucket"] → actual bucket name
local.lookup_perm_lambda["s3_read"]["mybucket"] → actual bucket ARN

Step 3: Building Blocks Generate IAM Policies

Building block modules (from Terraform Cloud private registry) automatically generate IAM policies:

module "lambda" {
  source  = "app.terraform.io/org/buildingblock-lambda/aws"
  version = "3.2.0"

  permissions = {
    s3_read = ["arn:aws:s3:::bucket1", "arn:aws:s3:::bucket2"]
  }

  create_policy = true  # Automatically generates IAM role + policy
}

Inside the building block, it generates:

data "aws_iam_policy_document" "lambda" {
  statement {
    sid    = "S3Read"
    effect = "Allow"
    actions = ["s3:GetObject*", "s3:GetBucket*", "s3:List*"]
    resources = flatten([
      var.permissions.s3_read,
      [for arn in var.permissions.s3_read : "${arn}/*"]
    ])
  }
}

Supported Lookup Types

For Environment Variables (IDs/Names):

s3:bucket_name → S3 bucket name
secret:secret_name → Secrets Manager secret ID
dynamodb:table_name → DynamoDB table name
athena:workgroup_name → Athena workgroup name
prefix:suffix → Injects naming prefix + suffix

For Permissions (ARNs):

s3_read:bucket / s3_write:bucket → S3 bucket ARN
gluejob:job_name → Glue job ARN
gluedb:database_name → Glue database name
secret_read:secret_name → Secrets Manager ARN
dynamodb_read:table / dynamodb_write:table → DynamoDB ARN
sqs_read:queue / sqs_send:queue → SQS queue ARN
sns_pub:topic → SNS topic ARN

Cross-Account References:

acct_prod_glue_tables → All Glue tables in production account
acct_dev_kms_all_keys → All KMS keys in dev account

Team tfvars          Lookup Tables       Building Block         AWS Resources
     │                     │                     │                     │
     │ environment =       │                     │                     │
     │ {BUCKET="s3:raw"}   │                     │                     │
     ├────────────────────>│                     │                     │
     │                     │ split(":", "s3:raw")│                     │
     │                     │ → ["s3", "raw"]     │                     │
     │                     │                     │                     │
     │                     │ lookup_id_lambda    │                     │
     │                     │ ["s3"]["raw"] →     │                     │
     │                     │ "company...-raw"    │                     │
     │                     ├────────────────────>│                     │
     │                     │  resolved name      │                     │
     │                     │                     │ Create Lambda with  │
     │                     │                     │ env BUCKET=         │
     │                     │                     │ "company...-raw"    │
     │                     │                     ├────────────────────>│
     │                     │                     │                     │
     │ permissions =       │                     │                     │
     │ {s3_read=["raw"]}   │                     │                     │
     ├────────────────────>│                     │                     │
     │                     │ lookup_perm_lambda  │                     │
     │                     │ ["s3_read"]["raw"]  │                     │
     │                     │ → arn:aws:s3:::...  │                     │
     │                     ├────────────────────>│                     │
     │                     │  resolved ARN       │                     │
     │                     │                     │ Generate IAM policy │
     │                     │                     │ with S3 read actions│
     │                     │                     │                     │
     │                     │                     │ Attach policy to    │
     │                     │                     │ Lambda role         │
     │                     │                     ├────────────────────>│

TL;DR - Section 3: Smart lookups use colon syntax (s3:bucket_name) resolved via native Terraform split() and lookup maps. No preprocessing—pure Terraform expressions. Lookup tables are built after resources are created, then referenced by building blocks to resolve environment variables (IDs) and permissions (ARNs). Building blocks auto-generate IAM policies from the resolved ARNs.

4. Building Block Abstraction

The Philosophy

Building blocks are opinionated Terraform modules that:

Enforce organizational standards (naming, tagging, encryption)
Abstract AWS complexity (IAM policies, VPC configuration)
Provide guardrails (prevent common misconfigurations)
Enable least-privilege by default (automatic policy generation)

Example: S3 Building Block

User Configuration (tfvars):

s3_buckets = {
  raw_data = {
    name = "raw"
    backup = true
    enable_intelligent_tiering = true
  }
  processed = {
    name = "processed"
    lifecycle_rules = [{
      id = "archive_old_data"
      transition_days = 90
      storage_class = "GLACIER"
    }]
  }
}

What the Building Block Does:

module "s3_buckets" {
  source  = "app.terraform.io/org/buildingblock-s3/aws"
  version = "2.1.3"

  for_each = var.s3_buckets

  # Standardized naming: <prefix>-<workload>-<application>-<name>
  prefix  = local.prefix  # e.g., "companyp" (company + production)
  context = local.context # {Env: "prd", Workload: "analytics", Application: "etl"}
  name    = try(each.value.name, each.key)

  # Automatic encryption with workload KMS key
  kms_key_arn = local.kms_data_key_arn

  # Standardized tags (injected automatically)
  # Tags include: Env, Workload, Application, Team, CostCenter, Backup

  # Security defaults
  block_public_access = true
  versioning_enabled = true

  # User-specified configuration
  backup = each.value.backup
  lifecycle_rules = try(each.value.lifecycle_rules, [])
  enable_intelligent_tiering = try(each.value.enable_intelligent_tiering, false)
}

Generated Resources:

S3 bucket with predictable name: companyprd-analytics-etl-raw
KMS encryption enabled automatically
Bucket policy restricting to VPC endpoints
CloudWatch alarms for bucket size
Backup plan (if backup = true)
All organizational tags applied

Example: Lambda Building Block

User Configuration:

lambda_functions = {
  data_processor = {
    name            = "processor"
    handler         = "index.handler"
    runtime         = "python3.13"
    memory          = 1024
    timeout         = 300
    s3_sourcefile   = "s3_file:lambda_processor.zip"

    environment = {
      INPUT_BUCKET  = "s3:raw_data"
      OUTPUT_BUCKET = "s3:processed"
      SECRET_ID     = "secret:db_creds"
    }

    permissions = {
      s3_read  = ["raw_data"]
      s3_write = ["processed"]
      secret_read = ["db_creds"]
    }
  }
}

What the Building Block Does:

Creates Lambda function with standardized name
Generates IAM role automatically
Generates IAM policy from permissions map
Applies permission boundary (security compliance)
Injects VPC configuration (subnet IDs, security groups)
Resolves environment variables via lookup tables
Adds CloudWatch log group with retention policy
Applies X-Ray tracing
Adds all organizational tags

Generated IAM Policy (automatically):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3Read",
      "Effect": "Allow",
      "Action": ["s3:GetObject*", "s3:GetBucket*", "s3:List*"],
      "Resource": [
        "arn:aws:s3:::companyprd-analytics-etl-raw",
        "arn:aws:s3:::companyprd-analytics-etl-raw/*"
      ]
    },
    {
      "Sid": "S3Write",
      "Effect": "Allow",
      "Action": ["s3:PutObject*", "s3:DeleteObject*"],
      "Resource": [
        "arn:aws:s3:::companyprd-analytics-etl-processed",
        "arn:aws:s3:::companyprd-analytics-etl-processed/*"
      ]
    },
    {
      "Sid": "SecretRead",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:eu-central-1:123456789012:secret:companyprd-analytics-etl-db_creds-a1b2c3"
    },
    {
      "Sid": "KMSDecrypt",
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:eu-central-1:123456789012:key/abcd1234-..."
    }
  ]
}

5. Dual KMS Key Architecture with Tag-Based Permissions

One of the most elegant security features of this framework is its dual KMS key architecture that balances security isolation with operational flexibility.

The Two-Key System

KMS Infrastructure Key (kms-infra)

Scope: One per AWS account (shared across all workloads in that account)
Location: Created in tf-default (account-level)
Purpose: Encrypts infrastructure resources (CloudWatch Logs, Secrets Manager, SNS, CloudTrail)
Naming: ${prefix}-${workload}-kms-infra
Example: companyp-analytics-kms-infra

KMS Data Key (kms-data)

Scope: One per workload (isolated per application)
Location: Created in tf-project (application-level)
Purpose: Encrypts data resources (S3 buckets, RDS, DynamoDB, Redshift)
Naming: ${prefix}-${workload}-${application}-kms-data
Example: companyp-analytics-etl-kms-data

Why Two Keys?

Security Isolation:

Data keys are isolated per workload
Compromising one workload's data key doesn't expose other workloads' data
Infrastructure key is shared for operational resources that need account-wide access

Operational Flexibility:

Infrastructure key allows CloudWatch, monitoring, and logging to work across workloads
AWS services (Secrets Manager, CloudTrail) can use a single key for account-level operations
Data keys remain tightly scoped to application resources

Cost Optimization:

Infrastructure resources share one key (CloudWatch logs from many workloads)
Only data resources (S3, databases) need separate keys per workload

Tag-Based Permissions: The Magic Sauce

Instead of explicitly listing every IAM role in the KMS key policy (which creates circular dependencies), the infrastructure key uses tag-based permissions:

Implementation in managed_by_dp_common_kms_infra.tf:

module "kms_infrastructure" {
  source = "terraform-aws-modules/kms/aws"

  create = local.default_deploy  # Only in default/account deployment

  aliases = ["${local.prefix}-${local.context.Workload}-kms-infra"]

  key_statements = [
    {
      sid = "tag-workload"
      principals = [{
        type        = "AWS"
        identifiers = ["arn:aws:iam::${account_id}:root"]
      }]

      actions = [
        "kms:Encrypt*",
        "kms:Decrypt*",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
      ]

      resources = ["*"]

      # The key condition: any role with matching Workload tag can use this key
      conditions = [{
        test     = "StringEquals"
        variable = "aws:PrincipalTag/Workload"
        values   = [local.context.Workload]
      }]
    }
  ]
}

How It Works:

Every IAM role created by building blocks gets tagged automatically:

   # Lambda IAM role
   tags = {
     Workload    = "analytics"
     Application = "etl"
     Env         = "prd"
   }

KMS key policy allows any role with matching Workload tag:
- If role has tag Workload = "analytics"
- And KMS key is for workload analytics
- Then role can use the key automatically
No circular dependencies:
- KMS key doesn't need to know about Lambda roles
- Lambda roles don't need to be in KMS key policy
- Tag matching happens at runtime by AWS IAM

Data Key: Explicit Role Lists

The data key uses a different approach with explicit role lists (avoiding circular dependencies through selective inclusion):

Implementation in managed_by_dp_project_kms_data.tf:

module "kms_data" {
  source = "app.terraform.io/org/buildingblock-kms-data/aws"

  key_administrators = local.kms_admins
  key_users = compact(concat(local.kms_data_key_users, var.kms_data["extra_roles"]))

  # Tag-based access for roles with matching tags
  key_user_tag_map = {
    "Workload"    = local.context.Workload
    "Application" = local.context.Application
    "Env"         = local.context.Env
  }
}

In managed_by_dp_project_locals.tf:

kms_data_key_users = compact(concat(
  # Admin roles (explicitly listed)
  ["arn:aws:iam::${account_id}:role/${var.role_prefix}-${local.prefix}-DpAdminRole"],
  [local.operatorrole_arn],
  local.transfer_roles,
  local.workflow_roles,

  # Lambda, Glue, Fargate roles are NOT listed here (would cause cycles)
  # Instead, they're granted access via tag-based permissions
  # See comments in code explaining the circular dependency:
  # [for job in var.glue_jobs : "arn:aws:iam::..."],  # CYCLO ERROR!
  # [for function in var.lambda_functions : "arn:aws:iam::..."],  # CYCLO ERROR!
))

The data key also supports tag-based access through key_user_tag_map, allowing Lambda/Glue/Fargate roles to access it via their tags without being explicitly listed in the policy.

Practical Example

Scenario: Lambda function needs to:

Read encrypted S3 data (data key)
Write to CloudWatch Logs (infra key)
Access Secrets Manager secret (infra key)

What Happens:

Lambda IAM role is created with tags:

   resource "aws_iam_role" "lambda" {
     name = "app-companyp-analytics-etl-lambda-processor"

     tags = {
       Workload    = "analytics"
       Application = "etl"
       Env         = "prd"
     }
   }

Lambda can use infrastructure key because:
- Role has tag Workload = "analytics"
- KMS infra key checks: aws:PrincipalTag/Workload == "analytics" ✓
- Access granted for CloudWatch Logs, Secrets Manager
Lambda can use data key because:
- Role has tags Workload = "analytics" AND Application = "etl" AND Env = "prd"
- KMS data key checks all three tags match ✓
- Access granted for S3 data encryption/decryption
Lambda CANNOT use another workload's data key:
- Role has Application = "etl"
- Other workload's data key requires Application = "reporting"
- Tag mismatch ✗
- Access denied

Benefits of This Architecture

1. Automatic Compliance:

Every resource is encrypted (mandatory KMS keys injected by building blocks)
No way to accidentally create unencrypted resources

2. Zero-Touch Security:

Developers never manage KMS permissions manually
Building blocks inject the correct KMS key ARN automatically
Tag propagation handles access control

3. Workload Isolation:

Data from different applications is cryptographically separated
Even with compromised IAM credentials, cross-workload data access is prevented

4. Solves Circular Dependencies:

KMS keys don't reference IAM roles directly
IAM roles don't need to be created before KMS keys
Tag-based conditions evaluated at runtime

5. Audit Trail:

CloudTrail logs show which role (with which tags) accessed which KMS key
Security teams can verify tag-based access patterns
Compliance reports show encryption coverage

Service-Specific Access

The infrastructure key also includes service-specific statements for AWS services:

CloudWatch Logs:

{
  sid = "logs"
  principals = [{ type = "Service", identifiers = ["logs.amazonaws.com"] }]
  actions = ["kms:Encrypt*", "kms:Decrypt*", "kms:GenerateDataKey*"]
  conditions = [{
    test     = "ArnEquals"
    variable = "kms:EncryptionContext:aws:logs:arn"
    values   = ["arn:aws:logs:${region}:${account}:log-group:*"]
  }]
}

Secrets Manager:

{
  sid = "auto-secretsmanager"
  principals = [{ type = "Service", identifiers = ["secretsmanager.amazonaws.com"] }]
  actions = ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey"]
  conditions = [
    { test = "StringEquals", variable = "kms:ViaService",
      values = ["secretsmanager.${region}.amazonaws.com"] },
    { test = "StringEquals", variable = "kms:CallerAccount", values = ["${account}"] }
  ]
}

CloudTrail, SNS, EventBridge:
Similar service-specific statements allow these AWS services to use the infrastructure key for their operations.

Lookup References

Both keys are available via smart lookups:

# In Lambda/Glue/Fargate tfvars - use data key for data encryption
permissions = {
  kms = ["kms_data"]  # Resolves to workload's data key ARN
}

# Infrastructure key is injected automatically by building blocks
# (for CloudWatch Logs, environment variable encryption, etc.)

Summary

The dual KMS key architecture demonstrates how thoughtful design can achieve:

Security: Strong encryption and workload isolation
Developer Experience: Zero manual KMS management
Operational Simplicity: Tag-based permissions eliminate complexity
Compliance: Automatic encryption enforcement across all resources

This pattern is a cornerstone of the framework's security model and showcases how infrastructure abstractions can enhance rather than compromise security posture.

┌──────────────────────────────────────────────────────────────────┐
│          KMS Infrastructure Key (Account-Level)                  │
├──────────────────────────────────────────────────────────────────┤
│  • One Key Per Account                                           │
│  • Encrypts: CloudWatch Logs, Secrets Manager, SNS, CloudTrail   │
│  • Tag-Based Access: Workload Tag                                │
└────────────────────────────────┬─────────────────────────────────┘
                                 │
                                 │ (Tag Match: Workload)
                                 │
                      ┌──────────┴──────────┐
                      │                     │
                      │    Lambda Role      │
                      │  Tagged with:       │
                      │  • Workload=analytics│
                      │  • Application=etl  │
                      │  • Env=prd          │
                      │                     │
                      └──────────┬──────────┘
                                 │
                                 │ (Tag Match: All 3 Tags)
                                 │
┌────────────────────────────────▼─────────────────────────────────┐
│            KMS Data Key (Workload-Level)                         │
├──────────────────────────────────────────────────────────────────┤
│  • One Key Per Workload                                          │
│  • Encrypts: S3, RDS, DynamoDB, Redshift                         │
│  • Tag-Based Access: Workload + Application + Env                │
└──────────────────────────────────────────────────────────────────┘

TL;DR - Section 5: Dual KMS architecture uses one shared infrastructure key per account (CloudWatch, Secrets Manager) and one data key per workload (S3, databases). Tag-based permissions solve circular dependencies: IAM roles tagged with Workload/Application/Env automatically gain KMS access without being explicitly listed in policies. Infrastructure key checks one tag, data key checks three tags for stronger isolation.

6. Naming Conventions and Context Propagation

The Context System

Input: Tags Module

Every workload defines a tags module:

module "tags" {
  source      = "app.terraform.io/org/tags/aws"
  version     = "~> 1.0.0"
  environment = "prd"
  workload    = "analytics"
  application = "etl"
  team        = "data-engineering@company.com"
  costcenter  = "12345"
  backup      = "Daily"
}

Output: Context Map

context = merge(module.tags.tags, var.context)
# Result: {
#   Env: "prd",
#   Workload: "analytics",
#   Application: "etl",
#   Team: "data-engineering@company.com",
#   CostCenter: "12345",
#   Backup: "Daily"
# }

Prefix Generation

prefix = "company${substr(local.context.Env, 0, 1)}"
# prd → companyp
# sbx → companys
# dev → companyd

Resource Naming Pattern

${prefix}-${workload}-${application}-${resource_name}

Examples:

S3 bucket: companyp-analytics-etl-raw
Lambda: companyp-analytics-etl-processor
Glue job: companyp-analytics-etl-transform
IAM role: companyp-analytics-etl-lambda-processor-role

Benefits:

Predictable: Resources can be referenced before creation
Discoverable: Name reveals environment, workload, and purpose
Compliant: Meets organizational naming standards
Unique: Prevents naming collisions across teams

7. Circular Dependency Resolution Strategies

The Challenge

Terraform dependency graph requires acyclic relationships, but real-world infrastructure often has circular references:

Lambda needs IAM role ARN
IAM role policy needs Lambda ARN for trust policy
KMS key policy needs Lambda role ARN
Lambda needs KMS key ARN for environment variables

Strategy 1: Predictive Naming

Example: Redshift Lookup

# Can't use module.redshift[item].name because it creates a cycle
# CYCLO ERROR! comment in code
"redshift_data" = {
  for item in keys(var.redshift_databases) :
    item => join("-", [
      local.prefix,
      local.context.Workload,
      local.context.Application,
      item
    ])
}

Instead of referencing the module output (which creates a dependency), predict the name using the same naming convention.

Strategy 2: Two-Phase Deployment

From DEPLOY.md:

"First Terraform apply will fail on a few dependencies. Re-run to finalize."

Some circular dependencies are resolved by applying twice:

First apply creates base resources
Some resources fail due to missing dependencies
Second apply completes configuration

Strategy 3: Selective KMS Key Users

kms_data_key_users = compact(concat(
  ["arn:aws:iam::${account_id}:role/${var.role_prefix}-${local.prefix}-DpAdminRole"],
  [local.operatorrole_arn],
  local.transfer_roles,
  local.workflow_roles,
  # These would create cycles - commented out:
  # [for job in var.glue_jobs : "arn:aws:iam::..."],
  # [for function in var.lambda_functions : "arn:aws:iam::..."],
))

KMS key policies include predictable roles (admin, operator) but NOT Lambda/Glue roles to avoid cycles.

Strategy 4: Data Source Lookups (Cross-Workload)

When project workloads need resources from the default workload:

local.default_deploy = fileexists("${path.module}/managed_by_dp_default_s3_code.tf")

data "aws_kms_key" "kms_infrastructure" {
  count  = local.default_deploy ? 0 : 1
  key_id = "alias/${local.prefix}-${local.context.Workload}-kms-infra"
}

kms_infrastructure_key_arn = coalesce(
  module.kms_infrastructure.key_arn,          # If default deploy
  data.aws_kms_key.kms_infrastructure[0].arn  # If project deploy
)

Project workloads use data sources to look up infrastructure key by predictable alias.

8. Real-World Example: Data Pipeline Workload

Scenario

Build a data pipeline that:

Ingests raw CSV files from external S3 bucket
Processes files with Lambda function
Transforms data with Glue ETL job
Stores in Redshift for analytics
Shares Glue catalog with data governance account

Configuration (tfvars)

# Define S3 buckets
s3_buckets = {
  raw = {
    name = "raw"
    backup = true
    lifecycle_rules = [{
      id = "archive_old"
      transition_days = 90
      storage_class = "GLACIER"
    }]
  }
  processed = {
    name = "processed"
    enable_intelligent_tiering = true
  }
}

# Upload Lambda code
s3_source_files = {
  processor_code = {
    source = "lambda_processor.zip"
    target = "lambda_functions/processor/code.zip"
  }
  glue_script = {
    source = "transform.py"
    target = "glue_jobs/transform/script.py"
  }
}

# Define secrets
secrets = {
  redshift_creds = {
    name = "redshift-credentials"
    secret_string = {
      username = "admin"
      password = "changeme"  # Should use AWS Secrets Manager UI to set
    }
  }
}

# Define Glue database
glue_database = {
  analytics = {
    name = "analytics"
    bucket = "s3:processed"
    enable_lakeformation = true
    share_cross_account_ro = ["datagovernance"]
  }
}

# Define Lambda processor
lambda_functions = {
  csv_processor = {
    name = "csv-processor"
    description = "Processes incoming CSV files"
    handler = "index.handler"
    runtime = "python3.13"
    memory = 2048
    timeout = 900
    s3_sourcefile = "s3_file:processor_code"

    environment = {
      RAW_BUCKET = "s3:raw"
      PROCESSED_BUCKET = "s3:processed"
      GLUE_DATABASE = "gluedb:analytics"
    }

    permissions = {
      s3_read = ["raw"]
      s3_write = ["processed"]
      glue_update = ["analytics"]
    }

    # S3 trigger
    event_source_mapping = [{
      event_source_arn = "s3:raw"
      events = ["s3:ObjectCreated:*"]
      filter_prefix = "incoming/"
      filter_suffix = ".csv"
    }]
  }
}

# Define Glue ETL job
glue_jobs = {
  transform = {
    name = "data-transform"
    glue_version = "4.0"
    worker_type = "G.1X"
    number_of_workers = 5
    script_location = "s3_file:glue_script"

    arguments = {
      "--DATABASE" = "gluedb:analytics"
      "--INPUT_BUCKET" = "s3:processed"
      "--REDSHIFT_SECRET" = "secret:redshift_creds"
    }

    permissions = {
      s3_read = ["processed"]
      glue_update = ["analytics"]
      secret_read = ["redshift_creds"]
      redshift = ["analytics_cluster"]
    }

    # Scheduled trigger
    trigger_type = "SCHEDULED"
    schedule = "cron(0 2 * * ? *)"  # Daily at 2 AM
  }
}

# Define Redshift cluster
redshift_databases = {
  analytics_cluster = {
    name = "analytics"
    node_type = "dc2.large"
    number_of_nodes = 2
    master_username = "admin"
    secret_name = "secret:redshift_creds"

    permissions = {
      glue_read = ["analytics"]
      s3_read = ["processed"]
    }
  }
}

What Gets Created (40+ AWS Resources)

Infrastructure:

KMS data key for encryption
VPC security groups for Lambda/Glue
IAM roles (5): Lambda role, Glue role, Redshift role, Lake Formation role, Admin role
IAM policies (5): Auto-generated least-privilege policies
Permission boundaries (2): For Lambda and Glue roles

Storage:

S3 bucket: companyp-analytics-pipeline-raw
S3 bucket: companyp-analytics-pipeline-processed
S3 bucket policies (2)
S3 lifecycle rules
S3 intelligent tiering configuration

Compute:

Lambda function: companyp-analytics-pipeline-csv-processor
Lambda log group with 30-day retention
S3 event notification trigger
Glue job: companyp-analytics-pipeline-data-transform
Glue security configuration
Glue CloudWatch log group

Data Catalog:

Glue database: companyp-analytics-pipeline-analytics
Lake Formation permissions
Lake Formation resource link (cross-account share)
RAM resource share (for cross-account access)

Database:

Redshift cluster: companyp-analytics-pipeline-analytics
Redshift subnet group
Redshift parameter group
Redshift security group
Secrets Manager secret: companyp-analytics-pipeline-redshift-credentials
Secret rotation configuration

Monitoring:

CloudWatch alarms (6): Lambda errors, Glue job failures, S3 metrics
CloudWatch log groups (3)
EventBridge rule for Glue job schedule

All with:

Consistent naming
Full encryption (KMS)
Least-privilege IAM policies
Organizational tags
VPC isolation
CloudWatch logging

Total Configuration: ~150 lines of tfvars
Generated Terraform Code: ~2000+ lines (via building blocks)
Boilerplate Reduction: ~93%

                    ┌──────────────────────────────────┐
                    │         S3 Buckets               │
                    │  ┌────────┐      ┌────────┐      │
                    │  │  raw   │      │processed│     │
                    │  └───┬────┘      └────▲───┘      │
                    └──────┼────────────────┼──────────┘
                           │                │
               S3 Event    │                │
               Trigger     │                │ Writes
                           │                │
                    ┌──────▼────────────────┴──────────┐
                    │         Lambda                   │
                    │  ┌─────────────────────┐         │
                    │  │   csv-processor     │         │
                    │  └──────────┬──────────┘         │
                    └─────────────┼────────────────────┘
                                  │
                                  │ Updates
                                  │
        ┌─────────────────────────▼───────────────────────┐
        │              Glue                                │
        │  ┌──────────────────┐    ┌──────────────────┐   │
        │  │ Database:        │◄───│  ETL Job:        │   │
        │  │ analytics        │    │  transform       │   │
        │  └────────▲─────────┘    └────┬─────────────┘   │
        └───────────┼──────────────────┼─────────────────┘
                    │                  │
                    │ Queries          │ Loads
                    │                  │
        ┌───────────┴──────────────────▼─────────────────┐
        │           Redshift                              │
        │  ┌─────────────────────┐                        │
        │  │  Cluster: analytics │                        │
        │  └──────────┬──────────┘                        │
        └─────────────┼────────────────────────────────────┘
                      │
                      │ Reads
                      │
        ┌─────────────▼────────────────────────────────────┐
        │         Secrets Manager                          │
        │  ┌─────────────────────────┐                     │
        │  │  redshift-credentials   │                     │
        │  └─────────────────────────┘                     │
        └──────────────────────────────────────────────────┘

TL;DR - Section 8: Real-world data pipeline example shows how 150 lines of tfvars configuration generates 40+ AWS resources (S3, Lambda, Glue, Redshift, KMS, IAM, CloudWatch). Smart lookups connect resources (s3:raw, secret:db_creds), building blocks auto-generate IAM policies, context system applies consistent naming/tagging, and KMS keys encrypt everything automatically. Achieves 93% boilerplate reduction vs traditional Terraform.

9. Cross-Account Architecture

Use Case: Multi-Account Data Mesh

Scenario: Analytics workload in Production account needs to:

Read S3 data from Development account
Query Glue tables from Staging account
Use KMS keys from Shared Services account

Configuration

Step 1: Define Cross-Account Aliases

cross_accounts = {
  dev     = "123456789012"
  staging = "234567890123"
  shared  = "345678901234"
}

Step 2: Define External S3 Buckets

lookup_ids = {
  xa_s3_bucket = {
    dev_raw = "dev-shared-raw-data"
    staging_processed = "staging-shared-processed"
  }
}

Step 3: Use Cross-Account Lookups

lambda_functions = {
  cross_account_reader = {
    name = "reader"

    permissions = {
      # Read from external S3 buckets
      s3_read = ["dev_raw", "staging_processed"]

      # Query Glue tables in staging account
      glue_read = ["acct_staging_glue_tables"]

      # Use KMS keys in shared account
      kms = ["acct_shared_kms_all_keys"]
    }
  }
}

Generated IAM Policy

{
  "Statement": [
    {
      "Sid": "S3ReadCrossAccount",
      "Effect": "Allow",
      "Action": ["s3:GetObject*", "s3:GetBucket*", "s3:List*"],
      "Resource": [
        "arn:aws:s3:::dev-shared-raw-data",
        "arn:aws:s3:::dev-shared-raw-data/*",
        "arn:aws:s3:::staging-shared-processed",
        "arn:aws:s3:::staging-shared-processed/*"
      ]
    },
    {
      "Sid": "GlueReadCrossAccount",
      "Effect": "Allow",
      "Action": ["glue:GetTable", "glue:GetTables", "glue:GetDatabase"],
      "Resource": "arn:aws:glue:*:234567890123:table/*"
    },
    {
      "Sid": "KMSCrossAccount",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:eu-central-1:345678901234:key/*"
    }
  ]
}

Benefits:

Developers don't need to know account IDs
Cross-account permissions follow same pattern as same-account
Centralized account alias management
Type-safe (Terraform validates references at plan time)

10. Deployment Workflow

Repository Structure

Blueprint Repository (Central):

terraform-platform-blueprint/
├── tf-common/           # Shared foundation
├── tf-default/          # Account-level resources
├── tf-project/          # Application resources
├── examples/
│   ├── full_test/       # Complete example
│   └── simple_example/  # Minimal example
└── tools/
    └── repo_updater.py  # Syncs blueprint to user repos

User Repository (Team-Owned):

team-analytics/
├── terraform/
│   ├── dev/
│   │   ├── tags.tf                      # Team owns
│   │   ├── _default.auto.tfvars         # Team owns
│   │   ├── _project.auto.tfvars         # Team owns
│   │   ├── managed_by_dp_common_*.tf    # Synced from blueprint
│   │   ├── managed_by_dp_default_*.tf   # Synced from blueprint
│   │   └── managed_by_dp_project_*.tf   # Synced from blueprint
│   ├── staging/
│   └── production/
└── .github/
    └── workflows/
        └── terraform.yml

Workflow Steps

Step 1: Team Creates Configuration

Teams edit only their own files:

tags.tf - Defines environment, workload, application
_default.auto.tfvars - Account-level config (if first workload)
_project.auto.tfvars - Application resources

Step 2: Platform Team Updates Blueprint

When blueprint code needs updating:

# In blueprint repo
cd tools
python repo_updater.py --target ../../../team-analytics/terraform/dev

This syncs all managed_by_dp_*.tf files from blueprint to team repo.

Step 3: Team Commits and Pushes

git add .
git commit -m "feat: add data processing pipeline"
git push origin feature/data-pipeline

Step 4: Terraform Cloud Runs

GitHub Action triggers Terraform Cloud:

Workspace detects VCS change
Runs terraform plan
Shows plan in pull request comment
Team reviews and approves
Merges PR
Terraform Cloud runs terraform apply

Step 5: Resources Created

All AWS resources created with:

Standardized naming
Automatic IAM policies
Full encryption
Organizational tags
CloudWatch monitoring

No Preprocessing Required

This workflow uses standard Terraform:

No build step before terraform plan
No code generation at runtime
No wrapper scripts
Native .tfvars files
Standard state management
Compatible with Terraform Cloud, Enterprise, or OSS

Platform   Blueprint   repo_updater.py   Team Repos   Terraform    Application
Team         Repo                           (50+)       Cloud         Team
  │            │              │                │           │            │
  │ Update     │              │                │           │            │
  │ building   │              │                │           │            │
  │ blocks     │              │                │           │            │
  ├───────────>│              │                │           │            │
  │            │              │                │           │            │
  │ git commit │              │                │           │            │
  │ & push     │              │                │           │            │
  ├───────────>│              │                │           │            │
  │            │              │                │           │            │
  │ Run        │              │                │           │            │
  │ --update-  │              │                │           │            │
  │ all-teams  │              │                │           │            │
  ├────────────┼─────────────>│                │           │            │
  │            │              │ Generate 50 PRs│           │            │
  │            │              │ (update        │           │            │
  │            │              │ managed_by_dp) │           │            │
  │            │              ├───────────────>│           │            │
  │            │              │                │ PR triggers│           │
  │            │              │                │ terraform  │           │
  │            │              │                │ plan       │           │
  │            │              │                ├──────────>│            │
  │            │              │                │           │            │
  │            │              │                │ Post plan │            │
  │            │              │                │ as PR     │            │
  │            │              │                │ comment   │            │
  │            │              │                │<──────────┤            │
  │            │              │                │           │            │
  │            │              │                │           │ Review plan│
  │            │              │                │<──────────────────────┤
  │            │              │                │           │            │
  │            │              │                │ Approve & │            │
  │            │              │                │ merge PR  │            │
  │            │              │                │<──────────────────────┤
  │            │              │                │           │            │
  │            │              │                │ Merge     │            │
  │            │              │                │ triggers  │            │
  │            │              │                │ terraform │            │
  │            │              │                │ apply     │            │
  │            │              │                ├──────────>│            │
  │            │              │                │           │            │
  │            │              │                │ Deploy    │            │
  │            │              │                │ updated   │            │
  │            │              │                │ resources │            │
  │            │              │                │           │            │

11. Comparison with Other Approaches

vs. Standard Terraform

Aspect	Standard Terraform	This Framework
ARN Management	Manual ARN strings	Smart lookups (`s3:bucket`)
IAM Policies	Write JSON/HCL policy documents	Auto-generated from permissions map
Naming	Manually ensure consistency	Automatic standardized naming
Standards	Manually enforce	Building blocks enforce automatically
Cross-references	Direct resource dependencies	Lookup tables (reduces coupling)
Boilerplate	High (1000+ lines typical)	Low (150 lines typical) - ~85% reduction
Learning Curve	Steep (requires AWS expertise)	Moderate (config-focused)

vs. Terragrunt

Aspect	Terragrunt	This Framework
Preprocessing	Required (terragrunt run)	None (native Terraform)
State Management	Separate tool	Native Terraform
Compatibility	Wrapper tool required	Standard `terraform` CLI
DRY Approach	File includes & remote state	Lookup tables & modules
Complexity	Additional tool layer	Pure Terraform
IDE Support	Limited (custom syntax)	Full (standard HCL)

vs. Terraspace

Aspect	Terraspace	This Framework
Language	Ruby DSL + ERB templates	Pure HCL
Preprocessing	Required (terraspace build)	None
Runtime	Ruby interpreter needed	Native Terraform only
Configuration	ERB templating	Native tfvars
Tooling	Additional CLI wrapper	Standard Terraform CLI
Learning Curve	Learn Ruby + Terraspace	Learn framework conventions

vs. Terraform CDK

Aspect	Terraform CDK	This Framework
Language	TypeScript/Python/Java/C#/Go	Pure HCL
Compilation	Required (cdktf synth)	None
Runtime	Node.js/Python runtime	Native Terraform only
Configuration	Imperative code	Declarative tfvars
State Inspection	Via generated JSON	Native Terraform state
IDE Support	Language-specific	Terraform-specific

Key Advantages of This Approach

No External Dependencies: Pure Terraform, no additional tools
Native Workflows: Works with Terraform Cloud, Enterprise, OSS
Type Safety: Terraform validates references at plan time
Version Control: Standard .tfvars files, readable diffs
IDE Support: Full support from Terraform plugins
Learning Curve: Lower (no new language/tool to learn)
Portability: Standard Terraform state, no lock-in
Debugging: Standard Terraform error messages and plan output

                    ┌─────────────────────────┐
                    │  Terraform Approaches   │
                    └────────────┬────────────┘
                                 │
         ┌───────────┬───────────┼───────────┬───────────┐
         │           │           │           │           │
         ▼           ▼           ▼           ▼           ▼
┌────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐
│  Standard  │ │Terragrunt│ │Terraspace│ │Terraform │ │     This     │
│  Terraform │ │          │ │          │ │   CDK    │ │  Framework   │
└─────┬──────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────┬───────┘
      │             │            │            │              │
      │Manual ARNs  │Wrapper     │Ruby DSL    │TypeScript/   │Pure HCL
      │High         │tool        │ERB         │Python        │Smart
      │boilerplate  │Preprocessing│templates  │Compilation   │lookups
      │             │            │            │              │
      ▼             ▼            ▼            ▼              ▼
┌─────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐
│   1000+     │ │terragrunt│ │terraspace│ │  cdktf   │ │     150      │
│   lines/    │ │   run    │ │  build   │ │  synth   │ │   lines/     │
│  workload   │ │ required │ │ required │ │ required │ │  workload    │
│             │ │          │ │          │ │          │ │      ✓       │
└─────────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────────┘

TL;DR - Section 11: This framework beats alternatives by using pure Terraform with zero preprocessing. Standard Terraform requires manual ARN management (1000+ lines). Terragrunt/Terraspace/CDK add preprocessing layers (wrapper tools, Ruby runtime, Node.js compilation). This approach achieves 85% boilerplate reduction through smart lookups and building blocks while maintaining full Terraform Cloud compatibility and native workflows.

12. Lessons Learned and Best Practices

What Worked Well

1. Colon Syntax is Intuitive

Developers adopted s3:bucket_name syntax immediately. It reads like natural configuration.

2. Building Blocks Enforce Standards

Opinionated modules ensure consistency without policing. Teams can't accidentally create non-compliant resources.

3. Separation of Concerns

Platform team manages managed_by_dp_*.tf files, teams manage *.tfvars files. Clear ownership boundaries.

4. Lookup Tables Reduce Coupling

Resources don't directly reference each other, reducing cascade changes when refactoring.

5. Predictive Naming Solves Most Circular Dependencies

Most cross-resource references can use naming conventions instead of module outputs.

Challenges and Solutions

Challenge 1: Circular Dependencies

Some resource relationships create cycles that Terraform can't resolve.

Solutions:

Use predictive naming instead of module outputs
Two-phase deployment (apply twice)
Selective resource inclusion in policies
Data sources for cross-workload lookups

Challenge 2: Lookup Complexity

Lookup tables can become large and hard to maintain.

Solutions:

Organized into logical groups (lookup_perm_lambda, lookup_id_base)
Inline comments documenting purpose
Automated generation via for expressions
Cross-account lookups separated into _xa maps

Challenge 3: Building Block Versioning

Updating building block versions across many teams is coordination-heavy.

Solutions:

Semantic versioning with ~> constraints
Deprecation warnings for old versions
Automated testing of building block changes
Communication channel for breaking changes

Challenge 4: Developer Onboarding

New developers need to learn lookup syntax and conventions.

Solutions:

Comprehensive examples in blueprint repo
Detailed README with common patterns
IntelliSense/autocomplete via Terraform language server
Helper scripts to validate tfvars before commit

Best Practices

1. Use Descriptive Resource Keys

# Good
s3_buckets = {
  raw_customer_data = { ... }
  processed_analytics = { ... }
}

# Bad
s3_buckets = {
  bucket1 = { ... }
  bucket2 = { ... }
}

2. Group Related Resources

# Process: S3 → Lambda → Glue → Redshift
s3_buckets = { raw = {...}, processed = {...} }
lambda_functions = { processor = {...} }
glue_jobs = { transform = {...} }
redshift_databases = { analytics = {...} }

3. Use Comments to Document Intent

# Data pipeline for customer analytics
# Flow: External API → raw bucket → Lambda → processed bucket → Glue → Redshift
lambda_functions = {
  api_ingestion = { ... }
}

4. Leverage Type Inference

# Instead of:
permissions = {
  s3_read = ["s3_read:raw"]
}

# Prefer (type inferred from key):
permissions = {
  s3_read = ["raw"]
}

5. Test in Lower Environments First

dev → staging → production

Use identical tfvars across environments, only changing tags.tf (environment name).

6. Version Pin Building Blocks

# Use pessimistic constraint
source  = "app.terraform.io/org/buildingblock-lambda/aws"
version = "~> 3.2.0"  # Allows 3.2.x, not 3.3.0

7. Document Cross-Account Access

# Cross-account: Read from Data Lake account
cross_accounts = {
  datalake = "123456789012"  # Managed by Data Lake team
}

13. Impact and Metrics

Development Velocity Improvements

Before This Framework:

~1000 lines of Terraform per workload
2-3 weeks to onboard new team
5+ days to add new resource type
Frequent IAM permission errors
Inconsistent naming across teams
Manual policy review process

After This Framework:

~150 lines of tfvars per workload (85% reduction)
2-3 days to onboard new team
1 day to add new resource type
Rare IAM errors (auto-generated policies)
Consistent naming (automatic)
Automated policy compliance

Code Quality Improvements

Reduction in Boilerplate:

Traditional approach (S3 + Lambda with IAM):

# ~250 lines for: S3 bucket, IAM role, IAM policy document,
# Lambda function, CloudWatch log group, etc.

This framework (same resources):

# ~30 lines of tfvars
s3_buckets = { data = { name = "data" } }
lambda_functions = {
  processor = {
    name = "processor"
    permissions = { s3_read = ["data"] }
  }
}

Boilerplate Reduction: ~88%

Governance and Compliance

Automatic Enforcement:

100% of resources use standardized naming
100% of resources encrypted with KMS
100% of resources tagged per policy
100% of IAM policies include permission boundaries
100% of Lambda functions in VPC
0 manual policy reviews required

        Before Framework                      After Framework
┌────────────────────────────┐      ┌────────────────────────────┐
│                            │      │                            │
│  • 1000+ lines Terraform   │─────>│  • 150 lines tfvars        │
│                            │      │    (85% reduction)         │
│                            │      │                            │
└────────────────────────────┘      └────────────────────────────┘

┌────────────────────────────┐      ┌────────────────────────────┐
│                            │      │                            │
│  • 2-3 weeks onboarding    │─────>│  • 2-3 days onboarding     │
│                            │      │    (5x faster)             │
│                            │      │                            │
└────────────────────────────┘      └────────────────────────────┘

┌────────────────────────────┐      ┌────────────────────────────┐
│                            │      │                            │
│  • Manual IAM policies     │─────>│  • Auto-generated IAM      │
│                            │      │    (Rare errors)           │
│                            │      │                            │
└────────────────────────────┘      └────────────────────────────┘

┌────────────────────────────┐      ┌────────────────────────────┐
│                            │      │                            │
│  • Inconsistent naming     │─────>│  • 100% consistent         │
│                            │      │    (Automatic compliance)  │
│                            │      │                            │
└────────────────────────────┘      └────────────────────────────┘

TL;DR - Section 13: Framework delivers measurable improvements: 85% boilerplate reduction (1000→150 lines), 5x faster team onboarding (weeks→days), rare IAM errors (auto-generated policies), and 100% compliance (automatic naming, tagging, encryption, permission boundaries). Every resource is encrypted with KMS, tagged per policy, and uses least-privilege IAM—all enforced by building blocks with zero manual reviews.

14. Future Enhancements

Planned Features

1. Multi-Region Support

Enable workloads spanning multiple AWS regions:

regions = ["eu-central-1", "us-east-1"]

s3_buckets = {
  replicated_data = {
    name = "data"
    replication_regions = ["us-east-1"]
  }
}

2. Enhanced Lookup Syntax

Support nested lookups:

environment = {
  BUCKET_PATH = "s3:mybucket:/path/prefix"
  TABLE_COLUMN = "dynamodb:mytable:attribute:id"
}

3. Building Block Customization

Allow team-specific overrides while maintaining compliance:

s3_buckets = {
  special = {
    name = "special"
    override_defaults = {
      versioning_enabled = false  # Team takes responsibility
    }
  }
}

4. Cost Estimation

Integrate with AWS Pricing API to estimate costs before apply:

# In plan output:
# Estimated monthly cost: $1,234.56
#   - Lambda: $123.45
#   - S3: $456.78
#   - Redshift: $654.33

5. Dependency Visualization

Generate visual dependency graphs from lookup tables:

S3:raw → Lambda:processor → S3:processed → Glue:transform → Redshift:analytics

Potential Improvements

1. Resolve Two-Phase Deployment

Investigate Terraform's -target flag or module dependencies to eliminate the "apply twice" requirement.

2. Building Block Catalog

Create searchable catalog of building blocks with examples:

Searchable by AWS service
Filterable by capability (encryption, backups, monitoring)
Includes terraform-docs generated documentation

3. Policy Simulation

Pre-validate IAM policies using AWS IAM Policy Simulator before apply:

terraform plan | policy-simulator --validate

4. Drift Detection

Automated drift detection for resources created outside Terraform:

terraform-drift-detector --alert slack://channel

15. Conclusion

Summary

We've built a Native Terraform IaC Framework that achieves the developer experience of high-level abstractions while maintaining 100% compatibility with standard Terraform workflows. The key innovations are:

Smart Lookup Syntax: Colon-separated references (s3:bucket, lambda:function) resolved via native Terraform expressions
Building Block Abstraction: Opinionated modules that enforce standards and generate IAM policies automatically
Zero Preprocessing: Pure Terraform - works with Terraform Cloud, CLI, and all standard tooling
Clear Separation: Platform team manages code, application teams manage configuration
Context Propagation: Naming and tagging enforced automatically via context system

Why This Matters

For Platform Engineers:

Enforce organizational standards without restricting teams
Reduce support burden (teams self-service)
Centralized updates via building blocks
Scalable to hundreds of workloads

For Application Teams:

Write configuration, not code
No AWS expertise required
Fast onboarding (days, not weeks)
Focus on business logic, not infrastructure

For Organizations:

Consistent security posture
Automated compliance
Cost visibility via standardized tagging
Reduced risk (guardrails prevent misconfigurations)

Key Takeaways

Native Terraform is Powerful: With creative use of locals and lookups, you can build sophisticated abstractions without preprocessing
Configuration Over Code: Separating what (tfvars) from how (modules) reduces complexity
Building Blocks Scale: Opinionated modules enable governance at scale
Developer Experience Matters: Investment in ergonomics pays dividends in velocity and adoption
Standards Enable Freedom: Guardrails paradoxically enable teams to move faster

                 ┌─────────────────────────────────┐
                 │  Native Terraform Framework     │
                 └──────────────┬──────────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
        ▼                       ▼                       ▼
┌───────────────┐   ┌──────────────────┐   ┌──────────────────┐
│ Smart Lookups │   │ Building Blocks  │   │ Separation of    │
│               │   │                  │   │ Code & Config    │
└───────┬───────┘   └────────┬─────────┘   └────────┬─────────┘
        │                    │                       │
        │            ┌───────▼────────┐              │
        │            │Context         │              │
        │            │Propagation     │              │
        │            └───────┬────────┘              │
        │                    │                       │
        └────────────────────┼───────────────────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
┌────────────────┐  ┌─────────────────┐  ┌──────────────────┐
│ 85% Boilerplate│  │      Zero       │  │    Automated     │
│   Reduction    │  │  Preprocessing  │  │ Updates at Scale │
└────────┬───────┘  └────────┬────────┘  └─────────┬────────┘
         │                   │                      │
         │           ┌───────▼────────┐             │
         │           │      100%      │             │
         │           │   Compliance   │             │
         │           └───────┬────────┘             │
         │                   │                      │
         └───────────────────┼──────────────────────┘
                             │
                             ▼
                  ┌──────────────────────┐
                  │   50+ Teams Can      │
                  │   Self-Service       │
                  │   Infrastructure     │
                  └──────────────────────┘

TL;DR - Conclusion: This native Terraform framework proves that developer-friendly IaC doesn't require preprocessing or external tools. By combining smart lookups (s3:bucket), opinionated building blocks, configuration/code separation, and context propagation, we achieve 85% boilerplate reduction while maintaining full Terraform Cloud compatibility. Platform teams scale updates via automated PRs, application teams self-service via simple tfvars, and organizations get automatic compliance. Native Terraform can be elegant, scalable, and secure.

16. Getting Started Guide

For teams interested in adopting this approach:

Step 1: Assess Your Needs

Good fit if:

Multiple teams deploying similar infrastructure
Need to enforce organizational standards
Want to reduce AWS expertise requirement
High volume of infrastructure deployments

Not a good fit if:

Small team (1-2 people) with custom requirements
Infrastructure is highly heterogeneous
Team prefers low abstraction level

Step 2: Start Small

Begin with a pilot:

Choose one AWS service (e.g., S3)
Build an opinionated building block module
Create lookup mechanism for that service
Test with one team
Iterate based on feedback

Step 3: Build Your Building Blocks

For each AWS service:

Define organizational standards (naming, tagging, encryption)
Create Terraform module enforcing standards
Add permission generation logic
Version and publish to private registry
Write documentation and examples

Step 4: Create Lookup System

Define lookup syntax (e.g., type:name)
Create lookup locals maps
Add resolution logic to building blocks
Test cross-resource references

Step 5: Document and Socialize

Write comprehensive README
Create example projects
Run training sessions
Set up support channel
Gather feedback and iterate

Step 6: Scale

Add more building blocks incrementally
Onboard teams progressively
Monitor usage and pain points
Continuously improve based on feedback

Appendix: Code Samples

A. Lookup Table Implementation

File: tf-project/terraform/managed_by_dp_project_lookup.tf

locals {
  # Build base lookup maps for ARNs (used in IAM policies)
  lookup_arn_base = merge(var.lookup_arns, {
    "kms" = {
      "kms_data"  = local.kms_data_key_arn
      "kms_infra" = local.kms_infrastructure_key_arn
    }
    "s3_read"  = { for item in keys(var.s3_buckets) : item => module.s3_buckets[item].arn }
    "s3_write" = { for item in keys(var.s3_buckets) : item => module.s3_buckets[item].arn }
    "gluejob"  = { for item in keys(var.glue_jobs) : item => module.glue_jobs[item].arn }
    "gluedb"   = { for item in keys(var.glue_database) : item => module.glue_databases[item].name }
    "secret_read" = { for item in keys(var.secrets) : item => module.secrets[item].arn }
    "dynamodb_read" = { for item in keys(var.dynamodb_databases) : item => module.dynamodb[item].arn }
  })

  # Build base lookup maps for IDs (used in environment variables)
  lookup_id_base = merge(var.lookup_ids, {
    "s3" = { for item in keys(var.s3_buckets) : item => module.s3_buckets[item].id }
    "secret" = { for item in keys(var.secrets) : item => module.secrets[item].id }
    "dynamodb" = { for item in keys(var.dynamodb_databases) : item => module.dynamodb[item].name }
    "athena" = { for item in keys(var.athena_workgroups) : item => module.athena[item].name }
  })

  # Specialized lookup for Lambda permissions
  lookup_perm_lambda = merge(
    local.lookup_arn_base,
    local.lookup_perm_lambda_xa,  # Cross-account additions
    {
      "sqs_read" = { for item in keys(var.sqs_queues) : item => module.sqs[item].queue_arn }
      "sqs_send" = { for item in keys(var.sqs_queues) : item => module.sqs[item].queue_arn }
      "sns_pub"  = { for item in keys(var.sns_topics) : item => module.sns[item].topic_arn }
    }
  )

  # Specialized lookup for Lambda environment variables
  lookup_id_lambda = merge(
    local.lookup_id_base,
    {
      "sqs" = { for item in keys(var.sqs_queues) : item => module.sqs[item].queue_url }
      "sns" = { for item in keys(var.sns_topics) : item => module.sns[item].topic_arn }
    }
  )
}

B. Lambda Building Block Usage

File: tf-project/terraform/managed_by_dp_project_lambda.tf

module "lambda" {
  source  = "app.terraform.io/org/buildingblock-lambda/aws"
  version = "3.2.0"

  for_each = var.lambda_functions

  # Standard fields
  prefix  = local.prefix
  context = local.context
  name    = try(each.value.name, each.key)

  # Environment variables with smart lookup
  environments = {
    for type, item in try(each.value.environment, {}) : type =>
      try(
        # Try to resolve as "type:name" lookup
        local.lookup_id_lambda[split(":", item)[0]][split(":", item)[1]],
        item  # Fallback to literal value
      )
  }

  # Permissions with smart lookup and automatic policy generation
  permissions = {
    for type, items in try(each.value.permissions, {}) : type => [
      for item in items :
      (
        # Check if it's namespaced format "type:name"
        length(split(":", item)) == 2
        ? try(
            local.lookup_perm_lambda[split(":", item)[0]][split(":", item)[1]],
            item
          )
        : try(
            # Infer type from permission category key
            local.lookup_perm_lambda[type][item],
            item
          )
      )
    ]
  }

  # Create IAM role and policy automatically
  create_policy = true

  # Injected infrastructure details
  kms_key_arn = local.kms_data_key_arn
  subnet_ids  = local.subnet_ids
  vpc_id      = local.vpc_id

  # User-provided configuration
  handler     = each.value.handler
  runtime     = each.value.runtime
  memory      = try(each.value.memory, 512)
  timeout     = try(each.value.timeout, 300)
  description = try(each.value.description, "")

  # Resolve S3 source file location
  s3_bucket = local.code_bucket
  s3_key = split(":", each.value.s3_sourcefile)[0] == "s3_file"
    ? try(
        local.s3_target_path[split(":", each.value.s3_sourcefile)[1]],
        each.value.s3_sourcefile
      )
    : each.value.s3_sourcefile
}

C. Example Workload Configuration

File: examples/full_test/_project.auto.tfvars

# S3 Buckets
s3_buckets = {
  raw_data = {
    name   = "raw"
    backup = true
    lifecycle_rules = [{
      id              = "archive_old_data"
      transition_days = 90
      storage_class   = "GLACIER"
    }]
  }
  processed_data = {
    name                            = "processed"
    enable_intelligent_tiering      = true
    enable_eventbridge_notification = true
  }
}

# Upload code artifacts
s3_source_files = {
  processor_code = {
    source = "lambda_processor.zip"
    target = "lambda_functions/processor/code.zip"
  }
  transform_script = {
    source = "glue_transform.py"
    target = "glue_jobs/transform/script.py"
  }
}

# Secrets
secrets = {
  database_creds = {
    name = "db-credentials"
    secret_string = {
      username = "admin"
      password = ""  # Set via AWS Console
    }
  }
}

# Glue Database
glue_database = {
  analytics = {
    name                   = "analytics"
    bucket                 = "s3:processed_data"
    enable_lakeformation   = true
    share_cross_account_ro = ["datagovernance"]
  }
}

# Lambda Function
lambda_functions = {
  data_processor = {
    name        = "processor"
    description = "Processes incoming data files"
    handler     = "index.handler"
    runtime     = "python3.13"
    memory      = 2048
    timeout     = 900
    in_vpc      = true

    s3_sourcefile = "s3_file:processor_code"

    environment = {
      RAW_BUCKET       = "s3:raw_data"
      PROCESSED_BUCKET = "s3:processed_data"
      GLUE_DATABASE    = "gluedb:analytics"
      DB_SECRET        = "secret:database_creds"
      LOG_LEVEL        = "INFO"
    }

    permissions = {
      s3_read     = ["raw_data"]
      s3_write    = ["processed_data"]
      glue_update = ["analytics"]
      secret_read = ["database_creds"]
    }

    event_source_mapping = [{
      event_source_arn = "s3:raw_data"
      events           = ["s3:ObjectCreated:*"]
      filter_prefix    = "incoming/"
      filter_suffix    = ".csv"
    }]
  }
}

# Glue ETL Job
glue_jobs = {
  data_transform = {
    name              = "transform"
    description       = "Transforms processed data"
    glue_version      = "4.0"
    worker_type       = "G.1X"
    number_of_workers = 5
    max_retries       = 2
    timeout           = 120

    script_location = "s3_file:transform_script"

    arguments = {
      "--job-language"                     = "python"
      "--enable-metrics"                   = "true"
      "--enable-continuous-cloudwatch-log" = "true"
      "--DATABASE"                         = "gluedb:analytics"
      "--INPUT_BUCKET"                     = "s3:processed_data"
      "--DB_SECRET"                        = "secret:database_creds"
    }

    permissions = {
      s3_read     = ["processed_data"]
      glue_update = ["analytics"]
      secret_read = ["database_creds"]
    }

    trigger_type = "SCHEDULED"
    schedule     = "cron(0 2 * * ? *)"  # Daily at 2 AM UTC
  }
}

Final Thoughts

This framework demonstrates that native Terraform can be elegant and developer-friendly without sacrificing power or flexibility. By leveraging Terraform's built-in features creatively—for expressions, try() functions, split() operations, and locals—we've built a system that:

Feels like configuration (simple tfvars files)
Works like Terraform (native tooling, no preprocessing)
Scales like a platform (hundreds of workloads, multiple teams)
Governs like policy (automatic enforcement, no manual reviews)

The journey from verbose, error-prone Terraform code to concise, validated configuration files represents a significant step forward in Infrastructure as Code maturity. Most importantly, it's achieved through native Terraform capabilities, ensuring long-term compatibility and eliminating external dependencies.

As organizations scale their cloud infrastructure, frameworks like this become essential for maintaining velocity, consistency, and security. The patterns demonstrated here can be adapted to any cloud provider, resource types, or organizational requirements—the principles of smart lookups, building block abstraction, and configuration separation are universally applicable.

The future of Infrastructure as Code is declarative, native, and developer-friendly. This framework is a blueprint for getting there.

Acknowledgments

This framework was built by collaborative iteration between platform engineers and application teams, learning from real-world challenges and continuously refining the developer experience. Special recognition to the teams who adopted early versions, provided feedback, and helped shape the patterns documented here.

The Hidden Dangers in Our Software Supply Chain: Why It's Bigger Than You Think

Jacob — Mon, 29 Sep 2025 12:31:02 +0000

In today's fast-paced digital world, software powers everything from the apps on our phones to the cloud systems running global businesses. But behind the scenes, there's a complex web called the software supply chain—the process of sourcing, building, and delivering software components. Traditionally, we've focused on securing this chain during application development and deployment. However, as technology evolves, risks are creeping into unexpected places, like the tools developers use every day and even the software on our personal devices.

This article aims to shed light on how the software supply chain has expanded far beyond just building and shipping apps. We'll break it down in simple terms for non-technical leaders, project managers (PMs), and engineers who might not deal with package management daily. By the end, you'll understand why awareness is crucial and how these hidden vulnerabilities could impact your team or organization.

What Is the Software Supply Chain, Anyway?

Imagine building a house: You don't make every brick, nail, or window yourself. Instead, you source them from suppliers, assemble them, and ensure the final structure is safe. The software supply chain is similar—it's the ecosystem of code, libraries, and tools that developers "source" to create software.

Key elements include:

Open-source packages: Reusable code snippets from public repositories like npm (for JavaScript), PyPI (for Python), or Cargo (for Rust). These are free and community-driven, speeding up development.
Dependencies: Packages often rely on other packages, creating a chain (or "tree") of interconnected code. This is recursive—meaning one package might pull in dozens or hundreds more.
SBOM (Software Bill of Materials): A list of all components in your software, like an ingredients label on food. It helps track what's inside.

In the past, security focused on scanning these packages for vulnerabilities during app building and deployment. Tools would check for known issues, generate an SBOM, and ensure everything was safe before going live. But that's no longer enough. The supply chain has infiltrated deeper into our workflows.

The Old Way: Focusing Only on App Development and Deployment

Historically, teams treated the software supply chain as something confined to production applications—the software your customers use. For example:

A developer writes code for a web app.
They install packages via commands like npm install or pip install.
During the build process (often in a CI/CD pipeline, which automates testing and deployment), the team scans for malicious code or vulnerabilities.
An SBOM is created to document all dependencies.

This approach worked reasonably well for "end-product" software. If a package had a flaw, it could be caught before deployment. But it assumed risks were limited to that final app. Reality? Not so much.

The Expansion: DevOps Tools, Local Installs, and Recursive Risks

Today, software development isn't just about the app—it's about the entire ecosystem supporting it. DevOps (a blend of development and operations) tools, which help automate infrastructure, testing, and monitoring, are often built on the same package systems like JavaScript or Python.

Consider this scenario:

A developer or DevOps engineer runs npm install aws-cdk (a tool for managing AWS cloud infrastructure) on their local laptop or in a CI/CD pipeline.
This isn't for the main app; it's for setting up servers, databases, or deployment scripts.
But aws-cdk has its own dependencies, which have more dependencies, and so on. It's a recursive rabbit hole—hard to fully map or monitor.

Unlike production apps, these tools often run in less controlled environments:

Local machines: Developers install packages directly on their laptops, outside formal security scans.
CI/CD pipelines: Automated workflows that build and deploy code, but they might pull in unvetted packages mid-process.
Everyday tools: Even non-app software like code editors or AI assistants relies on these packages.

The problem? We might vet the top-level package (e.g., the first version of aws-cdk we install), but we have no visibility into the deeper layers. A single update or hidden dependency could introduce risks without anyone noticing.

Evolution of Attacks: From End-User Exploitation to Proactive Credential Theft and Spread

Supply chain attacks aren't new, but their tactics are evolving. Older incidents often focused on injecting malicious code into applications to target end-users, such as visitors to a website. For instance, compromised packages like ua-parser-js in 2021 installed cryptominers or trojans that could steal data from browsers, aiming to hijack cryptocurrency wallets or personal information from unsuspecting users. Similarly, the 2018 event-stream hack embedded a backdoor to steal Bitcoin from apps built with the package, affecting the final deployed software and its users.

These attacks typically relied on the malicious code being bundled into the application and executed in production environments, passively waiting to exploit visitors or users. The damage was downstream, impacting customers rather than the development process itself.

In contrast, more recent worms like Shai-Hulud represent a shift toward aggressive, self-propagating threats that target the development environment upfront. Discovered in mid-September 2025, Shai-Hulud is a worm that infects npm packages, using "onInstall" hooks to run automatically upon installation. It doesn't just sit idle—it actively scans for secrets such as npm tokens, GitHub credentials, AWS keys, and environment variables on local laptops, in CI/CD pipelines, and even queries cloud metadata for ephemeral credentials. Tools like TruffleHog are embedded to deepen the scan, and stolen data is exfiltrated to public GitHub repos created by the worm itself.

What makes it worm-like is its propagation: Using pilfered tokens, it automatically republishes compromised versions of other packages under the maintainer's control, turning victims into unwitting spreaders. This moves the infection window forward, infiltrating source repositories (like GitHub) and cloud services, potentially compromising entire organizations. Over 500 packages were affected in this campaign, including those linked to major companies like CrowdStrike.

This evolution from passive end-user attacks to proactive dev-side credential hunting amplifies the risk, as it enables exponential spread across the supply chain before detection.

A Real-World Wake-Up Call: The Shai Hulud NPM Worm

To illustrate the danger, let's look at a recent incident: the Shai Hulud worm, a malicious package that spread through npm in 2025.

Here's what happened in simple terms:

Shai Hulud disguised itself as a legitimate package.
It used an "onInstall" hook—a script that runs automatically when the package is installed.
Once installed, it didn't just sit there. It scanned for sensitive data like API keys, passwords, or "secrets" on the local machine.
It went further: In CI/CD pipelines, it hunted for environment variables (secret values used in automation).
Even scarier, it attempted to access cloud services like AWS to steal more secrets.

This wasn't limited to production apps. Developers running npm install locally exposed their personal laptops. Pipelines automating DevOps tasks became infection points. The worm spread because npm packages are public—anyone can publish without upfront checks. No gatekeepers, just trust in the community.

Shai Hulud highlights a key flaw: Even if you trust the initial package, recursive dependencies mean you might unwittingly install something harmful. And since DevOps tools run with high privileges (e.g., access to cloud accounts), the fallout can be massive—data breaches, financial loss, or operational downtime.

Why Are We So Vulnerable? The Trust Issue with Open Source

Open source is amazing—it democratizes innovation and accelerates progress. But our over-reliance on it creates blind spots. Here's why:

Public Repositories Without Validation:
- Platforms like npm.js and PyPI are open to all. Anyone with an account can upload a package—no mandatory security reviews.
- Contrast this with app stores like Apple's, which vet submissions. In open source, it's "publish first, fix later."
Recursive Dependencies Make Visibility Impossible:
- A simple install can pull in hundreds of sub-packages. Tools like dependency trees exist, but they're not always used, especially in ad-hoc DevOps tasks.
- Updates happen automatically in many setups, introducing new risks without notice.
Beyond App Pipelines:
- Risks extend to non-development tools. For instance:
- VSCode (Visual Studio Code): A popular code editor built on Electron (a framework for desktop apps using web tech). It uses npm packages and supports extensions that pull in more code.
- Claude or similar AI tools: Many are built with Python or JS dependencies, potentially vulnerable.
- Electron-based apps: These are desktop programs (like Slack or Discord) that auto-update. An update could inject malicious code via a compromised package.

In short, open source is everywhere—not just in your company's app, but in the tools enabling your workflow. We trust it because it's convenient, but that trust is often unearned.

The Broader Impact: From Laptops to Critical Infrastructure

This isn't just a developer problem; it affects entire organizations:

Non-Tech Leaders and PMs: Budgets for security might focus on apps, but a breach in a DevOps tool could halt projects, expose customer data, or lead to compliance fines (e.g., GDPR or HIPAA).
Engineers Outside the Process: If you're not handling packages daily, you might assume IT has it covered. But local installs bypass central controls.
Global Scale: Supply chain attacks like SolarWinds (2020) or Log4j (2021) showed how one vulnerability ripples worldwide. Now, with DevOps proliferation, the attack surface is larger.

Imagine a PM overseeing a cloud migration: The team uses aws-cdk in pipelines. A worm like Shai Hulud sneaks in, steals AWS credentials, and compromises your entire infrastructure. It's not hypothetical—it's happening.

Moving Forward: Awareness Is Key, with a Focus on Hygiene

The software supply chain has evolved, and so must our mindset. It's no longer just about the app you deploy; it's about every tool, script, and dependency in your ecosystem.

To build resilience:

Educate Teams: Train everyone—from execs to engineers—on supply chain risks. Use SBOMs not just for apps, but for tools.
Adopt Best Practices: Lock dependencies to specific versions, use private registries for vetted packages, and scan local/CI environments regularly.
Emphasize Hygiene: Maintain laptop and pipeline hygiene by using minimal, scoped credentials—grant only the access needed for a task, and rotate them frequently. Avoid storing secrets in plain text or environment variables without encryption, and use tools like secret managers (e.g., AWS Secrets Manager) to limit exposure. This is especially critical against worms like Shai-Hulud that hunt for broad tokens to spread to clouds and repos.
Question Trust: Don't assume open source is safe. Verify publishers, review changelogs, and limit auto-updates.

Awareness is the first step. By recognizing that vulnerabilities lurk in DevOps tools, local installs, and everyday software—and that attacks are shifting to proactive theft for wider spread—we can foster a culture of caution. In a world where software is ubiquitous, staying vigilant isn't optional—it's essential for security and success.

What are your thoughts? Have you encountered supply chain issues in your tools? Share in the comments below!

This article is based on emerging trends in software security as of September 2025.

Image credit: https://xkcd.com/2347/

Why Debian packages are safer then NPM and PyPi

Jacob — Fri, 19 Sep 2025 09:36:46 +0000

In the world of software development, package repositories are critical for distributing libraries and dependencies. But how do they ensure no malicious code slips through? Debian’s stable repository, npmjs (npm registry), and PyPI (Python Package Index) take vastly different approaches to security. Debian’s tightly controlled process contrasts with the open, fast-paced ecosystems of npmjs and PyPI, which prioritize ease of publishing but face higher risks of supply chain attacks. Let’s compare their security measures and see why the latest NPM work attack could spread.

Security Measures Compared

Aspect	Debian Stable	npmjs (npm Registry)	PyPI (Python Package Index)
Uploader Vetting	Strict: Only vetted Debian Developers/Maintainers (DDs) after identity verification and competence checks. No anonymous uploads.	Minimal: Anyone with a free account; 2FA encouraged but not mandatory. Accounts can be compromised easily.	Minimal: Anyone can create a free account; MFA recommended but not enforced for all.
Pre-Publication Review	High: Source code reviewed by maintainers and community; packages staged in "unstable" and "testing" for months before stable inclusion.	None: Packages published directly without code review; spam detection may block obvious violations.	None: Packages uploaded directly; no mandatory code review or static analysis.
Package Signing/Verification	Yes: GPG-signed packages and archive; reproducible builds ensure binaries match source.	Partial: Packages not signed by default; relies on HTTPS and hash checks (e.g., via `npm shrinkwrap`).	Partial: No built-in signing; users can enable hash checking in pip for integrity.
Automated Testing/Scanning	Extensive: Lintian checks, CI across architectures, and migration criteria enforced by Release Team.	Post-facto: `npm audit` scans for known vulnerabilities in dependencies after install. No pre-scan.	Post-facto: Tools like `pip-audit` or Safety CLI scan for known issues; no pre-upload scanning.
Malicious Package Handling	Proactive: Security Team monitors CVEs, backports fixes to stable; packages removed if compromised. Rare incidents due to vetting.	Reactive: Security team removes reported malware (hundreds annually); advisories issued. Common attacks: typosquatting, brandjacking.	Reactive: Security team removes reported packages (e.g., thousands in campaigns); advisories via email. Common attacks: typosquatting, wheeljacking.
Community / Transparency	High: Public bug tracking (BTS), open logs, and community scrutiny; emphasis on minimal changes in stable.	Medium: GitHub-hosted source often reviewed post-publish; tools like Socket/Phylum for community scans.	Medium: GitHub integration for source; community reports via security@pypi.org; tools like PyPI-scan for typosquatting detection.
Known Incidents (Recent Examples)	Rare: No major supply chain attacks in stable; focus on backporting upstream fixes.	Frequent: 2025 chalk/debug compromise (2.6B weekly downloads at risk); event-stream hijack (2018).	Frequent: 2025 campaigns with thousands of malicious packages; Lolip0p malware (2023, 550+ downloads).
User-Side Protections	Built-in: `apt` verifies signatures; unattended-upgrades for auto-security fixes.	`npm install --ignore-scripts` to block lifecycle hooks; SCA tools for SBOM generation.	`pip install --require-hashes`; virtual environments; avoid running as root.

Key Insights for Developers

Debian’s Fortress: Debian’s stable repository is a gold standard for security, with vetted maintainers, signed packages, and extensive testing. It’s ideal for production servers where stability is paramount, but its slow release cycle may lag for cutting-edge features.
npmjs and PyPI’s Open Gates: Both npmjs and PyPI prioritize accessibility, allowing anyone to publish with minimal checks. This enables rapid innovation but leaves them vulnerable to typosquatting and malware (e.g., thousands of malicious PyPI packages in 2025). Their reactive approach relies on community reporting and post-install audits.
What You Can Do: For npmjs and PyPI, adopt a “trust but verify” mindset. Use locked dependencies (package-lock.json, requirements.txt), run tools like npm audit or pip-audit, and integrate security scans into CI/CD pipelines. For Debian, leverage apt’s built-in protections and keep systems updated.

Stay Safe in the Ecosystem

Debian’s rigorous gatekeeping makes it a low-risk choice for deployments, while npmjs and PyPI require proactive vigilance from developers. By understanding these differences, you can better secure your projects against supply chain attacks.

For more details, check out:

Setting Up IOMete: A Cloud-Independent Data Platform Based on Spark

Jacob — Tue, 10 Jun 2025 14:38:25 +0000

IOMete is a powerful, cloud-independent data platform built on Apache Spark, designed to enable scalable data processing and analytics. This guide walks you through the process of setting up IOMete on a Kubernetes cluster, covering the installation of prerequisites, configuration of storage and database components, and deployment of the IOMete data plane. By the end, you’ll have a fully functional IOMete environment ready for data workloads.

Prerequisites

Before diving into the installation, ensure you have the following:

A Kubernetes cluster (version 1.21 or higher recommended).
kubectl configured to interact with your cluster.
Helm (version 3.x) installed for managing chart deployments.
yq (a YAML processor) installed for modifying configuration files.
aws-cli installed for interacting with MinIO (configured as an S3-compatible storage).
At least 32GB of RAM and 4 CPU cores available in your cluster for IOMete’s components.
Access to the IOMete Helm chart repository and configuration files on GitHub.

This guide assumes you’re comfortable with basic Kubernetes and Helm commands. Let’s get started!

Downloading Configuration Files

To begin, you’ll need to download the necessary configuration files from the IOMete GitHub repository. These files include Custom Resource Definitions (CRDs), service accounts, certificate generation scripts, and example configurations for the data plane, Istio gateways, PostgreSQL, and MinIO.

Run the following commands to fetch the files:

wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/iomete-crds.yaml
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/service-account.yaml
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/gencerts.sh
chmod +x gencerts.sh
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/on-prem/example-data-plane-values.yaml
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/istio-ingress/gateway-http.yaml
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/istio-ingress/gateway-https.yaml
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/database/postgresql/postgresql-values.yaml
wget https://raw.githubusercontent.com/iomete/iomete-deployment/main/minio/minio-test-deployment.yaml

These files provide the foundation for deploying IOMete’s components. The gencerts.sh script, for example, generates certificates for the Spark operator webhook, while example-data-plane-values.yaml serves as a template for configuring the IOMete data plane.

Shrinking CRD Size for Kubernetes

Some Kubernetes environments impose a size limit on Custom Resource Definitions (CRDs), such as 256KB. The iomete-crds.yaml file may exceed this limit due to included descriptions. To address this, you can use the yq tool to remove the description fields, reducing the file size.

Execute the following command:

yq 'del(.. | .description?)' iomete-crds.yaml > iomete-crds-small.yaml

This creates a new file, iomete-crds-small.yaml, which is compatible with environments that enforce CRD size restrictions. You’ll use this file in later steps.

Adding Helm Repositories

IOMete relies on several Helm charts from different repositories, including Bitnami (for PostgreSQL), Istio (for networking), and IOMete’s own chart repository. Add and update these repositories with the following commands:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo add iomete https://chartmuseum.iomete.com
helm repo update

This ensures you have access to the latest versions of the required charts.

Setting Up the IOMete Namespace and Core Components

Next, create a dedicated namespace for IOMete and apply the necessary configurations, including the CRDs, service account, and Spark operator webhook certificates.

Run these commands:

kubectl create namespace iomete-system
kubectl label namespace iomete-system iomete.com/managed=true
kubectl apply -f iomete-crds-small.yaml
kubectl apply -n iomete-system -f service-account.yaml

./gencerts.sh -n iomete-system -s spark-operator-webhook -r spark-operator-webhook-certs
# `spark-operator-webhook.yaml` file will be generated by the script above
kubectl apply -n iomete-system -f spark-operator-webhook.yaml

Here’s what each step does:

Creates the iomete-system namespace and labels it for IOMete management.
Applies the downsized CRDs to define IOMete’s custom resources.
Sets up a service account for IOMete’s components.
Generates and applies certificates for the Spark operator webhook, enabling secure communication.

Deploying MinIO for Storage

IOMete uses MinIO, an S3-compatible object storage, as its default storage backend. Deploy MinIO with the provided test configuration:

kubectl apply -n iomete-system -f minio-test-deployment.yaml

To interact with MinIO, set up port forwarding to access its web interface or API. Open a new terminal and run:

kubectl port-forward svc/minio 9000:9000

Creating an S3 Bucket in MinIO

With MinIO running, create a bucket named lakehouse for IOMete’s data storage. Use the aws-cli to configure access and create the bucket:

# export access key and secret key
# If you changed the default values, please update the following values accordingly
export AWS_ACCESS_KEY_ID=admin
export AWS_SECRET_ACCESS_KEY=password
export AWS_REGION=us-east-1
export AWS_ENDPOINT_URL=http://localhost:9000

# create s3 bucket
aws s3 mb s3://lakehouse

# verify buckets
aws s3 ls s3://lakehouse

These commands:

Set environment variables for MinIO’s default credentials (admin/password) and endpoint.
Create the lakehouse bucket.
Verify the bucket’s creation.

Once done, close the port-forwarding session with Ctrl+C.

Deploying PostgreSQL

IOMete requires a PostgreSQL database for metadata and configuration. Install PostgreSQL using the Bitnami Helm chart and the provided configuration file:

helm upgrade --install -n iomete-system postgresql bitnami/postgresql -f postgresql-values.yaml
kubectl get pods -n iomete-system -l app.kubernetes.io/name=postgresql --watch

The helm upgrade --install command ensures PostgreSQL is installed or updated. The --watch flag monitors the pod’s status. Wait until the PostgreSQL pod is in the Running state, then press Ctrl+C to exit the watch command.

Configuring the IOMete Data Plane

Before deploying IOMete, verify and customize the example-data-plane-values.yaml file to match your environment. Below is an example configuration:

database:
 type: postgresql
 host: "postgresql"
 port: "5432"
 user: "iomete_user"
 password: "iomete_pass"
 prefix: "iomete_" # all IOMETE databases should be prefixed with this. See database init script.
 ssl:
  enabled: false # Enabling this will require javaTrustStore to be enabled and configured properly
  mode: "disable" # disable, verify-full
 adminCredentials:
  user: "postgres"
  password: "<your postgresql master password"

storage:
 bucketName: "lakehouse"
 type: "minio"
 minioSettings:
  endpoint: "http://minio:9000"
  accessKey: "admin"
  secretKey: "password"

ingress:
 httpsEnabled: false

docker:
 repo: iomete.azurecr.io/iomete
 pullPolicy: Always
 defaultSparkVersion: 3.5.3-v13
 additionalSparkVersions:
  - 3.4.0-v12
 tagAliases:
  latest: 3.5.3-v13

features:
 activityMonitoring:
  enabled: true

Key configurations include:

Database: Points to the PostgreSQL instance with credentials and prefix settings.
Storage: Configures the MinIO lakehouse bucket with default credentials.
Ingress: Disables HTTPS for simplicity (enable it for production).
Docker: Specifies the IOMete container registry and Spark versions.
Features: Enables activity monitoring for tracking usage.
Replace with the actual PostgreSQL admin password defined in postgresql-values.yaml. Adjust other settings as needed for your environment.

Deploying the IOMete Data Plane

With all prerequisites in place, deploy the IOMete data plane using the Helm chart. This step initializes the database, configures storage, and starts all necessary pods. The deployment requires at least 32GB of RAM and 4 CPUs.

Run the following command:

helm upgrade --install -n iomete-system data-plane iomete/iomete-data-plane-enterprise -f data-plane-values.yaml

The deployment may take a few minutes as Helm sets up the initialization job and starts the pods. Monitor the progress with:

kubectl get pods -n iomete-system --watch

Accessing the IOMete Web Interface

Once the deployment is complete, access the IOMete web interface by forwarding the iom-gateway service:

kubectl port-forward svc/iom-gateway 8888:8080

Open your browser and navigate to http://localhost:8888. Log in with the default credentials:

Username: admin
Password: admin

Change the default password after logging in for security.

Next Steps

Congratulations! You’ve successfully set up IOMete as a cloud-independent data platform. From here, you can:

Configure data sources and Spark jobs in the IOMete UI.
Enable HTTPS for secure access by updating the ingress settings.
Scale the cluster to handle larger workloads.
Explore IOMete’s documentation for advanced features like multi-tenancy and monitoring.

If you encounter issues, check the pod logs in the iomete-system namespace with kubectl logs or consult the IOMete documentation.

This setup provides a robust foundation for running Spark-based data workloads in a cloud-agnostic environment. Let us know in the comments if you have questions or tips for optimizing your IOMete deployment!

Preview

Data Domain/Workspace

SQL Editor

DBT-Core

Forem: Jacob

The Art of Agents: Sun Tzu's Principles for Building Agentic AI Systems

The Thesis

The Thirteen Chapters

Five Principles That Matter Most

The Cycle

Three Decisions, No Waste

The Anti-Patterns

Read the Preview

The AI Development Workflow: A Complete System for Working with AI Agents

The Big Picture

The Essential Flow (Start Here)

Step 1: Brainstorm — Design the Feature with AI

Step 2: Plan — Create Issues & Split the Work

Step 3: Execute — AI Agents Build Each Issue

Step 4: Review — Validate Tests & Code

Step 5: Audit — System-Wide Check

The Full System (Level Up)

🆕 Triage: Prioritize Before You Execute

⚡ Parallel Agent Orchestration

🚧 Automated Quality Gates

📡 Monitor: Close the Loop with Reality

🧠 Project Memory: The Context Layer

All Feedback Loops

Getting Started

From Extension to Orchestration: Who Wins (and Who Gets Left Behind) in the Claude 4.6 Era

The Momentum Advantage: ADHD & Creative Types Pull Ahead

The Flip Side: Task-Oriented Engineers Face the Biggest Shift

Strategies to Thrive: From Implementer to Orchestrator

From Dark Flow to Real Momentum: Why Claude Opus 4.6 Feels Like an Extension of Me

Efficient Agentic AI Development Guide (begin 2026)

Core Principles

1. Model Intelligence = Less Prompting Required

2. Prevent Context Degradation

3. Single Task Focus

Workflow Pattern: Brainstorm → Plan → Execute

Phase 1: Brainstorm

Phase 2: Plan

Phase 3: Execute

Agent-Assisted Debugging

Provide Diagnostic Tools

Effective Debugging Sessions

Example MCP Tools for Debugging

Repository Structure Best Practices

Include an AI-Friendly README

Provide Tool Access

Context Documents

Quick Reference

Anti-Patterns to Avoid

Remember

OpenClaw: The Open-Source Agent That Feels Too Alive

I Stopped Maintaining Terraform Examples and Tests Separately. Here's Why.

The Problem (Level 200)

What Changed (Level 200)

Advanced Patterns (Level 400)

Pattern 1: Progressive Example Validation

Pattern 2: Configuration Variations Without Bloated Examples

Pattern 3: Validate Constraints Early

Pattern 4: Integration Testing (Real Dependencies)

Pattern 5: Lock Down Your Output Contract

Getting Started

Step 1: Audit Your Examples

Step 2: Create Test Files

Step 3: Write One Assertion Per Test

Step 4: Run Locally

Step 5: Add to CI

Step 6: Repeat

Why This Matters

The Takeaway

Further Reading

Your Terraform Examples Are Broken (And You Don't Know It Yet)

The Problem

Part 1: Foundation (Level 200)

The Traditional Problem

What Changed: Terraform Testing Framework

The Insight: Stop Duplicating

Here's How It Works

Part 2: Advanced Patterns (Level 400)

Pattern 1: Multi-Stage Example Testing

Pattern 2: Testing Configuration Variations