Forem: Michael Tuszynski

38% of AI Answers Are Wrong — And It's Your Prompt's Fault

Michael Tuszynski — Tue, 14 Apr 2026 15:16:26 +0000

Every week someone posts about AI hallucination like it's a mystery. It's not. A 2025 Frontiers in AI study measured it: vague, multi-objective prompts hallucinate 38.3% of the time. Structured, single-focus prompts? 18.1%. That's a 20-point accuracy gap from how you write the prompt — not which model you pick.

Everyone's debating GPT vs. Claude vs. Gemini. Nobody's talking about the fact that prompt structure matters more than model selection for most use cases.

The $0 Fix Nobody Uses

Research from SQ Magazine breaks it down further: zero-shot prompts (no examples, no structure) hallucinate at 34.5%. Add a few examples and that drops to 27.2%. Add explicit instructions: 24.6%. Simply adding "If you're not sure, say so" cuts hallucination by another 15%.

That last one is worth repeating. One sentence — "If you're not confident, say you don't know" — is worth more than upgrading your model tier. And it costs nothing.

Why Multi-Task Prompts Are the Worst Offender

"Summarize this doc, extract the key risks, and draft a response email" feels like one task. It's three. And each additional objective gives the model more room to fabricate connections between things that don't connect.

Language models are next-token predictors. Single task = narrow probability distribution = the model knows where it's headed. Three tasks stacked together = triple the surface area for error. A small fabrication in the summary becomes a stated fact in the risk analysis becomes a confident assertion in the draft email.

Longer, multi-part prompts increase error rates by roughly 10%. In legal contexts, hallucination rates run between 58% and 88%. That's not an AI problem. That's a prompting problem.

What Actually Works (With Numbers)

One prompt, one job. Summarize the doc. Stop. Review it. Then extract risks from the verified summary. Then draft the email from verified risks. Three prompts, each building on confirmed output.

Constrain the output. JSON, numbered lists, specific templates. Structured prompts cut medical AI hallucinations by 33%. The less room to improvise, the less it fabricates.

Give examples. Zero-shot to few-shot: 34.5% → 27.2%. Two examples costs you 30 seconds and buys a 7-point accuracy gain.

Set refusal conditions. "If confidence is below 70% or no evidence supports the claim, say 'insufficient data.'" You're not weakening the model. You're giving it a pressure valve so it doesn't fill gaps with fiction.

The Model Isn't the Bottleneck

The best models went from 21.8% hallucination in 2021 to 0.7% in 2025 on benchmarks. But benchmarks test clean, single-objective tasks. Real-world, multi-step workflows — the kind actual professionals run — depend more on how you ask than what you ask.

You wouldn't hand a contractor one work order that says "remodel the kitchen, fix the plumbing, and repaint the exterior." You'd scope each job, inspect the work, then move on.

The people getting the best results from AI already know this. Everyone else is blaming the model.

Building a Plugin Marketplace for AI-Native Workflows

Michael Tuszynski — Fri, 10 Apr 2026 22:55:16 +0000

Most AI coding tools ship as monoliths. One big system prompt, one set of capabilities, one-size-fits-all. That works fine for general software engineering. It falls apart the moment you need domain-specific workflows that vary by role, by team, and by engagement.

I build presales systems at Presidio — client research, SOW generation, meeting capture, deal operations. The kind of work where a solutions architect needs different tools than a deal desk analyst, and where loading everything into every session wastes tokens and degrades output quality.

So I built a plugin marketplace for Claude Code. Nine modular plugins, independently installable, composable by role. Here's what I learned shipping it to a team.

Why Plugins Instead of One Big Workspace

The original system was a monolith — 56 commands, 14 skills, 36 tools, all loaded into every session. It worked for me as the sole operator. The moment I tried to share it with other consultants, three problems surfaced immediately:

Context pollution. A consultant doing SOW work doesn't need the meeting transcription pipeline, the competitive intel framework, or the deal management commands cluttering their context window. Every irrelevant token degrades the model's attention on the task at hand.

Onboarding friction. "Clone this repo, read 200 lines of docs, configure 18 environment variables, and learn 56 commands" is not an adoption strategy. People need to install what they need and ignore what they don't.

Maintenance coupling. A bug fix in the meeting recorder shouldn't require every user to pull an update that also touches their SOW pipeline. Independent versioning matters when your users are busy consultants, not developers.

The fix was decomposition. Break the monolith into plugins that can be installed, updated, and removed independently.

The Architecture That Emerged

fulcrum-plugins/
├── plugins/
│   ├── core/          # Foundation: persona, auth, OneDrive, shared rules
│   ├── intel/         # Client research pipeline
│   ├── meeting/       # Silent recording + transcription
│   ├── discovery/     # Call prep, qualification, opportunity analysis
│   ├── sow/           # SOW drafting, review, QA, redlines
│   ├── proposal/      # Solution decks and pricing workbooks
│   ├── ops/           # Daily briefing, weekly review, context switching
│   ├── engage/        # Deal management, delivery handoff
│   └── util/          # Freshness audits, triage, workspace maintenance
├── shared/            # Frameworks and templates used across plugins
└── docs/

Each plugin is a self-contained directory with commands, scripts, rules, and hooks. One required dependency — core — provides identity, authentication, and the shared file system. Everything else is optional.

Installation is one command: /plugin install sow@fulcrum-plugins. Uninstall is equally clean. No cross-plugin imports, no shared state beyond what core provides.

Recommended Sets Over Required Bundles

Rather than prescribing one configuration, the marketplace offers recommended sets:

Minimal (SOW-focused): core + sow — for consultants who only write statements of work
Meeting-heavy: core + meeting + discovery — for consultants running client calls all day
Full presales: all 9 plugins — for people like me who touch every phase

This respects how people actually work. Nobody uses every tool every day. The consultant who installs just core + sow gets a focused, fast experience. The one who installs everything gets the full operating system. Both are first-class citizens.

Namespace Isolation Matters More Than You Think

When I first decomposed the monolith, every plugin had commands like /draft, /review, /status. Collisions everywhere. The fix was namespace syntax: /sow:draft, /intel:company-intel, /ops:weekly-review.

This felt verbose at first. It turned out to be the single most important design decision. Namespaces do three things:

Eliminate ambiguity. /draft could mean a SOW draft, a proposal draft, or an email draft. /sow:draft is unambiguous.
Enable discovery. A new user can type /sow: and see every SOW command without memorizing a list. The namespace IS the documentation.
Preserve plugin independence. Two plugin authors can independently create a status command without coordinating. /engage:status and /ops:status coexist without conflict.

I renamed all plugins in v1.1 specifically to get shorter prefixes — sow-pipeline became sow, research-intel became intel, engagement-lifecycle became engage. Every keystroke matters when you're typing these dozens of times a day.

Shared Lessons: Institutional Memory Without a Database

The feature I'm most proud of has zero lines of application code. It's a folder on OneDrive.

When a consultant learns something the hard way — a Salesforce field that's locked to AMs, a client's preferred meeting format, a compliance requirement that isn't documented anywhere — they drop a markdown file into a shared folder:

shared-lessons/jane-doe/2026-04-09-sfdc-stage-lock.md

At session start, a hook script scans all lesson files from the team, deduplicates by title, and renders them into a cached rule that loads into every session. The team's collective knowledge grows without anyone maintaining a wiki, attending a knowledge-sharing meeting, or filing a ticket.

The constraints are deliberate: one lesson per file, no client-confidential data, attribution required, dates required (stale lessons get pruned). The format is simple enough that non-developers can contribute. The mechanism — a OneDrive folder sync'd through Microsoft Teams — requires no new tools or logins.

This is the pattern I keep returning to: use infrastructure people already have. OneDrive, git, markdown files. Not a custom database, not a new SaaS tool, not an API integration. The best systems are the ones that disappear into workflows people already follow.

What Didn't Work

Plugin dependencies. The original design had plugins declaring dependencies on each other — sow depends on intel, engage depends on discovery. In practice, this created install-order headaches and made it harder to reason about what was loaded. The v1.3 architecture dropped all inter-plugin dependencies. Each plugin is fully self-contained. If sow needs client context, it reads the client's context file directly — it doesn't import a function from intel.

Automatic plugin updates. The original design auto-pulled updates on session start. In practice, this broke people mid-workflow when a command signature changed. The fix was making updates explicit — /core:update when you're ready, with a statusline indicator showing when updates are available. Users update on their own schedule, not yours.

Granular versioning. I initially tried to version each plugin independently. After three days of version coordination across nine plugins, I switched to a single marketplace version. All plugins share the version in VERSION. SemVer at the marketplace level, not the plugin level. Simpler to reason about, simpler to communicate ("update to 1.3.1"), simpler to tag.

The Adoption Signal

The most telling metric isn't installs — it's which recommended set people choose. When most of your users install the minimal set and gradually add plugins over weeks, you've built something that earns trust incrementally. When they install everything on day one and complain it's overwhelming, you've just shipped a monolith with extra steps.

So far, the pattern is healthy. New consultants start with core + sow (the job requirement), then add discovery after their first client call, then meeting after they see someone else's AI-generated meeting notes. Pull, not push.

Plugin architectures aren't new. What's new is applying them to AI agent context — treating the model's knowledge, rules, and capabilities as composable modules rather than a static system prompt. The tools exist today in Claude Code's plugin system. The hard part isn't the architecture. It's the discipline to decompose.

Context Engineering Is the New Prompt Engineering

Michael Tuszynski — Wed, 08 Apr 2026 22:06:58 +0000

Everyone's writing better prompts. Few are building better context.

That's the gap. Prompt engineering treats AI like a search box — craft the perfect query, get the perfect answer. Context engineering treats AI like a new team member — give them the right docs, the right access, and a clear understanding of how work actually gets done. As Andrej Karpathy put it, the hottest new programming language is English — but the program isn't the prompt. It's the context surrounding it.

I've spent the last six months building AI-native workflows at Presidio, where I'm a Principal Solutions Architect. Not chatbots. Not demos. Production systems where Claude Code agents run real presales operations — client research, proposal generation, meeting analysis, deal tracking. The kind of work that used to live in someone's head and a dozen browser tabs.

Here's what I learned about making AI actually useful.

Prompts Are Requests. Skills Are Frameworks.

The first mistake everyone makes: stuffing domain knowledge into prompts. "You are an expert in enterprise sales. When analyzing a deal, consider these 47 factors..."

That breaks immediately. Prompts are ephemeral — they disappear when the conversation ends. Domain knowledge needs to persist across sessions, get version-controlled, and evolve as you learn what works.

The pattern that works: skills files — what Claude Code's plugin architecture calls reusable domain knowledge. Markdown documents that encode decision frameworks, not instructions. A skill isn't "analyze this deal." A skill is the 5-gate qualification framework your team actually uses, written as structured markdown with decision criteria, red flags, and exit conditions. The AI reads it and applies it. You update the framework once, every future session uses the new version.

.claude/skills/
├── qualification-framework.md    # Decision gates with criteria
├── pricing-strategy.md           # Margin rules, discount authority
├── sow-review-rubric.md          # Evaluation checklist
└── competitive-positioning.md    # Differentiators by competitor

Skills are reusable. Prompts are disposable. That distinction matters more than any prompting technique.

Context-as-Code: Version Control Your AI's Brain

Every client engagement in my system has a single markdown file that serves as the AI's working memory for that account. Contacts, scope decisions, meeting history, action items, competitive intel — one file, version-controlled in git.

Why markdown? Three reasons:

It's grep-searchable. When an agent needs to find every mention of a specific technology across all accounts, grep -r "Kubernetes" clients/ works instantly. Try that with a vector database.
It diffs cleanly. Git shows you exactly what changed in the AI's understanding of an account. Who updated it, when, and why.
It's token-efficient. Structured markdown compresses well in context windows. A 200-line context file gives an agent everything it needs to operate on an account without RAG retrieval latency.

The anti-pattern is treating AI memory as a black box — embeddings you can't inspect, vector stores you can't diff, context you can't version. If you can't git blame your AI's knowledge, you don't control it.

One God-Agent Is a Trap

I started with one agent that did everything. It was mediocre at all of it.

The fix was domain specialization. Four agents, each with a clear role: one handles discovery and qualification, one handles technical design and proposals, one handles deal operations and pricing, one handles workspace maintenance. Each agent has its own tools, its own context, and a defined handoff protocol for passing work to another agent.

This mirrors how real teams work. Your sales engineer doesn't do contract redlines. Your deal desk doesn't design architecture. Specialization isn't just about accuracy — it's about cost. A maintenance agent running on Haiku costs 95% less than an Opus agent doing the same file cleanup.

Model routing by task complexity is the easiest money you'll save in AI:

Task Type	Model	Why
File organization, validation	Haiku	Structured, predictable
Research, summarization	Sonnet	Good reasoning, fast
Strategy, complex writing	Opus	Needs deep reasoning

Mistakes Must Become Infrastructure

Every production AI system has failure modes you won't predict. The question is whether failures teach the system or just annoy you.

My approach: every time an agent makes a mistake that I have to correct, it becomes a numbered rule in the system's configuration file. Not a mental note. Not a prompt tweak. A permanent, version-controlled rule that every future session reads on startup.

After six months, I have 39 of these. Examples:

"State files in .gitignore can vanish silently during merges — default to safe fallback values"
"Never infer what a client said in a meeting — only quote from the transcript or flag it as an assumption"
"Contract reverts produce the same error on every RPC — don't retry, they're non-recoverable"

These aren't prompt engineering. They're institutional memory encoded as code. The system gets smarter every time it fails, without retraining, fine-tuning, or hoping the model "remembers."

Don't Dump Everything Into Context

The biggest performance killer in AI systems isn't the model — it's context pollution. Loading every piece of knowledge into every session degrades output quality and burns tokens.

The pattern that works: modular context loading. My system has 14 skills, 56 commands, and context files for dozens of accounts. But any given session loads only what's relevant — the specific client context, the specific workflow skills, the specific agent role. Everything else stays on disk until needed.

Think of it like imports in code. You wouldn't import * from every module in your codebase. Don't do it with AI context either.

This also means your context files need to be current state, not changelogs. A context file that accumulates three months of historical notes becomes noise. Describe what the system is right now in 150 scannable lines. Put the changelog somewhere else.

The Stack That Actually Works

After building this across multiple enterprise engagements, here's the architecture I'd recommend for anyone building AI-native workflows:

Structured context files (markdown, git-tracked) over vector databases for domain knowledge
Skills (persistent frameworks) over prompts (ephemeral instructions) for domain expertise
Specialized agents with handoff protocols over one general-purpose agent
Cost-aware model routing — match model capability to task complexity
Error-to-rule pipelines — every failure becomes a permanent system improvement
Modular loading — only load context relevant to the current task

None of this requires a framework. No LangChain, no LlamaIndex, no orchestration layer. It's markdown files, a CLI, and good engineering discipline. The AI does the reasoning. You do the architecture.

The tools for building AI-native workflows exist today. The bottleneck isn't model capability — it's context architecture. Start treating your AI's knowledge like code: structured, versioned, reviewed, and intentionally loaded. That's the difference between a chatbot and a system.

AWS Frontier Agents: What $50/Hour Pen Testing and $30/Hour SRE Means for Platform Teams

Michael Tuszynski — Mon, 06 Apr 2026 00:57:56 +0000

AWS just launched two autonomous AI agents — Security Agent and DevOps Agent — and they're both generally available now. These aren't chatbots with polished wrappers. They're persistent, autonomous systems that run for hours or days without human oversight, doing work that previously required dedicated teams.

Here's what caught my attention, and why platform engineers should be paying close attention.

Two Agents, Two Big Problems

AWS Security Agent handles penetration testing. Not the "run a scanner and hand you a PDF" kind — it ingests your source code, architecture diagrams, and documentation, then operates like a human pen tester. It identifies vulnerabilities, builds attack chains, and validates that findings are real exploitable risks. Bamboo Health reported it "surfaced findings that no other tool has uncovered." HENNGE K.K. cut their testing duration by over 90%.

AWS DevOps Agent handles incident response and operational tasks. It correlates telemetry, code, and deployment data across your stack — AWS, Azure, hybrid, on-prem — and integrates with the observability tools you already use: CloudWatch, Datadog, Dynatrace, New Relic, Splunk, Grafana. Western Governor's University cut mean time to resolution from two hours to 28 minutes during a production incident by pinpointing a Lambda configuration issue that had been buried in undiscovered internal docs.

The preview numbers are worth noting: up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy.

The Pricing Makes the Strategy Obvious

This is where it gets interesting.

DevOps Agent costs $0.0083 per agent-second — roughly $29.88 per hour. AWS's own pricing examples show a small team running 10 investigations per month pays about $40. An enterprise running 500 incidents per month pays around $2,300. For context, a single on-call SRE costs you $150k-$200k/year fully loaded.

Security Agent runs at $50 per task-hour. A small API test costs about $173. A full application pen test runs around $1,200. Compare that to external pen testing firms charging $15k-$50k per engagement, with weeks of lead time and limited scope.

Both agents include a 2-month free trial. AWS is clearly betting on adoption velocity — get teams hooked on the speed and economics, then make it sticky through integration depth.

The DevOps Agent pricing also ties into existing AWS Support plans. Enterprise Support customers get 75% of their support charges back as DevOps Agent credits. Unified Operations customers get 100%. AWS is effectively saying: your support spend now buys you autonomous operations capacity.

What This Actually Means for Platform Teams

AWS calls these "frontier agents" — autonomous systems that work independently, scale massively across concurrent tasks, and run persistently. The framing matters because it signals a product category, not a one-off feature.

Three implications stand out:

Security becomes continuous, not periodic. Most organizations pen test their top 5-10 applications once or twice a year because of cost and staffing constraints. At $50/task-hour, you can afford to test everything, continuously. The security posture shift from "we tested our critical apps last quarter" to "every app gets tested every sprint" is significant.

Incident response gets a tireless first responder. The DevOps Agent doesn't replace your SRE team — it augments the 3am on-call rotation. It can start investigating before a human even picks up the page, correlating signals across your entire stack. By the time your engineer opens their laptop, the agent has already identified probable root cause with 94% accuracy.

The multicloud angle is deliberate. AWS built the DevOps Agent to work with Azure DevOps, GitHub, GitLab, and non-AWS observability tools. This isn't altruism — it's a land-and-expand play. Once your operational intelligence lives in AWS, migrating workloads away gets harder. But for teams running hybrid environments today, having a single agent that understands your entire topology is genuinely useful.

The Bigger Picture

Google and Microsoft have their own agentic plays, but AWS shipping two production-ready autonomous agents with per-second billing and free trials is a concrete move. The fact that both agents work with tools like Claude Code and Kiro for generating validated fixes signals that AWS sees these agents as part of a broader autonomous development loop — not isolated point solutions.

For platform engineering teams, the takeaway is practical: evaluate these agents against your current pen testing costs and incident response metrics. The economics alone justify a proof of concept. The free trial removes any excuse not to try.

The real question isn't whether AI agents will handle DevOps and security tasks. It's how fast your organization adapts its processes, roles, and trust models to work alongside them.

The systems behind enterprise AI adoption success - IBM

Michael Tuszynski — Sat, 28 Mar 2026 17:10:20 +0000

Everyone's Buying GPUs. Almost Nobody's Ready to Feed Them.

The enterprise AI conversation has a blind spot the size of a data center. Every budget meeting I've sat in over the past 18 months has the same shape: GPU allocation gets 70% of the discussion time, model selection gets 20%, and the data infrastructure that actually feeds those models gets whatever's left over. Usually about ten minutes and a vague reference to "we'll figure out storage later."

This is why most enterprise AI deployments stall after the proof of concept.

The Bottleneck Nobody Budgets For

Here's what happens in practice. A team spins up a promising AI workload — retrieval-augmented generation, a fine-tuning pipeline, an inference service. It works great on a curated dataset in a dev environment. Then they try to run it against production data at scale and everything falls apart. Not because the model is wrong, but because the storage layer can't deliver data fast enough, the pipeline can't unify sources across hybrid environments, and nobody planned for the I/O characteristics of AI workloads.

AI training and inference workloads have fundamentally different storage profiles than traditional enterprise applications. Training jobs need sustained sequential throughput across massive datasets. Inference needs low-latency random reads. Fine-tuning needs both, sometimes simultaneously. Your SAN that runs ERP just fine will choke on a distributed training job that's trying to saturate eight GPUs.

IBM's recent framing of AI-ready infrastructure gets this right: the systems layer — storage, compute fabric, automation — is where enterprise AI succeeds or dies. Not in the model layer.

The Data Gravity Problem

The reason storage matters so much for AI isn't just throughput. It's data gravity.

Enterprise data doesn't live in one place. It's spread across on-prem databases, cloud object stores, SaaS platforms, edge devices, and that one team's PostgreSQL instance that nobody wants to touch. IBM defines enterprise AI as the integration of AI across large organizations — but integration implies the data is accessible. In most companies, it isn't. Not in any unified, performant way.

This creates a cascading failure. Your RAG pipeline needs product data from SAP, customer interactions from Salesforce, and technical documentation from Confluence. Each source has different access patterns, different latency profiles, different security boundaries. Stitching them together with API calls and batch ETL jobs introduces hours of lag and creates brittle pipelines that break every time someone changes a schema.

The companies I've seen succeed at enterprise AI solve this problem first. They build a unified storage layer that can serve multiple AI workloads without requiring six different integration patterns. IBM's approach with Storage Fusion and FlashSystem targets exactly this — high-performance, unified storage that can handle the mixed I/O profiles of AI workloads across hybrid environments. Whether you're on their stack or not, the architectural principle holds: if your AI workloads can't access unified data at the speed they need it, no amount of GPU spend will fix your pipeline.

Hybrid Cloud Is the Reality, Not the Exception

There's still a persistent fantasy in some planning meetings that AI workloads will live entirely in one public cloud. Maybe someday. Right now, for regulated industries, for companies with significant on-prem investments, and for anyone who's done the math on data egress costs, hybrid is the reality.

And hybrid AI infrastructure is hard. You need consistent orchestration across environments. You need storage tiering that can move hot data close to compute without manual intervention. You need security and governance that doesn't collapse the moment data crosses a network boundary.

IBM identifies inadequate infrastructure as one of the top five AI adoption challenges — and in my experience, "inadequate" usually means "designed for a different era." The infrastructure that runs your web applications, your CI/CD pipelines, your traditional analytics workloads — it wasn't built for the throughput patterns, the data volumes, or the operational demands of production AI.

This isn't a rip-and-replace argument. Nobody's going to throw out their storage infrastructure overnight. But you need a plan for how your existing infrastructure evolves to support AI workloads, and that plan needs to happen before you commit to production deployments.

What Actually Works: Three Patterns From the Field

After spending time with organizations that have moved past the POC phase into production AI, I see three common patterns:

1. Storage-first capacity planning. Successful teams model their data pipeline throughput requirements before they size GPU clusters. They ask: "How fast can we feed data to training jobs?" and "What's our p99 latency for inference-time retrieval?" If the answers don't match the model's appetite, they fix storage before buying more compute.

2. Unified data access across environments. Whether it's IBM Storage Fusion, a well-architected MinIO deployment, or a managed cloud storage layer with on-prem caching, the pattern is the same: AI workloads get a single namespace to read from, regardless of where the source data physically lives. This eliminates the integration tax that kills most pipelines.

3. Automation of the data lifecycle. Production AI generates enormous amounts of intermediate data — checkpoints, embeddings, feature stores, evaluation datasets. Teams that automate tiering, retention, and cleanup avoid the "we ran out of storage on a Friday night" incident that's practically a rite of passage.

The Uncomfortable Math

Here's a rough calculation that sobers up most planning conversations. A mid-size enterprise running a fine-tuning pipeline on proprietary data with a 70B parameter model needs approximately 500TB of accessible, high-performance storage just for the training data, checkpoints, and model artifacts. That's before you add your RAG corpus, your vector store, and your evaluation datasets.

Now multiply that by the number of AI initiatives in your roadmap. Most enterprises I talk to have between five and fifteen active AI projects. The storage footprint adds up fast, and it needs to perform — not just exist.

The GPU shortage got all the headlines in 2024. The storage and data infrastructure gap is the quieter crisis that will define which companies actually ship production AI in 2026.

What CTOs Should Do Next Week

Stop treating infrastructure as a downstream consequence of model selection. Flip it around.

Audit your current storage throughput against the I/O demands of your planned AI workloads. Map where your training data lives and how many network hops separate it from your compute. Calculate the real cost of your data integration layer — not just the cloud bill, but the engineering hours spent maintaining brittle pipelines.

Then have an honest conversation about whether your infrastructure roadmap matches your AI ambitions. If there's a gap — and there almost certainly is — close it before you scale your GPU footprint. The fastest accelerator in the world is useless if it's starving for data.

The companies that figure this out won't just run AI. They'll run AI that actually works in production, at scale, without the 2 AM pages. That's a meaningful competitive advantage — and it starts with the infrastructure layer that nobody wants to talk about at the budget meeting.

Why Enterprise AI Infrastructure is Going Hybrid – and Geographic

Michael Tuszynski — Thu, 26 Mar 2026 04:39:27 +0000

The Cloud Repatriation Nobody Expected: Why Enterprise AI Is Pulling Compute Back from the Cloud

The original pitch for cloud computing was simple: stop buying servers, rent someone else's. For most workloads over the past fifteen years, that trade worked. But AI infrastructure has rewritten the economics, and enterprises are responding by doing something few predicted — they're moving compute closer to the data, not further away.

A recent DataBank survey found that 76% of enterprises plan geographic expansion of their AI infrastructure, while 53% are actively adding colocation to their deployment strategies. This isn't a minor adjustment. It's a structural shift in how organizations think about where AI workloads should run.

The Economics Changed Before the Strategy Did

Running inference on a large language model in a hyperscaler region costs real money. Not "line item you can bury in OpEx" money — more like "the CFO is asking questions in the quarterly review" money. GPU instance pricing on AWS, Azure, and GCP has remained stubbornly high because demand outstrips supply, and the cloud providers know it.

The math gets worse when you factor in data gravity. Most enterprises generate data in dozens of locations — retail stores, manufacturing plants, regional offices, edge devices. Shipping all that data to us-east-1 for processing, then shipping results back, creates latency and egress costs that compound as AI adoption scales.

Colocation flips this equation. You place GPU-dense compute in facilities close to where data originates, connect to cloud services where they make sense (object storage, managed databases, identity), and keep the expensive part — inference and fine-tuning — on hardware you control or lease at predictable rates.

Why "Cloud-Smart" Beats "Cloud-First"

The industry is moving toward what Seeking Alpha describes as a "cloud-smart" strategy — using public cloud, private cloud, and edge computing based on the workload profile rather than defaulting to one deployment model for everything.

This makes sense when you break down what AI workloads actually need:

Training still belongs in the cloud for most organizations. You need massive, bursty GPU capacity for weeks or months, then nothing. Buying that hardware outright is a terrible investment unless you're running training continuously. Hyperscaler reserved instances or on-demand capacity work fine here.

Inference is the opposite profile. It's steady-state, latency-sensitive, and runs 24/7. The cost-per-token adds up fast at scale. Running inference on colocated or on-premises hardware — especially with purpose-built accelerators — can cut costs 40-60% compared to cloud GPU instances, depending on utilization rates.

Fine-tuning sits in the middle. You need GPU capacity for days, not months, and the data involved is often sensitive enough that you don't want it leaving your network. A colocated setup with good connectivity to your data sources handles this well.

The Geography Problem Nobody Planned For

Data sovereignty and residency requirements are accelerating the geographic distribution of AI infrastructure in ways that pure cloud strategies can't easily accommodate.

The EU's AI Act imposes requirements on where and how AI systems process data. Healthcare organizations in the US deal with HIPAA locality requirements. Financial services firms face data residency rules that vary by jurisdiction. When your AI model needs to process customer data from Germany, running inference in a Virginia data center creates compliance headaches that no amount of architectural cleverness fully solves.

Enterprises are responding by deploying AI infrastructure across multiple geographies — not because they want the operational complexity, but because regulators and customers demand it. The 76% planning geographic expansion aren't chasing some multicloud vision. They're meeting regulatory reality.

The Edge Dimension

Hybrid edge-cloud architectures add another layer. Manufacturing plants running quality inspection models can't tolerate 200ms round-trip latency to a cloud region. Autonomous systems need inference at the point of action. Retail environments process customer interactions in real time.

These use cases demand on-site or near-site compute with cloud connectivity for model updates, monitoring, and periodic retraining. The architecture looks less like "cloud with edge caching" and more like "distributed compute with cloud coordination." The control plane lives in the cloud. The data plane runs where the data lives.

This is a harder architecture to build and operate than a cloud-native deployment. It requires teams who understand networking, hardware lifecycle management, and distributed systems — skills that many organizations let atrophy during the cloud migration years.

What This Means for Infrastructure Teams

If you're an infrastructure leader planning AI capacity for the next 2-3 years, here's the framework I'd use:

Audit your inference costs first. Most organizations are surprised by how much they're spending on cloud GPU instances for inference once they aggregate across teams and projects. This number is your baseline for a hybrid business case.

Map data gravity. Where does your training data originate? Where do inference requests come from? Where do results need to arrive? If the answer to all three is "the same cloud region," stay in the cloud. If it's "twelve different locations across three countries," you need a distributed strategy.

Don't build a GPU data center. Colocation with GPU leasing gives you the economics of owned hardware without the capital expenditure and refresh cycles. Companies like DataBank, Equinix, and CoreWeave are building exactly this model — dense GPU compute in colocation facilities with direct cloud interconnects.

Plan for heterogeneous accelerators. NVIDIA's dominance in training is real, but inference has viable alternatives — AMD Instinct, Intel Gaudi, AWS Inferentia, Google TPUs. A hybrid strategy lets you match accelerators to workload profiles instead of paying the NVIDIA tax on everything.

Invest in platform engineering. Hybrid AI infrastructure without a solid platform layer becomes an operational nightmare. You need consistent deployment pipelines, observability, and model lifecycle management that works across cloud regions, colo facilities, and edge locations. Kubernetes helps here, but it's the starting point, not the whole answer.

The Uncomfortable Reality

Going hybrid is operationally harder than going all-in on a single cloud provider. Anyone who tells you otherwise is selling colocation space. You'll manage more vendor relationships, more network paths, more failure modes.

But the economics and the regulatory environment have shifted enough that "just put it all in AWS" is no longer a defensible strategy for AI-heavy workloads. The organizations figuring out hybrid now — while GPU supply is still constrained and cloud pricing remains elevated — will have a meaningful cost advantage over those who wait.

The cloud isn't going away. It's just no longer the default answer for every AI workload. And the sooner infrastructure teams internalize that distinction, the better positioned they'll be when AI spending goes from "experimental budget" to "largest line item on the infrastructure bill."

Enterprise AI has an 80% failure rate. The models aren't the problem. What is?

Michael Tuszynski — Sat, 21 Mar 2026 18:27:58 +0000

Enterprise AI Fails at 80% — And the Models Have Nothing to Do With It

Most enterprise AI projects die quietly. No dramatic failure, no post-mortem email chain. They just... stop. The prototype gets a demo, leadership nods approvingly, and then six months later the Slack channel goes silent and the budget gets reallocated.

Roughly 80% of enterprise AI projects fail — double the failure rate of traditional software projects. That number has held steady for years now, even as the models themselves have gotten dramatically better. GPT-4, Claude, Gemini — pick your favorite. They all work. They work remarkably well, actually.

So why does the enterprise keep fumbling the ball?

The Model Isn't Your Problem. Your Plumbing Is.

Here's what I keep seeing from the architecture side: teams treat AI like a features problem when it's actually an infrastructure problem. They spin up a proof of concept using an API, get promising results in a notebook, and then hit a wall when someone asks "OK, how do we run this in production?"

That wall has a name. It's called the deployment gap — the distance between a working model and a production system that real users depend on. And it's enormous.

A platform engineer on Reddit put it bluntly: the failure pattern repeats across organizations regardless of size or industry. The models work fine. The org doesn't.

The Five Ways Smart Companies Still Blow It

I've watched this play out at dozens of enterprise accounts. The failure modes are predictable enough to catalog.

1. They Solve the Wrong Problem

This is the most common and most expensive mistake. A team picks a use case because it sounds impressive in a board deck, not because it maps to an actual operational bottleneck. "We'll use AI to predict customer churn!" Great. Do you have clean customer data? Do you have a process to act on those predictions? Is churn actually your biggest revenue leak?

Companies that fail at AI overwhelmingly choose the wrong problems first. They optimize for what's exciting instead of what's painful. The successful projects I've seen start with someone saying "this manual process costs us 200 hours a month and we keep getting it wrong."

2. They Live in Data Fantasy Land

Every AI project starts with an assumption about data quality that turns out to be wildly optimistic. The data exists, sure. It's in four different systems, three different formats, with no consistent identifiers, maintained by teams who don't talk to each other.

I worked with an enterprise that wanted to build an AI-powered inventory optimization system. The model was straightforward. The data pipeline took eleven months — not because the engineering was hard, but because getting three business units to agree on what "inventory" meant took that long.

3. They Skip the Platform Layer

This one hits close to home. Teams build AI applications without investing in the platform that supports them. No model registry. No feature store. No monitoring for drift. No rollback mechanism. No cost controls.

Then the model goes sideways in production — and it will — and there's no way to detect it, debug it, or revert it. You're flying blind with a system that's making real decisions.

The production gap isn't a model problem — it's a platform engineering problem. The organizations that ship AI successfully treat ML infrastructure with the same rigor they'd apply to any other production system: observability, CI/CD, access controls, cost management.

4. They Start With Tech Instead of Humans

I've seen teams spend months evaluating which LLM to use, which vector database to pick, whether to fine-tune or RAG, which embedding model performs best on their benchmark — and zero time figuring out who will actually use this thing and how it fits into their workflow.

The best AI system in the world is worthless if the end user Alt-F4s out of it because it adds three clicks to their existing process. Starting with tech instead of humans is the classic engineering trap: we build what's interesting to build, not what's useful to use.

5. They Treat AI as a Project, Not a Product

AI models degrade. The world changes, user behavior shifts, data distributions drift. A model that was 94% accurate in January might be 71% accurate by June. Traditional software doesn't do this. You deploy a calculator app and it keeps calculating correctly forever.

AI requires ongoing investment: retraining, monitoring, evaluation, data quality maintenance. When leadership treats AI as a one-time project with a ship date and a done state, they're guaranteeing that the system will rot.

What Actually Works

After 25 years in tech — the last several spent watching enterprise AI projects succeed and fail — here's what separates the 20% that ship from the 80% that don't.

Start with the workflow, not the model. Find a process where humans are doing repetitive cognitive work, making inconsistent decisions, or drowning in volume. Build AI into that workflow. Not as a standalone app — as an augmentation of what people already do.

Invest in platform before product. You need model serving infrastructure, monitoring, cost tracking, and rollback capabilities before you need a sophisticated model. A simple model on a solid platform beats a sophisticated model on nothing.

Set a 90-day production deadline. If your AI project hasn't touched a real user in 90 days, it probably never will. Scope ruthlessly. Ship something small. Learn from real usage. The organizations that perpetually prototype never ship.

Budget for operations, not just development. AI is more like a garden than a bridge. You don't build it and walk away. Plan for ongoing model evaluation, data quality work, and retraining cycles. If your budget only covers development, you're planning to fail.

Make the ROI case boring and specific. Not "AI will transform our customer experience." Instead: "This model will reduce manual review time from 6 hours to 45 minutes per day for the claims processing team, saving $280K annually." When the value is that concrete, the project survives leadership changes and budget cuts.

The Uncomfortable Truth

The enterprise AI failure rate represents roughly $154 billion in wasted spend. That money didn't evaporate because GPT wasn't smart enough. It evaporated because organizations treated AI adoption as a technology challenge when it's actually an organizational design challenge.

The models are good enough. They've been good enough for a while now. The question was never "can AI do this?" — it's "can your organization support AI doing this?"

If you can't answer yes to that second question, no amount of model capability will save you. Fix the plumbing. Define the problem. Invest in the platform. Then — and only then — worry about which model to use.

Beyond Comprehension Debt: Why Context Architecture Is the Real AI Moat

Michael Tuszynski — Thu, 19 Mar 2026 03:00:36 +0000

Addy Osmani dropped a piece last week that's been making the rounds: "Comprehension Debt — The Hidden Cost of AI-Generated Code." His thesis is sharp. Teams are shipping AI-generated code faster than anyone can understand it. Tests pass, PRs look clean, and nobody notices the growing gap between what's been deployed and what any human actually comprehends. When something breaks at 3am, that gap becomes the bill.

He's right. And he's not seeing the whole picture.

Osmani diagnosed one symptom of a larger condition. Comprehension debt — the gap between shipped code and understood code — is real, and it matters. But it's one line item on a ledger that most engineering organizations haven't even opened yet. If you're a CTO or VP of Engineering adopting AI-assisted development, comprehension debt is the problem you can see. The ones that will actually sink you are the ones you can't.

The AI Debt Paradox

There's a seductive argument circulating in engineering circles right now: AI makes rewriting cheap, so tech debt doesn't matter anymore. Why maintain a crumbling monolith when you can regenerate services in an afternoon?

It's about 40% right, which makes it dangerous.

Yes, the cost curve of rewriting code has collapsed. For bounded, well-specified modules, AI absolutely turns "rewrite" from a quarter-long initiative into a day's work. The classic excuse — "we can't touch that, it'll take months" — is dying. That's real progress.

But here's the paradox: every time you "start from scratch," you throw away embedded knowledge about why decisions were made. The 47 edge cases handled one by one over 18 months. The compliance requirement someone baked in after an audit. The OAuth flow three partners hardcoded against. AI can regenerate the code layer fast. It cannot regenerate the institutional context that made that code correct.

Tech debt didn't disappear. It shape-shifted. And the new forms are harder to detect, harder to measure, and harder to pay down.

Three Debts, Not One

Osmani gave us a name for the first one. Here are the other two.

1. Comprehension Debt (Osmani's Contribution)

This is the gap between the code your team ships and the code your team understands. Osmani nailed the mechanics: AI-generated code passes review because it looks right, engineers approve PRs they haven't fully internalized, and the organizational assumption that "reviewed = understood" quietly breaks down.

The insight is correct. But the prescription — slow down, review more carefully, quiz your engineers — is a cultural intervention. It treats comprehension debt as a discipline problem. For Google-scale teams with senior engineers and strong review culture, that might work. For the other 95% of engineering organizations? You need more than discipline. You need infrastructure.

2. Context Debt

This is the one nobody's naming clearly, and it's the most dangerous.

Context debt is the accumulated loss of institutional knowledge about why systems are built the way they are. It's not about whether engineers understand the code in front of them — it's about whether anyone understands the decisions, constraints, trade-offs, and edge cases that shaped it.

Consider what lives in a mature codebase beyond the code itself:

Architectural rationale. Why this service exists as its own deployment rather than a module in the monolith. Why the database schema looks the way it does. Why that particular API contract was chosen over three alternatives that were debated for weeks.
Boundary knowledge. Which downstream consumers depend on specific response shapes. Which partner integrations are fragile. Which compliance requirements are baked into the data flow and why.
Failure memory. The incident that revealed a race condition nobody anticipated. The scaling problem that drove the caching strategy. The security audit finding that explains the seemingly redundant validation layer.

None of this lives in the code. Very little of it lives in documentation. Most of it lives in the heads of engineers who were there when the decisions were made. When AI regenerates a service from scratch, it produces code that compiles, passes tests, and handles the happy path. What it cannot produce is the scar tissue — the hard-won understanding of what goes wrong and why.

Context debt accumulates every time a team rewrites without capturing context first. Every time an AI-generated solution replaces a human-authored one without preserving the reasoning behind the original. Every time an engineer leaves and their knowledge of why things are the way they are walks out with them.

This isn't new — context loss has always been a risk in software organizations. What's new is the velocity at which AI-assisted development can destroy context. When rewriting is cheap, the incentive to understand before replacing drops to zero.

3. Integration Debt

The third form is architectural. Integration debt is the growing inconsistency between AI-generated components that were each built in isolation without awareness of the broader system.

AI coding assistants operate within a context window. They see the file you're working on, maybe some adjacent files, maybe a system prompt describing your stack. What they don't see is the full topology of your system — every service, every contract, every shared assumption that holds your architecture together.

When three different engineers use AI to independently build three services that interact, each service might be internally excellent. Clean code, good patterns, thorough tests. But the interfaces between them — data formats, error handling conventions, retry semantics, authentication flows — will diverge unless someone is deliberately enforcing coherence.

The numbers back this up. CodeRabbit's 2026 analysis found teams merged 98% more PRs that were 154% larger year-over-year, while 61% of developers reported that AI produces code that "looks correct but is unreliable." The generation pressure is real. The verification pressure is downstream — and growing.

Integration debt compounds quietly. It shows up as unexpected failures during deployment. As subtle data inconsistencies between services. As "it works on my machine" problems that are actually contract mismatches between components that were never designed to work together, despite technically needing to.

The faster you generate components, the faster integration debt accumulates. And unlike code quality — which AI can actually help improve — integration coherence requires exactly the kind of big-picture architectural thinking that AI tools are worst at.

The Composition Shift

Here's why this matters strategically, not just technically.

For two decades, "tech debt" mostly meant code-level debt: poor abstractions, missing tests, duplicated logic, outdated dependencies. That's the debt AI is genuinely good at paying down. Refactoring, test generation, dependency updates, code modernization — these are tasks where AI excels. If your tech debt balance sheet was entirely code-level debt, the "AI makes debt irrelevant" crowd would be right.

But in any system of real complexity, code-level debt was always just the visible portion. The deeper liabilities — context, comprehension, integration — were always there. They were just overshadowed by the sheer volume of code-level problems.

AI didn't eliminate tech debt. It paid down the most visible kind, revealing the structural kinds that were hiding underneath. The balance sheet didn't shrink. The composition changed. And most organizations are still using the old chart of accounts.

The 2024 DORA report hints at this shift: despite widespread AI adoption, throughput dipped 1.5% and stability dropped 7.2% across 39,000 respondents. Teams are generating more code and shipping it less reliably. The metrics that matter — lead time, change failure rate, recovery time — aren't improving with velocity alone.

Context as Infrastructure, Not Culture

This is where I part ways with the prevailing conversation.

Osmani and others are framing these new debts as cultural and organizational challenges. Slow down. Review more carefully. Keep humans in the loop. These aren't wrong, but they're incomplete — and for many teams, impractical. You can't tell a startup burning runway to slow down their AI-assisted velocity for the sake of comprehension hygiene. You can't tell a team of five to institute Google-style code review rituals.

What you can do is treat context as infrastructure.

I've been arguing for a while now that context management is the real skill gap in AI-assisted development — that getting value from AI tools is less about prompt engineering and more about maintaining rich, current, accessible context that those tools can leverage. (I wrote about this in "This Above All: To Thine Own Context Be True" earlier this year.)

The same principle applies to the debt problem, but at organizational scale. Context debt isn't an inevitable consequence of AI adoption. It's an infrastructure failure — a failure to build systems that capture, maintain, and surface the institutional knowledge that makes code comprehensible and architecturally coherent.

What does context infrastructure look like in practice?

Living architectural decision records. Not dusty wiki pages nobody updates, but actively maintained documents that live alongside the code and get updated as part of the development workflow. Michael Nygard formalized the ADR pattern back in 2011 — Title, Status, Context, Decision, Consequences. The format is fifteen years old. What's changed is that AI makes the cost of not having ADRs catastrophically higher. When AI generates a new implementation, the context for why the old implementation existed this way needs to be right there — available to the engineer reviewing the change and to the AI generating it.

Structured project memory. Tools and conventions so the reasoning behind decisions persists beyond the individual who made them. This means treating context documents — system descriptions, constraint inventories, edge case catalogs — as first-class artifacts that get versioned, reviewed, and maintained with the same rigor as code.

Integration contracts as explicit artifacts. Rather than letting service interfaces emerge organically from individually-generated components, defining and maintaining explicit contracts that AI tools can reference during generation. The contract becomes the source of truth for integration coherence, not the individual developer's mental model of how everything fits together.

Context-aware generation workflows. Configuring AI tools to ingest project context before generating code, rather than generating in a vacuum and hoping for coherence. This means investing in the scaffolding — the context files, the system prompts, the reference documents — that turn AI from a talented but amnesiac intern into a contributor who understands the system they're working within.

None of this is revolutionary. It's the kind of engineering discipline that good teams have always practiced. What's different is the urgency. When humans wrote all the code, context accumulated naturally — slowly, and with gaps, but it accumulated. When AI generates code at 10x the velocity, context dissipates at 10x the rate unless you deliberately counteract it.

The Assessment Gap

The biggest opportunity right now isn't in tooling — it's in assessment.

Most engineering organizations have no way to measure their exposure to these new forms of debt. They can tell you their test coverage percentage, their deployment frequency, their mean time to recovery. They cannot tell you how much institutional context has been lost in the last six months of AI-assisted development. They cannot quantify how many of their AI-generated components have integration assumptions that conflict with each other. They cannot assess whether their team's comprehension of the codebase has kept pace with its growth.

This is the gap that needs to be closed first. Before you can manage context debt, comprehension debt, and integration debt, you need to be able to see them. And right now, almost nobody can.

What This Means

If you're leading an engineering organization that's adopting AI-assisted development — and at this point, that's nearly everyone — the question isn't whether these new forms of debt are accumulating. They are. The question is whether you're managing them deliberately or discovering them during incidents.

The organizations that will thrive in the AI era aren't the ones that generate code fastest. They're the ones that maintain the richest context while moving fast. That's a different capability than what most teams are building right now, and it's one that compounds over time. The team that invests in context architecture today will be moving faster and more safely a year from now. The team that optimizes only for generation velocity will be drowning in debt they can't see and can't name.

The old tech debt conversation was about code quality. The new one is about knowledge architecture. And it's just getting started.

Michael Tuszynski is the founder of MPT Solutions, where he writes about AI strategy, cloud architecture, and engineering leadership. With 25 years in software — including six years as a Senior Solutions Architect at AWS and a stint as CTO at Fandor — he focuses on helping teams adopt AI-assisted development without sacrificing the institutional context that makes their systems work.

Previously: This Above All: To Thine Own Context Be True — on why context management matters more than prompt engineering.

Why does nobody teach the infrastructure problems that destroy developer productivity before production breaks

Michael Tuszynski — Wed, 18 Mar 2026 02:36:58 +0000

The Production Gap: Why Nobody Teaches the Infrastructure That Actually Matters

Every bootcamp, CS program, and YouTube tutorial series teaches you how to build features. Almost none of them teach you what happens when those features meet real traffic, real failure modes, and real users who do things you never anticipated.

The result is predictable: developers ship code that works on their laptop, passes CI, and then falls apart the moment it hits production at scale. Not because the logic is wrong — because nobody taught them about connection pooling, graceful degradation, or what happens when your database runs out of connections at 2 AM on a Saturday.

The Curriculum Blind Spot

A thread on r/ExperiencedDevs captured this frustration perfectly: educational content focuses almost entirely on writing code and building features, while operational concerns — monitoring, error handling, memory management, rate limiting — only become relevant when applications break in production. By then, you're learning under fire.

This isn't a minor gap. It's the gap between "I can build software" and "I can build software that stays running." And it's enormous.

Think about what a typical full-stack course covers: React components, REST APIs, database queries, authentication flows. Maybe some Docker basics. Now think about what actually causes production incidents: thread pool exhaustion, cascading failures from a single downstream dependency, memory leaks that only manifest after 72 hours of uptime, DNS resolution failures, certificate expiration, connection storms after a deploy.

These aren't edge cases. They're Tuesday.

Why This Gap Exists

Three forces keep operational knowledge out of the curriculum.

First, it's hard to teach without real systems. You can't simulate connection pool exhaustion on a laptop running SQLite. You can't demonstrate cascading failures with a single-service tutorial app. The infrastructure problems that destroy productivity only emerge at a certain scale of complexity, traffic, and time — none of which exist in a classroom.

Second, it's not glamorous. "Build a full-stack app in 30 minutes" gets clicks. "Understanding TCP keepalive settings and why they matter for your connection pool" does not. Content creators optimize for engagement, and operational topics feel boring until the moment they're the only thing that matters.

Third, the people who know this stuff learned it the hard way and are too busy to teach it. The senior SRE who understands why your Kubernetes pods are getting OOMKilled at 3x expected memory usage is probably dealing with an incident right now, not writing blog posts. Operational knowledge lives in war stories, incident retrospectives, and tribal knowledge passed between teammates — not in structured curricula.

The Real Cost

This isn't just an education problem. It's a productivity problem that masquerades as a people problem. When teams complain about slow velocity, the instinct is to look at process, hiring, or morale. But often the real bottleneck is that developers spend hours debugging infrastructure issues they were never trained to anticipate.

A developer who doesn't understand connection pooling will open a new database connection per request, wonder why the app works in dev but times out under load, and then spend two days tracking down the issue. A developer who doesn't understand backpressure will build a message consumer that looks correct but silently drops events when the queue backs up. A developer who doesn't understand DNS caching will deploy a service that works perfectly until the load balancer rotates IPs.

Each of these costs days — sometimes weeks — of debugging time. Multiply that across a team, and the infrastructure gap becomes the single largest drag on developer productivity. Not the tools, not the process, not the sprint ceremonies. The fact that half the team has never been taught how production systems actually behave.

The Specific Knowledge That's Missing

Here's my list of operational topics that every developer should understand before they're responsible for a production system. None of these show up in a typical CS degree or bootcamp:

Connection management. How connection pools work, why they have limits, what happens when you exhaust them, and how to size them for your workload. This single topic prevents more production incidents than any framework feature.

Graceful degradation. What your application should do when a dependency is slow or unavailable. The answer is never "throw a 500 and hope for the best," but that's what most tutorial code does.

Observability fundamentals. Not "install Datadog" — actual understanding of what metrics matter, how to correlate logs across services, what a useful alert looks like vs. one that wakes you up for nothing.

Memory and resource management. How garbage collection actually works in your runtime. What causes memory leaks in languages that claim to manage memory for you. Why your Node.js service uses 2GB of RAM after running for a week.

Rate limiting and backpressure. How to protect your service from being overwhelmed, and how to be a good citizen when calling someone else's service. This is the difference between a service that handles traffic spikes and one that cascades failures across your entire platform.

Failure modes of distributed systems. Partial failures, network partitions, split-brain scenarios, exactly-once delivery myths. You don't need a PhD in distributed systems theory, but you need to understand that the network is not reliable, clocks are not synchronized, and retries without backoff are a denial-of-service attack on your own infrastructure.

What Should Change

I'm not expecting universities to overhaul their CS curricula overnight. But a few things would help.

Bootcamps should include a "production readiness" module. Before graduation, every student should deploy an app, load test it until it breaks, diagnose the failure, and fix it. That single exercise teaches more about real-world engineering than a semester of algorithm problems.

Senior engineers need to write down what they know. The gap persists partly because operational knowledge stays locked in people's heads. Incident retrospectives should be shared broadly. Internal tech talks on "how we debugged X" are worth 10x more than another talk on the latest framework.

Companies should invest in structured onboarding for production systems. Don't throw a new hire at the codebase and hope they figure out the monitoring stack. Walk them through the architecture, show them where things break, explain the failure modes you've already seen. This is not hand-holding — it's preventing the next incident.

Platform teams should build paved roads. If connection pooling is tricky, provide a standard library that does it correctly. If observability requires too much configuration, bake it into the deployment pipeline. Don't rely on every developer independently learning every operational concern — make the right thing the easy thing.

The Uncomfortable Truth

The industry has a weird relationship with operational knowledge. We celebrate feature velocity and treat infrastructure work as unglamorous plumbing. We promote the developer who shipped the flashy new feature and overlook the one who quietly prevented 47 production incidents through better error handling and circuit breakers.

Until we value the skills that keep systems running as much as the skills that build new ones, the production gap will persist. New developers will keep learning the hard way — at 2 AM, on a Saturday, with a Slack channel full of escalations and no idea why the connection pool is exhausted.

The fix starts with acknowledging that knowing how to write code and knowing how to run code are two different skills. We teach the first one extensively. The second one, we mostly leave to chance. That's a choice, and it's the wrong one.

The AI CapEx Arms Race Is Coming for Your Cloud Bill

Michael Tuszynski — Mon, 16 Mar 2026 13:34:54 +0000

The three major cloud providers are spending money like it's going out of style. Oracle is financing a $300 billion deal with OpenAI through $50 billion in stock sales and debt. Google just pledged to double its capital spending. And while the press releases talk about "meeting demand," some providers have quietly raised prices on existing services.

Someone has to pay for all that GPU infrastructure. If you're running workloads on any of the major clouds, that someone is you.

The Numbers Don't Add Up for Customers

Let's put Oracle's deal in perspective. $300 billion is roughly the GDP of Ireland. Financing $50 billion of that through stock dilution and debt means Oracle needs massive returns from AI infrastructure to justify the capital structure. The same logic applies to Google doubling its CapEx — these aren't charity projects, and the ROI has to come from somewhere.

That somewhere is cloud pricing.

Here's what makes this cycle different from previous infrastructure buildouts. When AWS, Azure, and GCP built out their initial cloud regions, they were competing for greenfield workloads. Prices trended down because the providers were buying market share. The AI infrastructure buildout flips that dynamic. Providers are spending enormous sums on specialized hardware — GPUs, custom AI chips, liquid cooling systems — that serves a narrow set of workloads. And they're doing it while demand appears insatiable.

When demand outstrips supply and the capital costs are this high, prices go up. Not just for AI services — for everything running on the same infrastructure.

The Quiet Price Creep Nobody's Tracking

The stealth price increases are the part that should worry enterprise IT leaders most. GPU instance pricing gets the headlines, but the real cost pressure shows up in the boring stuff: egress fees, storage tiers, network transit, and support plans.

Cloud providers have a well-documented playbook here. They absorb you with competitive initial pricing, build switching costs through proprietary services, then adjust pricing once you're locked in. The AI spending spree accelerates this pattern because the providers need to recoup capital faster.

I've watched this movie before. In 2022-2023, all three major clouds quietly adjusted reserved instance pricing, modified savings plan terms, and restructured support tiers. Most enterprises didn't notice until their next true-up. The AI CapEx cycle will produce the same pattern, just bigger.

The Hybrid Tax Gets More Expensive

AWS recently entered the hybrid AI infrastructure market with AI Factories, joining an already crowded field. Every major provider now offers some flavor of "run AI on your hardware, managed by our control plane." The pitch sounds good: keep sensitive data on-prem, use cloud for burst capacity, get the best of both worlds.

The reality is more complicated. These hybrid offerings create a new dependency layer. You're not just buying compute — you're buying into an orchestration framework, a model serving stack, and a monitoring stack that ties back to the provider's cloud. The more AI infrastructure you deploy through these hybrid products, the harder it becomes to move workloads between providers or back to fully self-managed infrastructure.

This matters because when the provider raises prices — and they will — your negotiating position is weaker than it was before you adopted their hybrid AI stack.

What Smart Teams Are Doing Right Now

The enterprises handling this well share three characteristics:

They're tracking AI infrastructure costs separately from general cloud spend. Most FinOps practices lump GPU instances, AI API calls, and model training costs into their general cloud bill. That makes it impossible to see the AI cost trajectory independently. Break it out. You need a clear trendline on AI-specific spending to make informed build-vs-buy decisions.

They're building abstraction layers before they need them. The teams that survived the last round of cloud price adjustments had already abstracted their workloads away from provider-specific services. The same principle applies to AI infrastructure. If your inference pipeline is hard-coded to SageMaker or Vertex AI, you have zero negotiating power when pricing changes. Tools like KServe, Ray Serve, or even a simple API gateway in front of your model endpoints give you options.

They're doing the math on owned infrastructure. For sustained AI workloads — inference serving at steady-state volume, fine-tuning jobs on a regular cadence — the economics of owned GPU clusters have shifted significantly. An NVIDIA H100 that costs $30,000 to buy will cost you $40,000+ per year to rent from a cloud provider at current rates. If your utilization stays above 60%, owned hardware wins on a 2-year horizon. That calculation gets even more favorable as cloud prices creep up.

The Consolidation Nobody's Talking About

There's a second-order effect of this spending race that deserves attention. Not every cloud provider can sustain this level of capital investment. Oracle's $50 billion financing structure is aggressive. Smaller cloud providers and regional players simply can't compete on AI infrastructure spending.

This means the market is consolidating around fewer providers with the capital to build AI-scale infrastructure. Fewer providers means less competition. Less competition means higher prices. The AI CapEx arms race is, paradoxically, reducing the competitive pressure that kept cloud pricing in check for the last decade.

Enterprise architects need to plan for a world where cloud infrastructure costs 15-25% more than it does today, with the increases concentrated in compute and networking. That's not a pessimistic estimate — it's what happens when three companies collectively spend hundreds of billions on infrastructure that needs to generate returns.

The Bottom Line

The cloud providers aren't wrong to invest in AI infrastructure. The demand is real, and the companies that build capacity now will capture enormous markets. But don't confuse their strategic interests with yours.

Your job is to use AI infrastructure cost-effectively, not to subsidize someone else's capital buildout. That means tracking costs obsessively, maintaining architectural flexibility, and being willing to own hardware when the math supports it.

The AI infrastructure spending spree will produce better, faster, more capable cloud services. It will also produce higher bills. Plan for both.

Your Platform Team Needs an Agent Policy — Yesterday

Michael Tuszynski — Sun, 15 Mar 2026 02:28:26 +0000

On March 3rd, an attacker compromised the Xygeni GitHub Action by poisoning a mutable tag. Every CI runner referencing xygeni/xygeni-action@v5 quietly started executing a reverse shell to a C2 server. The exposure window lasted a week. 137+ repositories were affected.

The root cause wasn't exotic. A GitHub App private key with overly broad permissions got compromised. Combined with a maintainer's personal access token, the attacker could create a PR and move the tag — no human review required.

This is what happens when automated actors run without governance. And it's about to get much worse.

Agents Are a New User Persona

Your platform team already manages identities for developers, service accounts, and CI bots. But AI agents are a fundamentally different category.

A developer reads docs, thinks, and opens a PR. A service account runs a fixed script. An AI agent does something in between — it reasons about what to do, then acts. It might create infrastructure, modify configurations, call APIs, or chain together a dozen tools. The blast radius of a compromised or misconfigured agent is closer to a rogue admin than a broken cron job.

Yet most organizations treat agents like any other service account. Same IAM roles. Same broad permissions. Same lack of runtime monitoring.

The numbers back this up. A 2026 Gravitee report found that 80.9% of technical teams have pushed agents into active testing or production, but only 14.4% went live with full security and IT approval. And here's the kicker: 82% of executives feel confident their existing policies cover unauthorized agent actions, while only 21% have actual visibility into what their agents access, which tools they call, or what data they touch.

That's not a gap. That's a canyon.

What an Agent Policy Actually Looks Like

An agent policy isn't a PDF that legal signs off on. It's a set of enforced constraints that your platform team builds into the golden path. Here's what that means in practice:

Identity and RBAC

Every agent gets a dedicated identity — not a shared service account, not a developer's credentials. Each identity maps to a role with explicitly scoped permissions. If an agent writes Terraform, it gets write access to the specific modules it manages and nothing else.

This sounds obvious. In practice, most teams hand agents the same broad IAM role they use for local development because it's faster to ship.

Runtime Boundaries

Static permissions aren't enough. Agents make decisions at runtime, and those decisions need guardrails:

Rate limits on API calls and resource creation
Allowlists for which tools and endpoints an agent can invoke
Cost ceilings per execution (an agent that spins up 50 GPU instances because the prompt was ambiguous is an expensive mistake)
Mandatory human-in-the-loop for destructive operations — deleting resources, modifying security groups, pushing to main

Audit and Observability

Every agent action should produce a trace. Not just logs — structured traces that capture the reasoning chain, the tools invoked, the data accessed, and the outcome. When something goes wrong (and it will), you need to reconstruct exactly what the agent did and why.

The CNCF's 2026 forecast frames this well: the enterprise shift to autonomy will be defined by four control mechanisms — golden paths, guardrails, safety nets, and manual review workflows. All four apply to agents.

Supply Chain Verification

The Xygeni attack was a supply chain attack on an automated actor. Your agent policy needs to cover the agents' own dependencies: pinned versions (not mutable tags), signature verification, and provenance checks for any action or tool an agent consumes. If your CI agent references some-action@v3, you're trusting that the tag hasn't been moved. Pin to a commit SHA instead.

Start With the Blast Radius

You don't need to boil the ocean. Start by answering one question for every agent in production: what's the worst thing this agent could do with its current permissions?

If the answer makes you uncomfortable, you've found your first policy item.

From there:

Inventory your agents. You can't govern what you can't see. OneTrust, CloudEagle, and similar platforms now offer agent discovery — continuously scanning for AI agents, their ownership, integrations, and data access.
Scope permissions to the task. Apply least-privilege like you would for any identity. An agent that summarizes Jira tickets doesn't need write access to your infrastructure.
Add runtime guardrails before production. Galileo's open-source Agent Control and Palo Alto's agentic governance tools are both worth evaluating. The pattern is the same: intercept agent actions at runtime, check them against policy, and block or escalate violations.
Pin your dependencies. Mutable tags are a liability. Every action, plugin, or tool your agents consume should be pinned to an immutable reference.
Build the audit trail now. Retroactively reconstructing what an agent did is painful. Instrument from day one.

This Is a Platform Problem

Some teams try to solve agent governance at the application layer — each team building their own guardrails. That doesn't scale, and it doesn't produce consistent policy enforcement.

This is a platform engineering problem. The same team that builds your internal developer platform, manages your golden paths, and enforces your deployment policies should own agent governance. They have the infrastructure context. They have the policy enforcement mechanisms. And they're already thinking about developer experience, which matters because overly restrictive agent policies that slow teams down will just get bypassed.

The Xygeni attack was a preview. The attack surface for AI agents in CI/CD, infrastructure management, and code generation is growing fast. Your platform team needs an agent policy — not next quarter, not after the first incident. Yesterday.

Building Resilient Microservices: Lessons from Production

Michael Tuszynski — Mon, 09 Dec 2024 21:10:45 +0000

Introduction

In today's distributed systems landscape, building resilient microservices isn't just about writing code—it's about preparing for failure at every level. After years of managing production microservices at scale, I've learned that resilience is more about architecture and patterns than individual lines of code. Let me share some battle-tested insights that have proven invaluable in real-world scenarios.

Key Resilience Patterns

Circuit Breakers: Your First Line of Defense

Circuit breakers are essential in preventing cascade failures across your microservices architecture. Think of them as electrical circuit breakers for your code—they automatically "trip" when they detect potential problems, preventing system overload.

In my experience, implementing circuit breakers has saved our systems countless times, especially during unexpected downstream service failures. The key is to configure them with sensible thresholds:

A failure count threshold (typically 5-10 failures)
A reset timeout (usually 30-60 seconds)
A half-open state to test recovery

class CircuitBreaker {
  private failures = 0;
  private lastFailureTime?: Date;
  private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';

  constructor(
    private readonly failureThreshold = 5,
    private readonly resetTimeout = 60000, // 60 seconds
  ) {}

  async execute<T>(operation: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      if (this.shouldAttemptReset()) {
        this.state = 'HALF_OPEN';
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    this.failures = 0;
    this.state = 'CLOSED';
  }

  private onFailure(): void {
    this.failures++;
    this.lastFailureTime = new Date();
    if (this.failures >= this.failureThreshold) {
      this.state = 'OPEN';
    }
  }

  private shouldAttemptReset(): boolean {
    return this.lastFailureTime! &&
           Date.now() - this.lastFailureTime.getTime() > this.resetTimeout;
  }
}

Retry Strategies: Smart Persistence

While retry logic seems straightforward, implementing it correctly requires careful consideration. Exponential backoff with jitter has proven to be the most effective approach in production environments. Here's why:

It prevents thundering herd problems during recovery
It accounts for transient failures that resolve quickly
It gracefully handles longer-term outages

class RetryWithExponentialBackoff {
  constructor(
    private readonly maxAttempts = 3,
    private readonly baseDelay = 1000,
    private readonly maxDelay = 10000
  ) {}

  async execute<T>(operation: () => Promise<T>): Promise<T> {
    let lastError: Error | undefined;

    for (let attempt = 0; attempt < this.maxAttempts; attempt++) {
      try {
        return await operation();
      } catch (error) {
        lastError = error as Error;
        if (attempt < this.maxAttempts - 1) {
          await this.delay(attempt);
        }
      }
    }

    throw lastError;
  }

  private async delay(attempt: number): Promise<void> {
    const jitter = Math.random() * 100;
    const delay = Math.min(
      this.maxDelay,
      (Math.pow(2, attempt) * this.baseDelay) + jitter
    );

    await new Promise(resolve => setTimeout(resolve, delay));
  }
}

Service Discovery and Health Checks

Robust health checking is fundamental to maintaining system reliability. A comprehensive health check should:

Verify connectivity to all critical dependencies
Monitor system resources (memory, CPU, disk)
Check application-specific metrics
Report detailed status information

I've found that implementing different health check levels (liveness vs readiness) provides better control over container orchestration and load balancing decisions.

interface HealthStatus {
  status: 'healthy' | 'unhealthy';
  checks: Record<string, boolean>;
  metrics: {
    memory: number;
    cpu: number;
    disk: number;
  };
}

class HealthChecker {
  async check(): Promise<HealthStatus> {
    const [dbStatus, cacheStatus, metrics] = await Promise.all([
      this.checkDatabase(),
      this.checkCache(),
      this.getMetrics()
    ]);

    return {
      status: this.isHealthy(dbStatus, cacheStatus, metrics) ? 'healthy' : 'unhealthy',
      checks: {
        database: dbStatus,
        cache: cacheStatus
      },
      metrics
    };
  }

  private async checkDatabase(): Promise<boolean> {
    try {
      // Implement actual DB check
      return true;
    } catch {
      return false;
    }
  }

  private async checkCache(): Promise<boolean> {
    try {
      // Implement actual cache check
      return true;
    } catch {
      return false;
    }
  }

  private async getMetrics(): Promise<{ memory: number; cpu: number; disk: number }> {
    // Implement actual metrics collection
    return {
      memory: process.memoryUsage().heapUsed,
      cpu: process.cpuUsage().user,
      disk: 0 // Implement actual disk usage check
    };
  }

  private isHealthy(dbStatus: boolean, cacheStatus: boolean, metrics: any): boolean {
    return dbStatus && cacheStatus && metrics.memory < 1024 * 1024 * 1024; // 1GB
  }
}

Handling Cascading Failures

The Bulkhead Pattern

Named after ship compartmentalization, the bulkhead pattern is crucial for isolation. In our production systems, we implement bulkheads by:

Separating critical and non-critical operations
Maintaining separate connection pools
Implementing request quotas per client
Using dedicated resources for different service categories

Rate Limiting and Load Shedding

One often-overlooked aspect of resilience is knowing when to say "no." Implementing rate limiting at service boundaries helps maintain system stability under load. Consider:

Per-client rate limits
Global rate limits
Adaptive rate limiting based on system health
Graceful degradation strategies

class RateLimiter {
  private readonly requests: Map<string, number[]> = new Map();

  constructor(
    private readonly limit: number = 100,
    private readonly windowMs: number = 60000 // 1 minute
  ) {}

  async isAllowed(clientId: string): Promise<boolean> {
    this.clearStaleRequests(clientId);

    const requests = this.requests.get(clientId) || [];
    if (requests.length < this.limit) {
      requests.push(Date.now());
      this.requests.set(clientId, requests);
      return true;
    }

    return false;
  }

  private clearStaleRequests(clientId: string): void {
    const now = Date.now();
    const requests = this.requests.get(clientId) || [];
    const validRequests = requests.filter(
      timestamp => now - timestamp < this.windowMs
    );

    if (validRequests.length > 0) {
      this.requests.set(clientId, validRequests);
    } else {
      this.requests.delete(clientId);
    }
  }
}

Monitoring and Observability

Distributed Tracing

In a microservices architecture, distributed tracing isn't optional—it's essential. Key aspects to monitor include:

Request paths across services
Latency at each hop
Error propagation patterns
Service dependencies and bottlenecks

Here is an example using OpenTelemetry:

import { trace, context } from '@opentelemetry/api';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { JaegerExporter } from '@opentelemetry/exporter-jaeger';

export function setupTracing() {
  const provider = new NodeTracerProvider({
    resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: 'my-service',
    }),
  });

  const exporter = new JaegerExporter();
  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
  provider.register();

  return trace.getTracer('my-service-tracer');
}

// Usage example
async function tracedOperation() {
  const tracer = setupTracing();
  const span = tracer.startSpan('operation-name');

  try {
    // Your operation logic here
    span.setAttributes({ 'custom.attribute': 'value' });
  } catch (error) {
    span.recordException(error as Error);
    throw error;
  } finally {
    span.end();
  }
}

Metrics That Matter

Focus on these key metrics for each service:

Request rate
Error rate
Latency percentiles (p95, p99)
Resource utilization
Circuit breaker status
Retry counts

Lessons Learned

Start Simple : Begin with basic resilience patterns and evolve based on actual failure modes
Test Failure : Regularly practice chaos engineering to verify resilience
Monitor Everything : You can't improve what you can't measure
Document Decisions : Keep records of why certain resilience patterns were chosen
Review Incidents : Learn from every failure and adjust patterns accordingly

Conclusion

Building truly resilient microservices is an iterative process that requires constant attention and refinement. The patterns described above have proven their worth in production environments, but they must be adapted to your specific context.

Remember: resilience is not a feature you add—it's a property you build into your system from the ground up.

Next Steps

In my next post, we'll explore performance comparisons between Rust and Node.js implementations of these resilience patterns, with a focus on real-world benchmarks and trade-offs.