Forem: Wolyra

Prompt Engineering at Scale: When It Becomes Software Engineering

Wolyra — Sun, 26 Apr 2026 09:15:02 +0000

In the first months of an AI initiative, prompt engineering is something an individual engineer does in an afternoon. By the time a dozen features are in production, the prompts have accumulated across files, the behavior they encode is load-bearing, and nobody on the team can confidently say why a particular instruction is phrased the way it is. The notebook-level activity has quietly become software, and the team is maintaining it the way it was before the stakes went up.

This post is about the moment prompt engineering becomes software engineering, what changes, and the disciplines that make the transition manageable.

The symptoms of outgrown tooling

Three signals tell you the prompt layer has outgrown informal management. An engineer cannot reproduce an incident from last week because the prompt has changed and there is no record of what it was. A quality regression appears after a deploy, but nobody is sure whether it came from a prompt change, a model version update, or a retrieval change, because all three happen through the same code path. A new team member asks why a particular phrase is in a system prompt, and the answer is “I think Alex added it during the customer escalation in March.” These are not failures of talent. They are the normal consequence of treating production prompts as configuration that lives where it fits.

Prompts are code, but a peculiar kind of code

The first move is to treat prompts as first-class artifacts: stored in version control, reviewed like code, deployed on a schedule, rolled back when they regress. This is the easy part — it only requires discipline.

The harder part is that prompts are not exactly code. A change to a prompt does not produce an obvious diff in behavior. A carefully A/B-tested prompt may perform brilliantly on one workload and poorly on an adjacent one. The feedback loop between a prompt change and its full production impact can be days, not minutes. This means the software engineering disciplines that work for traditional code — fast local tests, deterministic behavior, clear ownership — need adaptation.

The four disciplines

1. Templates, not strings

The first step away from informal prompting is separating prompt templates from the values substituted into them at runtime. A prompt template — with named variables for user input, retrieved context, role, language, and any other dynamic pieces — can be versioned, reviewed, and diffed meaningfully. A formatted string concatenated from three files cannot.

This also enables the next discipline: evaluation.

2. Every change passes an evaluation

No prompt change reaches production without running against a representative evaluation set. The evaluation does not have to be elaborate — a hundred curated examples with expected outcomes, scored automatically, is enough to catch ninety percent of regressions. What matters is that the evaluation runs before the change lands, the results are visible on the pull request, and a regression blocks the merge the way a broken unit test would.

Teams that adopt this discipline once will not give it up. The first time a well-intended cleanup of a system prompt would have broken production and the evaluation catches it instead, the cultural argument is over.

3. Prompts carry provenance

Every instruction in a production prompt should be traceable to the reason it exists. The easiest way to do this is comments in the template itself — a short note next to each paragraph explaining what it is defending against, what incident or review prompted it, and under what conditions it could be removed. This sounds bureaucratic. It is not. It is the only way a prompt that has accumulated over eighteen months remains legible to the team maintaining it, and the only way a new engineer can safely change it without undoing silent guardrails.

4. Observability lands prompts in production traces

When a production incident happens, the on-call engineer needs to be able to see the exact prompt that was sent, the values substituted in, the response received, and which version of the template was active. This requires logging the template ID, the variables, and a hash or version of the template with every call. Without this, incident response becomes guesswork, and guesswork on a stochastic system is slow.

Organizational shifts

Beyond the technical disciplines, prompt engineering at scale tends to force two organizational decisions.

Ownership. Individual engineers cannot be the sole owners of production prompts. The person who wrote an original prompt is often not the right person to maintain it eighteen months later, and informal ownership invites the “I think Alex added it” problem. Explicit ownership, usually at the feature-team level, with a named reviewer for prompt changes, closes this gap.

The prompt review. Code review catches bugs. Prompt review catches subtler problems: instructions that contradict each other, edge cases the prompt does not handle, tone drift, bias risks, compliance implications. Teams that run a light prompt-review process alongside code review tend to produce noticeably better prompts than teams that do not, because two people thinking about the same prompt for ten minutes is a surprisingly effective quality gate.

The escape-hatch question

None of this discipline matters if the team cannot change a prompt quickly when production behavior is wrong. Build the fast path. A prompt edit that has passed evaluation should deploy within minutes, not hours. A rollback to a previous version should be a single command. The rigor around evaluation and review exists to make the fast path safe, not to slow it down.

The pattern that works is strict on correctness and permissive on speed. A prompt change that fails evaluation cannot land, full stop. A prompt change that passes evaluation should land as fast as the team can push the commit. This is the same pattern mature software teams apply to traditional code — strict quality gates, fast everything else — adapted to a domain where the quality gates have to understand stochastic outputs instead of deterministic ones.

Where to start

A team that has accumulated informal prompts in production and wants to move to a more disciplined operating model usually benefits from a specific sequence. Extract the prompts from code into named templates. Build an evaluation set for each template, even a small one. Wire the evaluation into the pull request workflow. Add provenance comments as the team touches each prompt for other reasons. Turn on template-version logging so production traces are actionable.

This is a quarter of steady work, not a single project. The payoff is that the prompt layer stops being a source of mysterious regressions and starts behaving like the rest of the production codebase — changeable, observable, and defended by the same kind of quality gates every other critical system has.

Fine-tuning vs. RAG: A Cost-Benefit Framework

Wolyra — Sat, 25 Apr 2026 15:22:29 +0000

Two common questions show up within the first month of any serious AI initiative. Should we fine-tune a model on our data? Should we build a retrieval system on top of a general model instead? The two approaches solve overlapping problems, cost very different amounts, and require very different operational discipline. Teams that pick wrong usually do not find out for six to twelve months.

This post is the cost-benefit frame we walk clients through when the decision is still open.

What each approach actually does

Fine-tuning changes the weights of a model using a curated dataset, so the model behaves differently on future inputs. The new behavior is baked into the model. You do not need to ship your data at inference time. You do need to ship new models whenever your data changes.

Retrieval-augmented generation leaves the model unchanged. At inference time, a separate system retrieves relevant context from a corpus and inserts it into the prompt. The model reasons over content it has never seen before, using its general capabilities. Your data stays in your corpus; the model is disposable.

The common confusion is that both approaches can produce the same surface behavior — a system that answers questions about your domain. They differ in where the domain knowledge lives, how quickly it can be updated, and what it costs to keep running.

When fine-tuning is the right answer

Fine-tuning is correct when one or more of the following is true:

The desired behavior is a style or format rather than a set of facts — the model needs to write like a specific voice, follow a specific output schema, or apply a specific classification scheme consistently.
The task requires the model to internalize a large number of examples to generalize correctly, and prompting with examples at inference time is prohibitively expensive or exceeds the context window.
Latency is critical and the retrieval step would add unacceptable overhead.
The data is relatively stable — it does not need to be updated more than quarterly, so the cost of retraining does not dominate the lifecycle.

In these cases, fine-tuning produces a tighter, cheaper-to-operate system than RAG, with better consistency across responses.

When RAG is the right answer

RAG is correct when any of the following is true:

The knowledge base changes frequently — daily, weekly, or monthly — and waiting for a new fine-tuned model each time would be impractical.
The system needs to cite its sources, show provenance, or be auditable. RAG makes this natural; fine-tuning makes it almost impossible.
The knowledge base is large, such that fitting it into a fine-tuned model is either technically infeasible or creates a model that is expensive to serve.
Different users should see different slices of the knowledge, and that scoping has to happen at query time. Fine-tuning a model per user or per role does not scale; retrieval filtering does.

For most enterprise knowledge workloads — documentation, support, research, regulatory lookup — RAG is the default, and the case for fine-tuning has to be argued for.

The cost curves

Fine-tuning has a high upfront cost — dataset curation, training runs, evaluation — and a low per-inference cost. A fine-tuned model answers a query without reaching for external context, which makes inference cheap and fast. The trap is that fine-tuning costs appear to be “done” after the training run, but they recur every time the data shifts meaningfully. Teams that fine-tune quarterly often find that the total cost over two years exceeds what a well-tuned RAG system would have cost.

RAG has a low upfront cost — set up a vector index, wire up retrieval — and a higher per-inference cost. Every query pays for an embedding lookup and additional input tokens from retrieved context. The trap here is that per-inference costs compound at scale, and a system that feels cheap at a thousand queries a day becomes a budget line item at a million.

A useful rule of thumb: below a few hundred thousand queries a month, RAG is almost always cheaper in total cost of ownership. Above a few million queries a month on stable data, fine-tuning starts to pay for itself. In the middle, the decision usually comes down to how fast the data changes.

The operational burden

Fine-tuning adds an ML operations discipline your team may not currently have: dataset versioning, training pipeline management, model evaluation, deployment and rollback of model versions. If your team does not already operate ML models in production, adopting fine-tuning is committing to building this capability.

RAG adds an information-retrieval operations discipline: corpus ingestion, chunking strategy, embedding freshness, vector index maintenance, retrieval quality measurement. This is closer to what a data engineering team already knows, but it is still a non-trivial system to keep healthy.

Neither is free. The right question is not “which is simpler,” but “which operational burden does our team already know how to carry?”

The hybrid pattern

The mature answer in most enterprise deployments is hybrid. Use RAG as the default for anything that looks up factual knowledge. Use fine-tuning selectively, for the narrow parts of the system where style, format, or classification discipline are not reliably achievable through prompting alone.

A customer support agent, for example, might use a fine-tuned classifier to route tickets by intent (a problem where fine-tuning excels), a fine-tuned response generator to match the company’s tone (style), and a RAG system to pull the current documentation into the answer (factual knowledge). Each sub-component gets the approach suited to its problem, and the system as a whole is more accurate, cheaper to operate, and easier to update than any pure-RAG or pure-fine-tuning design would be.

How to decide

Three questions usually resolve the decision:

How often does the underlying knowledge change? If more than quarterly, start with RAG.
Does the system need to cite sources or be auditable? If yes, RAG.
Is the desired behavior a style or a format rather than a set of facts? If yes, fine-tuning is worth evaluating.

Everything else is refinement. The worst outcome is not picking one and building something that works. It is building a system that mixes the approaches without clear reasoning and then spending the next year confused about why quality is uneven.

AI Observability: Monitoring Agent Failures in Production

Wolyra — Sat, 25 Apr 2026 15:21:53 +0000

Teams that ship AI-powered features often discover, six months in, that their observability stack was designed for a different kind of software. Traditional monitoring tells you when a service returns a five-hundred, when latency spikes, when a queue backs up. These signals are still necessary for AI workloads. They are no longer sufficient.

The distinctive failure modes of production AI systems — silent regressions, confident wrong answers, cost blowouts from a single looping agent, drift introduced by an upstream model update — all happen inside the boundary where traditional monitoring stops looking. A system can be ninety-nine point nine percent available and still be wrong half the time. That is a category of failure most monitoring stacks were never designed to detect.

This post is a practical framework for the observability layer you actually need around language models and agents in production.

The three signal categories

AI observability divides naturally into three categories. Treating them as one stack is how teams end up with dashboards that look thorough and miss every real incident.

Operational signals are the ones your existing monitoring already handles: request volume, error rates, latency percentiles, token throughput, cost per request. These tell you whether the system is running. They do not tell you whether it is working.

Quality signals measure whether the answers the system produces are correct or useful for the task. This is the category most teams skip or implement weakly because it requires thinking clearly about what “correct” means for each workload.

Behavioral signals capture how the agent or model is making decisions over time. Are tool calls succeeding? Are multi-step reasoning chains getting longer or shorter? Is the model increasingly routing through expensive paths? These are the signals that detect drift before it becomes a quality problem.

Building the quality signal

The hardest part of AI observability is measuring quality in production without human reviewers in the loop for every request. Three patterns tend to work.

Golden set regression. Maintain a curated set of representative inputs with known good outputs. Run the production system against the golden set on a schedule — daily is usually enough — and alert on regression. This catches the case where an upstream model update or a prompt change silently degrades quality on inputs your team cares about.

Downstream signals. If the AI system feeds a workflow that eventually produces an observable outcome — a ticket gets resolved, a document gets approved, a recommendation gets accepted — track the outcome rate over time, segmented by whether the AI path was involved. A ten percent drop in resolution rate on tickets handled by the agent, while the non-agent path is stable, is a quality regression even if every operational signal is green.

Model-as-judge scoring. For a sample of production traffic, have a separate model score the primary model’s output against a rubric. This is less reliable than the first two patterns and is prone to its own drift, but it scales to workloads where ground-truth labels are expensive. Use it to detect large regressions, not small ones.

Tracing an agent is not tracing a request

A request through a traditional microservice stack produces a trace with tens of spans. A request through an agent can produce a trace with hundreds, spread over minutes. The planning loop fires repeatedly, each tool call is its own sub-request, memory is read and written, and failures at any step can trigger a fallback path that looks nothing like the original request.

Two practical consequences. First, the shape of a useful agent trace is hierarchical rather than flat — you want to see the planning decisions as parent spans, with tool calls and model invocations nested beneath them. Flat tracing views turn an agent trace into a wall of text that nobody reads. Second, sampling strategy matters differently. Head-based sampling (one-in-N requests traced in full) loses the signal on the rare, expensive agent runs — which are precisely the ones worth investigating. Tail-based sampling, where the decision to keep a trace is made after the run completes based on its characteristics, is more appropriate for agent workloads.

Cost is a first-class signal

A bug in a traditional service might leak memory or retry a failing request too many times. The cost of those bugs is real but bounded. A bug in an agent — a planning loop that decides to gather more context repeatedly, a tool call that returns noise and triggers more tool calls — can burn through months of budget in hours. We have seen real incidents where a single runaway agent consumed a six-figure budget over a weekend.

This is why cost per request, not just aggregate cost, has to be an alerting signal for AI workloads. Alert on requests that exceed a threshold of tokens consumed, tool calls made, or wall-clock duration. These are cheap alarms to set up and they catch a category of incident that traditional APM will not.

The drift problem

Model providers update their models. Sometimes they announce this. Sometimes they do not. Sometimes the announcement is in a changelog nobody reads, and the first signal your team has that the underlying model has shifted is when production behavior changes in a way that no diff in your own codebase can explain.

The defense is layered. Pin model versions explicitly where the provider supports it. Monitor behavioral signals — average response length, tool-use rates, refusal rates — over time, so that a shift is visible before it becomes a quality problem. Keep the golden-set regression running so that when a shift does happen, you can quantify its impact on workloads that matter.

The providers are getting better about this, but the responsibility for noticing drift remains with the consumer. A team that does not instrument for drift will learn about it from a customer complaint.

What a useful AI observability dashboard looks like

Three panels tend to be the minimum for a dashboard that a team will actually look at:

Today’s quality: golden-set pass rate, downstream outcome rate, sampled model-as-judge score — each trended against last week and last month.
Today’s cost: total spend, spend per feature, distribution of cost per request, a count of requests that exceeded the cost alarm threshold.
Today’s behavior: average tool calls per request, distribution of response length, refusal rate, top ten slowest or most expensive runs linked to their traces.

The operational dashboard stays where it is. These three panels go next to it. When someone says “is the AI feature healthy?”, all three have to be green.

Where to start

If your AI workloads are in production and your observability answer is “we have the same dashboards we had before AI,” there is a gap. The highest-leverage first investment is a golden-set regression, because it catches the broadest class of silent failures and it keeps working even when the team that set it up has moved on. Cost-per-request alerting is a close second. Everything else is refinement on top of those two.

Visibility into AI systems is not optional infrastructure. It is the difference between a feature you can trust in production and one you are hoping is still working.

RAG Architecture for Regulated Industries

Wolyra — Sat, 25 Apr 2026 15:21:17 +0000

Retrieval-augmented generation has moved from a research curiosity to the default pattern for grounding large language models on enterprise data. A model on its own hallucinates; a model equipped with retrieval over a curated corpus does not, or at least does so far less often. This has made RAG the operating pattern for internal search, customer support, legal research, and regulatory lookup across most sectors.

For regulated industries, the pattern is more interesting and more constrained. Finance, healthcare, legal, and any organization operating under data-residency obligations cannot adopt a generic RAG pipeline without thinking carefully about where documents are indexed, where embeddings are computed, where queries are logged, and what the model sees when it produces an answer. Nearly every default in a typical RAG stack is a compliance decision in disguise.

This post walks through the architectural decisions that matter when a RAG system has to be defensible to an auditor, not just useful to a user.

The parts of a RAG system that a regulator cares about

A RAG pipeline has five components that are worth naming individually because each one has its own compliance profile: the corpus (the documents the system can see), the index (the embeddings and metadata derived from those documents), the retrieval path (how a query finds relevant chunks), the generation step (how the model composes an answer), and the telemetry (what is logged, where, and for how long).

In an unregulated setting, the interesting engineering lives in retrieval quality. In a regulated setting, the interesting engineering lives in the boundaries between those five components — which one crosses a trust boundary, which one leaves your control, which one persists data beyond the query that generated it.

Where documents live, and what that forces

Start with the corpus. If documents contain customer personal data, protected health information, privileged legal material, or controlled unclassified information, the first question is not about models. It is about jurisdiction. Which regions are authorized to hold these documents, and are those the regions where your cloud provider actually stores them?

Most cloud object stores let you pin data to a specific region. Many do not let you make equivalent guarantees about derived artifacts — embeddings, for example, can be computed in a region that is different from the region where the source data is stored, depending on how the embedding service routes traffic. Audit this path. The embedding of a document is still, from a regulatory perspective, a derivative of that document.

The embedding boundary is the real trust boundary

For RAG pipelines that use a hosted embedding API, the embedding call is the moment your data leaves your control. Two questions decide whether this is acceptable:

Does the provider train on your embedding inputs? Most enterprise tiers disable training on customer data, but the default API tier often does not. Verify the contract, not the marketing page.
Does the provider log the content of embedding requests? Retention policies on embedding logs vary. Some providers retain for thirty days for abuse monitoring; some offer zero-retention modes; some log by default and expect you to request an exception.

If either answer is unsatisfactory, the embedding step has to move inside your own infrastructure. This is more tractable than it was two years ago — strong open-weight embedding models are now competitive with hosted ones for most retrieval tasks — but it changes the cost and operational profile of the system.

The retrieval path and metadata leakage

Retrieval quality in production depends heavily on metadata filtering. You almost never want pure semantic search over the whole corpus; you want semantic search scoped by department, document type, access control, date range, or jurisdiction. Every one of these scoping filters is both a quality lever and a compliance requirement.

The failure mode to design against is metadata leakage through the generation step. If retrieval pulls a chunk that a user should not have seen, but the generation step incorporates content from that chunk into the answer, you have built a system that can leak through the language model rather than through the index. The fix is access-control-aware retrieval — applying user permissions at the retrieval layer, before the model sees the results — combined with tight prompting that instructs the model to quote rather than paraphrase sensitive content.

The generation step and model choice

For regulated workloads, the generation model is subject to the same constraints as any other cloud service your organization uses: data residency, encryption in transit and at rest, contractual terms on training, and auditability of responses. The frontier-model providers have dedicated enterprise endpoints that address these constraints; verify that the endpoint, not just the brand, is covered.

A question that often goes unasked: can you reproduce the answer the model gave a user last Tuesday? Reproducibility requires pinning the model version and storing enough of the prompt and retrieval context to replay the call. In regulated environments where answers inform decisions, this reproducibility is part of the audit obligation. Build it in from the start. Retrofitting it later is painful.

What to log, and where

Telemetry is where many RAG systems acquire compliance debt without noticing. A typical observability stack will log the query, the retrieved chunks, the prompt, the response, and the user identity. Each of these, combined with the others, is a data product that may itself be regulated.

Decide deliberately which of these to retain, for how long, in which region, and with what access controls. A useful default for regulated deployments is to log aggregated metrics freely, but to retain prompt and response content only long enough to investigate incidents, and to encrypt those logs with keys your security team controls rather than the observability vendor’s.

A reference posture

The architectures we see succeed in regulated RAG deployments converge on a few common decisions:

Documents and embeddings stored in the same region, with derived artifacts explicitly treated as subject to the same controls as source data.
Embedding either performed on a private endpoint of a vetted provider, or using a self-hosted open-weight model where the contract with a hosted provider is not acceptable.
Retrieval that enforces user-level access controls before results reach the model, with the permission check logged as part of the audit trail.
A pinned generation model version, with the option to fall back to a prior version if behavior regresses on a regulated workload.
Logs that separate operational telemetry (always kept) from content telemetry (retained briefly, encrypted independently, accessible to a narrow set of roles).

None of this is exotic. It is the same set of decisions that any well-run enterprise service makes. What changes in RAG is the number of places where data crosses a boundary, and the subtlety of how it does so — through embeddings, through retrieval context, through logged prompts, through cached answers. Each of those boundaries is where compliance succeeds or quietly fails.

Where to start

If you are building a RAG system for a regulated workload right now, the most valuable early exercise is not picking a vector database. It is mapping the five components above against your data classification and residency policy, and identifying the two or three points where the defaults of the tooling you are about to adopt do not match what your compliance team will ultimately require. Fixing those mismatches before the system is in production is an order of magnitude cheaper than fixing them after.

A retrieval system that is defensible from day one is a retrieval system that earns trust. Everything downstream gets easier from there.

Post-Quantum Cryptography Migration: Why 2026 Is the Year Enterprises Must Act

Wolyra — Sat, 25 Apr 2026 15:20:41 +0000

In August 2024, the National Institute of Standards and Technology finalized the first three post-quantum cryptographic standards: ML-KEM for key establishment, ML-DSA for digital signatures, and SLH-DSA as a hash-based signature alternative. Eighteen months later, most enterprise cryptographic inventories still look exactly as they did before the announcement. RSA-2048 and ECDSA remain the workhorses of TLS termination, code signing, VPN tunnels, database encryption, and every internal service that speaks TLS to every other internal service.

This is not a crisis today. It is a problem with a long fuse, a hard deadline, and a class of attackers who are already preparing for the transition by doing something that sounds theoretical and is not: they are collecting encrypted traffic now, storing it, and planning to decrypt it later, once a sufficiently capable quantum computer exists. The cryptography community calls this harvest now, decrypt later. For any data that still needs to be secret in 2035, the attack is happening in 2026 whether we have noticed it or not.

This post is a practical guide to where post-quantum cryptography migration sits for enterprises today, what the realistic timeline looks like, and what your organization should be doing in the next twelve months to avoid being caught unprepared.

Why now, when the threat is not now

Two dates matter. The first is the date a cryptographically relevant quantum computer exists — a machine capable of running Shor’s algorithm on keys of the size currently in production. No one knows this date. Credible estimates from cryptography researchers cluster in a window from the early 2030s to the late 2030s, with meaningful probability mass on either side.

The second date is the one that matters more: the date by which any given piece of data must already be protected by post-quantum cryptography. That date is not the date the quantum machine arrives. It is the date the data first leaves your control, minus the length of time the data must remain confidential.

A medical record with a thirty-year confidentiality requirement, transmitted in 2026 over conventional TLS, has to be considered compromised if a quantum computer exists in 2035. The encrypted packets were captured in 2026. The decryption just happens nine years later. The same logic applies to government communications, financial records subject to long retention, intellectual property, and any data governed by long-horizon compliance obligations.

This is why NIST, ENISA, and major regulators are pushing for migration to begin now rather than in the late 2020s. The deadline is not the arrival of the threat. The deadline is several years before the arrival of the threat.

What changed in the standards

The 2024 NIST standards replaced the entire asymmetric cryptography stack, not a layer of it. Three algorithms are now designated for production use:

ML-KEM (formerly Kyber) replaces RSA and elliptic-curve Diffie-Hellman for key establishment. This is the algorithm TLS 1.3 clients and servers will use to negotiate a shared session key.
ML-DSA (formerly Dilithium) replaces RSA and ECDSA for digital signatures. This is what certificates, code signing, and firmware signing will use.
SLH-DSA (formerly SPHINCS+) is a hash-based signature alternative, with different performance and size characteristics, suitable for long-lived signatures and applications where the mathematical assumptions underpinning ML-DSA are considered insufficient.

The practical consequence for enterprise systems is that every place RSA or ECDSA currently appears in production will, eventually, need a replacement or a hybrid construction. Certificate chains, TLS configurations, signed binaries, signed firmware, JWT signing keys, SSH host keys, VPN pre-shared keys and ephemeral exchanges — all of them.

Hybrid constructions — where a classical algorithm and a post-quantum algorithm are used in parallel, so a session is secure if either holds — are the expected transition mechanism. The major TLS libraries and cloud providers are rolling out hybrid modes through 2026 and 2027. This matters because it means migration does not have to be a flag day. It can be a gradual, measured shift.

What enterprises should do in the next twelve months

The first step is not deploying new algorithms. The first step is knowing where the old ones are. Most enterprises drastically underestimate the number of places in their infrastructure where asymmetric cryptography is in use, because most of those places are buried inside operating systems, libraries, appliances, and vendor products that were configured once and forgotten.

1. Build a cryptographic inventory

Identify every system that uses asymmetric cryptography. TLS-terminating load balancers, internal service meshes, certificate authorities, code-signing pipelines, firmware update channels, VPN gateways, database client authentication, SSH access, signed webhooks, JWT-based authentication between services. For each, record the algorithm, the key sizes, the expiry, and the owner.

This inventory is the single most valuable artifact of the migration. Everything else depends on it. Organizations that have done this exercise consistently find two or three times more cryptographic dependencies than they expected, most of them inside vendor products the security team did not know used public-key cryptography at all.

2. Classify data by confidentiality lifetime

Not all data needs to be migrated on the same schedule. Data with a short confidentiality lifetime — operational telemetry, transient session tokens, short-lived audit logs — is not meaningfully at risk from harvest-now-decrypt-later, because by the time a quantum computer exists, the data no longer matters.

Data with a long confidentiality lifetime — customer financial records, healthcare data, intellectual property, government communications, legally privileged material — must be prioritized. This is the data for which the transition window is already closing.

3. Audit your vendor dependency tree

Ask every critical vendor for their post-quantum cryptography roadmap in writing. Cloud providers, TLS CA vendors, identity providers, HSM and key-management vendors, VPN vendors, code-signing infrastructure. Expect a mix of confident answers, vague answers, and silence. The pattern of responses tells you which vendors will migrate on time and which will become a constraint on your own migration.

A vendor that cannot articulate a credible roadmap in 2026 will be a blocker in 2028.

4. Pilot hybrid TLS in a non-critical path

Most major TLS libraries and several cloud load balancers now support hybrid key exchange combining a classical algorithm with ML-KEM. Stand up a non-critical internal service using hybrid TLS. Measure the latency impact, the certificate handling, the compatibility with your observability tooling, the behavior of older clients that do not understand the new ciphersuites.

This is cheap learning. The point is not to protect anything. The point is to identify the operational problems — the unexpected log format changes, the tooling that does not parse the new algorithm names, the legacy systems that negotiate down — before you need to run hybrid on the paths that matter.

5. Build cryptographic agility into new systems

Every new system your organization builds in 2026 should treat the choice of cryptographic algorithm as a configuration, not a baked-in assumption. This is good architecture regardless of the post-quantum transition. With the transition on the horizon, it becomes essential. A system that hard-codes RSA key sizes into its data model will be migrated by rewriting. A system that reads its algorithm choices from configuration will be migrated by restart.

What to watch over the next two years

Three signals will shape how fast the rest of the industry moves.

Browser and operating system defaults. Chrome, Firefox, and Edge have all begun enabling hybrid key exchange by default for TLS. The schedule on which those defaults become mandatory, rather than opt-in, is a strong leading indicator of when the broader web will have migrated.

Public certificate authorities. The CA/Browser Forum is working on the rules under which public CAs will issue certificates using post-quantum algorithms. When those rules stabilize and the first public CAs begin issuing ML-DSA certificates, the migration of public-facing TLS will accelerate quickly.

Regulator guidance. NSA Suite CNSA 2.0 already mandates post-quantum cryptography for US national security systems by 2035, with internal milestones before then. EU, UK, and Japanese regulators are on similar trajectories. For regulated industries, the regulatory deadline will arrive before the technical deadline, and the migration will need to be complete earlier than the threat model alone would require.

The executive view

Post-quantum migration is not a security team project. It is an infrastructure program that touches networking, identity, application architecture, vendor management, and regulatory reporting. Budgeting for it, planning it, and staffing it is a 2026 decision, even though the execution will run for several years.

The organizations that start this year will migrate calmly. The ones that wait will migrate in a hurry, under regulatory pressure, with vendors who are themselves scrambling, and with a cryptographic inventory they built in panic rather than in deliberation. The cost difference between those two migrations is not small.

If post-quantum cryptography is not yet on your engineering or compliance roadmap for 2026, it is time to put it there.

Managed AI Agents: When to Build, When to Buy, When to Orchestrate

Wolyra — Sat, 25 Apr 2026 15:20:05 +0000

The announcement cycle has shifted. A year ago, the new thing was a more capable chat model. Then it was tool use and function calling. Now it is managed agents — pre-built, vendor-hosted systems that take a goal, plan a sequence of steps, call tools, and report back with results. Anthropic’s Managed Agents, Google’s agent runtime, OpenAI’s Assistants and its successors, and a crowded long tail of startup offerings all promise the same thing: the productive part of an autonomous AI workflow, without the engineering cost of building one.

For enterprise leaders watching this wave, the question is the same one that surfaced with every prior category of enterprise software: build our own, buy a managed service, or orchestrate across both? We have written before about the framework for build versus buy in general software. Agents add a new dimension, because the decision is not only about cost and differentiation. It is also about where the intelligence lives — and therefore where the risk, the data, and the long-term leverage live.

What a managed agent actually is

Strip the marketing away and a managed agent offering usually includes four things: a hosted planning loop (the model that decides what to do next), a tool registry (the catalog of functions the agent can call), a memory and state layer (so the agent remembers what it did across steps), and a supervision and observability layer (so you can see what happened, intervene, and audit later).

Building these four things yourself is not impossible. Every serious AI platform team has built at least a rough version. But each of them is a non-trivial engineering commitment on its own, and together they are the difference between a demo that works in a notebook and a production system that runs reliably at three in the morning when a tool call times out halfway through a multi-step task.

This is why managed offerings are getting traction. They are not selling the model. They are selling the plumbing around the model.

The three postures

Build

You own the agent framework. You write the planning loop, you register the tools, you manage memory, you build observability. The model vendor becomes one component among several, and you can swap it without rewriting your agent layer.

Correct when the agent is core to how you deliver value and the workflow is specific enough that no off-the-shelf framework models it naturally. A fintech compliance engine that reviews transactions against internal policies, calls proprietary risk services, and files regulatory reports is probably a build. The specific tool graph and the data sensitivity do not map cleanly to a generic agent product, and the investment compounds because the framework gets reused across adjacent workflows.

Cost profile: high upfront, moderate ongoing, maximum flexibility. Realistic timeline for a first production deployment: three to six months with a capable platform team.

Buy (managed)

You use a vendor’s managed agent runtime. The planning loop, state, and observability come pre-built. You configure tools, connect data sources, and deploy. The vendor operates the system. You operate the configuration.

Correct when the workflow is general-purpose and not a source of competitive differentiation — internal research assistants, meeting-notes summarization with calendar and document access, internal support triage, developer productivity helpers. The value is in deploying quickly, not in owning the agent platform.

Cost profile: low upfront, usage-based ongoing, limited flexibility. Realistic timeline to first production deployment: weeks. The risk profile is not zero, though. You are pushing intermediate reasoning state — which often contains the sensitive parts of your queries — through a vendor’s infrastructure. And when the vendor updates the planning model, your agent’s behavior can shift in ways you only discover in production.

Orchestrate

You maintain a thin internal orchestration layer that can route individual workflows to either a managed agent service or to internal agents, depending on the workload. The layer owns identity, policy enforcement, logging, and cost accounting; the agents underneath can be a mix.

Correct at scale, once you have more than a handful of agent-based workflows in production and the cost of running everything on one vendor, or rebuilding everything in-house, has become visible. Orchestration gives you the ability to put sensitive workflows on internal agents while keeping general ones on a managed runtime, and to switch vendors behind the layer without asking the consuming teams to change anything.

Cost profile: moderate upfront on the layer, variable ongoing depending on where workloads land, maximum strategic flexibility. Realistic timeline: this is rarely a first step. It is the posture you migrate to once your first build or buy decision has run for a year and the constraints have become obvious.

The deciding questions

Before debating the three postures, answer four questions about the specific workflow you have in mind. The answers usually resolve the debate.

How sensitive is the data the agent will touch?

If an agent reasons over customer personal data, contracts, source code, or financial records, the intermediate reasoning traces are themselves sensitive. A managed agent often logs those traces on vendor infrastructure for a period of time. Some vendors offer zero-retention modes; many do not. If your compliance posture cannot accept vendor-side logging of reasoning state, the managed option narrows sharply, and build becomes the realistic path regardless of cost.

How bespoke is the tool graph?

Count the tools the agent needs to call. If most of them are generic — web search, calendar, document store, email — a managed agent is probably enough. If more than a third are internal services with non-standard interfaces, authentication requirements, or rate-limit behaviors that need careful handling, the managed option starts to creak. You end up writing and maintaining a large adapter layer, and at that point you have already paid most of the cost of building the agent yourself.

How critical is reproducibility?

Managed agents update their planning models on the vendor’s schedule. For an internal research helper, slight behavior drift between versions is tolerable. For an agent that makes customer-visible decisions, fills regulatory forms, or calculates anything downstream systems rely on, drift is a liability. Build gives you the ability to pin a model version and roll forward on your schedule. Buy means accepting that a model you never asked to change will get smarter, stranger, or simply different one day.

How likely is this workflow to proliferate?

One agent in production is a pilot. Ten agents in production is an operations problem. If the organization’s realistic two-year view includes many agent-based workflows, some sensitive and some not, the orchestration posture becomes attractive earlier than the cost curve alone would suggest. The decision is not per-workflow; it is about where the platform boundary lives across the whole portfolio.

A pragmatic sequence

For most mid-market and enterprise organizations we work with, the correct sequence is not “pick one of the three and commit.” It is a staged path that starts small, learns fast, and preserves optionality.

Start with buy for the first one or two workflows, on an uncontroversial internal use case. The goal is not to ship the flagship agent. The goal is to learn how agents fail, how they get measured, what they cost, and what your governance process actually needs to look like.
Build selectively for the second or third workflow, once a differentiation or compliance reason has made the case. By this point your team has operational scar tissue from the first deployments and knows what to build.
Orchestrate once you have three or more workflows in production across at least two sources. The orchestration layer pays for itself the moment the cost of maintaining separate governance per vendor exceeds the cost of maintaining one thin layer.

The organizations that get into trouble with agents are usually the ones that skipped stage one and tried to build their own platform before anyone had run a real production agent for six months. The ones that get stuck are usually the ones that stayed in stage one too long and ended up with a sprawl of agents on three different managed vendors with no common policy layer.

The broader point

Managed agents are a genuinely useful development. They compress the cost of getting a first agent into production from months to weeks. They are also not, by themselves, a strategy. The strategic decision is where the reasoning, the data, and the control plane live over the next three years — and that decision is the same one enterprises have made about every prior category of infrastructure: build the differentiating parts, buy the commodity parts, and be honest about which is which.

The build-versus-buy question does not get easier when intelligence becomes part of the stack. It gets more consequential. The companies that reason about it clearly now will spend the next three years shipping. The ones that let the vendor announcement cycle drive their architecture will spend the next three years migrating.

Enterprise LLM Selection in 2026: A Framework That Outlasts the Benchmarks

Wolyra — Sat, 25 Apr 2026 15:12:18 +0000

By the time this post goes live, the published benchmarks for the three top-tier frontier language models will already be stale. Kimi K2.6 will have claimed a lead on some reasoning evaluation, GPT-5.4 will have responded with a coding benchmark, and Gemini 3.1 will have quietly taken a multimodal crown that nobody noticed because the relevant press cycle had already moved on. Inside a lot of enterprises, a procurement committee will be staring at a comparison slide built on whichever numbers were current the morning the deck was made.

This is not how durable decisions are made.

The model you choose for customer support summarization, internal knowledge retrieval, or regulated document processing is going to touch production systems, move data across boundaries, and accumulate vendor lock-in for years. Benchmark league tables answer a much smaller question than the one you are actually asking. This post lays out the framework we use when a client sits down and says, “We need to pick an LLM. Help us think about it properly.”

Why benchmarks answer the wrong question

A benchmark score tells you one thing: how this model performed on a fixed set of questions, measured by whoever published the score, on the date the test was run. It does not tell you how the model will behave on your support tickets, your contract language, your codebase, or your internal jargon. It does not tell you what the model costs at your request volume. It does not tell you what happens when the provider deprecates the model version you spent six months tuning a workflow around.

Worse, benchmarks are a lagging signal of a vendor’s ability to ship. The leader on last quarter’s benchmark is often not the leader on this quarter’s. If your selection criterion is “pick the one at the top of the leaderboard,” you will be re-running this decision every eight months, and each re-run will cost you migration effort, retraining effort, and reputation inside the organization.

The question to ask is not “which model is best?” It is “which model is best for this workload, at our scale, under our constraints, from a vendor we can keep betting on?”

The six-axis evaluation

Across client engagements we have converged on six axes that deserve weight in any enterprise LLM selection. None of them is optional. How you weight them depends on your industry and your risk appetite.

1. Task-fit capability

The only meaningful capability test is a private evaluation on representative samples of your data. Assemble fifty to two hundred real examples of the task you intend to run — redacted if necessary — and score the candidate models against them. Measure accuracy, but also measure the shape of the failures. A model that is eighty-five percent correct but wrong in spectacular, unpredictable ways is often worse for production than a model that is eighty percent correct with failures that cluster around a few predictable patterns you can detect and route around.

Run the same evaluation quarterly. This is the only number that tells you whether the model is getting better or worse for your workload, independent of whatever the vendor is marketing.

2. Total cost at realistic volume

Vendors publish per-token prices. Per-token prices are not your cost. Your cost is the full loaded rate: input tokens plus output tokens, multiplied by request volume, plus the retries you will incur on timeouts and safety filters, plus the fine-tuning or prompt-engineering budget required to reach acceptable accuracy, plus the egress and observability costs of piping traffic through your own infrastructure.

Model this out for twelve months at projected volume, not current volume. The frontier-model tier whose pricing looks manageable at ten thousand requests a day often prices a mid-market team out of the market at one million requests a day. A slightly less capable mid-tier model, used with better prompt engineering, is frequently the correct answer on total-cost grounds alone.

3. Data residency and compliance

Where does the model run? Where is the inference request logged? Is the provider contractually forbidden from training on your inputs, and is that enforceable across every region you operate in? For regulated industries — finance, healthcare, anything touching EU personal data — these questions eliminate candidates before capability is even discussed.

The answer is increasingly provider-specific and region-specific. A model that is cleared for enterprise use in the United States may not have equivalent controls available in the EU or in Turkey. Verify in writing, and verify for the specific deployment region your workload will run in.

4. Latency and reliability under load

A model that is fast during your evaluation is not necessarily fast during a peak event on a Monday morning. Stress-test at projected peak throughput. Measure p95 and p99 latency, not just averages. Check the vendor’s published uptime numbers, but also check the incident history on their status page. A model three hundred milliseconds faster on average but with twice as many hour-long outages per quarter is not faster in any sense that matters to a customer-facing workload.

5. Ecosystem and integration surface

Which SDKs does it support? Does it expose native tool-use and structured-output modes that match your workflow, or will you be writing adapters? Is there an observability story — traces, token accounting, prompt diffing — that your platform team can actually use, or are you building that layer yourself? Does the model support the context-window size your longest document actually requires, without aggressive truncation?

The ecosystem around a model often matters more than the model itself. A second-place model with first-class tooling will ship to production faster and more reliably than a first-place model whose integration surface you have to build.

6. Vendor trajectory

This is the axis enterprises underweight, and it is the one that determines whether you will be running this selection process again in eighteen months. Look past the current model to the provider’s financial position, release cadence, enterprise commitments, and the clarity of their public roadmap. A vendor burning cash on a price war, or whose enterprise support team you cannot reach for a production incident, is not a partner you can build on regardless of how strong this quarter’s benchmarks are.

The hidden cost of choosing wrong here is not the model cost. It is the migration cost when you have to leave.

How the axes interact

The six axes are not independent. A model that is strong on capability and ecosystem but weak on vendor trajectory is a trap: you will build on it, love it, and then spend a painful year migrating when the provider pivots or prices you out. A model that is strong on vendor trajectory and compliance but weaker on capability is often the correct choice for regulated workloads, because the gap on capability can be closed with prompt engineering and domain context, while the gap on compliance cannot be closed at all.

In practice, we see three common profiles:

Frontier-first: Pick the capability leader, accept the vendor and cost risk, and expect to re-evaluate every six to twelve months. Correct for small pilot workloads and high-value, low-volume use cases.
Enterprise-stable: Pick a provider with strong compliance, predictable pricing, and clear enterprise support, even if it trails the frontier by a model generation. Correct for regulated industries and workloads you intend to operate for years.
Portfolio routing: Use multiple providers, routing each workload to the model best suited for it. Correct at scale, once you have enough volume to justify the routing layer and enough in-house capability to maintain it.

A decision cadence that survives the news cycle

Enterprise LLM selection is not a decision you make once. It is a process you institutionalize.

We recommend a quarterly review rhythm: re-run the private evaluation, refresh the cost model against actual invoiced usage, and revisit the vendor-trajectory view. The point is not to switch providers every quarter. The point is to always know what switching would cost, so that when the decision actually becomes necessary, you have already done the homework.

The companies that handle this well treat model selection the way they treat cloud-provider selection: as a long-horizon, reviewed-on-schedule, architectural decision. The companies that handle it poorly treat it as a one-time procurement, and find themselves surprised every time the landscape shifts.

Where this leaves you

The honest answer to “which LLM should we use?” in 2026 is: probably not the one currently at the top of whichever benchmark made the news this week. The answer is the one that scores acceptably on a private evaluation of your workload, remains affordable at your real volume, fits inside your compliance envelope, integrates cleanly with the tooling your team already operates, and comes from a provider you are willing to bet will still be shipping in three years.

That model is rarely the most exciting one. It is usually the one you can ship on, measure honestly, and replace calmly when the time comes.

If you are evaluating options right now and want a second opinion on the framework, we are happy to walk through it with your team. The worst time to discover that a selection was made on the wrong axis is after the contract is signed.

Build vs. Buy: A Framework for Choosing Between Custom Software and SaaS

Wolyra — Sat, 25 Apr 2026 15:12:14 +0000

Every few months, a familiar scene repeats inside growing companies. A department head arrives at the CTO’s office with a SaaS subscription bill that has quietly tripled over two years. The CFO asks why the support team is using three different tools to answer the same customer question. An engineering lead points at an integration backlog and says, “We have thirty connectors, and not one of them does exactly what operations needs.”

That is the moment the question surfaces: build or buy?

It is rarely a question with a clean answer, and the framing is often wrong. The real question is not which is cheaper. It is which decision compounds in our favor over five years, and which one quietly erodes our margin, our data, or our ability to change direction?

This article lays out the framework we use with clients when the decision is still open — before the spreadsheet fight, before the vendor demo, before an internal champion has already picked a side.

Why the question is harder in 2026 than it was in 2020

Three shifts have changed the arithmetic.

SaaS pricing is no longer the default bargain. Per-seat inflation, AI feature surcharges, and consolidation-driven repricing have pushed many category leaders into territory where a mid-market company can pay more for a CRM than for the engineers who would build a competent alternative. We have watched companies spend close to half a million dollars a year on tooling that a small internal team could replace in under a year — and that team would still be there, shipping new features, after the next renewal arrived.

AI assistance has compressed custom software cost. The same greenfield operational application that would have cost several hundred thousand dollars and eight months in 2021 now ships in four months for a fraction of the price when an experienced team uses modern tooling. The labor cost curve bent downward at the exact moment the SaaS cost curve bent up.

Data portability is now a competitive concern. Customer data, operational data, and model training data have become durable business assets. When those assets live inside a third-party platform, a future pricing change or acquisition can put them behind a wall. Custom systems are not automatically better, but they keep the question of portability under your own control.

None of this means custom always wins. It means the old heuristic — “buy unless you absolutely have to build” — no longer maps cleanly onto the cost curves most companies actually face.

The four criteria that actually matter

Ignore cost for a moment. The following four questions usually resolve the decision before a spreadsheet is opened.

1. Is this function a source of differentiation, or is it table stakes?

If a function is what makes the business distinct — the way you route service tickets, the way you score leads, the way you price a bespoke quote — SaaS will almost always force you to either conform to an industry-standard workflow or fight the tool. The first option erodes your differentiation. The second erodes your margin on support and configuration.

If a function is table stakes — payroll, expense reports, general-ledger accounting — SaaS is almost always correct. You are not going to out-innovate an established payroll provider. Pay them and move on.

2. How complex is the integration surface?

Modern businesses rarely use one tool. They use forty. When a SaaS product does seventy percent of what you need but requires custom glue code to connect to the rest of your stack, the hidden cost is not the subscription — it is the headcount whose full-time job becomes keeping integrations alive through vendor API changes, outages, and schema drift.

A useful rule of thumb: if the integration effort to make a SaaS tool work in your environment exceeds roughly forty percent of what it would cost to build the function internally, the build option has likely already won on total cost of ownership. You simply have not finished the calculation.

3. What is the lifecycle cost, not the first-year cost?

Every SaaS contract looks reasonable in year one. The question is what it looks like in year five. Model three lines on the same chart: the SaaS cost curve (inflation plus seat growth plus feature upsell), the custom build curve (upfront cost plus ongoing maintenance at roughly fifteen to twenty percent of the build cost per year), and the opportunity-cost curve (what does the SaaS not let you do, and what is that worth?).

On a five-year horizon, a well-built custom system with a competent internal or partner team tends to win for high-use, high-customization workflows. SaaS tends to win for low-use, low-customization ones. The middle is where most internal arguments happen.

4. How fast is the underlying process changing?

If the business process is volatile — it changes with every regulation update, every new product line, every acquisition — custom software lets you change the system as fast as the process changes. SaaS will let you change it as fast as the vendor’s roadmap allows.

Stable processes belong in SaaS. Volatile, competitive processes belong in systems you control.

When SaaS is the right answer

Several situations reliably favor buying:

The function is mature, well-understood, and standardized — payroll, accounting, email marketing.
Your usage pattern falls comfortably inside the vendor’s intended use cases.
The integration surface is narrow — two or three systems, not twenty.
You do not have, and cannot affordably hire, the engineering capacity to maintain a custom alternative.
The process is not a source of competitive differentiation.

In these cases, custom software is rarely a good investment. The engineering hours are worth more elsewhere.

When custom software pays off

The inverse is also clear. Custom is typically the correct answer when:

The function is central to how you make money or deliver value.
The process is specific enough that SaaS forces workarounds every quarter.
Integration complexity is high — the custom system will be an orchestration layer as much as a feature layer.
Lifecycle cost modeling shows SaaS overtaking custom within three to five years.
Data ownership or portability is a board-level concern.

The signal we watch for most: the team has already built extensive internal tooling on top of the SaaS product to make it work. That tooling is a quiet admission that the SaaS choice was wrong. It is also usually sixty to eighty percent of what a purpose-built replacement would cost, delivered in a less maintainable shape.

The hybrid model, and why it wins more often than people think

For most mid-market and enterprise companies, the correct answer is not build or buy. It is: buy the commodity layers, build the differentiation layer, and integrate them deliberately.

In practice, this means:

Use SaaS for HR, finance, standard CRM, standard support, and standard analytics.
Build custom for your pricing engine, your customer onboarding flow, your proprietary workflows, and your internal operational dashboards.
Invest deliberately in the integration layer. This is where most hybrid models fail, and it is worth the engineering rigor.

Done well, the hybrid model gives a company the cost discipline of SaaS and the differentiation of custom software, without the downside of either.

A decision framework you can apply this week

For any function where the question is open, answer these five prompts honestly:

Differentiation test. If this function worked thirty percent better than every competitor’s, would customers notice?
Process volatility test. How many times has this process changed in the last twenty-four months? How many times will it change in the next twenty-four?
Integration depth test. How many other systems does this function need to read from or write to?
Five-year cost test. What is the modeled total cost of ownership of each option in year five, not year one?
Capability test. Do we have — or can we reliably hire — the team needed to maintain a custom system over its lifecycle?

If the first two answers point to yes, this matters and the last three point to we can handle the complexity, custom is usually right. If they point the other direction, SaaS is usually right. If the answers are mixed, the hybrid model is almost always the answer.

How we approach this with clients

At Wolyra, we start engagements where the build-versus-buy question is still open by running a structured assessment against this framework. The deliverable is not a pitch for custom software development. In roughly a third of cases, our recommendation is to stay on SaaS, fix the integration layer, and revisit in eighteen months. In another third, we recommend a hybrid model. The remaining cases are where custom software genuinely pays off, and we design the build around lifecycle cost rather than launch cost.

The discipline matters more than the outcome. Companies that decide build-versus-buy deliberately — with the right criteria, on the right horizon — outperform companies that drift into one or the other. The decision is less about the software and more about the clarity of the thinking behind it.

If you are facing this decision now and want an outside perspective grounded in lifecycle economics rather than vendor preference, get in touch. We will work through the framework with you and tell you what we actually see — including the times the right answer is to do nothing.

The Hidden Cost of AI: A TCO Framework for Enterprise Leaders

Wolyra — Sat, 25 Apr 2026 15:00:15 +0000

The first invoice from a frontier AI provider is rarely the one that surprises a finance team. The fiftieth one is. By the time a company has several AI features in production, the monthly line item has often grown past what anyone budgeted for, and the breakdown has become opaque in a way that traditional software spend rarely is. Nobody on the engineering team can easily explain which feature is consuming which fraction of the bill, or what a twenty-percent reduction would require.

This is not a failure of discipline. It is a structural consequence of how AI systems price, how they consume resources, and how much of their cost is buried in adjacent infrastructure that was not obviously part of the AI stack. Understanding total cost of ownership for AI features is a different exercise than understanding it for SaaS or for internal services, and the finance teams that treat it the same way find themselves without clear levers when cost becomes a problem.

This post is the TCO framework we use with clients when AI spend becomes visible enough that the CFO starts asking questions.

The categories of AI cost

AI spend breaks into five categories. A complete TCO model has to account for all five. Most budgets only see the first.

Direct model costs. The per-token price multiplied by volume. This is the invoice the finance team sees. It is also usually the smallest of the five line items over the full lifecycle of a serious deployment, though it is the only one that gets attention.

Supporting infrastructure. Vector databases, embedding services, orchestration layers, observability tools, caching, queueing, rate limiting. A RAG system is not just a model; it is a small platform. The monthly cost of that platform — including the engineers who keep it running — often exceeds the model invoice.

Data preparation and evaluation. The dataset curation for fine-tuning, the golden sets for evaluation, the human review of samples, the red-team testing before release. These costs are concentrated at the start of a feature’s life and recur every time the underlying model or data changes materially. Teams that skip them pay instead in incidents and rework.

Operational headcount. The engineers who maintain the AI features, the platform team that supports them, the data team that curates inputs, the security team that reviews new capabilities. AI features tend to be staff-intensive in a way that SaaS features are not, because there is no vendor operating them on your behalf. You are the vendor for your own AI systems.

Incident cost. The business impact of a failed or degraded AI system — incorrect customer responses, lost sales, trust damage, regulatory exposure. This is the category that accountants struggle with and that risk teams care about most. It is harder to quantify, but ignoring it is the reason companies that underinvest in evaluation eventually overpay in reputation.

Where the hidden cost usually hides

Four patterns produce most of the cost overruns we see in practice.

Unbounded context growth. A feature launches with short prompts. Over six months, product teams add “helpful” context, system prompts grow, retrieval returns larger chunks, conversation history accumulates. Per-token costs double or triple without any announcement of a price change. The fix is boring and effective: context budget reviews, where every token in the prompt has to justify itself.

Retries and fallbacks. The invoice reflects tokens billed, not tokens useful. A feature that retries on safety filters, falls back to a more expensive model when the cheap one fails, or re-runs when the output format is invalid is paying for failures in addition to successes. At scale, the multiplier can be thirty or forty percent. Instrument retry rates as a cost signal, not just a reliability one.

Agentic workflows. An agent that takes ten tool calls and three planning rounds to answer a question costs roughly ten times what a single-shot model call would. The answers are often better, which is the point, but teams underestimate the cost multiplier. Tracking cost per user-facing outcome, not cost per model call, is the only way to see this clearly.

Over-capable models. The frontier-tier model is necessary for perhaps fifteen percent of the queries a feature handles. The other eighty-five percent could be handled by a mid-tier model for a fraction of the cost. Teams routinely send everything to the frontier tier because it is simpler, then discover six months later that a routing layer would have saved half the bill.

The accounting discipline

A TCO model that is actually useful requires a few operational disciplines that most organizations have to build deliberately.

Attribute cost to features, not teams. A provider invoice rolls up to one account. Useful cost analysis requires knowing that feature X consumed forty-seven percent of the spend this quarter, that feature Y is the fastest-growing line item, and that feature Z is the most expensive per customer interaction. This requires tagging every request with feature identifiers at the gateway layer.

Track cost per outcome, not per call. Cost per model call is a technical metric. Cost per resolved support ticket, cost per approved document, cost per qualified lead — those are business metrics that connect AI spend to value. If you cannot compute these, you cannot tell whether an AI feature is earning its cost.

Review monthly, model quarterly, redesign annually. A monthly review catches drift. A quarterly modeling exercise refreshes the TCO against actual usage and renegotiated rates. An annual redesign asks whether the architecture is still right — whether the routing, the model mix, the retrieval strategy, the caching layer are still fit for purpose. Three cadences, three different questions, each useful on its own.

The levers that actually move the number

When cost reduction becomes a priority, the levers with the highest impact tend to be the same across deployments:

Model routing. Sending simple queries to cheap models and hard queries to expensive ones. The easiest way to cut thirty to fifty percent of a frontier bill, with minimal quality impact when implemented carefully.
Prompt compression. Shorter system prompts, tighter retrieval chunks, deduplicated context. Often removes fifteen to twenty-five percent of input tokens without changing behavior.
Caching. For queries with overlapping contexts or answers, a cache layer that survives for seconds to minutes. Effective ratios vary wildly by workload, but a well-placed cache can remove twenty to sixty percent of calls.
Provider negotiation. Enterprise tiers, committed-use discounts, and regional pricing that are not advertised. At serious volume, this is a budgeted procurement activity, not a one-time conversation.

The strategic question

AI cost, handled well, is boring operational discipline. Handled badly, it becomes a strategic constraint — the reason a product cannot scale to the next tier of customers, or the reason a promising feature gets shut down for reasons the business never understood. The difference is whether the organization has built the instrumentation and the accounting discipline to see its AI spend the way it sees any other significant cost category.

If the honest answer to “what does our AI cost and why?” is a shrug, the fix is not more budget. It is the instrumentation that makes cost visible before it becomes a problem.

Why Scalable Architecture Matters for Growing Businesses

Wolyra — Tue, 14 Apr 2026 21:24:21 +0000

Most companies don't think about their software architecture until something breaks.

I've seen it happen more times than I can count. A business starts with a simple setup — maybe a single server,
a basic database, a monolithic application that handles everything. And for a while, it works fine. The team

is small, traffic is manageable, and nobody needs to worry about what happens when the number of users doubles
or triples.

Then it happens. Growth kicks in. Suddenly the system that handled 500 requests per minute is getting 5,000.

Pages load slower. The database starts choking. Deployments become risky because touching one part of the
codebase might break three others. The engineering team spends more time firefighting than building.

This is the cost of ignoring architecture early on.

## The Monolith Trap

There's nothing inherently wrong with monolithic applications. For early-stage products, they're often the

right choice — simple to build, simple to deploy, simple to reason about. The problem is that monoliths don't
age well under pressure.

When everything lives in one place, scaling means scaling everything. You can't independently scale the part of
your system that handles payments without also scaling the part that sends emails. Resources get wasted.
Bottlenecks become harder to isolate. A bug in one module can take down the entire application.

I'm not saying every company needs microservices on day one — that would be over-engineering. But every company
needs to think about what happens next.

## What Scalable Architecture Actually Looks Like

News. It's about making intentional decisions that give your system room to grow.

In practice, this means a few things:

Separation of concerns. Your authentication logic shouldn't be tangled with your billing system. When
components are loosely coupled, you can update, scale, or replace them independently.

Horizontal scaling capability. Instead of buying a bigger server every time traffic increases, your system
should be designed to run across multiple smaller instances. Load balancers distribute traffic. Stateless

services make this possible.

Database strategy. A single relational database can take you far, but there's a ceiling. Read replicas,
caching layers, and knowing when to introduce a different data store for specific workloads — these decisions

matter more than most teams realize.

Infrastructure as code. If your deployment process involves someone manually configuring a server, you have
a problem waiting to happen. Reproducible, automated infrastructure isn't a luxury — it's a baseline.

## The Real Cost of Getting It Wrong

Bad architecture doesn't just cause technical headaches. It costs real money.

Downtime during peak traffic means lost revenue. Slow page loads drive customers to competitors. Engineers
spending 70% of their time on maintenance instead of new features means your product falls behind. And when the
system finally needs a rewrite, that's months of work that could have been avoided with better decisions

upfront.

The tricky part is that architecture problems are invisible until they're not. Everything looks fine at low
scale. The cracks only show when the load increases — and by then, fixing them is expensive and disruptive.

## When to Start Thinking About This

The honest answer is: earlier than you think.

You don't need to build for a million users on day one. But you should build with the assumption that your
system will need to handle significantly more than it does today. That means making choices that don't paint

you into a corner — choosing technologies that support horizontal scaling, keeping components modular, and

investing in monitoring so you can see problems before your users do.

The companies that handle growth well aren't the ones with the most engineers or the biggest budgets. They're
the ones that made smart architectural decisions early, even when the immediate payoff wasn't obvious.

If your system is starting to show cracks, or if you're building something new and want to get the foundation

right, it's worth having that conversation sooner rather than later. At Wolyra, this is

exactly what we help organizations figure out — building digital systems that don't just work today, but hold
up as the business evolves.