Forem: Ashok Kanjarla

AI Will Not Replace Engineers — But It Will Expose Shallow Thinkers.

Ashok Kanjarla — Thu, 05 Mar 2026 04:24:02 +0000

Introduction

Over the past few years, artificial intelligence has rapidly entered the daily workflow of software engineers. Tools capable of generating code, explaining complex algorithms, debugging issues, and even proposing architectural solutions have changed the development landscape dramatically. As these tools improved, a new narrative began to dominate discussions across technology forums, social media, and industry conferences: AI will replace software engineers.

At first glance, the claim appears convincing. AI systems can generate working code within seconds, automate repetitive tasks, and significantly reduce the time required to build features. To many observers, it seems inevitable that automation will reduce the need for human developers.

However, this conclusion misunderstands the real shift happening inside engineering teams.

Artificial intelligence is not replacing engineers. Instead, it is revealing a deeper distinction that has always existed in the industry: the difference between engineers who understand systems and engineers who only understand syntax.

The true disruption is not the elimination of coding jobs. It is the automation of shallow work—tasks that rely on pattern recognition rather than deep reasoning. As AI becomes more capable of generating code, the value of engineers who can reason about architecture, constraints, and long-term system behavior increases dramatically.

The future of engineering will not be defined by who can write code the fastest. It will be defined by who can evaluate, design, and sustain systems under real-world complexity.

The Anti-Pattern: Tool Dependency Without Understanding

The early adoption of AI coding tools has introduced a dangerous pattern in some engineering environments. Developers increasingly treat large language models as authoritative sources of solutions rather than as productivity accelerators.

The workflow often becomes simple and repetitive: describe the problem, receive generated code, paste it into the project, and move forward. For small scripts or isolated utilities, this approach can appear efficient and harmless. However, the problems emerge when the same workflow is applied to complex systems.

AI-generated solutions frequently work under ideal conditions but fail under real-world constraints. Generated database queries may appear correct yet introduce performance bottlenecks at scale. Suggested concurrency implementations may ignore subtle race conditions. Generated API patterns may overlook security considerations such as injection vulnerabilities or authentication weaknesses.

When these issues arise in production environments, someone must diagnose the root cause. Someone must understand the underlying architecture well enough to identify the flaw and design a stable correction.

If the developer relying on AI lacks that depth of understanding, debugging becomes slow and unpredictable. The tool that was intended to accelerate productivity instead multiplies complexity and risk.

Artificial intelligence can increase speed, but speed alone does not guarantee correctness. In distributed systems, speed without architectural judgment often leads to instability.

The Core Engineering Requirement in the AI Era

Before AI-assisted development became common, the primary limitation of less experienced engineers was often their ability to implement code efficiently. Writing large amounts of working code required time, concentration, and practice.

Today, that limitation has shifted dramatically.

AI systems can generate entire modules, refactor legacy functions, and suggest implementations within seconds. This means the bottleneck in engineering workflows is no longer code production—it is code evaluation.

Engineers must now determine whether generated solutions are safe, scalable, and maintainable. They must evaluate whether an AI-generated algorithm will degrade under heavy load, whether a proposed architecture will survive service failures, and whether a generated pattern aligns with long-term system goals.

This shift increases the importance of foundational engineering knowledge. Skills such as distributed system reasoning, database modeling, concurrency control, and failure-mode analysis become even more valuable in an AI-assisted environment.

When solutions become abundant, judgment becomes the scarce resource.

The engineers who thrive in the AI era will not be those who generate the most code. They will be those who can identify which generated code should never reach production.

The Mental Model Shift

As artificial intelligence becomes embedded in development workflows, two distinct mindsets are emerging among engineers.

The first group can be described as tool operators. These developers focus primarily on maximizing output by using AI prompts to generate solutions quickly. Their primary goal is speed and immediate functionality.

The second group can be described as system designers. These engineers use AI tools as accelerators, but they remain focused on the structural integrity of the system being built. They evaluate solutions based on constraints such as reliability, scalability, maintainability, and operational complexity.

Tool operators typically ask a simple question: “What prompt will generate the code I need?”

System designers ask a far more important question: “Under what conditions will this solution fail?”

This difference in perspective becomes increasingly important as systems grow in complexity. Artificial intelligence can generate structure, but it cannot assume responsibility for production incidents, service outages, or security breaches.

Engineers still own those outcomes.

The Trade-Off Reality

Software engineering has always been about trade-offs. Every system design must balance competing priorities such as performance, reliability, cost, and maintainability. Artificial intelligence does not eliminate these trade-offs—it simply accelerates the creation of solutions that must still be evaluated against them.

AI-generated code often prioritizes functional correctness, meaning the code compiles and produces the expected output for common inputs. However, non-functional requirements are frequently overlooked.

These requirements include system scalability, latency tolerance, security constraints, data consistency, and operational observability. A piece of code that functions perfectly in isolation may fail when deployed into a distributed environment handling thousands of requests per second.

The responsibility for identifying these risks remains with engineers.

AI can suggest solutions, but it cannot fully reason about complex operational environments or anticipate every edge case that arises in production systems.

Engineering maturity is therefore not measured by how quickly code can be produced. It is measured by how accurately engineers can anticipate where that code might fail.

Responsible Use of AI in Engineering

Integrating artificial intelligence into development workflows requires discipline. Engineers must treat AI-generated output as suggestions rather than authoritative answers.

In practice, this means applying several principles when working with AI-assisted tools.

AI-generated code should always undergo the same level of review as code written manually. Architectural decisions should be validated against real-world system constraints rather than accepted at face value. Generated solutions must be evaluated for security vulnerabilities, performance implications, and maintainability.

AI is extremely effective at accelerating repetitive or boilerplate tasks. It can also assist with brainstorming implementation strategies and explaining unfamiliar concepts. However, the responsibility for evaluating correctness, safety, and long-term viability must remain with the engineer.

The difference between responsible AI usage and careless reliance lies in whether the engineer remains in control of the reasoning process.

Observed Impact in Engineering Teams

Organizations that adopt AI tools responsibly often experience measurable improvements in productivity. Engineers can reduce time spent on repetitive coding tasks, accelerate debugging processes, and prototype solutions more rapidly.

When used correctly, AI becomes a powerful assistant that frees engineers to focus on higher-level design and decision-making.

However, teams that rely on AI without sufficient engineering discipline often encounter the opposite effect. Debugging cycles become longer as poorly understood generated code introduces hidden complexity. Systems become harder to maintain as inconsistent patterns emerge throughout the codebase. Security vulnerabilities appear in places where generated solutions were accepted without proper review.

The difference between these outcomes is not the technology itself but the level of engineering maturity applied to its use.

Final Thought

Artificial intelligence will not eliminate the need for engineers. What it will eliminate is the illusion that writing code alone defines engineering expertise.

As code generation becomes easier, the value of architectural thinking becomes greater. Engineers who understand distributed systems, performance constraints, and long-term design trade-offs will become even more essential in the development process.

The competitive advantage in the AI era will not belong to those who generate the most code. It will belong to those who can evaluate systems deeply, design resilient architectures, and make sound technical decisions under uncertainty.

AI will not replace engineers.

But it will expose the difference between those who merely produce code and those who truly understand how software systems behave. And in the long run, depth always compounds.

Building a Production-Ready AI Gateway in ASP.NET Core

Ashok Kanjarla — Tue, 17 Feb 2026 17:22:13 +0000

Introduction

AI integration in backend systems looks deceptively simple at first glance. You inject an OpenAI client, call GenerateAsync, and return the result. In development, everything behaves predictably. Under light load, the system performs well. The abstraction feels clean.

As usage increased within our ASP.NET Core services, AI quickly evolved from a helpful feature into a systemic risk. It became a latency bottleneck during peak traffic, a cost multiplier as token usage scaled, a failure amplifier whenever the provider experienced issues, and a debugging challenge when unexpected behavior surfaced in production.

We were handling thousands of concurrent requests per hour. External LLM providers introduced unpredictable latency spikes, occasional rate limits, and intermittent outages. Each of these external behaviors directly affected our API performance because we had tightly coupled AI calls to our request pipeline.

Calling providers directly from controllers created architectural fragility. What looked convenient initially was in fact exposing our core system to external volatility.

We redesigned the system around one core principle:

AI is not a feature. It is a distributed dependency.

The Anti-Pattern: Direct Controller Calls

Our initial implementation followed a common pattern:

[HttpPost("generate")]
public async Task<IActionResult> Generate([FromBody] PromptRequest request)
{
    var result = await _openAiService.GenerateAsync(request.Prompt);
    return Ok(result);
}

From a surface-level perspective, this approach appears clean and efficient. However, at scale, this design introduces multiple systemic weaknesses.

There were no retry policies to handle transient network failures. There was no circuit breaker to protect the system during sustained provider outages. The implementation lacked provider abstraction, meaning vendor changes would ripple across the codebase. There was no rate limiting to protect upstream dependencies. Token cost tracking was absent, making budgeting reactive rather than proactive. Observability was fragmented, making incident investigation difficult.

If OpenAI throttled requests, our API would fail immediately. If latency spiked, request threads remained blocked longer than expected. If pricing models changed, we had no structured mechanism to monitor impact.

This design violated fundamental distributed systems resiliency principles.

Architectural Goal

Before rewriting anything, we defined explicit engineering requirements.

We needed the AI provider to be abstracted so that business logic remained vendor-agnostic. Failures from external providers could not cascade into the API layer. Token usage needed to be measurable to support cost governance and forecasting. Providers had to be swappable to support future multi-provider strategies. Latency required containment mechanisms. Observability needed to be built into the architecture rather than added later.

In short, AI integration had to be treated as infrastructure, not as a helper library.

High-Level Architecture

Client Request
      │
      ▼
ASP.NET Core API
      │
      ▼
AI Gateway Layer
 ├── Provider Abstraction
 ├── Retry Policy
 ├── Circuit Breaker
 ├── Rate Limiter
 ├── Token Metering
 ├── Structured Logging
 └── Fallback Strategy
      │
      ▼
AI Provider(s)

The key shift was introducing a dedicated AI Gateway layer. This layer became the containment boundary between core business logic and external AI providers. It absorbed volatility, enforced governance, and centralized resilience patterns.

Step 1 — Provider Abstraction

We began by defining a strict contract for AI providers:

public interface IAIProvider
{
    Task<AIResponse> GenerateAsync(AIRequest request, CancellationToken ct);
}

A concrete implementation looked like this:

public class OpenAIProvider : IAIProvider
{
    private readonly HttpClient _client;

    public OpenAIProvider(HttpClient client)
    {
        _client = client;
    }

    public async Task<AIResponse> GenerateAsync(AIRequest request, CancellationToken ct)
    {
        var response = await _client.PostAsJsonAsync(
            "/v1/chat/completions",
            request,
            ct);

        response.EnsureSuccessStatusCode();

        return await response.Content.ReadFromJsonAsync<AIResponse>(cancellationToken: ct);
    }
}

This abstraction decouples business logic from the vendor, enables a multi-provider strategy, improves unit testing, and prevents long-term vendor lock-in. More importantly, it creates a seam in the architecture where resilience policies can be applied consistently.

Step 2 — Resilience with Polly

External AI providers are network dependencies. Network dependencies fail. Therefore, resilience policies must be applied systematically.

We introduced exponential backoff retry policies:

builder.Services.AddHttpClient<IAIProvider, OpenAIProvider>()
    .AddPolicyHandler(
        Policy.Handle<HttpRequestException>()
        .WaitAndRetryAsync(3,
            retry => TimeSpan.FromSeconds(Math.Pow(2, retry)))
    );

We also implemented circuit breakers:

.AddPolicyHandler(
    Policy.Handle<HttpRequestException>()
    .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30))
);

Retries handle transient failures. Circuit breakers prevent cascading system degradation during sustained outages. Without a breaker, retry mechanisms can unintentionally amplify failure by increasing pressure on an already struggling provider.

Together, they create controlled resilience.

Step 3 — Token Cost Governance

AI usage scales cost non-linearly. Without visibility, cost becomes reactive and unpredictable.

We implemented middleware to track token usage centrally:

public class TokenTrackingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<TokenTrackingMiddleware> _logger;

    public TokenTrackingMiddleware(RequestDelegate next, ILogger<TokenTrackingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task Invoke(HttpContext context)
    {
        await _next(context);

        if (context.Items.TryGetValue("TokenUsage", out var tokens))
        {
            _logger.LogInformation("Token usage: {Tokens}", tokens);
        }
    }
}

This enabled cost forecasting, per-tenant billing capabilities, and anomaly detection. AI usage stopped being opaque and became measurable infrastructure.

Step 4 — Rate Limiting

To prevent AI overload and protect upstream providers, we introduced rate limiting:

builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter("aiPolicy", limiterOptions =>
    {
        limiterOptions.PermitLimit = 100;
        limiterOptions.Window = TimeSpan.FromMinutes(1);
    });
});

Applied at the endpoint level:

[EnableRateLimiting("aiPolicy")]

This provided guardrails that preserved both system stability and provider relationships.

Step 5 — AI Gateway Orchestrator

The gateway orchestrates provider selection and failover logic:

public class AIGateway
{
    private readonly IEnumerable<IAIProvider> _providers;
    private readonly ILogger<AIGateway> _logger;

    public AIGateway(IEnumerable<IAIProvider> providers, ILogger<AIGateway> logger)
    {
        _providers = providers;
        _logger = logger;
    }

    public async Task<AIResponse> GenerateAsync(AIRequest request, CancellationToken ct)
    {
        foreach (var provider in _providers)
        {
            try
            {
                return await provider.GenerateAsync(request, ct);
            }
            catch (Exception ex)
            {
                _logger.LogWarning(ex, "Provider failed. Trying next.");
            }
        }

        throw new Exception("All AI providers unavailable.");
    }
}

This design implements controlled failover and prepares the architecture for future multi-provider routing strategies.

What We Explicitly Rejected

We deliberately avoided patterns that appear convenient but degrade system integrity over time. We rejected direct SDK calls in controllers, blocking calls such as .Result, static singletons without resilience policies, hard-coded API keys, and ignoring cancellation tokens.

These approaches are acceptable in prototypes. They are not acceptable in production-grade distributed systems.

Operational Impact

Within three months of implementing the AI Gateway architecture, we observed measurable improvements. AI-related failures dropped by 42%. Token costs were reduced by 31% due to better visibility and governance. There were zero cascading API outages during provider incidents. Response latency stabilized within SLA thresholds.

Most importantly, AI became predictable infrastructure rather than a systemic risk.

Final Thoughts

External dependencies will fail. Latency will spike. Costs will fluctuate. These realities do not disappear with better SDKs or cleaner controller methods.

Architecture does not eliminate volatility. It contains it.

By introducing an AI gateway layer with abstraction, resilience policies, rate limiting, and observability, we transformed AI from an unstable integration into governed infrastructure. The difference is not merely technical elegance — it is operational confidence.

Treat AI as infrastructure, not as a feature, and your systems will scale with clarity rather than uncertainty.