Expectations vs Reality: Adopting AI (LLM) in Product Engineering

gauravsoni — Sat, 05 Apr 2025 11:28:29 +0000

As LLM become part of everyday engineering workflows, it's critical to understand what they can and cannot do — especially in product engineering, where business context matters deeply.

When you ask LLM to generate code for a particular business context (something only you know entirely), LLM don't "know" your business.

LLM can only predict based on:

Common patterns from the code that was trained on
Your prompt (whatever hints and context you give to LLM)
General best practices

BUT — LLM cannot magically "know" your private business rules unless you clearly describe them.

LLM predict code that sounds correct. LLM don't understand or validate your real-world needs unless you explain them.

How LLM work when you ask for code

Prediction: It predicts the most likely following tokens that look like good code for the problem you described.
Pattern matching: LLM copies and adapts patterns from its training data that seem to fit your request.
Filling gaps: If you miss details, it's based on common sense, but that guess might be wrong.

So, how do you get reliable code from an LLM?

Give very detailed prompts Include your business rules, data structures, constraints, and edge cases.
Use critical review after generation After LLM give you the code, you must review and test it to ensure it fits your business.
Set up test cases Always ask for code plus tests that verify the logic according to your business needs.
Iterative correction You might need to correct, guide, or refine the code a few times by giving feedback ("No, this function also needs to handle XYZ").

Analogy

Asking an LLM to write perfect business code without full context is like asking a lawyer who never heard your case to write a court judgment — they'll guess based on experience, but the real facts must come from you.

Here's a simple reality check for anyone adopting AI into their engineering teams:

Expectations vs Reality

LLM writes perfect business-specific code
➡️ LLM predicts likely code patterns based on general training data.

LLM understands my business domain
➡️ LLM does not know your private business context unless you explain it explicitly.

LLM validates outputs against business rules
➡️ LLM only predicts outputs — you must validate and test separately.

LLM reduces need for strong specs
➡️ LLM needs more precise and stricter specifications to perform well.

LLM saves time without supervision
➡️ LLM amplify productivity but requires human review to ensure correctness.

LLM innovate or invent new ideas
➡️ LLM recombine existing knowledge; they don't invent beyond what they have seen.

LLM replace developers or product engineers
➡️ LLM are assistants, not replacements — judgment and domain expertise stay human-driven.

Bigger models are always better
➡️ Smaller fine-tuned models often perform better for specific business use cases.

Key Mindset Shift for Product Engineers

Think of LLM as prediction engines, not knowledge engines.
Use them for draft generation, automation, and exploration, but own the final quality yourself.
Training data ≠ for your company’s context.
- If it’s not in the prompt, it’s not in the model’s head.
Good prompts = good outputs.
- Better input leads to much better results.

Tasks where LLM are very effective:

Drafting code templates
➡️ Fast at generating boilerplate, CRUD operations, and API wrappers.

Creating documentation
➡️ Summarises, formats, and structures text very quickly.

Generating test cases
➡️ Can suggest unit/integration tests if logic is clear.

Exploring alternatives
➡️ Provides multiple ways to solve a coding or design problem.

Speeding up research
➡️ Quickly summarises concepts, tools, libraries, and frameworks.

Idea expansion
➡️ Good at suggesting more use cases, edge cases, or features.

Writing first drafts of emails, specs, and user stories
➡️ Useful for early rough drafts to save time.

Basic data transformation scripts
➡️ Good at SQL queries, simple ETL scripts, and data formatting.

Tasks that require human ownership:

Understanding deep business context
➡️ LLM can't "know" your company’s strategy, policies, or customer expectations.

Validating correctness
➡️ AI-generated code, tests, or documents still need human review and adjustment.

Architectural decisions
➡️ LLM can suggest, but real-world trade-offs must be handled by experienced engineers.

Security and compliance
➡️ LLM may miss critical risks unless you guide them specifically.

Creative product thinking
➡️ True innovation — new product ideas and differentiation — still requires human creativity.

Prioritisation and trade-offs
➡️ AI doesn't "feel" urgency, politics, or customer pain points like humans do.

Cultural and communication nuances
➡️ Writing for internal stakeholders, clients, or executives needs human judgment on tone and sensitivity.

One-line Summary

LLM are power tools — not decision-makers.
Use them to amplify your thinking, not replace it.

💬 I'd love to hear how you're blending LLM into your engineering workflow! What challenges have you faced, and what wins have you seen?

Solving 502 Errors in Microservices Node.js and AWS ALB

gauravsoni — Sat, 27 Apr 2024 11:18:07 +0000

Understanding 502 Errors
A 502 Bad Gateway error occurs when a server acting as a gateway or proxy (like AWS Application Load Balancer or ALB) receives an invalid response from the upstream server (e.g., your Node.js application server). In microservices architectures, where multiple servers communicate through a load balancer, this error is common when server-to-server communication is interrupted.

The Problem: Timeout Mismatch
In my setup, AWS ALB was configured with a 30-second timeout, and the Node.js servers were handling requests using their default timeout settings. However, these default server timeout settings were not aligned with the ALB configuration, leading to unexpected connection terminations and resulting in 502 errors.

To resolve this, the timeout settings on the Node.js servers must be configured to complement ALB’s timeout.

Key Timeout Settings in Node.js
There are two critical timeout settings in Node.js that affect its interaction with AWS ALB:

KeepAliveTimeout
This setting determines how long the server keeps a connection open to allow additional requests from the same client. By default, Node.js sets this to 5 seconds, which is too short compared to ALB’s 30 seconds.
HeadersTimeout
This setting specifies the maximum time the server waits for complete request headers before closing the connection. By default, it is 60 seconds, which can be optimized for consistency.

To prevent 502 errors, the KeepAliveTimeout should be set slightly longer than the ALB timeout, and the HeadersTimeout should exceed the KeepAliveTimeout.

Adjust Node.js Server Timeout Settings
Configure the timeout settings in your Node.js application server. Here’s an example of how to do this:

const http = require("http");
const app = require("./app"); // Your Express or custom app setup
const server = http.createServer(app);
// Configure timeouts
server.keepAliveTimeout = 35000; // 35 seconds
server.headersTimeout = 40000; // 40 seconds
server.listen(3000, () => {
console.log("Server running on port 3000");
});

Apply these timeout settings consistently across all Node.js servers involved in the microservices architecture to prevent unexpected close connections.

Why These Settings Work
The KeepAliveTimeout ensures the Node.js server keeps the connection alive long enough to match or slightly exceed ALB’s idle timeout.
The HeadersTimeout ensures the server has ample time to process and respond to headers, preventing premature connection termination.

Configuring the correct timeout settings in Node.js servers when used with AWS ALB is crucial in mitigating 502 errors.

Forem: gauravsoni

Expectations vs Reality: Adopting AI (LLM) in Product Engineering

Solving 502 Errors in Microservices Node.js and AWS ALB