Forem: Voskan Voskanyan

The 15-Minute Goroutine Leak Triage: Two Dumps, One Diff, Zero Guessing

Voskan Voskanyan — Tue, 06 Jan 2026 18:16:02 +0000

Goroutine leaks rarely announce themselves with a dramatic outage. They show up as "slowly getting worse":

p95/p99 creeps up over an hour or two
memory trends upward even though traffic is flat
goroutine count keeps climbing and doesn’t return to baseline

If you’ve been on-call long enough, you’ve seen the trap: people debate why it’s happening before they’ve proven what is accumulating.

This post is a compact, production-first triage that I use to confirm a goroutine leak fast, identify the dominant stuck pattern, and ship a fix that holds.

If you want the full long-form runbook with a root-cause catalog, hardening defaults, and a production checklist, I published it here:

https://compile.guru/goroutine-leaks-production-pprof-tracing/

What a goroutine leak is (the only definition that matters in production)

In production I don’t define a leak as "goroutines are high."

A goroutine is leaked when it outlives the request/job that created it and it has no bounded lifetime (no reachable exit path tied to cancellation, timeout budget, or shutdown).

That framing matters because it turns debugging into lifecycle accounting:

What started this goroutine?
What is its exit condition?
Why is the exit unreachable?

Minute 0–3: confirm the signature (don’t skip this)

Before you touch profiling, answer one question:

Is the system accumulating concurrency footprint without a matching increase in work?

What I look at together:

QPS / job intake (flat or stable-ish)
goroutines (upward slope)
inuse heap / RSS (upward slope)
tail latency (upward slope)

If goroutines spike during a burst and then gradually return: that’s not a leak, that’s load.

If goroutines rise linearly (or step-up repeatedly) while work is stable: treat it as a leak until proven otherwise.

Minute 3–10: capture two goroutine profiles and diff them

The key move is comparison. A single goroutine dump is noisy. Two dumps tell you what’s growing.

Option A (best for diffing): capture profile format and use `go tool pprof`

Capture twice, separated by 10–15 minutes.

curl -sS "http://$HOST/debug/pprof/goroutine" > goroutine.1.pb.gz

sleep 900

curl -sS "http://$HOST/debug/pprof/goroutine" > goroutine.2.pb.gz

Now diff them.

go tool pprof -top -diff_base=goroutine.1.pb.gz ./service-binary goroutine.2.pb.gz

What you want:

one (or a few) stacks that grow a lot
a clear wait reason: channel send/recv, network poll, lock wait, select, etc.

Option B (fastest human scan): `debug=2` text dumps

curl -sS "http://$HOST/debug/pprof/goroutine?debug=2" > goroutines.1.txt

sleep 900

curl -sS "http://$HOST/debug/pprof/goroutine?debug=2" > goroutines.2.txt

Then do a "poor man’s diff":

search for repeated top frames
count occurrences (even roughly)
focus on the stacks with the biggest growth

Minute 10-15: map the dominant stack to the first fix you should try

Once you have "the stack that grows," the fix is usually not mysterious. Here’s the mapping I use to choose the first patch.

1) Many goroutines blocked on `chan send` / `chan receive`

Interpretation: backpressure/coordination bug. Producers outpace consumers, or receivers are missing, or close ownership is unclear.

First fix:

add a cancellation edge to send/receive paths (select { case <-ctx.Done(): ... })
bound the queue/channel (and decide policy: block with timeout vs reject)

Example helper:

func sendWithContext[T any](ctx context.Context, ch chan<- T, v T) error {
    select {
    case ch <- v:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

2) Many goroutines stuck in `net/http.(*Transport).RoundTrip` / netpoll waits

Interpretation: outbound I/O without a real deadline or missing request context wiring. Slow downstream causes your service to "hold on" to goroutines.

First fix:

enforce timeouts at the client level (transport + overall cap)
always use http.NewRequestWithContext (or req = req.WithContext(ctx))
always close bodies and bound reads

3) Many goroutines waiting on `WaitGroup.Wait`, semaphores, or `errgroup`

Interpretation: join/cancellation bug or unbounded fan-out. Work starts faster than it completes; cancellation doesn’t propagate; someone forgot to wait.

First fix:

ensure there is exactly one "owner" that always calls Wait()
use errgroup.WithContext so cancellation is wired
bound concurrency explicitly (e.g., SetLimit)

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(16)

4) Many goroutines in timers/tickers / periodic loops

Interpretation: time-based resources not stopped, or loops not tied to cancellation/shutdown.

First fix:

stop tickers
stop + drain timers when appropriate
ensure the loop has a ctx.Done() exit

Where tracing fits (and why it’s worth it even if pprof "already shows the stack")

pprof tells you what is stuck. Tracing tells you:

which request/job spawned it
what deadline/budget it had
which downstream call/queue wait never returned

If you already have OpenTelemetry (or any tracing), the fastest win is:

put spans around anything that can block: outbound HTTP/gRPC, DB calls, queue publish/consume, semaphore acquire, worker enqueue
tag spans with route/operation, downstream name, and timeout budget

That way, when profiling says "these goroutines are stuck in RoundTrip," tracing tells you "95% of them are from /enrich, tenant X, calling payments-api, timing out at 800ms."

The patch that actually holds: ship "hardening defaults," not one-off fixes

If you only patch the one stack you saw today, the next incident will be a different stack.

The fixes that keep paying dividends are defaults:

timeout budgets at boundaries
bounded concurrency for any fan-out
bounded queues + explicit backpressure policy
explicit channel ownership rules
structured shutdown (stop admission → cancel context → wait with a shutdown budget)

I keep the complete hardening patterns + production checklist in the full post:

https://compile.guru/goroutine-leaks-production-pprof-tracing/

Prove it’s fixed (don’t accept vibes)

A real fix has artifacts:

goroutine slope stabilizes under the same traffic/load pattern
the dominant growing stack is gone (or bounded) in comparable snapshots
tail latency and timeout rate improve (or at least stop worsening)

Also watch out for "false confidence":

restarts and autoscaling can hide leaks without removing the bug
short tests miss slow leaks (especially timer/ticker issues)

Wrap-up

The fastest way to win against goroutine leaks is to stop guessing:

1) confirm the signature (slope + correlation)

2) take two goroutine captures and diff them

3) fix the dominant stack with lifecycle bounds (timeout/cancel/join/backpressure)

4) prove the fix with before/after slope and comparable snapshots

If you want the deeper catalog of leak patterns and the production checklist I use in reviews and incident response, here’s the complete runbook:

https://compile.guru/goroutine-leaks-production-pprof-tracing/

10 Advanced Prompting Techniques for Getting Better Results from ChatGPT

Voskan Voskanyan — Wed, 19 Nov 2025 18:27:37 +0000

Most people think ChatGPT gives random or shallow answers - but in reality, the quality of its output depends entirely on how you structure the input. Modern LLMs respond best when you treat them like highly capable assistants: give them a role, context, constraints, and a clear outcome.

In this post, I’ll walk through 10 advanced prompting techniques that consistently produce expert-level answers, cleaner code, deeper analysis, and more useful documents. These techniques are based on 2026-era model behavior and are designed for developers, founders, analysts, and anyone who uses AI for daily work.

For a deeper dive and more examples, check out my extended guide:
10 powerful prompts for ChatGPT

Why Advanced Prompting Matters

Large language models don’t "guess" what you want—they follow instructions.
The more structure you provide, the more predictable the result.

Here are four fundamentals behind every strong prompt:

1. Assign a Role

Starting with "Act as a..." frames the model's tone, depth, and domain knowledge.

2. Provide Clear Context

Give it the background, goals, constraints, examples, and assumptions upfront.

3. Encourage Reasoning

Use "think step-by-step" or "show your reasoning" for complex tasks.

4. Define the Output Format

Tell it exactly how to present the result: JSON, table, bullets, sections, etc.

Master these four elements, and the rest becomes easy.

Advanced Prompts for 2026

Below are 10 fully ready prompts you can paste directly into ChatGPT.

1. The Guided Learning Prompt

A powerful way to actually learn a complex subject, not skim it.

Prompt:

Act as a Socratic tutor for the topic of [your topic]. 
Start by asking me a foundational question to assess my knowledge. 
Do not explain anything directly—only guide me through questions. 
Wait for my response after each question.

2. The Code Refinement Expert

Great for improving messy or legacy code.

Act as a senior developer specializing in [language]. 
I will paste code. 
Refactor it for readability, performance, and best practices. 
Return the improved code first, then summarize all changes in bullets.

3. The On-Demand Data Analyst

Act as a data analyst. 
I will paste a CSV dataset. 
Your tasks:
1. Identify the top 3 insights.
2. Summarize the findings.
3. Present key metrics in a Markdown table.

4. The Multi-Format Content Transformer

Useful for content creators and marketers.

Take the core ideas from this article (text or link). 
Turn them into a 5-part Twitter/X thread. 
Each tweet must be under 280 characters and include a relevant hashtag.

5. The Devil’s Advocate Prompt

My argument is: [your argument]. 
Act as a devil’s advocate and generate 3 strong counterpoints. 
Identify the primary logical framework behind each one.

6. The Persona-Based Copywriter

Act as a world-class copywriter for a luxury automotive brand. 
Write a 150-word product description for an all-electric SUV. 
Focus on silent power, eco materials, minimalism. 
Audience: tech-savvy professionals 35-50.

7. The Meeting -> JSON Converter

Summarize the following meeting transcript. 
Output must be a JSON object with:
- key_decisions: string[]
- action_items: { owner: string, task: string }[]
- open_questions: string[]

8. The API Documentation Generator

Act as a technical writer. 
Generate OpenAPI 3.0 documentation for this Flask route. 
Include descriptions, parameters, and example responses for 200 and 404.

9. The Analogy Explainer

Explain [complex topic] using an analogy based on [simple topic]. 
Make the analogy detailed enough to teach a beginner.

10. The Strategic SWOT Architect

Act as a strategist. 
Create a SWOT analysis for a [business type] operating in a [market condition].
Organize each section into 3–5 concise points.

Troubleshooting: When Your Prompts Still Fail

If the results still feel off, check these common mistakes:

Start a new chat

Clears leftover context.

Break large tasks into smaller ones

LLMs handle sequential workflows better.

Give examples

Show the format you want.

Rephrase

Even small changes fix misinterpretations.

Good prompts aren’t "magic"-they’re structured instructions.
Once you learn how to communicate with AI in a role-context-reasoning-format style, you unlock capabilities most users never reach.

For expanded explanations, deeper examples, and fully customizable templates, take a look at my full guide:

Core Principles of Effective Prompting

Breaking the Monolith: How We Split a Node.js Backend into Go Microservices on AWS ECS-Without Stopping the World

Voskan Voskanyan — Sat, 25 Oct 2025 14:36:48 +0000

We didn’t "rewrite everything from scratch". We couldn’t. We have a product to ship and customers to support. What we did at SolarGenix.ai was more boring and more durable: we peeled a Node.js (TypeScript) monolith apart one seam at a time, stood up small Go services on AWS ECS, and changed the way services talk to each other-from synchronous REST fan-out to events on EventBridge.

This is the playbook we actually used. It’s not a template for everyone, but if you're staring at a similar migration, it should save you a few traps.

Why we changed

Our monolith wasn’t "bad". It was successful enough to accumulate real traffic and real teams. But success came with some sharp edges:

Tight coupling inside request/response. A single HTTP request often fanned out to three or more internal calls. When one downstream was slow, p95 went up for the entire user flow.
Cascading latency. Spiky load in one area caused queueing in unrelated areas because everything was tied to the same hot path.
Deploy risk. Shipping a "small" change could nudge a shared code path and cause unwanted side-effects. Rollbacks existed in theory; in practice, the blast radius was too big.

Our goals were mundane: safer deploys, clearer ownership, and predictable delivery. We wanted a system where one domain could evolve without touching six others, and where we could choose consistency vs. latency explicitly instead of inheriting it accidentally.

The plan (gradual, not big-bang)

We kept users on REST at the edge. Internals moved from sync calls to events.

REST at the boundary. Edge handlers stayed in the monolith for a while because the API surface was stable and familiar to clients.
Turn side-effects into events. Instead of calling N services synchronously, the handler published an event-proposal.published is the canonical example-and returned fast.
EventBridge for bus + routing. We picked Amazon EventBridge as the managed bus and routing layer. It let us route with simple patterns and avoid building our own "bus that only we understand."
Go for small services. New domain services were written in Go. Teams liked the simplicity, the small memory footprint, and the standard library. No framework rabbit hole.
Mirror prod names in staging. Staging mirrored production naming (with a -stg suffix), so cutovers were predictable. Versioned detail-type events (proposal.published:v2) gave us room to evolve.

We ran this migration as a rolling set of small moves, not a calendar-driven "big switch."

Contracts and compatibility

OpenAPI at the edge

Clients didn’t need to care that our internals changed. We published OpenAPI contracts for the edge endpoints and kept them stable. Where we knew we would break something, we versioned the endpoint explicitly or added a feature flag to control new behavior.

Event envelope

Every event followed the same envelope. It sounds nitpicky; it saved us a lot of confusion.

{
  "id": "01J9Q9W0R3W3S3DAXYVZ8R3M7V",
  "source": "app.solargenix",
  "type": "proposal.published",
  "version": "v2",
  "occurredAt": "2025-09-21T14:33:12Z",
  "data": {
    "proposalId": "pr_0f3b1e",
    "accountId": "acc_7a21",
    "publishedBy": "u_193",
    "currency": "USD"
  }
}

id is a stable ULID (more readable, roughly time-sortable).
type is a domain noun: proposal.published, account.updated, etc.
version is a major version only. Minor changes must be additive.
occurredAt is UTC. The producers set it once. Consumers don’t "fix" it.

Dual-publish for majors

Major changes published both v1 and v2 for a sprint. Consumers opted into v2 when ready. We measured v1 usage and removed it when zero. No hidden toggles, no guessing.

Idempotency, retries, DLQ

You don’t get exactly-once delivery. Assume duplicates; you’ll sleep better.

Idempotency table in DynamoDB. Each consumer writes a row keyed by eventId with a TTL at least equal to the retry window (we use 24h). If the key exists, the side-effect already happened; drop it.
Per-target DLQs and explicit retry policies. Every rule target has its own DLQ and RetryPolicy. When something breaks, it breaks locally. We can triage by DLQ name and owner.
At-least-once thinking. We stopped treating duplicates as "bugs" and started treating lack of idempotency as the bug.

Minimal consumer shape (SQS buffer -> Lambda):

// Skeleton; real code has tracing, structured logs, and metrics.
func Handle(ctx context.Context, e events.SQSEvent) error {
    for _, r := range e.Records {
        var ev Event
        if err := json.Unmarshal([]byte(r.Body), &ev); err != nil {
            return err // retried by SQS/Lambda, ends up in DLQ if persistent
        }

        // TTL ≥ retry window; we used 24h.
        seen, err := idem.Seen(ctx, ev.ID, 24*time.Hour)
        if err != nil { return err }
        if seen { continue }

        if err := apply(ev); err != nil {
            return err // retry; isolated to this target
        }
    }
    return nil
}

Data projections & reads

We kept records of truth where they fit best and projected what the UI needed.

DynamoDB Streams for projections. Where a table is authoritative (e.g., counters, idempotency, some hot entities), Streams emit change events that update search indexes, analytics, or a read-optimized view.
Hot read path. UI reads follow: cache -> key lookup (DynamoDB) -> fallback to source of truth (often Aurora or the monolith during transition). Cache TTLs match how stale each endpoint is allowed to be.
Cache busting. Any state transition that changes what the UI renders triggers cache invalidation. We use small, explicit helpers-no "magic auto-busting."

The result: p95 fast-path reads dropped from 48 ms to 11 ms. Most pages now hit a projection or cache instead of stitching five joins under pressure.

Release strategy on ECS

We didn’t forklift deploy. We pared the system into boundaries and rolled them out one at a time.

Service-by-service rollout. Each new Go service ran in ECS with its own task definition, autoscaling policy, and health checks. We didn’t cram everything into one cluster service.
Shadow consumers. Early consumers ran in shadow for a sprint. They processed the real event stream and wrote results next to the old path. We diffed until we trusted them.
Feature flags. When a consumer replaced an old synchronous call, we shipped the flag first, validated in staging, then flipped for a subset of tenants in production.
Rollback plan. We kept the synchronous path alive for a while. If the consumer misbehaved, we flipped the flag off and investigated with a hot DLQ replay.
Blast-radius limits. We routed by detail-type, source, and sometimes accountId or region to control who got the new behavior.

A sketch of the EventBridge rule we used repeatedly:

{
  "Name": "proposal-published-v2",
  "EventPattern": {
    "source": ["app.solargenix"],
    "detail-type": ["proposal.published:v2"]
  },
  "Targets": [{
    "Arn": "arn:aws:lambda:...:function:proposal-emailer",
    "RetryPolicy": { "MaximumRetryAttempts": 185, "MaximumEventAgeInSeconds": 86400 },
    "DeadLetterConfig": { "Arn": "arn:aws:sqs:...:dlq-proposal-emailer" }
  }]
}

Each target had its own DLQ (e.g., dlq-proposal-emailer). Ownership was never ambiguous.

The small Go publisher

Producers do one thing: put a clean event on the bus. No tight loops. No cleverness.

type Event struct {
    ID         string      `json:"id"`
    Source     string      `json:"source"`
    Type       string      `json:"type"`    // e.g., "proposal.published"
    Version    string      `json:"version"` // e.g., "v2"
    OccurredAt time.Time   `json:"occurredAt"`
    Data       interface{} `json:"data"`
}

func Publish(ctx context.Context, bus, region string, ev Event) error {
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion(region))
    if err != nil { return err }
    cli := eventbridge.NewFromConfig(cfg)

    detailType := fmt.Sprintf("%s:%s", ev.Type, ev.Version)
    payload, _ := json.Marshal(ev)

    _, err = cli.PutEvents(ctx, &eventbridge.PutEventsInput{
        Entries: []types.PutEventsRequestEntry{{
            EventBusName: &bus,
            Source:       aws.String(ev.Source),
            DetailType:   aws.String(detailType),
            Time:         aws.Time(ev.OccurredAt),
            Detail:       aws.String(string(payload)),
        }},
    })
    return err
}

That’s it. The repository or handler constructs the event with the right version and calls Publish.

Observability that catches real problems

We avoided dashboards that look great in demos and tell you nothing on-call. The few signals that consistently matter:

DLQ depth and age per target. Alerts when DLQ depth ≥ 5 for 3 minutes, and when max age > 10 minutes. That split catches both bursts and stuck poison messages.
EventBridge target failure rate. Metric math on Invocations vs. FailedInvocations per rule target. We page on a non-zero failure rate sustained for 5 minutes.
Read p95 for hot endpoints. Because that’s what users feel. We annotate deploys so we can correlate regressions with changes.
Projection lag. If the read model is stale beyond our acceptable TTL, we want to know before users do.

We also added one "boring but lifesaving" alarm on idempotency table write failures. If the table throttles or permissions drift, duplicates slip through.

Results and one scar-tissue story

Numbers first:

p95 fast-path reads: 48 ms -> 11 ms after shifting hot reads to a key lookup or cache.
DLQ rate: < 0.1% over the last 30 days. Replays are scripted and dull (which is the point).

Now the scar tissue.

During one cutover we assumed global ordering for a certain flow. It wasn’t. A burst produced duplicates in a consumer that sent emails. Some customers received two messages. The fix was straightforward once we accepted the delivery model: we keyed idempotency by eventId + recipient and set a 24h TTL. The consumer became "boring." On-call became boring with it.

What we’d do again

Clean envelopes. That little JSON contract carried more weight than any library choice.
type:version in detail-type. Routing by major version without parsing payloads is a gift to operations.
Per-target DLQs. Ownership and blame become obvious. It shortens incidents.
A few boring alarms. DLQ depth/age, failure rate per target, p95 reads, projection lag. No noise.

What we’d change next time

Earlier "projection first" thinking. We could have moved hot reads to projections sooner and earned the p95 win earlier.
Stricter contracts for optional fields. We allowed too many "maybe present" fields that crept into business logic. I’d lock those down sooner.
Cost/ops trade-off, acknowledged. Managed EventBridge beats self-hosted when you’re small or moving fast. At higher scale, some teams roll their own bus for cost and control. For us, the ops time saved was worth the bill. Your curve may differ.

Notes on Clean Architecture (brief, practical)

We didn’t treat Clean Architecture as a religion. We kept it to two rules:

Domain code doesn’t depend on the transport. Handlers call a use-case; use-cases call repositories; repositories hide storage and messaging.
No cross-domain imports. If two domains need to coordinate, they publish/consume events. They don’t import each other’s packages and reach in.

It kept the Go services small and readable, and it made moving logic between processes almost trivial.

We didn’t stop the world; we changed how it moved. The monolith is smaller now, the teams are less entangled, and the hot paths are faster. The nice part is that none of this requires heroics-just consistent patterns and the discipline to apply them.

If there’s interest, I’ll follow up with the exact fan-out patterns we use and how we handle backfill/replay safely. If this was useful, follow my profile and the SolarGenix.ai page to catch the next deep dives and benchmarks as they land.

Event-Driven Architecture in Production: Designing with AWS EventBridge and Go

Voskan Voskanyan — Thu, 23 Oct 2025 06:32:42 +0000

We recently finished moving a set of synchronous REST flows at SolarGenix.ai to an event-driven core. The goal wasn’t to chase buzzwords; we wanted fewer cross-service dependencies, clearer ownership, and predictable delivery under load. Below is what we actually shipped: EventBridge as the bus, Go for publishers/consumers, DynamoDB Streams for projections, and a small set of rules that kept us out of trouble.

Two quick numbers after cutover:

Fast-path reads p95: 48 ms -> 11 ms (UI paths now hit a keyed projection or cache).
DLQ rate: <0.1% over the last 30 days (and most of those were known poison messages we fixed and replayed).

Why we moved off "sync everything"

We had a handful of handlers that fanned out to multiple services. When one downstream slowed down, the whole user flow did too. We kept REST at the edge (users), but replaced side-effects with events:

Before: handler -> service A -> service B -> service C
After: handler (200 OK) -> publish proposal.published -> N consumers do their work independently

The day we shipped the first flow, on-call went noticeably quieter.

Bus and naming (obfuscated but real)

Bus: app-core-prod (staging mirrors prod names with -stg suffix)
Detail-type: proposal.published:v2 (type:major) so we can route by version without parsing JSON
Rule example: proposal-published-v2 -> targets emailer, projector, analytics indexer
DLQ: one per target, e.g. dlq-proposal-emailer (ownership is obvious)

Event envelope and versioning

We keep a small, stable envelope and version the detail-type:

{
  "id": "01J9Q9W0R3W3S3DAXYVZ8R3M7V",
  "source": "app.solargenix",
  "type": "proposal.published",
  "version": "v2",
  "occurredAt": "2025-10-10T14:33:12Z",
  "data": {
    "proposalId": "pr_0f3b1e",
    "accountId": "acc_7a21",
    "publishedBy": "u_193",
    "currency": "USD"
  }
}

Minor fields -> additive only.
Major changes -> dual-publish (v1 + v2) for a sprint; consumers opt in when ready.
Schemas live in the EventBridge Schema Registry and we generate Go bindings.

Idempotency: assume duplicates

EventBridge and Streams are at-least-once. We treat duplicates as normal:

Every event has a stable id (we use ULID).
Consumers write eventId into a small idempotency table (DynamoDB) with a TTL >= retry window.
If the key exists, we skip the side effect and move on.

Minimal consumer shape (SQS buffer -> Lambda):

func Handle(ctx context.Context, e events.SQSEvent) error {
    for _, r := range e.Records {
        var ev Event
        if err := json.Unmarshal([]byte(r.Body), &ev); err != nil { return err }

        seen, err := idem.Seen(ctx, ev.ID, 24*time.Hour)
        if err != nil { return err }
        if seen { continue }

        if err := apply(ev); err != nil { return err } // retried by SQS/Lambda
    }
    return nil
}

Delivery guarantees, retries, and DLQs

Each rule target sets an explicit RetryPolicy and a DLQ:

{
  "Name": "proposal-published-v2",
  "EventPattern": { "source": ["app.solargenix"], "detail-type": ["proposal.published:v2"] },
  "Targets": [{
    "Arn": "arn:aws:lambda:...:function:proposal-emailer",
    "RetryPolicy": { "MaximumRetryAttempts": 185, "MaximumEventAgeInSeconds": 86400 },
    "DeadLetterConfig": { "Arn": "arn:aws:sqs:...:dlq-proposal-emailer" }
  }]
}

This is boring on purpose. When something breaks, it breaks locally (one rule/target), not across a chain.

DynamoDB Streams for projections

Where the record of truth lives in DynamoDB (e.g., counters, hot entities), we project changes out with Streams:

Streams retention is 24h; consumers must catch up.
Ordering is per item, not global. If we care about causal order, we put a version and timestamps in the payload and validate before applying.

Observability: alarms we actually watch

We wired a few CloudWatch alarms that correlate with user pain. Two that paid off immediately:

DLQ depth / age

{
  "AlarmName": "dlq-proposal-emailer-depth",
  "MetricName": "ApproximateNumberOfMessagesVisible",
  "Namespace": "AWS/SQS",
  "Dimensions": [{"Name":"QueueName","Value":"dlq-proposal-emailer"}],
  "Statistic": "Sum",
  "Period": 60,
  "EvaluationPeriods": 3,
  "Threshold": 5,
  "ComparisonOperator": "GreaterThanOrEqualToThreshold"
}

EventBridge rule failure rate (per target) Metric math on Invocations vs FailedInvocations for proposal-published-v2 -> proposal-emailer. We page when failures > 0 for 5 minutes. That caught a bad template deploy before users did.

Migration notes (what actually happened)

We wrapped the old REST handler so it returns 200 quickly and publishes an event.
New consumers ran in shadow for a sprint and wrote their results next to the old path so we could diff.
We hit one snag: an emailer Lambda assumed global ordering; a burst of duplicate events produced repeat sends. Fix was trivial once we added idempotency with a 24h TTL and keyed by eventId + recipient.

Results after cutover

Fast-path p95 reads: 48 ms -> 11 ms (most UI paths now hit a projection or cache).
DLQ rate: <0.1% in 30 days; replays are scripted and boring.
Deploys: producers and consumers ship independently; on-call load is down because failures isolate to one rule/target.

Minimal Go publisher

func Publish(ctx context.Context, bus, region string, ev Event) error {
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion(region))
    if err != nil { return err }
    cli := eventbridge.NewFromConfig(cfg)

    detailType := fmt.Sprintf("%s:%s", ev.Type, ev.Version)
    payload, _ := json.Marshal(ev)

    _, err = cli.PutEvents(ctx, &eventbridge.PutEventsInput{
        Entries: []types.PutEventsRequestEntry{{
            EventBusName: &bus,
            Source:       aws.String(ev.Source),
            DetailType:   aws.String(detailType),
            Time:         aws.Time(ev.OccurredAt),
            Detail:       aws.String(string(payload)),
        }},
    })
    return err
}

EventBridge isn’t magic, but it let us decouple without inventing infrastructure. The patterns above-clean envelopes, type:version, idempotency, per-target DLQs, and a few boring alarms-were enough to make the system predictable.

If there’s interest, I’ll follow up with the exact fan-out rules we use and how we handle backfill/replay safely. If this was useful, follow my profile and the SolarGenix.ai page to catch the next deep dives and benchmarks as they land.

Hybrid Cloud Stack: Balancing Aurora PostgreSQL and DynamoDB for Optimal Performance

Voskan Voskanyan — Wed, 22 Oct 2025 03:45:37 +0000

At SolarGenix.ai, we are building an AI-driven platform that turns the slow, manual parts of solar proposals into a fast, reliable, and automated flow, from roof detection and shading analysis to financial modeling and polished customer-ready PDFs. We are a startup, and development is moving fast.

This article walks through how we split workloads between Amazon Aurora PostgreSQL and Amazon DynamoDB, what consistency/latency trade-offs we accept, and how a unified data-access layer in Go plus caching lets us keep developer ergonomics high without sacrificing performance or reliability.

Why we run both Aurora PostgreSQL and DynamoDB

Aurora PostgreSQL gives us strong consistency, relational integrity, and powerful SQL for reporting/joins-ideal for workflows that must be correct first and fast second (e.g., billing artifacts, subscription state, proposal audit trails, RBAC metadata). DynamoDB gives us predictable low-latency, elastic throughput, and effortless horizontal scale-ideal for high-velocity, key-based access with simple access patterns (e.g., proposal snapshots, step autosaves, layout/planning intermediates, idempotency records).

On the product side, this split reflects how our engine works: the AI pipeline produces intermediate states and final artifacts that are naturally document-like and accessed by key, while financials, user/org relationships, and compliance data benefit from transactions, joins, and SQL ergonomics. The outcome is a platform that feels instant without compromising accuracy in the places where accuracy is non-negotiable.

Decision guide - which workloads go where?

Aurora PostgreSQL (strongly consistent, relational, transactional)
DynamoDB (low latency, high throughput, partition-friendly)

A simple rule we use internally: If the access pattern is "get by key, occasionally update, serve fast," it probably belongs in DynamoDB. If we need cross-entity constraints, transactions, or ad-hoc queries, it’s an Aurora problem.

Consistency, latency, and the "fast path"

User actions write to Aurora in a transactional way when correctness matters (e.g., plan upgrades), but we project a derived, read-optimized view into DynamoDB (or cache) for the UI fast path.
Background processors hydrate DynamoDB with the most-needed fields for the next interaction, turning expensive joins into a single-digit-millisecond key lookup.
For rare, cross-cutting queries, we go straight to Aurora and treat the added latency as acceptable (and cache results aggressively).

This gives us strongly consistent writes where we need them and very low-latency reads where we want them, without confusing app developers: they talk to a single repository interface, and the implementation decides when/how to use each store.

Unified data-access layer in Go

We present storage behind a clean interface and keep policy (when to read/write which store) inside the repository, not the handlers. That means product teams ship features without learning two databases’ footguns.

Interface:

// pkg/proposals/repo.go
package proposals

import "context"

type ID string

type Proposal struct {
    ID          ID
    AccountID   string
    Version     int64
    State       string            // "draft", "ready" ...
    Snapshots   map[string]string // lightweight links to artifacts
    UpdatedAt   int64
}

type Repository interface {
    Get(ctx context.Context, id ID) (*Proposal, error)
    Save(ctx context.Context, p *Proposal) error
    Snapshot(ctx context.Context, id ID, label string, ref string) error
}

Implementation sketch (Aurora + DynamoDB + Cache)

// pkg/proposals/repo_hybrid.go
type hybridRepo struct {
    aurora  AuroraStore   // wraps pgx with ctx-aware tracing/retries
    ddb     DynamoStore   // wraps DynamoDB SDK with marshaling helpers
    cache   Cache         // Redis or in-memory with TTL + stampede control
    clock   Clock
    metrics Metrics
}

func (r *hybridRepo) Get(ctx context.Context, id ID) (*Proposal, error) {
    // Try cache
    if p, ok := r.cache.Get(string(id)); ok {
        return p, nil
    }

    // Hot path: DynamoDB by key
    if p, err := r.ddb.GetByID(ctx, string(id)); err == nil && p != nil {
        r.cache.Set(string(id), p, ttlFast())
        return p, nil
    }

    // Fallback: Aurora (authoritative), then project to DDB
    p, err := r.aurora.GetProposal(ctx, string(id))
    if err != nil {
        return nil, err
    }
    _ = r.ddb.Put(ctx, p)
    r.cache.Set(string(id), p, ttlSlow())
    return p, nil
}

func (r *hybridRepo) Save(ctx context.Context, p *Proposal) error {
    // Authoritative write to Aurora (transactional)
    if err := r.aurora.UpsertProposal(ctx, p); err != nil {
        return err
    }
    // Async or inline projection to DynamoDB for the fast path
    _ = r.ddb.Put(ctx, p)
    r.cache.Delete(string(p.ID))
    return nil
}

func (r *hybridRepo) Snapshot(ctx context.Context, id ID, label, ref string) error {
    // Snapshots are key-addressable, perfect for DynamoDB
    if err := r.ddb.AddSnapshot(ctx, string(id), label, ref, r.clock.Now()); err != nil {
        return err
    }
    r.cache.Delete(string(id))
    return nil
}

Aurora writes are the source of truth for mutable, relational entities.
DynamoDB holds read-optimized projections and append-only events (snapshots, idempotency, counters).
Cache shields both and absorbs spikes; cache TTLs reflect staleness tolerance per endpoint.

Read/write patterns that actually matter in production

Autosave drafts: write to DynamoDB (cheap, frequent); periodically consolidate into Aurora.
Publishing a proposal: transactional write in Aurora; project final state to DynamoDB; bust cache.
Fetching the latest proposal for UI: cache → DynamoDB by key → fallback to Aurora (and re-project).
Audit/exports: run directly on Aurora with SQL; results cached by hash of the query params.
Idempotent APIs: store request hashes in DynamoDB with short TTL; reject duplicates fast.
Rate limiting and quotas: DynamoDB counters (or Redis) with atomic increments and per-key TTL.

Caching strategy (simple rules)

Per-entity caches keyed by ID with short TTLs (seconds to a minute).
Per-query caches keyed by normalized params with longer TTLs only for read-only analytics views.
Stampede protection (singleflight) around cold misses; negative caching for known-absent keys.
Explicit cache busting on any state transition that affects the fast path.

Observability and operations

Emit storage labels on every call: store=aurora|ddb|cache, op=get|put|tx, plus latencies and error classes.
Keep service-level SLOs: p95 read latency, error rates, projection lag.
Run regular consistency checks that diff a sample of Aurora rows against their DynamoDB projections; alert on shape drift.
Backfills and schema evolution run behind feature flags; repositories expose read-only mode if we need to pause writes during critical migrations.

Using Aurora PostgreSQL and DynamoDB together isn’t about hedging bets-it’s about putting each workload where it performs best, then hiding that complexity behind a clean Go API and a disciplined caching layer. That’s how we keep the product feeling instantaneous while preserving the correctness guarantees we need for money, compliance, and trust.

5 Security Patterns Go Developers Should Avoid

Voskan Voskanyan — Sun, 03 Aug 2025 07:29:24 +0000

I’ve been writing and reviewing Go code for a while now - from building backend services to contributing to open-source projects. One thing I’ve learned the hard way is that insecure code doesn’t always look "wrong" - sometimes it looks completely ordinary.

In this post, I want to share five common patterns I keep seeing in real-world Go codebases that can easily turn into serious security issues. These aren’t exotic bugs — they’re patterns that show up quietly, subtly, and often go unnoticed in PRs, even by experienced teams.

1. Passing Raw Input Directly into Queries

Let’s start with the classic:
query := fmt.Sprintf("SELECT * FROM users WHERE email = '%s'", email)

Even if email is sanitized upstream, or “should never come from user input”, you’ve just opened the door for SQL injection. I’ve seen this exact line in production code. It’s easy to write, easy to miss, and dangerous.

What to do instead:
Use parameterized queries. Always.
With database/sql, it’s:
query := "SELECT * FROM users WHERE email = ?" row := db.QueryRowContext(ctx, query, email)

Also: be careful even with ORM “raw” queries — a lot of people think gorm.Raw() handles escaping. It doesn’t.

2. Constructing Shell Commands with Input Strings

Another one I keep seeing is dynamic command execution:
cmd := exec.Command("sh", "-c", "run-task "+userInput)

Here’s the problem: if userInput contains a semicolon, pipe, or any shell metacharacters — you’ve got command injection.

Better:
Avoid sh -c unless you really need it. Use exec.Command with explicit args:
cmd := exec.Command("run-task", userInput)

And validate or sanitize the input before it gets near the shell.

3. Overexposing `http.Request` Data to Internal Layers

This one’s sneakier.

I’ve seen APIs where r.URL.Query() or r.FormValue() are passed directly to service layers, logging systems, or even database access. The assumption is that somewhere down the line, someone will validate it. But that rarely happens.

The result?
Untrusted input ends up buried inside your logic, touching sensitive code.

Fix:
Validate early. Don’t pass raw http.Request data into your internal layers. Parse and validate in the handler/controller layer — not deeper.

4. Leaky Global State and Insecure Defaults

This one is more architectural, but still causes security problems.

Global configuration, shared variables, or init() functions that silently override behavior often lead to unexpected exposure. I've seen cases where a global debug = true flag was left on in production — leaking sensitive logs.

Tip:
Avoid global mutable state unless you really need it. Use dependency injection, context, and controlled configuration structs.

5. Ignoring Error Values — Especially from Critical Calls

We all know Go makes you write if err != nil. And yet, I’ve seen plenty of cases where it's quietly ignored:
token, _ := jwt.Parse(...)

That underscore hides a lot. If parsing fails and you continue anyway, that’s an auth bypass waiting to happen.

Always check your errors, especially in:

Auth-related code
Encryption/decryption
File system and network operations
Anything parsing untrusted data

It’s better to fail early than to keep going with a broken assumption.

None of these patterns are exotic. That’s what makes them dangerous — they blend in. They’re easy to write, easy to review past, and they compile just fine.

Over time, I realized I was repeating the same feedback in code reviews. That led me to build CodexSentinel, a static analyzer for Go that focuses on real security patterns like these — not just lint rules or formatting.

If you’re working in Go and care about security, give it a try or drop me a message.
And if you’ve seen other common anti-patterns — I’d love to hear them.

Thanks for reading. Stay safe

GitHub: https://github.com/Voskan/codexsentinel

Forem: Voskan Voskanyan

The 15-Minute Goroutine Leak Triage: Two Dumps, One Diff, Zero Guessing

What a goroutine leak is (the only definition that matters in production)

Minute 0–3: confirm the signature (don’t skip this)

Minute 3–10: capture two goroutine profiles and diff them

Option A (best for diffing): capture profile format and use go tool pprof

Option B (fastest human scan): debug=2 text dumps

Minute 10-15: map the dominant stack to the first fix you should try

1) Many goroutines blocked on chan send / chan receive

2) Many goroutines stuck in net/http.(*Transport).RoundTrip / netpoll waits

3) Many goroutines waiting on WaitGroup.Wait, semaphores, or errgroup

4) Many goroutines in timers/tickers / periodic loops

Where tracing fits (and why it’s worth it even if pprof "already shows the stack")

The patch that actually holds: ship "hardening defaults," not one-off fixes

Prove it’s fixed (don’t accept vibes)

Wrap-up

10 Advanced Prompting Techniques for Getting Better Results from ChatGPT

Why Advanced Prompting Matters

Advanced Prompts for 2026

Troubleshooting: When Your Prompts Still Fail

Breaking the Monolith: How We Split a Node.js Backend into Go Microservices on AWS ECS-Without Stopping the World

Why we changed

The plan (gradual, not big-bang)

Contracts and compatibility

OpenAPI at the edge

Event envelope

Dual-publish for majors

Idempotency, retries, DLQ

Data projections & reads

Release strategy on ECS

The small Go publisher

Observability that catches real problems

Results and one scar-tissue story

What we’d do again

What we’d change next time

Notes on Clean Architecture (brief, practical)

Event-Driven Architecture in Production: Designing with AWS EventBridge and Go

Why we moved off "sync everything"

Bus and naming (obfuscated but real)

Event envelope and versioning

Idempotency: assume duplicates

Delivery guarantees, retries, and DLQs

DynamoDB Streams for projections

Observability: alarms we actually watch

Migration notes (what actually happened)

Results after cutover

Minimal Go publisher

Hybrid Cloud Stack: Balancing Aurora PostgreSQL and DynamoDB for Optimal Performance

Why we run both Aurora PostgreSQL and DynamoDB

Decision guide - which workloads go where?

Consistency, latency, and the "fast path"

Unified data-access layer in Go

Read/write patterns that actually matter in production

Caching strategy (simple rules)

Observability and operations

5 Security Patterns Go Developers Should Avoid

1. Passing Raw Input Directly into Queries

2. Constructing Shell Commands with Input Strings

3. Overexposing http.Request Data to Internal Layers

4. Leaky Global State and Insecure Defaults

5. Ignoring Error Values — Especially from Critical Calls

Option A (best for diffing): capture profile format and use `go tool pprof`

Option B (fastest human scan): `debug=2` text dumps

1) Many goroutines blocked on `chan send` / `chan receive`

2) Many goroutines stuck in `net/http.(*Transport).RoundTrip` / netpoll waits

3) Many goroutines waiting on `WaitGroup.Wait`, semaphores, or `errgroup`

3. Overexposing `http.Request` Data to Internal Layers