Forem: u11d

Asset-Based Data Orchestration: Lessons from Building a Multi-State Social Data Platform

uninterrupted — Thu, 26 Mar 2026 07:21:25 +0000

Building reliable data platforms rarely fails because of scale alone.
More often, reliability collapses under heterogeneity: multiple data providers, inconsistent schemas, partial updates, and unclear ownership.

While building a multi-state social data platform ingesting resource data from dozens of organizations, we discovered that reliability is not a property of pipelines. It is a property of data artifacts and their relationships.

1. Why Reliability Becomes a Systems Problem at Scale

The accompanying essay frames trust as something earned through consistent system behavior under real-world pressure. What that framing leaves implicit, but what became unavoidable in practice, is that reliability stops being a property of individual components very early on. Once multiple organizations, jurisdictions, and publishing surfaces are involved, reliability becomes an emergent property of the entire system.

For us, this meant that no single pipeline, connector, or database could be made "reliable enough" in isolation. Failures were rarely total. They were partial, localized, and often silent. The engineering challenge was not preventing all failure, but designing the system so that failures were isolated, detectable, and explainable.

That requirement drove nearly every architectural decision that followed.

2. System Overview

Technically, the platform ingests social service resource data from independent organizations operating across multiple U.S. states. Each organization exposes data via a different source system (for example iCarol, WellSky, VisionLink, RTM), with varying schemas, update cadences, and quality guarantees.

At a high level, the system consists of:

Writers - per-tenant ingestion and transformation projects that:
- Fetch raw data from source systems via connector adapters
- Persist raw data into Snowflake source schemas
- Normalize and standardize data via DBT
- Apply enhancements such as geocoding or translations
Readers - publisher processes that:
- React to completed writer runs, either bulk or incremental
- Publish curated artifacts to OpenSearch, MongoDB, and optionally Postgres

Snowflake acts as the system of record for intermediate and normalized datasets. Dagster coordinates the execution and materialization of data assets. DBT is used explicitly for set-based transformations, not orchestration.

Scale is not extreme in raw volume, but complexity is high: dozens of tenants, hundreds of tables per tenant, and frequent partial updates.

High-Level Architecture

The following diagram illustrates the writer → Snowflake → DBT → asset orchestration → readers pattern.

The platform is designed around asset lineage and normalization contracts, not pipelines.

3. Source Heterogeneity as the Dominant Constraint

The hardest constraint was not throughput or storage. It was heterogeneity.

Each data provider differed along several axes simultaneously:

Schema shape: even when nominally "the same" entities existed, fields varied
Semantics: identical fields often meant subtly different things
Update cadence: some sources updated continuously, others weekly or ad hoc
Quality guarantees: missing fields, stale records, or partial exports were common

Early on, we underestimated how strongly these differences would dominate system design. Attempting to treat ingestion as a uniform, pipeline-shaped process led to brittle assumptions and cross-tenant coupling.

The system only became manageable once heterogeneity was treated as fundamental, not incidental.

4. Normalization (HSDS, SDOH or Equivalent) as an Architectural Contract

Normalization into an HSDS-like model was not implemented as a downstream convenience. It became an architectural contract.

All downstream consumers, internal and external, implicitly rely on the guarantees of the normalized model: stable fields, predictable relationships, and documented semantics. That meant normalization could not be "best effort" or delayed until the end of a pipeline.

In practice:

Raw source data is written verbatim into Snowflake source schemas
DBT ELT projects transform this raw data into standardized intermediate models
DBT STAGE projects apply tenant-specific adaptations while preserving the contract

This separation made it explicit where interpretation happens. If a field is wrong in the normalized model, the question becomes which contract was violated, not "what broke in the pipeline".

5. What We Got Wrong Initially

Several early assumptions did not survive contact with reality.

Pipeline-first thinking

We initially modeled work as long-running jobs. This obscured which intermediate datasets were durable, reusable, or safe to depend on. Debugging often meant rerunning more than necessary.

Manual validation

Data quality checks lived outside the orchestration layer. Engineers and analysts manually inspected outputs, which worked at small scale but failed under concurrency and time pressure.

Shared failure domains

Multiple tenants often shared execution paths. A failure in one tenant’s ingestion could block or delay others, even when their data was unrelated.

None of these issues were catastrophic individually. Together, they made reliability depend on human attention.

6. Transitioning to Asset-Based Data Orchestration with Dagster

The shift to asset-based orchestration was driven less by tooling preference and more by a change in mental model.

Instead of asking "what jobs should run?", we started asking:

What data artifacts must exist?
What do they depend on?
How fresh do they need to be?
What constitutes success or failure for this artifact?

Dagster assets provided a way to encode those questions directly.

A simplified example from a writer project shows how DBT models are treated as assets rather than opaque steps:

# writer-xyz/assets.py (excerpt)
from dagster_dbt import load_assets_from_dbt_project

dbt_elt_assets = load_assets_from_dbt_project(
    project_dir="dbt_elt",
    profiles_dir="dbt_elt",
)

This does not explain how DBT runs. It declares that the resulting tables are first-class assets with lineage and state.

Once assets replaced jobs as the primary abstraction, freshness, lineage, and partial recomputation became explicit rather than implicit.

7. Partitioning for Failure Isolation

Partitioning was critical for isolating failures.

We partitioned primarily along tenant and state boundaries, not time. This reflected operational reality: data issues almost always affected a single organization or region.

In Dagster terms, this meant:

Separate writer projects per tenant
Independent schedules and sensors
Asset materializations scoped to a tenant’s data domain

A failure in one writer no longer blocked publishing for others. More importantly, remediation could be targeted and auditable.

8. Data Quality Embedded in the Asset Graph

Data validation moved into the asset graph itself.

Instead of post-hoc checks, validations became explicit dependencies. If a validation asset failed, downstream assets simply did not materialize.

An example pattern used across writers:

@asset
def validate_staging_tables(staging_tables):
    assert staging_tables.count_missing_ids() == 0

This is intentionally simple. The key point is not the check itself, but that failure is structural. The system records that an expected artifact does not exist, rather than silently publishing bad data.

This shifted failure detection earlier and reduced the blast radius of errors.

9. Operational Outcomes

Day-to-day operations changed in several concrete ways:

On-call work shifted from rerunning pipelines to inspecting asset lineage
Partial backfills became routine rather than exceptional
Publishing delays were easier to attribute to specific upstream causes
New tenants could be added without increasing shared operational risk
None of this eliminated operational effort. It made that effort more focused and less reactive.

10. Open Trade-offs and Unresolved Questions

Some challenges remain unresolved:

Cross-tenant schema evolution still requires coordination and discipline
Observability across Snowflake, DBT, Dagster, and downstream stores is fragmented
Cost attribution at the asset level is still coarse-grained
Human review remains necessary for certain semantic validations

With more time, we would invest earlier in unified observability and more formal schema versioning.

11. Why These Lessons Matter Beyond This Platform

These lessons are not unique to this system.

Any platform operating in civic tech, govtech, or environmental data shares similar constraints: multiple data producers, uneven quality, and real-world consequences for failure.

The core takeaway is not "use asset-based orchestration", but treat data artifacts as obligations. Once that shift happens, many architectural decisions become clearer.

Reliability stops being something you hope for and becomes something you can reason about.

Final Thoughts

The biggest lesson from this platform was not about any particular tool. It was about how we model the system itself.

Once data artifacts became the core abstraction, many reliability problems became easier to reason about. Failures became visible, dependencies became explicit, and operational work shifted from firefighting pipelines to managing data contracts.

For data platforms operating across heterogeneous sources, this shift can be the difference between a system that merely runs - and one that can be trusted.

FAQ

What is asset-based data orchestration?

Asset-based data orchestration treats data artifacts (tables, datasets, models) as the primary units of orchestration instead of pipelines or jobs. Systems like Dagster allow teams to define dependencies between assets, enabling better lineage tracking, partial recomputation, and failure isolation.

Why is asset-based orchestration useful for complex data platforms?

In large systems with many data producers and consumers, failures are rarely binary. Asset-based orchestration makes dependencies explicit, allowing teams to:

isolate failures
recompute only affected datasets
track lineage across transformations
enforce data quality checks before publishing

How does Dagster differ from traditional pipeline orchestrators?

Traditional orchestrators schedule jobs or workflows. Dagster emphasizes data assets and lineage. This enables better visibility into:

what datasets exist
what produced them
what depends on them
whether they meet freshness or quality requirements

When should teams adopt asset-based orchestration?

Asset-based orchestration becomes particularly valuable when systems have:

many data sources
heterogeneous schemas
multiple downstream consumers
partial or incremental updates
strict reliability requirements

These conditions are common in multi-tenant data platforms and civic tech systems.

Does asset-based orchestration replace tools like DBT?

No. Asset-based orchestration complements transformation tools like DBT. In many architectures:

DBT performs set-based transformations
Dagster manages asset lineage, dependencies, scheduling, and validation

This separation keeps orchestration and transformation responsibilities clear.

Orchestrating Trust: Building Reliable Data Systems for Social Impact

Paweł Sławacki — Tue, 24 Mar 2026 08:54:40 +0000

How production-grade orchestration enables impact at scale

Systems built for social impact are often judged by their intent. The assumption is that good outcomes naturally follow from good goals, and that technical sophistication is secondary to mission. In practice, the opposite is often true. When information systems fail, the cost is not measured in lost revenue or delayed insights, but in missed opportunities for help, support, or timely intervention.

At scale, social impact is not a matter of aspiration. It is a matter of reliability.

Platforms that serve vulnerable populations operate under constraints that are both technical and ethical. Information must be accurate, current, and accessible. It must adapt as the world changes. It must withstand uneven demand and tolerate partial failure without collapsing. Most importantly, it must continue operating without constant human supervision, because manual intervention does not scale to moments of urgency.

These requirements are familiar to anyone who has built systems for finance, logistics, or large-scale commerce. What differs is the margin for error. In social contexts, latency is not an inconvenience. Inconsistency is not an annoyance. Failure is not an abstract metric. The system either delivers trustworthy information when it is needed, or it does not.

This is where data architecture quietly becomes social infrastructure.

The hidden fragility of well-intentioned systems

Many social platforms begin with pragmatic solutions. Data is collected from disparate sources, normalized through custom pipelines, and exposed through simple interfaces. Early success reinforces the approach: the system works, organizations adopt it, and impact grows.

Over time, however, complexity accumulates. Data sources evolve independently. Update cycles diverge. Quality varies across contributors. What once felt manageable starts to strain under its own assumptions.

In building social data platforms, we learned that fragility rarely appears all at once. It emerges gradually. Pipelines grow longer. Reprocessing becomes broader than necessary. Validation shifts from design to manual oversight. Eventually, the system still functions but confidence in its outputs begins to erode.

When correctness depends on human vigilance, availability depends on institutional memory. When updates become opaque, trust shifts away from architecture toward individual heroics. For systems intended to support people under real-world pressure, this is an unsustainable state.

The problem is not a lack of data or compute. It is a lack of structural guarantees.

From pipelines to obligations

Traditional data pipelines are designed around execution. They define a sequence of tasks that transform inputs into outputs. This model assumes that intermediate states are transient and that value resides primarily at the end of the flow.

In social data systems, this assumption does not hold.

Normalized datasets, enriched resources, derived aggregates-these are not disposable by-products. They are durable artefacts with meaning beyond a single run. They are reused, audited, compared over time, and relied upon by downstream organizations making decisions under uncertainty.

Once data outputs are treated as obligations rather than by-products, the role of orchestration changes fundamentally. The system’s responsibility is no longer to run jobs, but to ensure that specific states of data exist, remain current, and remain explainable.

This distinction matters because obligations persist. They require guarantees: freshness, lineage, and reproducibility. They require the system to know what it has produced, what it depends on, and what must change when assumptions shift.

In this framing, orchestration stops being an operational convenience and becomes a form of governance.

Reliability under real-world constraints

These principles became tangible while building Connect211, a modern search platform designed to support 211 organizations operating across multiple U.S. states. The platform aggregates resource data from independent organizations, each maintaining its own systems, taxonomies, and update rhythms.

What we learned early is that reliability in such an environment cannot be retrofitted. Data sources change independently. Failures are localized. Demand is uneven and often event-driven. Manual coordination quickly becomes the bottleneck.

Meeting these constraints required treating data artefacts as first-class citizens. Each normalized dataset, each enrichment step, each derived index represents a commitment: this information must exist, must be correct, and must be traceable back to its origins.

Asset-oriented orchestration provided a natural way to express these commitments. Instead of reasoning about execution order, the system reasons about data state. Instead of pushing data through pipelines, it ensures that required artefacts are materialized and kept current as upstream conditions change.

Dagster’s asset-based model aligned closely with this way of thinking. It allowed us to encode not only how data is processed, but what must be true for the system to be considered healthy. Orchestration became a mechanism for maintaining trust rather than merely coordinating tasks.

Automation without opacity

Automation is often presented as a universal solution to scale. In social systems, automation without structure can be as dangerous as manual fragility. When updates propagate automatically but their causes remain hidden, errors scale just as efficiently as value.

What distinguishes resilient systems is not the absence of automation, but the presence of clarity. Asset-based orchestration preserves the narrative of the data. Every artefact carries its provenance. Every update has a reason. When stakeholders ask why information changed, the answer is embedded in the structure of the system itself.

In environments where information influences real-world outcomes, this explainability underpins legitimacy. Trust is not established through assurances, but through the ability to demonstrate correctness when it matters.

Automation, in this sense, is not about removing humans from the loop. It is about ensuring that when humans intervene, they do so with understanding rather than guesswork.

Social impact as an emergent property

It is tempting to frame social impact in terms of outcomes alone. Did the platform help more people? Did it improve access? Did it reduce friction?

These questions are essential, but they are downstream. They describe effects, not causes.

At scale, social impact emerges from systems that behave predictably under stress. From platforms that continue operating as inputs change unexpectedly. From architectures that degrade gracefully rather than fail catastrophically. From data systems that are transparent by design rather than opaque by accident.

The same principles that govern production-grade platforms in commercial domains apply here, but with heightened stakes. Reliability is not an optimization. It is the foundation upon which impact rests.

In this light, orchestration is not a technical detail. It is part of the social contract embedded in the system-defining how obligations are met, how failures are contained, and how trust is maintained over time.

Beyond mission-driven engineering

There is a persistent tendency to treat social platforms as exceptional-worthy of different standards because their goals are noble. In practice, this often leads to underinvestment in architecture, justified by urgency or limited resources.

Our experience suggests the opposite conclusion. When tolerance for error is low and consequences are real, architectural rigor becomes more important, not less. Production-grade data systems are not at odds with social missions. They are prerequisites for sustaining them.

Asset-based orchestration, exemplified by tools like Dagster, provides a framework for expressing this rigor. It shifts focus from execution to responsibility, from pipelines to promises. It allows systems to scale not only in size, but in trustworthiness.

Social impact does not arise from technology alone. But without reliable systems, even the strongest intentions struggle to translate into lasting effect. When data platforms are designed as social infrastructure, reliability ceases to be a purely technical concern and becomes a public good.

FAQ: Production-Grade Orchestration for Social Impact Systems

What is asset-based orchestration in data engineering?

Asset-based orchestration is an architectural approach where data artefacts (datasets, models, indexes, aggregates) are treated as first-class citizens rather than by-products of pipeline runs.

Instead of defining execution steps, you define data states that must exist and their dependencies. The orchestration system ensures:

Correct dependency resolution
Incremental recomputation
Freshness guarantees
Lineage tracking
Failure isolation

This shifts orchestration from task coordination to state governance.

How is asset-based orchestration different from traditional pipelines?

Traditional pipelines are execution-oriented:

Step A → Step B → Step C
Outputs are transient
Reprocessing is often coarse-grained

Asset-based systems are state-oriented:

Explicit dependency graphs
Selective re-materialization
Persistent artefacts with lineage
Declarative data contracts

The key difference is that pipelines answer:

“What runs next?”

Asset-based orchestration answers:

“What must be true about the data?”

For systems operating under strict reliability constraints, that distinction is critical.

Why does data lineage matter in social impact systems?

In social systems, incorrect data can influence:

Access to services
Emergency response decisions
Resource allocation
Regulatory reporting

Lineage provides:

Auditability
Explainability
Reproducibility
Impact traceability

When stakeholders ask, “Why did this information change?”, lineage allows engineering teams to answer with certainty, not speculation.

What architectural risks do social data platforms typically face?

Common failure modes include:

Silent schema drift from independent data providers
Broad, expensive reprocessing triggered by minor upstream changes
Manual validation becoming a hidden operational dependency
Lack of observability into partial failures
Tight coupling between ingestion and serving layers

Without structural guarantees, reliability degrades gradually — often without obvious alerts.

How does orchestration contribute to data governance?

Orchestration becomes governance when it encodes:

Explicit data ownership boundaries
Dependency contracts
Freshness expectations
Failure domains
Version-aware updates

Rather than governance being a policy document, it becomes embedded in the system’s execution model.

This reduces reliance on institutional memory and tribal knowledge.

Is asset-based orchestration only relevant at large scale?

No. It becomes more visible at scale, but its benefits appear earlier:

Faster iteration cycles
Safer refactoring
More predictable deployments
Lower operational overhead
Clearer system reasoning

For mission-critical domains, reliability requirements often emerge before traffic scale does.

How does this relate to data observability and reliability engineering?

Asset-based orchestration complements:

Data observability (freshness, volume, schema monitoring)
Data reliability engineering practices
SLA/SLO enforcement
Incident response workflows

Because dependencies are explicit, blast radius and impact analysis become tractable. Observability signals can be tied directly to defined data obligations.

What role does automation play in maintaining trust?

Automation enables scale, but trust requires:

Transparency
Traceability
Deterministic recomputation
Controlled failure propagation

Well-structured orchestration ensures that automation is explainable, not opaque. Errors do not silently cascade across the system.

When should a team consider moving from pipelines to asset-oriented architecture?

Signals include:

Increasing reprocessing cost
Growing dependency complexity
Difficulty explaining data changes
Manual intervention becoming routine
Rising stakeholder sensitivity to correctness

If correctness is non-negotiable, state-aware orchestration becomes a strategic investment rather than an optimization.

About the authors / context

This article is based on our direct experience building production-grade data infrastructure for https://connect211.com, a modern search platform supporting 211 organizations across multiple U.S. states. The insights presented reflect real architectural decisions made while scaling a social impact system operating under strict reliability and data-quality constraints.

Next.js 16 Partial Prerendering (PPR): The Best of Static and Dynamic Rendering

uninterrupted — Thu, 05 Mar 2026 08:21:46 +0000

Next.js has long been a leader in giving developers flexible, high-performance rendering strategies - Static Site Generation (SSG), Server-Side Rendering (SSR), Incremental Static Regeneration (ISR), and Client-Side Rendering (CSR) all play roles depending on your use case. With Next.js 14, Partial Prerendering (PPR) was introduced as an experimental feature, starting from Next.js 16, PPR has become a stable and becomes a foundational part of this landscape by letting you blend static pre-rendering and dynamic behavior in the same route, improving perceived performance and SEO without sacrificing real-time data needs.

What Is Partial Prerendering (PPR)?

Partial Prerendering is a rendering strategy that lets Next.js deliver a static HTML “shell” immediately, then stream in dynamic content as it becomes available - all in the same server response. Instead of choosing static or dynamic at the page level, PPR combines both on a per-component basis.

The static shell (layout, logo, product titles, navigation) is pre-generated at build time or cached ahead of requests.
Dynamic parts (cart state, recommendations, session data) load later and are streamed to the client using React’s <Suspense> boundaries.
Users see meaningful UI immediately, with dynamic content filling in seamlessly.

This leads to faster Time to First Byte (TTFB) and a smoother perceived experience, while still allowing data personalization and real-time updates.

How Partial Prerendering Works in Next.js 16

Cache Components: The Heart of PPR

In Next.js 16, PPR isn’t just an experimental flag - it’s integrated into the Cache Components system. Cache Components lets you opt-in to cache certain components or data instead of rendering them on every request, making pre-rendering more predictable and performant.

cacheComponents: true in next.config.ts enables PPR and component-level caching.
The new "use cache" directive lets you mark data-fetching functions or components as cacheable.
Anything not cacheable or depending on request-specific data can be wrapped in a <Suspense> fallback.

This approach gives you fine-grained control: static UI is reused across requests, while dynamic parts rehydrate or stream in only when needed.

Streaming and Suspense: The Engine Under the Hood

React’s <Suspense> plays a key role in PPR:

Wrap dynamic components in <Suspense> with a fallback UI (like a skeleton loader).
During rendering, Suspense tells Next.js where to halt pre-rendering and stream dynamic content later.
The server sends the pre-rendered shell immediately and then streams dynamic sections in parallel as they’re ready.

This strategy avoids blocks in page delivery and reduces the “white screen” effect often seen in traditional SSR or CSR approaches.

Why PPR Matters for E-commerce

For e-commerce platforms, performance directly impacts sales and SEO - factors also emphasized in the U11D articles about rendering strategies. A slow page can hurt conversions and search rankings. Partial Prerendering improves this by:

Faster initial loads: Users see the page skeleton instantly, improving engagement and performance metrics.
SEO advantages: Static content is available for crawler indexing immediately.
Dynamic personalization: Cart contents, recommendations, user-specific prices or availability can appear without blocking the initial render.
More flexible than SSG/SSR alone: You’re not limited to fully static or completely dynamic pages - the best of both lives together.

In e-commerce apps where product detail pages might be static for most users but show personalized recommendations or inventory status, PPR is a compelling hybrid.

How to Enable and Use PPR in Next.js 16

Enable Cache Components:

In your next.config.ts:

import type { NextConfig } from 'next';

const nextConfig: NextConfig = {
  cacheComponents: true, // enables Partial Prerendering
};

export default nextConfig;

Wrap dynamic UI:

Use React’s Suspense for dynamic content:

import { Suspense } from 'react';

export default function ProductPage() {
  return (
    <>
      <Header />
      <Suspense fallback={<div>Loading product...</div>}>
        <ProductDetails />
      </Suspense>
    </>
  );
}

Use use cache for predictable dynamic data:

Inside dynamic data functions, prefix with "use cache" to mark them cacheable:
```
'use cache';
const product = await getProduct(id);
```
This tells Next.js that caching is safe for that data.

How PPR Fits with Other Rendering Strategies

Partial Prerendering doesn’t replace SSR, SSG, ISR, or CSR - but coordinates with them. While U11D’s articles give a great overview of traditional strategies like SSG, SSR, and ISR, PPR adds a hybrid strategy that sits between:

SSG/ISR: Good for full static pages or pages that regenerate occasionally.
SSR: Ideal for real-time personalized content.
CSR: Great for highly interactive client-heavy UI.
PPR (Next.js 16): Combines static cached shells with streaming dynamic content.

Think of PPR as a component-level SSG + dynamic hybrid - fast to load like SSG, flexible like SSR.

Wrapping Up

Next.js 16 elevates Partial Prerendering from an experimental concept to a practical, integrated performance strategy through Cache Components and React Suspense. It’s especially powerful for complex, dynamic sites like e-commerce stores, where you want the benefits of static pre-rendering and dynamic personalization without sacrificing UX or SEO.

By serving instantly usable HTML and streaming dynamic parts in parallel, PPR bridges the traditional divide between static and dynamic rendering - helping your app feel faster while staying robust and scalable.

If you’re building with Next.js 16, definitely explore PPR alongside other strategies like SSR, ISR, and CSR to find the most performance-optimized combination for your routes.

Secure RDS Access Without Bastion Hosts: Using ECS Containers and SSM

Bartek Gałęzowski — Wed, 25 Feb 2026 08:00:00 +0000

The Problem

Accessing RDS databases in production environments presents a common security challenge. Direct internet access to databases is a significant security risk, so most organizations place their RDS instances in private subnets without public endpoints. But how do you access these databases for debugging, data analysis, or administrative tasks?

Traditional approaches include:

Bastion hosts: Requires maintaining and securing additional EC2 instances
VPN connections: Complex setup and ongoing maintenance overhead
SSH tunneling: Still requires a jump server with SSH access

There's a more elegant solution: leveraging your existing ECS containers as secure tunnels to your RDS databases using AWS Systems Manager (SSM) Session Manager.

The Solution

This script creates a secure tunnel from your local machine to an RDS database through a running ECS container, using SSM Session Manager for the connection. No SSH keys, no bastion hosts, no exposed ports—just IAM-based authentication and encrypted connections.

Prerequisites

Before using this script, ensure you have:

AWS CLI installed and configured with appropriate credentials
Session Manager Plugin for AWS CLI
ECS Task Role with permissions to use SSM:

 {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "ssmmessages:CreateControlChannel",
           "ssmmessages:CreateDataChannel",
           "ssmmessages:OpenControlChannel",
           "ssmmessages:OpenDataChannel"
         ],
         "Resource": "*"
       }
     ]

ECS Exec enabled on your service (can be enabled with aws ecs update-service --cluster <cluster> --service <service> --enable-execute-command --force-new-deployment)

How It Works

The script performs the following operations:

1. Task Discovery

TASK_ID=$(aws ecs list-tasks \
  --cluster "$CLUSTER" \
  --service-name "$SERVICE_NAME" \
  --desired-status RUNNING \
  --query 'taskArns[0]')

Finds the first running task in the specified ECS service.

2. Container Runtime ID Retrieval

CONTAINER_RUNTIME_ID=$(aws ecs describe-tasks \
  --cluster "$CLUSTER" \
  --tasks "$TASK_ID" \
  --query 'tasks[0].containers[?name==`'"$SERVICE_NAME"'`].runtimeId | [0]')

3. Port Forwarding Session

aws ssm start-session \
  --target "ecs:${CLUSTER}_${TASK_ID}_${CONTAINER_RUNTIME_ID}" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["$DB_HOST"],"portNumber":["5432"], "localPortNumber":["$LOCAL_PORT"]}'

Establishes a secure tunnel through the ECS container to the database.

Usage

#!/usr/bin/env bash

set -eu

usage() {
  echo "Usage: $0 <CLUSTER> <SERVICE_NAME> <DB_HOST> <LOCAL_PORT> <REGION>"
  echo "  cluster: ECS cluster name"
  echo "  service: ECS service name"
  echo "  db_host: Database host"
  echo "  local_port: Local port to forward"
  echo "  region:  AWS region (e.g., us-east-2)"
  exit 1
}

if [ $# -ne 5 ]; then
  echo "Error: Expected 5 arguments, got $#" >&2
  usage
fi

CLUSTER="$1"
SERVICE_NAME="$2"
DB_HOST="$3"
LOCAL_PORT="$4"
REGION="$5"

get_first_task_id() {
  local FIRST_TASK_ID
  FIRST_TASK_ID=$(aws ecs list-tasks \
    --cluster "$CLUSTER" \
    --service-name "$SERVICE_NAME" \
    --desired-status RUNNING \
    --output text \
    --query 'taskArns[0]' \
    --region "$REGION" | sed 's/.*\///')

  if [ "$FIRST_TASK_ID" = "None" ] || [ -z "$FIRST_TASK_ID" ]; then
    echo "Error: No running tasks found for service '$SERVICE_NAME' in cluster '$CLUSTER'" >&2
    return 1
  fi

  echo "$FIRST_TASK_ID"
}

TASK_ID=$(get_first_task_id)

CONTAINER_RUNTIME_ID=$(aws ecs describe-tasks \
  --output text \
  --cluster "$CLUSTER" \
  --tasks "$TASK_ID" \
  --region "$REGION" \
  --query 'tasks[0].containers[?name==`'"$SERVICE_NAME"'`].runtimeId | [0]')

TARGET="ecs:${CLUSTER}_${TASK_ID}_${CONTAINER_RUNTIME_ID}"

aws ssm start-session \
  --target "$TARGET" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters "{\"host\":[\"$DB_HOST\"],\"portNumber\":[\"5432\"], \"localPortNumber\":[\"$LOCAL_PORT\"]}" \
  --region "$REGION"

Save the script as ecs-db-tunnel.sh and make it executable:

chmod +x ecs-db-tunnel.sh

Run the script with the required parameters:

./ecs-db-tunnel.sh <CLUSTER> <SERVICE_NAME> <DB_HOST> <LOCAL_PORT> <REGION>

Example:

./ecs-db-tunnel.sh \
  production-cluster \
  api-service \
  mydb.c9akciq32.us-east-2.rds.amazonaws.com \
  5432 \
  us-east-2

Once connected, you can access the database from your local machine:

psql -h localhost -p 5432 -U dbuser -d mydb

or using other tools such as DBeaver.

Key Benefits

1. No Infrastructure Overhead

No need to maintain bastion hosts or VPN servers. You leverage existing ECS containers.

2. Enhanced Security

No SSH keys to manage or rotate
No exposed ports or public endpoints
IAM-based authentication and authorization
All traffic encrypted through SSM
Audit trail through CloudTrail

3. Zero Configuration

If your ECS service is already running with ECS Exec enabled, you're ready to go.

4. Cost Effective

No additional EC2 instances to run. Session Manager has no additional cost.

5. Temporary Access

Connection exists only while the script is running—perfect for adhoc administrative tasks.

Security Considerations

Network Security

This approach works because:

The ECS container is in the same VPC as the RDS instance
The container's security group allows outbound connections to the RDS security group
The RDS security group permits inbound connections from the ECS container's security group

Access Control

Control who can establish tunnels using IAM policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ecs:ListTasks", "ecs:DescribeTasks", "ssm:StartSession"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "us-east-2"
        }
      }
    }
  ]
}

Troubleshooting

No running tasks found

Ensure the ECS service has at least one running task:

aws ecs list-tasks --cluster <cluster> --service-name <service> --desired-status RUNNING

Session Manager plugin not found

Install the Session Manager plugin following AWS documentation.

Connection refused

Verify:

RDS security group allows inbound from ECS container security group
Database endpoint is correct
ECS task has network connectivity to RDS

ECS Exec not enabled

Enable it on your service:

aws ecs update-service \
  --cluster <cluster> \
  --service <service> \
  --enable-execute-command \
  --force-new-deployment

Conclusion

Using ECS containers as secure tunnels to RDS databases is an elegant solution that leverages existing infrastructure while maintaining strong security posture. This approach eliminates the need for bastion hosts, reduces attack surface, and provides auditable, temporary access to private databases.

The script demonstrates that sometimes the best security solutions are those that work with your existing architecture rather than adding more complexity on top of it.

How to Deploy Next.js 16 SSG to AWS Amplify: Complete Guide

Michał Miler — Tue, 24 Feb 2026 07:02:03 +0000

Deploying Next.js Static Site Generation (SSG) applications to AWS Amplify can be challenging without the right configuration. While Vercel offers a seamless experience, AWS Amplify may provide a cost-effective alternative with up to 96% savings for smaller sites and 20-30% for medium-traffic applications. In this comprehensive guide, we'll walk through the exact steps to successfully deploy your Next.js 16 SSG site to AWS Amplify.

Why AWS Amplify for SSG?

AWS Amplify is particularly attractive for SSG deployments because:

Lower costs than Vercel, especially for commercial use (see our detailed cost comparison)
No per-seat pricing for team collaboration
Generous free tier for the first 12 months
Standard AWS infrastructure (S3 + CloudFront) with no vendor lock-in

Note on static export limitations: Static Site Generation pre-generates all pages at build time. This means no server-side rendering (SSR), no Incremental Static Regeneration (ISR), and no API routes. If you need these features, you'll need a different deployment strategy.

Prerequisites

Before we begin, ensure you have:

Node.js 22.x or later installed
Next.js 16 project using App Router
An AWS account (free tier eligible for 12 months)
Git repository (GitHub, GitLab, Bitbucket, or AWS CodeCommit)
Basic familiarity with Next.js App Router

Step 1: Configure Next.js 16 for SSG

Update `next.config.ts`

Create or modify your next.config.ts file to enable static export:

import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  output: "export",
  images: {
    unoptimized: true,
  },
};

export default nextConfig;

Key Points:

output: 'export' disables the Next.js runtime and enforces a full static export. This ensures your site can be deployed as plain static HTML/CSS/JS.
images.unoptimized: true disables Next.js Image Optimization API (required for static export)

Handle Image Optimization

Since Next.js Image Optimization doesn't work with static exports, choose one of these options:

Option 1: Unoptimized images (simplest for testing)

import Image from "next/image";

<Image src="/images/photo.jpg" width={500} height={300} alt="Description" />;

Option 2: Standard HTML img tags

<img src="/images/photo.jpg" width={500} height={300} alt="Description" />

Option 3: Third-party CDN (recommended for production)

Use services like Cloudflare Images, Cloudinary, or imgix. See our guide on optimizing S3 images with Cloudflare Images.

const imageLoader = ({
  src,
  width,
  quality,
}: {
  src: string;
  width: number;
  quality?: number;
}) => {
  return `https://your-cdn.com/${src}?w=${width}&q=${quality || 75}`;
};

<Image
  loader={imageLoader}
  src="photo.jpg"
  width={500}
  height={300}
  alt="Description"
/>;

Verify SSG Compatibility

Ensure your project doesn't use these incompatible features:

Cannot use:

Route Handlers (API Routes) - require server runtime
Server-Side Rendering (SSR) - e.g., cache: 'no-store' in fetches
Incremental Static Regeneration (ISR) - revalidate property
Dynamic data without generateStaticParams — including any data fetched from “dynamic APIs” that depend on runtime values (see Next.js guide on dynamic rendering)

Must implement:

All dynamic routes need generateStaticParams to pre-generate pages
Use cache: 'force-cache' for data fetching at build time

Example for dynamic routes:

// app/blog/[slug]/page.tsx
export async function generateStaticParams() {
  return [{ slug: "first-post" }, { slug: "second-post" }];
}

export default async function BlogPost({
  params,
}: {
  params: Promise<{ slug: string }>;
}) {
  const { slug } = await params;
  return <h1>Blog Post: {slug}</h1>;
}

Test Locally

Verify your static export works:

npm run build
npx serve out

Visit http://localhost:3000 to ensure everything renders correctly.

Step 2: Deploy to AWS Amplify

Push to Git

git add .
git commit -m "Configure Next.js for SSG deployment"
git push origin main

Access AWS Amplify Console

Log into AWS Console
Search for "Amplify" in the services search bar
Click "Get Started" under "Amplify Hosting"

Connect Your Repository

Select your Git provider (GitHub, GitLab, Bitbucket, or AWS CodeCommit)
Authorize AWS Amplify to access your repositories
Choose the repository and branch to deploy
Click "Next"

Configure Build Settings

Amplify should auto-detect Next.js, but verify your amplify.yml:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: out
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*

Critical settings:

baseDirectory: out - Next.js static export directory
npm ci - Consistent, reproducible builds
Cache node_modules for faster builds

Add Environment Variables (Optional)

If your app needs environment variables, add them in "Advanced settings":

NEXT_PUBLIC_API_URL=https://api.example.com
NEXT_PUBLIC_ANALYTICS_ID=G-XXXXXXXXXX

Remember: Only NEXT_PUBLIC_* variables are accessible in the browser. Server-side variables are available during build time only.

Deploy

Review settings
Click "Save and deploy"
Monitor the build progress (typically 3-5 minutes)

Once complete, you'll get a URL like: https://[branch].[app-id].amplifyapp.com

Step 3: Configure Redirects & Routing

Fix 404 Pages

Configure redirects in Amplify console under "Rewrites and redirects":

Rule 1: Custom 404 page

Source: </^[^.]+$|\\.(?!(css|gif|ico|jpg|js|png|txt|svg|woff|woff2|ttf|map|json|webp)$)([^.]+$)/>
Target: /404.html
Status: 404

Rule 2: Client-side routing fallback

Source: /<*>
Target: /index.html
Status: 200 (Rewrite)

Important: Place Rule 2 last - Amplify processes rules in order.

Step 4: Handle Common Considerations

Environment Variables

Client Components (with "use client"):

Can only access NEXT_PUBLIC_* variables
Variables are embedded in the build output

Server Components (default):

Can access all environment variables at build time
Never expose secrets in NEXT_PUBLIC_* variables

Forms and Interactive Features

SSG sites need external services for dynamic functionality:

Forms: Formspree, Web3Forms, AWS Lambda
Search: FlexSearch, Fuse.js, Algolia, Meilisearch
Comments: Giscus, Utterances, Disqus
Authentication: Clerk, Auth0, AWS Cognito

Next.js 16 App Router Features

Works perfectly with SSG:

Metadata API
Route Groups
Loading and Error UI
Server Components (data fetched at build time)

Requires pre-generation:

Dynamic routes (use generateStaticParams)
Parallel and Intercepting Routes (all combinations must be pre-generated)

Troubleshooting Common Issues

Build Fails: "Output directory not found"

Fix: Verify amplify.yml has baseDirectory: out and next.config.ts has output: 'export'

Images Not Loading

Fix: Ensure images.unoptimized: true in config. Images should be in public/ directory and referenced without the public prefix.

Blank Page After Deployment

Fix: Check browser console for errors. Usually means:

Missing NEXT_PUBLIC_ prefix on environment variables
Client-side JavaScript error
Incorrect build output

Dynamic Routes Return 404

Fix:

Implement generateStaticParams for all dynamic routes
Configure redirect rules in Amplify console

Environment Variables Not Working

Fix:

Client-side variables must use NEXT_PUBLIC_ prefix
Set in Amplify Console under "Environment variables"
Rebuild after adding new variables

When NOT to Use This Approach

Consider alternatives if you need:

Incremental Static Regeneration (ISR): Requires Node.js server
Server-Side Rendering (SSR): Real-time, per-request rendering
API Routes: Need serverless functions
On-demand Revalidation: Dynamic content updates without rebuilds
Edge Middleware: Complex routing logic at the edge

For these use cases, consider Vercel, AWS Amplify with SSR configuration, or AWS App Runner/ECS.

Next Steps

After deployment:

Add custom domain: In Amplify console under "Domain management"
Set up monitoring: Use CloudWatch for build and deployment alerts
Configure preview deployments: For pull requests and feature branches
Optimize costs: Review bandwidth usage and optimize images
Test thoroughly: Verify all pages and features work as expected

Conclusion

AWS Amplify provides a cost-effective platform for deploying Next.js 16 SSG applications with proper configuration. The key is understanding SSG limitations, configuring Next.js correctly, and setting up Amplify's build and routing properly.

While the setup requires more configuration than Vercel, the cost savings (up to 96% for small sites) and lack of per-seat pricing make it an excellent choice for businesses looking to optimize their hosting costs without sacrificing performance or reliability.

Partial Catchment Delineation for Inundation Modeling

Aleksy Bohdziul — Thu, 19 Feb 2026 07:34:53 +0000

A System-Oriented Approach to Scalable Hydrological Feature Engineering

Traditional watershed delineation wasn't designed for machine learning at scale. The standard approach treats each watershed as a complete, self-contained unit, which makes sense when you're studying individual rivers. But it creates real problems when you need to train models across hundreds of locations.

We ran into this on a recent flood prediction project. The study area had almost no hydrological data, no stream gauges, no riverbed surveys, nothing you'd typically use for hydraulic modeling. So we trained an LSTM model to predict discharge instead. It worked, but there was a catch: the model could only predict flow at watershed outlets. One outlet per watershed meant we didn't have enough training points.

The obvious solution was to create more outlets by subdividing watersheds along the river. But traditional catchment boundaries overlap heavily when you do this, which breaks parallel processing and makes selective updates nearly impossible. We needed many independent spatial units for the ML model, but we also needed to preserve the downstream flow aggregation that hydrological modeling depends on.

Our solution was to delineate watersheds at the reach level instead. Each reach gets its own partial catchment, smaller units that remain hydrologically valid but can be computed and updated independently. It's a compromise between what the data pipeline needs and what the hydrology requires.

Getting the geometry right was only part of the problem. We tested multiple DEM sources, modified the river network repeatedly, and needed to recompute catchments constantly during development. Treating delineation as a one-time preprocessing step wasn't viable. We needed to version every intermediate result, recompute selectively when inputs changed, and compare outputs across iterations.

This pushed us toward Dagster's asset model. Instead of treating catchments as temporary pipeline outputs, we manage them as persistent spatial assets with explicit dependencies and lineage tracking.

The following sections cover the hydrological rationale, the technical implementation, and how asset orchestration made this approach practical for production use.

Catchment Delineation in the Context of Inundation Modeling

Inundation modeling relies on an accurate representation of how water accumulates and propagates through a river network. Traditionally, this begins with watershed delineation derived from a digital elevation model, followed by hydraulic simulation over the resulting domain. When applied to large regions, however, this workflow introduces practical limitations. Entire catchments must be processed as single units, even when only a small portion of the river network is relevant for a given prediction or model update.

From a data engineering perspective, this creates an undesirable coupling between upstream and downstream regions. A change in DEM preprocessing, stream burning strategy, or river vector alignment forces recomputation of large spatial extents, even when the change is localized. This coupling becomes a bottleneck when experimenting with multiple configurations or when operating a system that must adapt continuously to new data.

The core insight behind partial catchment delineation is that hydraulic dependency flows downstream, but computational dependency does not need to. By separating catchments into smaller, reach-aligned units, it becomes possible to preserve hydrological correctness while dramatically improving computational flexibility.

The Core Idea: Reach-Level and Progressive Catchments

We segmented each river into short reaches (100 m) and delineated a catchment for each reach. From those building blocks we constructed larger, progressively downstream catchments.

The method introduced here distinguishes between two complementary spatial constructs: reach-level catchments and progressive catchments.

Reach Catchments (non-overlapping)

Reach-level catchments are defined for individual river segments, bounded upstream by the nearest confluence and downstream by the segment's outlet. These units do not overlap and collectively partition the drainage area of the river network. Their non-overlapping nature makes them well suited for parallel processing, independent feature extraction, and localized recomputation.

Visualize the landscape divided into narrow, adjacent drainage areas:

Each reach catchment drains only to its own 100 m river segment
None of them include upstream contributions
Their boundaries tile the basin without overlaps

Progressive Catchments (overlapping by design)

Progressive catchments, by contrast, represent the cumulative upstream area contributing to a given river reach. Each progressive catchment is constructed by aggregating all upstream reach-level catchments along the river network. This structure mirrors traditional hydrological reasoning and provides a direct bridge to downstream hydraulic modeling.

Now start combining those catchments as you move downstream:

Progressive Catchment 1 = Reach 1
Progressive Catchment 2 = Reach 1 + Reach 2
Progressive Catchment 3 = Reach 1 + Reach 2 + Reach 3

Visually:

the first progressive catchment is small and upstream
each subsequent one contains the previous
downstream catchments fully envelop upstream ones

This is why we call them progressive: each one represents the basin area contributing flow up to that point along the river.

By maintaining both representations explicitly, the system can operate at two levels simultaneously. Reach-level catchments support scalable computation and machine learning workflows, while progressive catchments preserve the physical context required for inundation modeling.

Why not delineate progressive catchments directly?

We could have. It would actually be simpler than whatever we're actually doing. But:

We also needed the reach catchments for inundation simulation later
Running delineation logic twice felt like a smell

So we delineate once, and compose later.

Tributaries

Where a tributary joins:

its reach catchments are merged into the progressive catchment only after the confluence
upstream progressive catchments on the main stem remain unaffected

In other words:

reach catchments are spatial building blocks
progressive catchments are cumulative assemblies of those blocks

What the Two Catchment Types Are Used For

Progressive catchments → model training

Our LSTM predicts discharge at outlet points. For each progressive catchment, we derive features such as precipitation, temperature, humidity, pressure, the aridity index, and others.

All inputs are provided as raster datasets. Catchment geometries are used as spatial masks to extract and aggregate pixel values.

This workflow requires repeated spatial joins, raster masking, and temporal aggregation over large geospatial datasets. We implement and orchestrate these pipelines using Dagster, which allows us to manage dependencies, partition computations, and scale processing across large spatial-temporal datasets.

Reach catchments → inundation mapping

Each reach gets its own discharge estimate (derived from differences between progressive catchments), which later feeds the inundation simulation.

Once reach-level and progressive catchments are established, they become the foundation for feature extraction. Terrain attributes, land cover statistics, soil properties, and hydrological indices can be computed independently for each reach-level catchment. These features serve as inputs to machine learning models predicting discharge or inundation extent.

Progressive catchments then provide a natural mechanism for aggregating upstream contributions. Features derived at the reach level can be accumulated downstream in a controlled, traceable manner. This separation simplifies both training and inference: models operate on consistent, non-overlapping units, while hydraulic context is reintroduced through aggregation.

At this stage, the delineation method transitions from a GIS exercise into a data orchestration problem. Each derived feature depends on specific preprocessing choices, spatial units, and upstream dependencies. Managing these relationships manually quickly becomes infeasible.

DEM (Digital Elevation Model) Preprocessing

Implementing partial catchment delineation at high spatial resolution exposes a range of practical challenges. Reliable catchment delineation depends far more on DEM preprocessing than on the delineation algorithm itself.

High-resolution DEMs (1 m × 1 m in our case) amplify artifacts that are negligible at coarser scales, including spurious sinks, artificial barriers, and noise-induced flow paths. Stream burning and sink filling become necessary, but their parameters introduce additional degrees of freedom that affect downstream results.

Below we summarize the preprocessing steps that proved essential for stable and repeatable results.

Depression filling

Raw DEMs frequently contain spurious sinks caused by:

measurement noise
vegetation and built structures
interpolation artifacts

Left untreated, these sinks interrupt downstream connectivity and lead to fragmented or incomplete catchments. We therefore applied depression filling prior to any flow calculations.

Our goal was not to aggressively flatten terrain, but to ensure continuous drainage paths with minimal elevation modification. Priority-flood-style algorithms worked well in practice and preserved overall terrain structure.

Stream burning

Even after sink removal, we observed inconsistencies between modeled flow paths and known river locations. To address this, we burned the vector river network into the DEM by lowering elevations along river centerlines.

Aligning raster-based flow accumulation with vector river networks proved particularly sensitive. Small positional discrepancies between datasets can lead to misaligned pour points, fragmented catchments, or unrealistic drainage patterns. These issues are not purely geometric; they directly influence the stability and reproducibility of downstream features.

This step serves two purposes:

it enforces hydrologically plausible drainage paths
it reduces sensitivity to small elevation errors in flat or low-gradient terrain

Stream burning significantly improved watershed stability, especially near confluences and in wide floodplains where DEM gradients are weak.

Flow accumulation and its limitations

We initially experimented with flow accumulation to:

identify channelized flow paths
snap pour points automatically to areas of high contributing area

However, the high spatial resolution of the DEM (1 m × 1 m) introduced significant noise into flow accumulation outputs. Minor elevation perturbations resulted in fragmented or unrealistic accumulation patterns, making automated snapping unreliable.

As a result, we limited the use of flow accumulation and instead relied more heavily on burned-in river vectors and explicit reach endpoints for pour point placement.

During later experiments we found that using D-infinity for calculating flow paths rather than D8 significantly improved flow accumulation calculation, but due to it being discovered too late, we weren't able to implement it before the end of first phase of the project.

Spatial alignment issues

During development we discovered small but still significant horizontal offsets between the DEM and the river vector dataset, on the order of a few meters. At some point we discovered that our river geometries were offset by a couple of meters relative to the DEM.

These discrepancies led to:

pour points falling outside effective drainage paths
unstable catchment boundaries
inconsistent results across neighboring reaches
weird catchment boundaries
several hours of existential doubt

While stream burning mitigated some of these effects, resolving DEM-vector alignment remains an important area for future improvement. This is still on our list of things to fix properly. For now, burning rivers into DEM somewhat alleviated the issue.

Rather than attempting to eliminate these uncertainties entirely, we treated them as explicit dimensions of experimentation. Different preprocessing strategies were preserved as separate artifacts, allowing their effects to be compared systematically. This approach only becomes feasible when intermediate results are treated as first-class entities rather than overwritten pipeline outputs.

Overall, careful DEM preprocessing proved essential not only for hydrologic correctness, but also for producing geometries stable enough to support downstream machine-learning workflows.

Implementation Outline

Below is a cleaned-up, simplified sketch of the workflow. The real code is longer, louder, and contains more comments written at 2 a.m.

# 1. Load DEM and preprocess
filled_dem = fill_depressions(dem)
burned_dem = burn_streams(filled_dem, river_lines)
flow_dir   = d8_flow_direction(burned_dem)

# 2. Split rivers into fixed-length reaches
reaches = split_lines(river_lines, segment_length=100)

# 3. Create pour points at reach outlets
pour_points = reaches.geometry.apply(get_downstream_endpoint)

# 4. Delineate reach catchments
reach_catchments = delineate_watersheds(
    flow_dir=flow_dir,
    pour_points=pour_points
)

# 5. Build progressive catchments
progressive = []
current = None
for reach in ordered_downstream(reach_catchments):
    current = reach if current is None else union(current, reach)
    progressive.append(current)

The devil, as always, lives in:

tributary joins
reach ordering
and spatial indexing performance

Joining tributaries means:

identifying parent-child relationships between reaches
merging reach catchments in the correct downstream order
avoiding double-counting areas

Asset-Based Orchestration of Spatial Dependencies

To make this workflow operational, we modeled reach-level catchments, progressive catchments, and derived features as explicit assets within Dagster. Each asset represents a durable spatial artifact with well-defined dependencies on upstream inputs. Changes in DEM preprocessing, river network alignment, or feature definitions propagate through the asset graph in a controlled way.

This asset-oriented approach allows recomputation to be both selective and explainable. When a preprocessing parameter changes, only the affected reach-level catchments and their downstream aggregates are recomputed. Historical artifacts remain available for comparison, enabling systematic evaluation of alternative configurations.

Dagster's lineage tracking plays a critical role here. Each feature can be traced back through the chain of spatial transformations that produced it, providing transparency during debugging and model validation. Rather than reasoning about pipeline execution order, the system reasons about data state and dependency.

Operational Implications

Treating partial catchment delineation as an orchestrated asset graph changes the operational profile of inundation modeling workflows. Iteration becomes cheaper because recomputation is localized. Failures become easier to diagnose because dependencies are explicit. Experimentation becomes safer because previous states are preserved rather than overwritten.

Perhaps most importantly, this approach aligns hydrological reasoning with modern data platform design. Physical dependencies are respected, but they no longer dictate computational coupling. The system can evolve incrementally, accommodating new data sources, preprocessing strategies, and modeling approaches without requiring full recomputation of the spatial domain.

Lessons Learned

DEM preprocessing is very important
1 m DEMs are great until you compute derivatives
River vectors and DEMs rarely agree — believe neither blindly

Conclusion

Segmenting rivers into reach-level catchments gave us:

more training points
spatially consistent features
and a clean bridge between ML discharge prediction and inundation modeling

Partial catchment delineation proved valuable not because it produced a single optimal representation of a watershed, but because it enabled a shift in how spatial dependencies are managed at scale. By decomposing watersheds into reach-level units and reconstructing downstream context through progressive aggregation, we gained a representation that supports both hydrological correctness and computational scalability.

The effectiveness of this approach ultimately depended on its orchestration. Without an asset-oriented framework, the complexity introduced by multiple delineation strategies and iterative experimentation would quickly become unmanageable. By modeling spatial artifacts explicitly and preserving their lineage, we were able to integrate hydrology, machine learning, and geospatial preprocessing into a coherent, production-ready system.

If nothing else, this workflow taught us humility, patience, and how many ways water can refuse to flow downhill.

While this article focused on inundation modeling, the underlying pattern extends to any domain where high-resolution geospatial data meets iterative, data-driven workflows. Partial decomposition of space, combined with asset-based orchestration, offers a practical path toward scalable and trustworthy spatial modeling systems.

Architecting and Operating Geospatial Workflows with Dagster

Daniel Kraszewski — Tue, 17 Feb 2026 12:00:00 +0000

This article is a technical companion to our earlier essay on geospatial data orchestration (link: https://u11d.com/blog/geospatial-data-orchestration/).

Geospatial data pipelines present a distinct set of challenges that traditional ETL frameworks struggle to address. Raster datasets measured in gigabytes, coordinate reference system transformations, temporal partitioning across decades of observations, and the need for reproducible lineage across every derived artifact-these requirements demand an orchestration model that treats data as the primary actor rather than a byproduct of task execution. This article describes the architecture we developed for water management analytics: a system that ingests elevation models, meteorological observations, satellite imagery, and forecast data to support flood prediction and hydrological modeling.

The System Boundary

The platform serves as a data preparation layer, not a model serving infrastructure. Its responsibility begins at external data sources-WFS endpoints, FTP servers, governmental APIs-and ends at materialized artifacts in object storage ready for consumption by downstream ML training pipelines and QGIS-based analysts. We deliberately exclude real-time inference, user-facing APIs, and visualization from this system's scope.

Two Dagster projects comprise the workspace. The landing-zone project handles all data ingestion and transformation: elevation tiles, meteorological station observations, satellite imagery, climatic indices, and forecast GRIB files. The discharge-model project consumes these prepared datasets to train neural network models for water discharge prediction using NeuralHydrology. Both projects share a common library under shared/ containing IO managers, resource definitions, helper utilities, and cross-project asset references.

The platform does not manage model deployment, handle user authentication, or serve predictions. Those responsibilities belong to separate systems that consume our outputs via S3-compatible object storage.

Data Types and Access Patterns

Three fundamental data types flow through the system, each with distinct storage and access characteristics.

Raster data-digital elevation models, satellite imagery, land cover maps-dominates storage volume. We store all rasters as Cloud Optimized GeoTIFFs (COGs) in S3, enabling range-request access for partial reads. The elevation pipeline alone processes tiles from multiple national geodetic services, converting formats like ARC/INFO ASCII Grid and XYZ point clouds into standardized COGs with consistent coordinate reference systems. A single consolidated elevation model can exceed several gigabytes.

Tabular data encompasses meteorological observations, hydrological measurements, station metadata, and computed indices. We standardize on Parquet with zstd compression, leveraging Polars for in-process transformations and DuckDB for SQL-based quality checks. A custom IO manager handles serialization of both raw DataFrames and Pydantic model collections, automatically recording row counts and column schemas as Dagster metadata.

Vector data-catchment boundaries, station locations, regional polygons-exists primarily as intermediate artifacts used for spatial joins and raster clipping operations. Shapely geometries serialize alongside tabular records in Parquet files, with coordinate transformations handled through pyproj.

All data resides in S3-compatible object storage. The storage layout follows a predictable convention: s3://{bucket}/{asset-path}/{partition-keys}/filename.{ext}, where asset paths derive directly from Dagster asset keys. This enables both programmatic access through IO managers and ad-hoc exploration via S3 browsers.

Asset Graph Design

Every durable artifact in the system maps to a Dagster asset. This is not a philosophical preference but a practical requirement: when a hydrologist questions why a model prediction differs from last month's, we need to trace backward through the exact elevation tiles, meteorological observations, and climatic indices that produced those training features.

Asset naming follows a hierarchical convention reflecting data lineage. Elevation data progresses through elevation/dtm/wfs_index → elevation/dtm/raw_tiles → elevation/dtm/converted_tiles → elevation/dtm/consolidated. Each stage represents a distinct, independently materializable artifact with its own partitioning strategy. The shared/assets/landing_zone.py module maintains a centralized registry of asset keys, enabling type-safe cross-project references:

ELEVATION_DSM = AssetKey(["elevation", "dsm"])
ELEVATION_DTM = AssetKey(["elevation", "dtm"])
FORECAST_GRIB_RAW = AssetKey(["forecast", "grib", "raw"])
CLIMATIC_INDICES_CATCHMENT = AssetKey(["climatic_indices", "catchment"])

Dependencies between assets express both data flow and materialization order. The converted elevation tiles asset explicitly declares its dependency on raw tiles through the ins parameter, ensuring Dagster's asset graph correctly represents the relationship and prevents stale data from propagating downstream.

We employ Dagster's component pattern for assets that share structural similarities but operate on different data. The elevation pipeline defines Converted as a component that can be instantiated for both DTM and DSM processing, sharing conversion logic while maintaining separate asset keys and partition spaces.

Partitioning Strategy

Partitioning serves two purposes: it bounds the scope of individual materializations to manageable sizes, and it enables incremental updates without full recomputation. We use different partitioning strategies depending on the data's natural structure.

Elevation data partitions spatially by tile grid and region. A MultiPartitionsDefinition combines a tile index dimension with a regional dimension, allowing selective materialization of specific geographic areas. Dynamic partition definitions enable the tile catalog to grow without code changes-sensors read from index parquet files and issue AddDynamicPartitionsRequest calls to register new partitions.

Satellite imagery partitions temporally using year and month dimensions:

def get_multipartitions_def(self) -> dg.MultiPartitionsDefinition:
    return dg.MultiPartitionsDefinition({
        "index": self.indices_partition,
        "region": self.regions_partition,
    })

New temporal partitions register automatically through a sensor that monitors the catalog file for previously unseen year/month combinations.

Meteorological observations partition by data source and processing stage rather than by time. Then pipeline uses a sync-plan/execute-plan pattern where a planning asset determines which source files need synchronization, and an execution asset processes only the delta. This approach handles the irregular update patterns of governmental data sources more gracefully than fixed temporal partitions.

Raster Processing Pipeline

The raster processing subsystem converts heterogeneous input formats into standardized COGs suitable for ML feature extraction. A dedicated module provides the core transformation utilities, built on GDAL and Rasterio.

The main COG writing function orchestrates the complete transformation: coordinate reprojection, resolution resampling, nodata gap filling, geometry clipping, and overview generation. Memory management is critical—we process large rasters block-wise to avoid loading entire datasets into RAM:

def _fill_nodata_gaps(
    dataset: DatasetWriter,
    max_search_distance: float,
    smoothing_iterations: int,
) -> None:
    """Fill nodata gaps block-wise to avoid loading the whole raster into memory."""
    for _, window in dataset.block_windows(1):
        block = dataset.read(1, window=window, masked=True)
        if not np.ma.is_masked(block) or not block.mask.any():
            continue
        mask = np.asarray(~block.mask).astype(np.uint8)
        filled = fillnodata(
            block.filled(fill_value=dataset.nodata),
            mask=mask,
            max_search_distance=max_search_distance,
            smoothing_iterations=smoothing_iterations,
        )
        dataset.write(np.asarray(filled, dtype=block.dtype), 1, window=window)

Format-specific converters handle the idiosyncrasies of source data. The XYZ converter detects axis ordering issues in point cloud data, the ASCII grid converter parses ESRI's legacy format, and the VRT builder creates virtual rasters for multi-file operations. Each converter produces consistent metadata that downstream assets can rely on.

For smaller rasters below a configurable threshold, we process entirely in memory using Rasterio's MemoryFile. Larger rasters write to temporary files before final COG copy to S3 via GDAL's /vsis3/ virtual filesystem driver. This dual-path approach optimizes for both small-tile throughput and large-raster memory safety.

Data Quality as Dependencies

Data quality checks execute as first-class Dagster asset checks, not as afterthoughts in logging statements. The climatic indices pipeline defines sixteen distinct checks covering structural integrity, range validation, and business logic constraints:

@dg.multi_asset_check(
    specs=[
        dg.AssetCheckSpec("non_empty_data", asset=CLIMATIC_INDICES_CATCHMENT),
        dg.AssetCheckSpec("primary_key_uniqueness", asset=CLIMATIC_INDICES_CATCHMENT),
        dg.AssetCheckSpec("pet_avg_valid_range", asset=CLIMATIC_INDICES_CATCHMENT),
        dg.AssetCheckSpec("aridity_index_avg_valid_range", asset=CLIMATIC_INDICES_CATCHMENT),
        # ... additional checks
    ],
    name="catchment_climatic_indices_checks",
)
def _catchment_climatic_indices_checks(
    context: dg.AssetCheckExecutionContext, duckdb: DuckDBResourceExtended
) -> Iterable[dg.AssetCheckResult]:
    # Checks execute SQL against DuckDB, returning structured results

Checks validate domain-specific constraints: potential evapotranspiration must fall within 0-15 mm/day, aridity indices between 0-5, precipitation seasonality between -1 and +1. Each check returns structured metadata-not just pass/fail, but the actual values found, enabling rapid diagnosis when checks fail.

A helper function loads the asset's parquet file into a DuckDB temporary table, enabling SQL-based validation without loading the entire dataset into Python memory. This pattern scales to multi-million row datasets while keeping check execution times reasonable.

Geometry validation occurs inline during raster conversion. A dedicated validation function compares source and destination bounds, skipping tiles where reprojection would introduce unacceptable distortion. Rather than failing the entire partition, we record skipped tiles with explicit reasons, allowing manual review of edge cases.

Execution and Elasticity

The platform runs locally for development using uv run dg dev and deploys to Kubernetes for production workloads. Resource requirements vary dramatically across asset types-a metadata sync might need 256MB of memory, while elevation tile conversion demands 16GB.

We express resource requirements through operation tags that map to Kubernetes node pools:

class K8sOpTags:
    @staticmethod
    def xlarge() -> dict[str, Any]:
        return K8sOpTags._config(Nodes.M6I_2XLARGE, divider=2)

    @staticmethod
    def gpu() -> dict[str, Any]:
        return K8sOpTags._config(Nodes.G4DN_2XLARGE, divider=1)

The divider parameter enables fractional node allocation-running two medium workloads on a single large instance, or dedicating an entire GPU node to model training. Assets declare their requirements via op_tags=K8sOpTags.xlarge(), and the Kubernetes executor schedules pods accordingly.

Concurrent processing within assets uses a custom worker pool implementation that handles the messy realities of I/O-bound geospatial work: network timeouts, partial failures, and graceful cancellation. The worker pool provides retry policies, progress logging, and fail-fast behavior while aggregating per-item metrics:

result = worker_pool.process(
    items=tiles,
    worker=worker,
    logger=context.log,
    max_workers=self.max_workers,
    cancellation_event=cancellation_event,
    metrics_extractor=lambda res: {
        size_key: int(res.metadata.get(size_key, 0)),
    },
)
result.raise_if_failures()

The worker pool tracks success, failure, skip, and cancellation states for each item, building aggregate statistics that appear in Dagster's materialization metadata. When processing fails partway through, we know exactly which items succeeded and which need retry.

Observability and Lineage

Every asset materialization records structured metadata: row counts for tabular data, dimensions and CRS for rasters, processing duration, and custom metrics like total bytes written. The custom IO manager automatically captures table schemas and row counts; raster assets explicitly record resolution, bounds, and file sizes.

Sensors provide the observability layer for external data sources. The satellite data partition sensor polls catalog files every 30 seconds, logging new partition discoveries and registration requests. Forecast schedules run four times daily at fixed UTC times, with run keys that encode the scheduled timestamp for easy identification:

@dg.schedule(
    cron_schedule="30 8,13,16,20 * * *",
    target=dg.AssetSelection.assets(FORECAST_GRIB_RAW),
)
def forecast_grib_raw_schedule(context: dg.ScheduleEvaluationContext) -> dg.RunRequest:
    scheduled_time = context.scheduled_execution_time
    return dg.RunRequest(
        run_key=f"forecast_grib_raw_{scheduled_time.strftime('%Y%m%d_%H%M')}",
        tags={"schedule": "forecast_grib_raw"},
    )

The declarative automation condition enables declarative freshness policies-assets can specify how stale they're allowed to become, and Dagster automatically triggers materializations to maintain freshness. We use this for assets that need to stay current with upstream changes without manual intervention.

Asset lineage flows automatically from dependency declarations. When investigating a model prediction, we can trace from the model asset back through training data, through climatic indices, through individual station observations, to the original source files. This lineage persists across runs, enabling historical comparisons when methodology changes.

Lessons from Production: Architectural Revisions

Three architectural decisions required significant revision after initial deployment.

First, we underestimated the memory requirements for raster operations. Early implementations loaded entire tiles into memory, which worked fine for small test datasets but caused OOM kills on production-scale elevation models. The fix required systematic refactoring to block-wise processing throughout the raster pipeline-reading, transforming, and writing in chunks that fit within pod memory limits. This added complexity but eliminated an entire class of production incidents.

Second, we initially implemented dynamic partitions without proper cleanup logic. Sensors would happily add new partitions as data arrived, but nothing removed partitions for data that had been superseded or corrected. Over time, the partition space accumulated stale entries that confused operators and wasted storage. We added explicit partition lifecycle management: sensors now compare desired partitions against current state and issue both AddDynamicPartitionsRequest and DeleteDynamicPartitionsRequest as needed.

Third, we placed too much validation logic inside asset compute functions rather than as separate asset checks. This made debugging failures difficult-a validation error would fail the entire materialization, losing the partial work already completed. Extracting validation into dedicated asset checks with structured metadata output dramatically improved debuggability and allowed us to materialize "known-bad" data when necessary for investigation, running checks separately.

Open Questions and Next Steps

Several architectural questions remain unresolved in the current implementation.

The platform lacks a unified approach to schema evolution. When upstream data sources change their formats-which governmental APIs do without warning-we currently handle it through ad-hoc converter updates. A more systematic approach might involve schema registries or versioned asset definitions, but the right pattern for geospatial data with complex nested structures remains unclear.

Cross-project asset dependencies work through shared asset key definitions, but the materialization coordination relies on manual scheduling or external triggers. A more elegant solution might use Dagster's cross-code-location asset dependencies, but this would require restructuring how projects deploy and discover each other's assets.

The Memory of Water: Why LSTMs Demand Polished Data

Paweł Sławacki — Wed, 04 Feb 2026 13:02:30 +0000

In the era of "Big Data," there is a pervasive myth in environmental science that quantity is a proxy for quality. We assume that if we have terabytes of telemetry logs from thousands of sensors, the sheer volume of information will overpower the noise. We assume that modern Deep Learning architectures—specifically Long Short-Term Memory (LSTM) networks—are smart enough to figure it out.

They are not.

In hydrology, raw data is not fuel; it is crude oil. It is full of impurities, gaps, and artifacts that, if fed directly into a neural network, will clog the engine. When building systems to predict flash floods or manage reservoir levels, the sophistication of your model architecture matters far less than the continuity and physical integrity of your input data.

We don't just need to "clean" data. We need to polish it.

The Illusion of Abundance

A modern hydrological sensor network is a chaotic environment. Pressure transducers drift as sediment builds up. Telemetry radios fail during the very storms we need to measure. Batteries die in the cold.

When you look at a raw dataset, you see a time series. But an LSTM sees a narrative. If that narrative is riddled with holes, spikes, and flatlines, the model cannot learn the underlying physics of the catchment.

We often see feeding raw sensor logs into training pipelines, hoping the neural network will learn to ignore the errors. This is a fundamental misunderstanding of how LSTMs work. A standard regression model might average out the noise. An LSTM, however, tries to learn the sequence of events. If we feed it noise, it doesn't just make a bad prediction for that timestep; it learns a false causal relationship that corrupts its understanding of future events.

The High Cost of Discontinuity

To understand why data polishing is critical, you have to understand the "Memory" in Long Short-Term Memory.

Unlike a standard feed-forward network that looks at a snapshot of data, an LSTM maintains an internal "cell state"—a vector that carries context forward through time. In hydrology, this cell state represents the physical state of the catchment: How saturated is the soil? How high is the groundwater? Is the river already swollen from yesterday's rain?

Data continuity is the lifeline of this cell state.

When a sensor goes offline for three hours, we don't just lose three hours of data. We sever the model's connection to the past. If we simply drop those rows and stitch the time series back together, we are teleporting the catchment three hours into the future instantly. The LSTM sees a sudden, inexplicable jump in state that violates the laws of physics.

It tries to learn a pattern to explain this jump. But there is no pattern—only a broken sensor. The result is a model that "hallucinates," predicting sudden floods or droughts based on data artifacts rather than meteorological forcing.

Building the Pipeline with Dagster

To solve this, we cannot rely on ad-hoc cleaning scripts scattered across Jupyter notebooks. We need a rigorous, reproducible engineering standard. This is where we leverage Dagster to orchestrate the transformation from chaos to clarity.

In our architecture, we treat data stages as distinct software-defined assets.

First, we define a raw_sensor_ingestion asset. Dagster pulls this directly from our telemetry APIs or S3 buckets. This asset is immutable; it represents the "ground truth" of what the sensors actually reported, warts and all. We never modify this layer, ensuring we always have a pristine audit trail.

Next, we define a downstream polished_timeseries asset. This is where the engineering happens. Dagster manages the dependency, ensuring that the polishing logic only runs when new raw data is available. Inside this asset, we execute our cleaning algorithms—removing outliers, handling gaps, and normalizing timestamps.

By using Dagster, we gain full lineage. If a model starts behaving strangely, we don't have to guess which cleaning script was run. We can look at the asset graph and see exactly which version of the code produced the training data, ensuring that our "polish" is as version-controlled as our model architecture.

Enforcing the Laws of Physics on Data

The logic inside that polished_timeseries asset is designed to enforce the laws of physics. A neural network starts as a blank slate; it doesn't know that water cannot flow uphill or that a river cannot dry up in seconds.

We must teach it these boundaries through rigorous checks:

Physical Bounds: A river stage cannot be negative. Soil moisture cannot exceed porosity. Precipitation cannot physically reach 500mm in 10 minutes. These aren't just outliers; they are impossibilities.
Temporal Consistency: Water has mass and momentum; it accelerates and decelerates according to gravity and friction. A reading that jumps from 1m to 5m and back to 1m in a single 15-minute interval is almost certainly a sensor glitch, not a flash flood.

If we leave these "ghost signals" in the training set, the LSTM wastes its capacity trying to model impossible physics. By removing them, we allow the model to focus its gradient descent on learning the actual behavior of water.

Filling the Void Without Lying to the Model

Once we identify the gaps and the ghosts, we face the hardest choice in data engineering: Imputation. How do we fill the silence without lying to the model?

This is where domain expertise becomes code.

Linear Interpolation might work for temperature, which changes gradually.
Forward Filling might work for a reservoir level that changes slowly.
Masking is often the most honest approach for precipitation. If we don't know if it rained, we shouldn't guess. We should explicitly tell the model, "I don't know," often by using a separate boolean channel in the input tensor indicating data validity.

The danger of aggressive polishing is creating a "perfect" dataset that doesn't exist in reality. If we smooth out every peak and fill every gap with a perfect average, we train a model that is terrified of extremes. It will under-predict floods because it has never seen the raw, jagged reality of a storm.

Respecting the Journey of the Data

In the rush to adopt the latest Transformer architectures or state-of-the-art LSTMs, it is easy to view data processing as a janitorial task—something to be automated away so we can get to the "real work" of modeling.

But in environmental science, the data is the real work.

The performance ceiling of any hydrological forecast is not determined by the number of layers in your neural network, but by the fidelity of the story your data tells. A simple model trained on polished, physically consistent data will outperform a complex model trained on raw noise every time.

We are not just training models to predict numbers. We are training them to understand the memory of water. And that memory must be clear.

Decoupling Ingress with TargetGroupBinding in EKS

Paweł Swiridow — Wed, 28 Jan 2026 11:00:00 +0000

As we scale our EKS clusters, relying solely on Kubernetes Ingress objects to provision AWS Application Load Balancers (ALBs) can become restrictive. Sometimes we need to attach an EKS Service to a pre-existing ALB managed by Terraform, or we need complex routing rules that are easier to manage in HCL (Terraform) than in K8s annotations.

The AWS Load Balancer Controller supports a custom resource called TargetGroupBinding (TGB). This allows us to provision the Load Balancer and Target Group in Terraform (Infrastructure layer) and simply "bind" our Kubernetes Service to it at runtime (Application layer).

This guide walks through how to set up an AWS Target Group in Terraform and register a Prometheus instance to it using Helm values.

Prerequisites

Before proceeding, ensure your environment meets the following requirements. This architecture relies on specific AWS components to route traffic directly to Pods:

Amazon EKS Cluster: A running EKS cluster is required.
AWS VPC CNI Plugin: We will be using target_type = "ip". This mode requires the AWS VPC CNI (the default networking plugin for EKS), which assigns native AWS VPC IP addresses to Pods. This allows the ALB to route traffic directly to the Pod IP, bypassing the worker node's kube-proxy.
AWS Load Balancer Controller: You must have the AWS Load Balancer Controller (v2.0+) installed and running in your cluster. This controller is responsible for installing the TargetGroupBinding CRD and actively managing the registration of targets.
- Note: Ensure the controller has the necessary IAM permissions to elasticloadbalancing:RegisterTargets and elasticloadbalancing:DeregisterTargets.

Part 1: Infrastructure (Terraform)

First, we need to create the Target Group. The critical setting here is target_type = "ip".

When using the AWS Load Balancer Controller with the AWS VPC CNI, we want the ALB to send traffic directly to the Pod IP addresses, bypassing NodePorts.

`main.tf`

resource "aws_lb_target_group" "prometheus_tg" {
  name        = "eks-prometheus-tg"
  port        = 9090
  protocol    = "HTTP"
  vpc_id      = module.vpc.vpc_id

  # CRITICAL: Must be 'ip' for direct Pod routing via AWS LB Controller
  target_type = "ip"

  health_check {
    path                = "/-/healthy"
    protocol            = "HTTP"
    matcher             = "200"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# We need to export this ARN to pass it to our Helm chart later
output "prometheus_tg_arn" {
  value = aws_lb_target_group.prometheus_tg.arn
}

Note: Do not use aws_lb_target_group_attachment in Terraform. The AWS Load Balancer Controller running inside the cluster will manage the targets dynamically as Pods come and go.

Part 2: Security (IAM Least Privilege)

By default, the generic AWS Load Balancer Controller policy is permissive (often using Resource: *). In a production environment - especially one with multiple teams sharing an AWS account - we should adhere to Least Privilege.

We must restrict the Controller's ability so it can only register/deregister targets for this specific Target Group, preventing it from accidentally modifying other Load Balancers.

Add this policy to the IAM Role used by your Load Balancer Controller ServiceAccount:

`iam.tf`

data "aws_iam_policy_document" "lb_controller_tgb_policy" {
  statement {
    sid       = "AllowRegisterTargets"
    effect    = "Allow"
    actions   = [
      "elasticloadbalancing:RegisterTargets",
      "elasticloadbalancing:DeregisterTargets"
    ]
    # Scope permissions strictly to the specific Target Group ARN created above
    resources = [aws_lb_target_group.prometheus_tg.arn]
  }

  statement {
    sid       = "AllowDescribeHealth"
    effect    = "Allow"
    actions   = [
      "elasticloadbalancing:DescribeTargetHealth"
    ]
    resources = [aws_lb_target_group.prometheus_tg.arn]
  }
}

resource "aws_iam_policy" "tgb_strict_policy" {
  name        = "eks-alb-controller-tgb-restricted"
  description = "Restricted access for TargetGroupBinding to specific TGs only"
  policy      = data.aws_iam_policy_document.lb_controller_tgb_policy.json
}

Part 3: The Glue (TargetGroupBinding CRD)

The TargetGroupBinding CRD tells the controller: "Watch this Kubernetes Service, and whenever its endpoints change, update this specific AWS Target Group."

A raw TGB manifest looks like this:

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: prometheus-tgb
  namespace: monitoring
spec:
  serviceRef:
    name: prometheus-k8s # The name of your K8s Service
    port: 9090           # The port defined in the Service
  targetGroupARN: <YOUR_TF_OUTPUT_ARN>

Part 4: Application Deployment (Prometheus Helm Values)

We don't want to apply that YAML manually. We want it version-controlled with our Prometheus deployment.

Most Prometheus Helm charts (like kube-prometheus-stack) support an extraManifests or additionalManifests property in their values.yaml. This allows us to inject arbitrary K8s objects-like our TGB-directly during the Helm install.

Here is how you configure your values.yaml to register Prometheus to the Terraform-managed Target Group.

`values.yaml`

# Configuration for kube-prometheus-stack
prometheus:
  service:
    port: 9090
    # Ensure the service selector matches what the TGB expects
    # usually standard, but good to verify.

# Injecting the Custom Resource Definition
extraManifests:
  - apiVersion: elbv2.k8s.aws/v1beta1
    kind: TargetGroupBinding
    metadata:
      name: prometheus-binding
      namespace: monitoring # Must match the release namespace
    spec:
      serviceRef:
        name: prometheus-kube-prometheus-prometheus # Default name in the stack
        port: 9090
      targetGroupARN: "arn:aws:elasticloadbalancing:us-east-1:1234567890:targetgroup/eks-prometheus-tg/6d0ecf831eec9f09"

Summary of Flow

Terraform creates the empty Target Group (Mode: IP).
Helm deploys Prometheus + the TargetGroupBinding CR.
AWS Load Balancer Controller sees the new Binding CR.
Controller looks up the Pod IPs backing the Prometheus Service.
Controller registers those Pod IPs into the AWS Target Group automatically.

This approach gives us the stability of Terraform-managed Infrastructure (the ALB and Listeners) with the flexibility of Kubernetes-managed endpoints.

AWS CloudFront Explained: How Cache, Origin, and Response Policies Supercharge Your CDN

Maciej Łopalewski — Wed, 21 Jan 2026 09:00:00 +0000

If you have configured Amazon CloudFront in the past, you might remember wrestling with "Cache Behaviors" - a monolithic setting where caching logic, origin forwarding, and header manipulation were all jumbled together.

Those days are over.

Modern CloudFront architecture uses a modular Policy System. This approach decouples caching (what is stored) from origin requests (what is sent to the backend) and response headers (security/CORS).

For DevOps engineers and cloud architects, understanding these three policy types is the key to building performant, secure, and scalable content delivery networks. This guide breaks down the ecosystem of CloudFront Managed Policies and helps you choose the right tools for the job.

What is CloudFront?

Before diving into policies, let’s ground ourselves in the basics. Amazon CloudFront is a global Content Delivery Network (CDN). Its primary job is to sit between your users and your infrastructure (the "Origin" - like an S3 bucket or an EC2 load balancer).

Latency: It serves content from "Edge Locations" physically closer to the user.
Security: It terminates TLS connections at the edge and blocks DDoS attacks.
Scale: It absorbs traffic spikes so your backend doesn't crash.

The Policy Trio: How They Work

In the modern CloudFront request flow, three distinct policies interact to process a user's request. Understanding the distinction between them is critical for avoiding common pitfalls like "cache misses" or CORS errors.

1. Cache Policy

Where it sits: At the very front of the flow.
What it does: It determines the Cache Key.

When a user requests content, CloudFront uses this policy to decide if it already has a copy. It defines which headers, cookies, or query strings make a request "unique."

Strict policies = Higher cache hit ratio.
Loose policies = Lower cache hit ratio (more load on origin).

2. Origin Request Policy

Where it sits: Between CloudFront and your Backend (Origin).
What it does: It determines what data is forwarded to the backend during a cache miss.

This is the most misunderstood policy. It allows you to send data (like user-specific cookies) to your backend without including that data in the Cache Key. This keeps your cache efficiency high while still giving your application the data it needs to process logic.

3. Response Headers Policy

Where it sits: On the way back to the user.
What it does: It injects specific HTTP headers into the response.

Regardless of what your backend sends, this policy ensures the browser receives the correct Security (HSTS, XSS protection) and CORS headers.

Top Managed Policies: A Cheat Sheet

AWS maintains a library of "Managed Policies" that cover about 90% of use cases. Using these is a best practice - they are rigorously tested, updated by AWS, and require zero maintenance.

Here are the most essential managed policies for each category.

A. Managed Cache Policies

Control the Cache Key and TTL (Time To Live).

CachingOptimized

Best For: S3 Buckets, Static Websites, Images/Assets.

How it works: It ignores almost all headers and cookies. It aggressively caches content based solely on the URL path.
Why choose it: This provides the highest possible cache hit ratio. If your content doesn't change based on who is viewing it, use this.

CachingDisabled

Best For: Dynamic APIs, WebSockets, Real-time data.

How it works: It sets the Time-To-Live (TTL) to 0. Every request bypasses the cache and goes straight to the origin.
Why choose it: Essential for endpoints where data changes every second, or for write operations (POST/PUT) where caching would break functionality.

UseOriginCacheControlHeaders

Best For: CMS (WordPress/Drupal), Hybrid Apps.

How it works: It defers the decision to your server. It looks for Cache-Control headers sent by your backend to decide how long to store the file.
Why choose it: Perfect if you have a mix of static and dynamic content and want your application code, rather than CloudFront configuration, to control cache duration.

B. Managed Origin Request Policies

Control what the backend sees (without breaking the cache).

AllViewer

Best For: Legacy Applications, Complex Dynamic Apps.

How it works: Forwards everything - every header, every cookie, every query string - to the origin.
Why choose it: If your application relies on specific, obscure headers or client-side cookies to function, this ensures nothing is stripped out. Warning: This may expose internal origin details.

CORS-S3Origin

Best For: S3 Buckets serving assets to other domains.

How it works: Specifically whitelists the headers S3 requires to process CORS checks (Origin, Access-Control-Request-Method, etc.).
Why choose it: S3 handles CORS differently than a standard web server. Standard forwarding often fails with S3; this policy fixes those specific issues instantly.

UserAgentRefererHeaders

Best For: Analytics, Hotlink Protection.

How it works: It specifically forwards the User-Agent and Referer headers while stripping others.
Why choose it: If your backend needs to block requests from specific sites (hotlinking) or serve different content to mobile vs. desktop devices, but doesn't need full cookie visibility.

C. Managed Response Headers Policies

Control browser security and access.

SecurityHeadersPolicy

Best For: Everything. (Seriously, use this everywhere).

How it works: Automatically injects industry-standard security headers like:
- Strict-Transport-Security (HSTS)
- X-Frame-Options: DENY (prevents clickjacking)
- X-Content-Type-Options: nosniff
Why choose it: It instantly hardens your application against common web attacks without requiring code changes on your server.

CORS-with-preflight-and-SecurityHeadersPolicy

Best For: Single Page Apps (React, Vue, Angular) calling APIs.

How it works: Combines the security headers above with a permissive CORS configuration. It handles the OPTIONS pre-flight requests that modern browsers send before making API calls.
Why choose it: It solves the dreaded "CORS Error" in browser consoles for modern web applications.

SimpleCORS

Best For: Public, read-only data feeds.

How it works: Adds Access-Control-Allow-Origin: * to the response.
Why choose it: If you are hosting public data (like a weather feed or public JSON file) that you want anyone to be able to use on their website, this is the quickest way to enable it.

Common CloudFront Misconfigurations and How Managed Policies Fix Them

Even experienced DevOps teams run into the same CloudFront issues over and over. Almost all of them trace back to legacy cache behaviors or overly customized settings. Here’s how CloudFront Managed Policies solve the most common problems.

1. “My cache hit ratio is terrible.”

Cause: Your Cache Key is too loose - it includes unnecessary headers, cookies, or query strings.
Symptom: Every request is seen as "unique," forcing a constant stream of cache misses.
The Fix: Use the CachingOptimized managed policy. It strips almost everything from the Cache Key, restoring high hit ratios - perfect for static assets, SPAs, and S3 origins.

2. “CloudFront keeps forwarding too many headers to my origin.”

Cause: Legacy behaviors often forward all headers by default.
Impact: Increased origin load, slower responses, and potential "Request Header Too Large" errors on the backend.
The Fix: Switch to an Origin Request Policy like UserAgentRefererHeaders or CORS-S3Origin. This ensures you forward only what your backend actually needs to function.

3. “I’m still getting CORS errors in the browser.”

Cause: Missing or inconsistent Access-Control-* headers.
The Fix: Apply the CORS-with-preflight-and-SecurityHeadersPolicy response policy. It handles OPTIONS preflight requests and injects all required CORS headers at the edge - even if your backend forgets them.

4. “S3 CORS works on localhost, but not in CloudFront.”

Cause: S3 requires specific headers to process CORS. If CloudFront strips them, S3 treats the request as standard and omits the CORS response headers.
The Fix: Use the CORS-S3Origin Origin Request Policy. This explicitly forwards Origin, Access-Control-Request-Method, and Access-Control-Request-Headers so S3 knows to respond correctly.

5. “My dynamic API is being cached when it shouldn’t be.”

Cause: Your API path (/api/*) is falling through to a default behavior that has caching enabled.
The Fix: Create a specific behavior for your API path and attach CachingDisabled. This guarantees every request bypasses the edge and reaches your application.

Summary

Moving to Managed Policies allows you to operate with "Intent-Based Configuration." Instead of tweaking individual settings, you select a policy that matches your architectural intent.

Intent	Recommended Policy Combo
Static Website	Cache: `CachingOptimized` Origin Request: `None` (or `CORS-S3Origin`) Response: `SecurityHeadersPolicy`
Dynamic API	Cache: `CachingDisabled` Origin Request: `AllViewer` Response: `SimpleCORS`
Modern Web App (SPA)	Cache: `CachingOptimized` (for assets) Origin Request: `None` Response: `CORS-with-preflight`

Geospatial Data Orchestration: Why Modern GIS Pipelines Require an Asset-Based Approach

Paweł Sławacki — Thu, 15 Jan 2026 07:41:53 +0000

In the world of data, true turning points are rare—moments when a technology originally designed for one category of problems turns out to be the missing piece in a completely different domain. This is precisely what is happening now in geospatial data. Workflows traditionally rooted in the GIS niche have become one of the most demanding components of contemporary AI systems and environmental analytics.

What once relied on manual work inside desktop tools must now meet requirements of scalability, reproducibility, and full automation. Models need to be retrained continuously, data arrives in real time, and every forecast must be explainable and fully reproducible.

These were exactly the challenges we faced while building a cloud-native hydrological and environmental data processing system — one that merges dynamic measurements, large raster datasets, machine learning, and GIS-based interpretation. That experience made one thing very clear: geospatial does not simply need “better workflows.” It needs an orchestration layer that treats data as the primary actor — not a byproduct.

Dagster became the natural choice for such an architecture. Dagster is increasingly used for geospatial data orchestration because its asset-based model aligns naturally with GIS datasets, raster processing pipelines, and reproducible environmental analytics.

What is geospatial data orchestration?

Geospatial data orchestration is the practice of managing, automating, and governing complex GIS and spatial data pipelines — including raster processing, feature engineering, machine learning training, and data publication — in a way that is scalable, reproducible, and fully traceable.

Unlike traditional GIS workflows that rely on manual execution inside desktop tools, geospatial orchestration treats datasets and derived artefacts as first-class assets with explicit dependencies, versioning, and lineage.

Why asset-based orchestration is the language geospatial systems speak intuitively

In geospatial projects, every element of the workflow — an elevation raster, a land-cover classification, a soil map, a catchment-level aggregation, a training tensor — exists as a meaningful artefact with its own purpose and lineage. These artefacts form the narrative spine of the entire system.

While building the hydrological platform, we quickly discovered that geospatial processing fundamentally conflicts with the task-oriented paradigm used by most workflow tools. In hydrology, meteorology, or environmental modelling, “a task” is merely a transient carrier of work. What matters is the end product: the raster, the derived feature set, the trained model, the forecast.

This is precisely why Dagster’s model — where the core unit is the asset, not the task — feels almost native to geospatial data.

When we convert a DEM to a tile-optimized raster format, we create an asset.

When we generate soil attributes or retention-capacity parameters for a catchment, we create assets.

When we produce features for training a model or build the final forecasts — those are assets as well.

Each of these objects has a life of its own, a history, and a network of dependencies. Dagster makes this structure visible, not as an incidental side effect of code, but as the logical architecture of the entire system.

Why traditional workflow orchestrators struggle with geospatial pipelines

Most workflow orchestrators were designed for task-centric ETL pipelines. In geospatial systems, this approach breaks down because tasks are transient, while spatial datasets — rasters, tiles, features, and models — are long-lived analytical artefacts.

As a result, task-based orchestration makes lineage harder to understand, reproducibility fragile, and debugging costly in GIS-heavy and environmental data pipelines.

In geospatial, transparency is not a nice-to-have — it is a necessity

One of the core lessons from developing environmental systems is simple: results must be explainable. A hydrologist, GIS analyst, or decision-maker responsible for assessing risk must understand where every value in the model comes from and what transformations shaped it.

The orchestration we implemented enforces this clarity. Every stage — from data ingestion, through raster processing, to modelling and publication — leaves behind a durable artefact. There are no hidden transformations, no opaque steps, no “magic.” If a forecast changes from one iteration to another, we can point to the reason. If an experiment needs to be repeated, we do it deterministically.

Dagster amplifies this transparency, because it expresses the system as a web of dependencies between artefacts. In the geospatial architecture we built, full lineage is visible: from raw rasters to intermediate steps to the final products consumed in QGIS. This is not optional — it is a foundational requirement for analytical responsibility.

The cloud removes infrastructure friction; Dagster gives the system its rhythm

Geospatial data is large, and its processing can be computationally intensive. That is why one of the priorities of our solution was to clearly separate data processing from the infrastructure that executes it.

A central object store in S3, container-based processing, demand-driven autoscaling, version control of experiments in MLflow, and a permanent division between ETL and model-training environments allowed us to simplify the entire ecosystem. The team could focus on data, not on the platform itself.

Dagster acted as the coordinator in this architecture. It defined the relationships between artefacts, governed how data was refreshed, and set the cadence for model training. It provided structure without imposing unnecessary constraints, enabling architectural decisions to be made at the level of data — not infrastructure.

This is one of the benefits that only becomes visible in large geospatial systems: orchestration should not be a heavyweight layer “on top” of the system but a lightweight skeleton on which the system naturally rests.

Engineered for elasticity: The physical architecture

Our implementation relies on AWS EKS to absorb the extreme variance in geospatial compute. We treat infrastructure as elastic capacity, not a fixed cluster: it expands and contracts in response to the asset graph.

The cluster is divided into specialized node pools. Core services run on steady instances, while processing tasks route to autoscaling groups sized for their load — from lightweight CPU jobs to memory‑heavy raster operations. For machine learning, GPU nodes are provisioned on demand; a Dagster asset declares its needs via tags, and the cluster autoscaler supplies them. We pay for high‑performance compute only during the minutes that model training runs.

Operational rigor comes from isolation and clear identity boundaries. We split the Dagster deployment into two code locations — Data Preparation and Machine Learning — because geospatial stacks like GDAL and Rasterio conflict with the numerical stacks behind PyTorch or TensorFlow. Collapsing them into one environment creates brittle builds and version lock. By keeping them separate, each location owns its dependencies, and Dagster orchestrates across the seam cleanly. Security uses AWS Pod Identity to avoid long‑lived credentials, and GitHub Actions maintains a clean lineage from commit to ECR images. CloudWatch then provides a unified view of infrastructure health and pipeline performance.

This architecture delivers elasticity as a daily operational fact. After a training run, GPU nodes drain and terminate within minutes, returning spend to baseline. When a large raster arrives, the relevant pool expands to meet it, then contracts once the asset materializes. The system breathes with the workload — responsive at peaks, economical in troughs — without manual intervention or capacity planning. Most importantly, infrastructure fades from view: an asset definition states what it needs, and the platform ensures those resources appear exactly when required.

GIS remains a first-class partner, not collateral damage of modernization

Many contemporary data platforms try to replace GIS with proprietary viewers or dashboards. Yet in practice — and especially in environmental and hydrological projects — GIS tools remain irreplaceable. In our approach, GIS is not a competitor to modern data architecture but its natural consumer.

Final datasets are exposed in formats analysts know: GeoTIFF or Cloud Optimized GeoTIFF. As a result, GIS becomes a direct extension of the orchestration layer. Dagster produces the data; GIS interprets it. This separation of roles not only simplifies the system but also increases acceptance among experts who rely on these outputs daily.

Business value emerges not from automation, but from reduced analytical risk

From a technological standpoint, Dagster streamlines the workflow.

From a business standpoint, it does something more important: it reduces operational and analytical risk, which in geospatial projects is distributed across data quality, model correctness, and the reliability of forecasts used in decision-making.

In the architecture we built, every decision about data is reflected in the structure of artefacts. Every change is visible. Every experiment is reproducible. This means more control, less uncertainty, and a system that is significantly more resilient to errors and shifts in external conditions.

That is why well-designed orchestration becomes a strategic component — not an accessory — in geospatial data platforms. In domains where forecasts influence infrastructure planning, risk mitigation, or public safety, this level of analytical control is not a technical luxury — it is an operational requirement.

Frequently asked questions

Is Dagster suitable for geospatial and GIS workloads?

Yes. Dagster’s asset-based orchestration model works particularly well with geospatial pipelines where rasters, features, and models must be versioned, traced, and recomputed deterministically.

How does geospatial orchestration differ from traditional ETL?

Geospatial orchestration focuses on managing spatial data artefacts and their lineage rather than executing isolated tasks. The goal is analytical transparency and reproducibility, not just automation.

Can cloud-native orchestration coexist with GIS tools like QGIS?

Yes. Orchestrated pipelines can publish standard formats such as GeoTIFF or Cloud Optimized GeoTIFF, allowing GIS tools to remain first-class consumers of cloud-native data platforms.

Conclusion: Geospatial is entering the era of orchestration — and Dagster is its natural foundation

Geospatial data has a unique character: it blends the physical world with the mathematical complexity of models and the interpretability of spatial visualization. When these three worlds meet in a single project, traditional approaches to data processing quickly show their limits.

Dagster, applied to geospatial systems, breaks through these limits. It enables an architecture in which large environmental datasets, machine learning models, and GIS-based analytics do not fight for dominance but coexist within a coherent ecosystem.

It is not a tool that promises magic. It offers something more valuable: clarity, reproducibility, and accountability.

This is why geospatial is increasingly gravitating toward orchestration.

And why Dagster, with its asset-oriented philosophy, is emerging as the most natural language for that transformation.

Forwarding Cookies Using CloudFront: A Workaround for AWS Cache Policy Limitations

Daniel Kraszewski — Wed, 07 Jan 2026 12:00:00 +0000

When building our Terraform module for deploying Medusa on AWS, we ran into an unexpected challenge with Amazon CloudFront. We wanted to use CloudFront as a simple way to provide HTTPS and a public URL without requiring users to bring their own domain or SSL certificate. However, we discovered that CloudFront's managed cache policies don't forward cookies, headers, and query parameters when caching is disabled - exactly what we needed for our backend API.

The Problem: Managed Cache Policies and CachingDisabled

AWS CloudFront offers managed cache policies that handle common caching scenarios. The "CachingDisabled" policy seems perfect for dynamic content that shouldn't be cached. However, this policy doesn't forward cookies, headers, or query parameters to your origin by default.

For e-commerce platforms like Medusa, this is a dealbreaker. The backend needs:

Cookies for session management and authentication
Headers for content negotiation and API functionality
Query parameters for filtering and pagination

We initially tried to create a custom cache policy with MinTTL=0 (no caching) while specifying header and cookie forwarding behaviors. AWS rejected this with an error:

operation error CloudFront: CreateCachePolicy, https response error StatusCode: 400,
InvalidArgument: The parameter HeaderBehavior is invalid for policy with caching disabled.

AWS's validation logic considers forwarding settings incompatible with disabled caching when using formal cache policies. The problem is clear: cache policies won't let you forward data without caching, but dynamic applications need that data forwarded to work properly.

Why We Use CloudFront

Before diving into the solution, let's clarify why we chose CloudFront in the first place:

Free HTTPS with Default Certificate - CloudFront provides a free SSL/TLS certificate via cloudfront_default_certificate = true, giving you a URL like https://d123456abcdef.cloudfront.net
No Domain Required - Users don't need to purchase a domain, manage DNS records, or provision ACM certificates
VPC Security - Our Application Load Balancer (ALB) stays in private subnets, accessible only through CloudFront's VPC Origin feature
Simple Setup - One Terraform resource provides HTTPS, DNS, and secure origin access without additional configuration

For a deployment-focused module, this convenience is valuable. Users get a working HTTPS endpoint immediately after terraform apply.

The Solution: Legacy `forwarded_values` Configuration

The workaround is to use CloudFront's legacy forwarded_values block instead of modern cache policies. While AWS recommends cache policies for new distributions, the forwarded_values configuration still works and allows zero-TTL caching with full data forwarding.

Here's the configuration we use in our backend module:

default_cache_behavior {
  target_origin_id       = local.origin_id
  viewer_protocol_policy = "redirect-to-https"

  # Disable caching by setting all TTLs to zero
  min_ttl     = 0
  default_ttl = 0
  max_ttl     = 0

  forwarded_values {
    query_string = true    # Forward all query parameters
    headers      = ["*"]   # Forward all headers to origin

    cookies {
      forward = "all"      # Forward all cookies to origin
    }
  }

  allowed_methods = ["GET", "HEAD", "POST", "PUT", "PATCH", "OPTIONS", "DELETE"]
  cached_methods  = ["GET", "HEAD", "OPTIONS"]
}

Key Configuration Elements

The heart of this solution is the TTL configuration. By setting min_ttl, default_ttl, and max_ttl all to 0, we're telling CloudFront "don't cache anything, ever." Every request goes straight through to the origin, which is essential for dynamic content like user sessions and real-time inventory updates.

Inside the forwarded_values block, we're basically saying "pass everything through." Setting query_string = true ensures that API parameters like ?page=2&limit=20 reach your backend. The headers = ["*"] configuration is particularly important-it forwards every header, including Authorization, Content-Type, and custom headers your application might use. And crucially, forward = "all" in the cookies block ensures that session cookies make the round trip from browser to CloudFront to your backend and back again.

The allowed_methods array supports the full spectrum of HTTP verbs (GET, POST, PUT, PATCH, DELETE) because Medusa's admin API needs them all. This configuration effectively turns CloudFront into a passthrough proxy with HTTPS termination-not a traditional CDN, but a secure front door for your API.

Trade-offs and Considerations

This approach shines when you're working with dynamic applications that maintain session state-think authentication systems, shopping carts, or any API where each request is unique. It's particularly valuable in rapid deployment scenarios where getting HTTPS working quickly matters more than squeezing out every bit of performance optimization. We've also found it perfect for development and staging environments where managing domains and certificates feels like overkill.

That said, this isn't a one-size-fits-all solution. If you're serving static content like CSS, JavaScript bundles, or images, you're missing out on CloudFront's real strength: global edge caching. Similarly, if you're running a high-traffic production service where caching could significantly reduce origin load and costs, the no-cache approach leaves performance on the table. For applications serving a global audience where edge caching could shave hundreds of milliseconds off response times, you'd want to reconsider this pattern.

For our Medusa module specifically, the no-cache approach makes sense because backend APIs are inherently dynamic-every request involves database queries, authentication checks, and business logic that can't be cached safely. Caching would actually break core functionality like session management and real-time inventory updates. The convenience of instant HTTPS deployment is worth the trade-off, and users always have the option to add a proper CDN layer in front for their static storefront assets if needed.

Conclusion

AWS CloudFront's managed cache policies work well for typical CDN use cases, but they have limitations when you need no caching with full data forwarding. The legacy forwarded_values configuration provides a reliable workaround that's been working in production for our Medusa. deployments.

While AWS's documentation encourages using modern cache policies, the forwarded_values approach remains supported and is sometimes the pragmatic choice for dynamic applications. As always in infrastructure engineering, the "right" solution depends on your specific requirements-in our case, deployment convenience and session state management won the day.

This article is based on our experience building the terraform-aws-medusajs module for deploying Medusa. e-commerce backends on AWS.

Forem: u11d

Asset-Based Data Orchestration: Lessons from Building a Multi-State Social Data Platform

1. Why Reliability Becomes a Systems Problem at Scale

2. System Overview

High-Level Architecture

3. Source Heterogeneity as the Dominant Constraint

4. Normalization (HSDS, SDOH or Equivalent) as an Architectural Contract

5. What We Got Wrong Initially

6. Transitioning to Asset-Based Data Orchestration with Dagster

7. Partitioning for Failure Isolation

8. Data Quality Embedded in the Asset Graph

9. Operational Outcomes

10. Open Trade-offs and Unresolved Questions

11. Why These Lessons Matter Beyond This Platform

Final Thoughts

FAQ

What is asset-based data orchestration?

Why is asset-based orchestration useful for complex data platforms?

How does Dagster differ from traditional pipeline orchestrators?

When should teams adopt asset-based orchestration?

Does asset-based orchestration replace tools like DBT?

Orchestrating Trust: Building Reliable Data Systems for Social Impact

How production-grade orchestration enables impact at scale

The hidden fragility of well-intentioned systems

From pipelines to obligations

Reliability under real-world constraints

Automation without opacity

Social impact as an emergent property

Beyond mission-driven engineering

FAQ: Production-Grade Orchestration for Social Impact Systems

What is asset-based orchestration in data engineering?

How is asset-based orchestration different from traditional pipelines?

Why does data lineage matter in social impact systems?

What architectural risks do social data platforms typically face?

How does orchestration contribute to data governance?

Is asset-based orchestration only relevant at large scale?

How does this relate to data observability and reliability engineering?

What role does automation play in maintaining trust?

When should a team consider moving from pipelines to asset-oriented architecture?

About the authors / context

Next.js 16 Partial Prerendering (PPR): The Best of Static and Dynamic Rendering

What Is Partial Prerendering (PPR)?

How Partial Prerendering Works in Next.js 16

Cache Components: The Heart of PPR

Streaming and Suspense: The Engine Under the Hood

Why PPR Matters for E-commerce

How to Enable and Use PPR in Next.js 16

How PPR Fits with Other Rendering Strategies

Wrapping Up

Secure RDS Access Without Bastion Hosts: Using ECS Containers and SSM

The Problem

The Solution

Prerequisites

How It Works

1. Task Discovery

2. Container Runtime ID Retrieval

3. Port Forwarding Session

Usage

Key Benefits

1. No Infrastructure Overhead

2. Enhanced Security

3. Zero Configuration

4. Cost Effective

5. Temporary Access

Security Considerations

Network Security

Access Control

Troubleshooting

No running tasks found

Session Manager plugin not found

Connection refused

ECS Exec not enabled

Conclusion

How to Deploy Next.js 16 SSG to AWS Amplify: Complete Guide

Why AWS Amplify for SSG?

Prerequisites

Step 1: Configure Next.js 16 for SSG

Update next.config.ts

Handle Image Optimization

Verify SSG Compatibility

Update `next.config.ts`

`main.tf`

`iam.tf`