Journey with Apache Flink on AWS EKS

afnan amin — Tue, 16 Dec 2025 10:00:43 +0000

When building streaming pipelines, one of the biggest challenges isn’t just processing data , it’s keeping jobs running reliably while data continuously flows. Apache Flink addresses this challenge effectively.

Discovering Flink

Apache Flink is a distributed stream-processing framework built for stateful, real-time data processing. Unlike batch-first frameworks, Flink treats streams as the core abstraction, with batch as just a special case.

Flink stands out due to its handling of state, time, and fault tolerance:

Exactly-once processing guarantees
Native state management for long-running jobs
Event-time processing with watermarks
Checkpointing that makes failures recoverable

This made Flink feel less like a tool and more like a platform for always-on systems.

Why We Chose Flink

Streaming pipelines evolve over time , logic changes, schemas expand, and performance tuning becomes necessary. Flink’s checkpoint-based execution model allowed us to:

Restart jobs without breaking downstream systems
Roll out logic or configuration changes safely
Treat streaming jobs as living systems, not one-off deployments
Its combination of stateful processing, exactly-once semantics, and safe evolution made Flink the clear choice for our production pipelines.

Running Flink on AWS EKS

Once we decided on Flink, we had to figure out where and how to run it reliably. AWS EKS provided:

A managed Kubernetes control plane
Native integration with AWS services
Consistent environments across dev, test, and prod

To make Flink truly Kubernetes-native, we adopted the Flink Kubernetes Operator.

After installing it via Helm, it introduced a FlinkDeployment CRD. From that point, our deployments became fully declarative. We define the desired state in YAML, and the operator continuously reconciles it.

The operator:

Launches the JobManager pod (hosting the Flink UI)
Scales TaskManagers as needed
Configures networking, volumes, and service accounts
Manages job restarts, upgrades, and recovery

This drastically reduced operational overhead and made Flink cloud-native and production-ready.

Deploying Flink Jobs

In practice, we separate cluster management from job management:

FlinkDeployment defines the session cluster (image, resources, Flink config, EFS mounts)

FlinkSessionJob defines the actual streaming job (entrypoint, arguments, parallelism, upgrade mode)

Most of our deployments happen via Terraform, rendering YAML templates and applying them using kubernetes_manifest.

Here’s a simplified example of a FlinkSessionJob:

apiVersion: flink.apache.org/v1beta1 kind: FlinkSessionJob metadata: name: flink-job-gg-pending-orders namespace: flink spec: deploymentName: flink-session-v2 job: entryClass: "org.apache.flink.client.python.PythonDriver" args: - "-py" - "/opt/flink/lib/src/s3_to_iceberg.py" parallelism: 1 upgradeMode: stateless

💡 Tip: To rerun a job from the last checkpoint, switch upgradeMode to last-state.

Operating Jobs Safely

This is where Flink’s strength shines. For initial runs, we typically start jobs stateless. But for restarts, backfills, or upgrades, we rely on checkpoint-based recovery using upgradeMode: last-state.

This ensures:

Jobs resume from the latest successful checkpoint
Downstream systems remain stable
Minimal gaps or duplicates, even for CDC sources

Evolving Jobs Over Time

We handle changes carefully:

Configuration Changes (parallelism, resources, checkpoint tuning):

Update the job or cluster spec
Apply via Terraform or kubectl
Operator restores state automatically

Business Logic Changes (Python, SQL, JARs):

Push updated code to S3
Sync to EFS via AWS DataSync
Verify files in Flink containers
Perform a rolling, stateful upgrade

This process allows us to iterate safely, even on critical streaming pipelines.

Lessons Learned

From choosing Flink for its stateful streaming model, to running it on AWS EKS, and finally operating jobs safely with the Flink Kubernetes Operator, this journey has shaped how we build and maintain streaming pipelines.

Flink is more than a processing engine ,it’s a platform for evolving, long-running data pipelines. And running it Kubernetes-native on AWS lets us balance operational safety, scalability, and flexibility.