Forem: gokcedemirdurkut

Why Your Nginx Security Headers Disappear (add_header Inheritance Explained)

gokcedemirdurkut — Thu, 12 Mar 2026 09:53:58 +0000

Security headers can disappear in Nginx even when they are correctly configured.

Some endpoints may silently drop important headers like:

Content-Security-Policy
Strict-Transport-Security
X-Frame-Options

This behavior is subtle and has existed in Nginx for a long time.

How the Problem Appears

Everything looks correct at first.

curl -I https://example.com/api/health

Response headers include security policies as expected.

But then another endpoint is checked:

curl -I https://example.com/

And suddenly:

❌ Content-Security-Policy missing
❌ Strict-Transport-Security missing
❌ X-Frame-Options missing

Tools like securityheaders.com will immediately detect this.

The confusing part is that the headers are clearly defined in the configuration.

A Common Configuration

A typical setup may look like this:

server {
    add_header Content-Security-Policy "...";
    add_header Strict-Transport-Security "max-age=31536000";

    location ~* \.html$ {
        add_header Cache-Control "no-store";
    }
}

The intention is simple:

disable caching for HTML responses
apply security headers globally

However, this configuration does not behave as expected.

What Actually Happens

Responses served from:

location ~* \.html$

will only include:

Cache-Control: no-store

All security headers defined in the server block disappear.

There are no warnings and no errors in the logs.
The headers are simply missing.

The Reason

This happens because of how Nginx handles header inheritance.

If a location block defines any add_header directive, it does not inherit add_header directives from parent blocks.

Even adding a single header like:

Cache-Control

causes Nginx to drop previously defined headers such as:

Content-Security-Policy
Strict-Transport-Security
X-Frame-Options
Referrer-Policy
Permissions-Policy

This behavior has existed for many years and has caused confusion in many configurations.

The Previous Workarounds

Before recently, the usual approaches were:

1. Duplicating headers

Define all security headers again in every location block.

2. Using includes

Create a file like:

include security_headers.conf;

and include it everywhere.

Both approaches work but are difficult to maintain and easy to forget.

The Fix in Nginx 1.29+

Recent Nginx versions introduced a better solution:

add_header_inherit merge;

This directive changes how headers are inherited.

Instead of replacing headers defined in parent blocks, child blocks merge them.

This feature was introduced in newer Nginx releases and is documented in the official release notes:

https://blog.nginx.org/blog/nginx-open-source-1-29-3-and-1-29-4

A Safer Configuration

With the new directive, the configuration can be written like this:

server {

    add_header_inherit merge;

    add_header Content-Security-Policy "...security policy..." always;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;

    location ~* \.html$ {
        add_header Cache-Control "no-store, no-cache, private, max-age=0" always;
    }

}

Now the behavior is predictable:

HTML responses are not cached
Security headers are applied consistently
Headers do not need to be duplicated

Final Takeaway

When using add_header inside location blocks in Nginx 1.29+, enabling the following directive is recommended:

add_header_inherit merge;

This prevents accidental loss of security headers and makes configurations easier to maintain.

Preventing Silent ECS Deployment Failures with Circuit Breaker

gokcedemirdurkut — Thu, 26 Feb 2026 10:57:55 +0000

AWS Elastic Container Service (ECS) provides a built-in feature called the deployment circuit breaker, designed to make service deployments safer and more resilient.

This feature continuously monitors the health of tasks during a deployment and automatically rolls back changes if newly launched tasks fail to become healthy. When enabled, it prevents failed deployments from leaving services in a degraded or non-functional state.

Without this safeguard, deployment failures can easily go unnoticed. For example, if new tasks fail to start or never pass health checks, the service may still appear to be running while it is effectively broken. These silent failures can result in data loss, financial impact, or operational issues depending on the workload.

In this post, I’ll walk through how to enable the ECS deployment circuit breaker using Terraform, how to observe deployment failures via EventBridge, and how to send real-time alerts to Slack.

Why the ECS Deployment Circuit Breaker Matters

Enabling the deployment circuit breaker provides several important benefits:

Automatic rollback – Failed deployments are reverted to the last known healthy service revision
Improved visibility – ECS emits structured events whenever a deployment fails or rolls back
Reduced operational overhead – Failures are mitigated automatically without immediate manual intervention

Together, these significantly reduce the risk of production incidents caused by faulty deployments.

Enabling the Circuit Breaker with Terraform

The deployment circuit breaker can be enabled directly in your ECS service definition. In Terraform, this is done using the deployment_circuit_breaker block:

resource "aws_ecs_service" "default" {
  name            = "tuve"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.default.arn
  desired_count   = 2

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  ...
}

With this configuration in place, ECS will automatically stop and roll back a deployment if the new tasks fail to reach a healthy state.

Once enabled, the AWS Management Console clearly indicates that the Deployment circuit breaker is turned on.

Observing Deployment Failures

Automatic rollback is useful, but visibility is just as important.

When the ECS deployment circuit breaker triggers, ECS emits events to Amazon EventBridge with the following detail type:

ECS Deployment State Change

Here is an example event payload:

{
  "version": "0",
  "id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE",
  "detail-type": "ECS Deployment State Change",
  "source": "aws.ecs",
  "account": "111122223333",
  "time": "2020-05-23T12:31:14Z",
  "region": "eu-central-1",
  "resources": [
    "arn:aws:ecs:eu-central-1:111122223333:service/default/servicetest"
  ],
  "detail": {
    "eventType": "ERROR",
    "eventName": "SERVICE_DEPLOYMENT_FAILED",
    "deploymentId": "ecs-svc/123",
    "updatedAt": "2020-05-23T11:11:11Z",
    "reason": "ECS deployment circuit breaker: task failed to start."
  }
}

Key Fields to Monitor

Some fields in this event are particularly useful for monitoring and alerting:

eventName
- SERVICE_DEPLOYMENT_FAILED
- SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED
reason – Explains why the deployment failed
resources – Identifies the affected ECS service
updatedAt – Indicates when the failure occurred

Tracking these fields ensures that deployment issues are visible immediately instead of being discovered hours later.

Deployment Rollback in the AWS Console

The AWS Management Console also provides clear visibility into rollback activity. After a failed deployment, the Deployments tab shows the rollback status along with the target service revision.

This view is particularly useful for confirming that the circuit breaker worked as expected.

Sending Deployment Alerts to Slack

To ensure deployment failures are noticed immediately, ECS deployment events can be routed to Slack using EventBridge and Lambda.

The overall flow looks like this:

ECS → EventBridge → Lambda → Slack

Lambda Handler Example

The Lambda function listens for ECS deployment state changes and sends notifications when a deployment fails or rolls back:

def lambda_handler(event, context):
    detail_type = event.get("detail-type", "")

    if detail_type == "ECS Deployment State Change":
        event_name = event.get("detail", {}).get("eventName")

        if event_name in [
            "SERVICE_DEPLOYMENT_FAILED",
            "SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED"
        ]:
            detail = event.get("detail", {})
            resources = event.get("resources", [])
            service_name = resources[0].split("/")[-1] if resources else "unknown"
            reason = detail.get("reason", "Unknown")
            updated_at = detail.get("updatedAt", "Unknown")

            send_slack_notification(
                service=service_name,
                reason=reason,
                event_type=event_name,
                timestamp=updated_at
            )

EventBridge Rule (Terraform)

The following EventBridge rule filters ECS deployment events and forwards them to the Lambda function:

resource "aws_cloudwatch_event_rule" "ecs_deployment" {
  name = "ecs-deployment-events"

  event_pattern = jsonencode({
    "source": ["aws.ecs"],
    "detail-type": ["ECS Deployment State Change"],
    "detail": {
      "eventName": [
        "SERVICE_DEPLOYMENT_FAILED",
        "SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED"
      ]
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda" {
  rule = aws_cloudwatch_event_rule.ecs_deployment.name
  arn  = aws_lambda_function.notification.arn
}

Final Outcome

After enabling the ECS deployment circuit breaker and adding Slack notifications:

Failed deployments automatically roll back
Silent service failures are eliminated
Deployment issues become visible in real time
ECS services are safer by default

By combining automated rollback with real-time alerts, you can significantly reduce operational risk and increase confidence in your ECS deployments.

Deploying and Customizing AWS ParallelCluster Service (PCS) for HPC Workloads

gokcedemirdurkut — Sun, 26 Oct 2025 09:15:02 +0000

I recently worked on a project involving AWS ParallelCluster Service (PCS).

The main goal was to build an HPC cluster that meets our specific requirements such as using an image with Python 3.10, installing the necessary dependencies and deploying a PCS cluster.
To achieve this, I built a custom AMI(Amazon Machine Image) using Packer, then launched a PCS cluster based on that image.
Throughout the process, I automated the workflow using Terraform, GitHub Actions and shell scripts.
After deploying the cluster, I verified that everything was working correctly by running several SLURM job commands, which confirmed that our setup was operational.

In this article, I’ll walk you through the entire process from building the custom AMI to running jobs on PCS.

What is AWS PCS?

AWS ParallelCluster Service (PCS) is a managed service that enables high-performance computing (HPC) on AWS. It’s designed for running parallel workloads such as simulations, ML training, or large-scale data analysis.
Using PCS, you can deploy and manage HPC clusters without manually configuring compute nodes, networking or schedulers. It integrates seamlessly with services like Amazon S3 and AWS Batch, supporting complex workloads efficiently.

Why Use AWS PCS?

Traditionally, deploying and managing HPC clusters required deep expertise in cluster configuration, job scheduling, and infrastructure management.
AWS PCS abstracts much of this complexity by offering:

Fully managed cluster orchestration
Integration with SLURM scheduler
Elastic scaling based on job demand
Infrastructure as Code (IaC) support with Terraform and CloudFormation

This makes AWS PCS an excellent choice for researchers, data scientists, and DevOps engineers who want to focus on workloads rather than infrastructure.

How Was the PCS Cluster Created?

The PCS cluster was provisioned using Terraform with the awscc
provider. The infrastructure is defined as code and includes the cluster, login/compute node groups and job queue.

Note: The awscc provider was required because PCS resources are not yet supported by the standard AWS provider. It uses the AWS Cloud Control API to manage newer services like PCS.

📢 Official Announcement — Terraform Support for AWS ParallelCluster Service (March 2025)

PCS Cluster Components

A PCS setup typically includes three main components:

Cluster: Defines general settings, scheduler configuration, and networking.
Node Groups: Specifies login and compute nodes.
Queue: Manages job scheduling and execution PCS Setup with Terraform.

Below is a simplified example of how a PCS cluster can be created using Terraform and the awscc provider:

# Create PCS Cluster using awscc provider
resource "awscc_pcs_cluster" "this" {
  count = var.enable_cluster ? 1 : 0
  name  = var.cluster_name
  size  = var.cluster_size

  scheduler = {
    type    = "SLURM"
    version = var.cluster_slurm_version
  }

  networking = {
    security_group_ids = [aws_security_group.this.id]
    subnet_ids         = var.subnet_ids
  }
}

Variables like cluster_name, subnet_ids, and cluster_slurm_version can be parameterized to adapt across environments.

Cost Awareness & Usage Control

Running a PCS cluster can become expensive depending on your configuration.
Each compute node in the cluster uses Amazon EC2 instances, and costs increase while those instances are running.

To manage this, we added a Terraform variable that controls whether the PCS cluster should be deployed:

# Control cluster deployment
enable_pcs_cluster = true   # create the cluster
# enable_pcs_cluster = false  # skip deployment

This allows you to enable or disable the PCS cluster as needed.
For example, during development or testing, you can set enable_pcs_cluster = false to avoid unnecessary charges.

Creating Custom AMIs with Packer

Custom AMIs let you pre-install software and dependencies on compute nodes.

Steps:

Create a Packer template based on a base AMI (e.x. Ubuntu or Amazon Linux 2).
Install the required software packages.
Build the AMI and use it in your PCS cluster.

Example Packer Template:

{
  "builders": [{
      "type": "amazon-ebs",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "{{user `instance_type`}}",
      "ssh_username": "{{user `ssh_username`}}",
      "ami_name": "{{user `ami_name_prefix`}}-{{user `distribution`}}-{{user `architecture`}}-{{isotime `2006.01.02-15.04`}}",
      "ami_description": "{{user `ami_description`}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sudo yum update -y",
      "sudo yum install -y gcc make python3"
      ...
    ]
  }]
}

This is just an example. You can add other dependencies as needed.

We used SLURM as the job scheduler within PCS to manage HPC workloads efficiently.
It handles queueing, job submission, and node allocation allowing multiple users or jobs to share compute resources dynamically.

During our setup, we required Python 3.10, but the default Amazon Linux 2 AMI provided only Python 3.7.
To solve this, we created a custom AMI using Packer, based on the following Ubuntu Marketplace image:

Ubuntu Server 22.04 LTS (arm64)
AMI ID: ami-0f45e2f16611e3139
Source: AWS Marketplace

This custom AMI included:

Python 3.10 preinstalled
Common HPC dependencies (gcc, make, python3-pip)
Additional Python libraries required for our workloads

By using a custom AMI, we ensured compatibility with our codebase and reduced setup time for new compute nodes.

Note: Custom AMIs are especially helpful when your HPC workloads require specific compiler versions, Python environments, or third-party libraries.

Testing the SLURM Cluster

Once the PCS cluster was up and running, we validated the environment by executing a few basic SLURM commands:

# Check the cluster and nodes
sinfo

# Submit a test job
echo "echo Hello from SLURM" > test.sh
sbatch test.sh

# View the job queue
squeue

These simple tests confirmed that the SLURM scheduler was active, compute nodes were responding, and jobs were successfully executed across the PCS cluster.

Conclusion

AWS PCS simplifies HPC cluster management with a managed, scalable, and elastic environment.
Terraform (awscc) enables modern Infrastructure as Code deployment.
SLURM provides flexible and familiar scheduling for HPC users.
Custom AMIs and bootstrap scripts enable deep customization and reproducibility.
Ideal for research, AI/ML training, and data-intensive simulations.

Thanks for reading!

#terraform #aws #devops #cloud #hpc #slurm #packer #ami #python