Forem: Akifumi Niida

How Cloudflare Proxy Silently Broke My Lambda ALB Communication

Akifumi Niida — Sun, 08 Mar 2026 01:10:40 +0000

TL;DR

Both api.hoge.com and backend.hoge.com had Cloudflare Proxy enabled.

Requests from the browser were already passing through Cloudflare Edge before reaching Lambda. When Lambda then called backend.hoge.com, DNS resolved to Cloudflare's Anycast IP — sending the request right back into Cloudflare Edge.

This created a Cloudflare loop: Cloudflare → Lambda → Cloudflare, resulting in:

Cloudflare Error 1000
DNS points to prohibited IP

flowchart LR
    Browser -->|①| CF1[Cloudflare Edge]
    CF1 -->|②| Lambda
    Lambda -->|③ backend.hoge.com = Cloudflare IP| CF2[Cloudflare Edge]
    CF2 -->|❌ Error 1000| ALB

As a quick fix, I turned Proxy OFF for backend.hoge.com. The long-term plan is to move Lambda → ALB communication inside the VPC.

The Error

Calling the API from the frontend returned a 403 Forbidden. The response body contained:

Cloudflare Error 1000
DNS points to prohibited IP

API Gateway and Lambda appeared to be working fine, and there were no logs on the ECS side at all. My first instinct was to blame the ALB or WAF.

Architecture

flowchart LR
    Browser -->|HTTPS| APIGW[API Gateway]
    APIGW --> Lambda
    Lambda -->|HTTPS backend.hoge.com| ALB
    ALB --> ECS
    ECS --> RDS

Lambda acts as a BFF (Backend for Frontend). The backend runs on ALB + ECS — a legacy constraint we hadn't moved away from yet. Lambda was calling that ALB over HTTPS using the backend.hoge.com domain.

Troubleshooting

Given the 403 status code, my initial suspects were:

WAF rules on the ALB
Security Group restrictions
Authorization logic in the ECS application
API Gateway authorizer configuration

No logs in ECS

As I dug deeper, something felt off. If the request was reaching the ALB, there should be something in the ECS logs. But there was nothing — not a single entry.

That pointed to the request never reaching the ALB in the first place.

Checking with curl

curl -v https://backend.hoge.com

Looking at the response headers:

server: cloudflare

And the response body contained DNS points to prohibited IP. That's when it clicked — Cloudflare itself was returning the 403, not our application.

Reading the docs

Cloudflare's official documentation covers the causes of Error 1000:

https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1000/

According to the docs, this error occurs when an A record points to a Cloudflare-owned IP, or when a request is routed through another reverse proxy and ends up back at Cloudflare a second time.

Since our ALB was configured via a CNAME (not an A record), I initially dismissed this as "not applicable." I knew how Proxy ON worked in theory, but the pressure of an ongoing incident made me overlook it. If I had checked the docs more carefully, I would have found the root cause much sooner.

Root Cause

Our Cloudflare DNS settings looked like this:

Record	Proxy
api.hoge.com	ON
backend.hoge.com	ON

It was a Cloudflare loop

Because api.hoge.com had Proxy ON, browser requests were already passing through Cloudflare Edge before arriving at Lambda.

When Lambda called backend.hoge.com — also Proxy ON — DNS returned Cloudflare's Anycast IP. The request looped back into Cloudflare Edge, creating a Cloudflare → Lambda → Cloudflare loop.

flowchart LR
    Browser -->|①| CF1[Cloudflare Edge api.hoge.com]
    CF1 -->|②| Lambda
    Lambda -->|③ backend.hoge.com = Cloudflare IP| CF2[Cloudflare Edge backend.hoge.com]
    CF2 -->|❌ Error 1000| ALB
    ALB --> ECS
    ECS --> RDS

Why does this trigger Error 1000?

Cloudflare returns Error 1000 when it detects a loop, or when the resolved origin IP falls into one of these categories:

A Cloudflare-owned IP (loop prevention)
RFC 1918 private addresses (10.x.x.x / 172.16.x.x / 192.168.x.x)
Loopback address (127.0.0.1)

In this case, backend.hoge.com resolved to a Cloudflare IP, so Cloudflare treated it as a request targeting itself and returned DNS points to prohibited IP.

Fix #1: Quick Fix

I turned Proxy OFF for backend.hoge.com.

Record	Proxy
backend.hoge.com	OFF

With Proxy OFF, DNS returns the actual origin CNAME (the ALB domain) instead of a Cloudflare IP. The loop was broken and requests started flowing normally again.

flowchart LR
    Lambda -->|backend.hoge.com = ALB domain| ALB
    ALB --> ECS
    ECS --> RDS

What I Learned

Cloudflare is more than just DNS

Cloudflare combines authoritative DNS, reverse proxy, CDN, and WAF into one. When Proxy is ON, all traffic is routed through Cloudflare Edge before reaching your origin.

This is great for browser-facing traffic — you get DDoS protection, caching, and WAF for free. But for server-to-server communication, it can cause unexpected behavior like this.

Proxy ON vs OFF changes the entire traffic path

flowchart LR
    subgraph Proxy OFF
        C1[Client] -->|ALB domain| ALB1[ALB]
    end
    subgraph Proxy ON
        C2[Client] --> CF[Cloudflare Edge] --> ALB2[ALB]
    end

Match Proxy settings to your use case

Use case	Proxy
Browser → API	ON (CDN + WAF benefits)
Server → Server	OFF (no benefit, potential loops)

If a server calls a Proxy ON domain while the calling service is itself behind Cloudflare, you risk creating exactly this kind of loop.

Fix #2: Long-term Fix

The quick fix works, but routing Lambda → ALB traffic through public DNS was never ideal. Moving this communication inside the VPC is the proper solution.

flowchart LR
    Browser -->|HTTPS| APIGW[API Gateway]
    APIGW --> Lambda
    subgraph VPC
        Lambda -->|VPC-internal| ALB
        ALB --> ECS
        ECS --> RDS
    end

A few things to keep in mind for this migration:

Lambda needs to be deployed inside the VPC (if it isn't already)
Cold start impact from VPC placement is minimal these days — this used to be a real concern, but AWS has significantly improved it
You'll need to explicitly allow Lambda → ALB traffic in your security groups

This approach eliminates the Cloudflare dependency entirely for internal traffic, reduces latency, and simplifies the network topology.

Why Did This Break Suddenly?

The Proxy ON configuration had been in place for a long time. So why did this only start failing now?

I checked Cloudflare's changelog but couldn't find any official announcement around this time about changes to Error 1000 detection or proxy behavior.

Searching the Cloudflare community, there are scattered reports of "Error 1000 suddenly appearing" — but in each case, the cause turned out to be a specific configuration change on the user's end (a DNS record update, an IP change on the hosting side, etc.).

I'm still looking into Cloudflare's Audit Log for anything that might explain this. I'll update this post if I find something.

Summary

If a request passes through a Proxy ON domain, and that request then calls another Proxy ON domain, you get a Cloudflare loop — and Error 1000.

For server-to-server communication, either turn Proxy OFF for internal domains, or better yet, keep that traffic inside the VPC entirely.

STOPSIGNAL is now available on Amazon ECS Fargate

Akifumi Niida — Sun, 14 Dec 2025 08:43:25 +0000

Fargate Now Supports “STOP SIGNAL”

https://aws.amazon.com/jp/about-aws/whats-new/2025/12/amazon-ecs-custom-container-stop-signals-fargate/

Fargate now sends the STOPSIGNAL command defined in your Dockerfile to containers.

Why is this great? Previous challenges

Previously, achieving a graceful shutdown on ECS Fargate required creating logic to safely stop after receiving SIGTERM.
https://aws.amazon.com/jp/blogs/news/graceful-shutdowns-with-ecs/

However, middleware often has fixed behavior when receiving a stop signal.
For example, nginx stops immediately upon receiving SIGTERM. This required workarounds like writing trap handling to retry SIGTERM as SIGQUIT.
https://linuxjm.sourceforge.io/html/nginx/man8/nginx.8.html

Improvements

By adhering to the OCI standard, specifying STOPSIGNAL arbitrary_signal in the Dockerfile alone guarantees graceful shutdowns on Fargate, Kubernetes, and local Docker, significantly improving portability.

Why Wasn't This Possible Before? (Analysis)

Fargate likely runs on AWS's proprietary managed host OS (microVM/Firecracker), which may have restricted flexible access. If anyone knows more, please share.

What I Tried

The source for verification is here:
https://github.com/nidcode/ecs-stop-signal-sample

I created a simple program like the one below for testing.

const express = require(‘express’);
const app = express();
const PORT = process.env.PORT || 8080;
const SHUTDOWN_DELAY_MS = parseInt(process.env.SHUTDOWN_DELAY_MS, 10) || 10000;

app.get(‘/’, (req, res) => {
    res.send(‘Running and waiting for stop signal...’);
});

const server = app.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`);
});

// --- Signal Handler Definition ---
// Handler for SIGTERM (default stop signal)
process.on(‘SIGTERM’, () => {
    handleShutdown(‘SIGTERM’);
});

// Handler for SIGINT (custom verification signal)
process.on(‘SIGINT’, () => {
    handleShutdown(‘SIGINT’);
});

// SIGKILL (force termination) cannot be caught!

function handleShutdown(signal) {
    console.log(`[${signal} RECEIVED] Graceful shutdown initiated.`);

    // 1. Stop accepting new connections to the server
    server.close(() => {
        console.log(‘HTTP server closed.’);
    });

    // 2. Simulate cleanup processes like writing logs and closing the DB
    console.log(`Starting cleanup. Waiting for ${SHUTDOWN_DELAY_MS / 1000} seconds...`);

    // Assumes cleanup completes within the delay period
    setTimeout(() => {
        console.log(`[${signal} SUCCESS] Cleanup complete. Exiting cleanly.`);
        process.exit(0);
    }, SHUTDOWN_DELAY_MS);
}

Test Application Behavior

Receives SIGINT and performs cleanup for 10 seconds
Records [SIGINT RECEIVED] logs in CloudWatch Logs

For verification, I set the ECS task definition stopTimeout to 15 seconds.

Behavior Without STOPSIGNAL

I added some comments to server.js to create a diff, then ran cdk deploy to force task recreation.

As expected, it receives SIGTERM

Setting STOPSIGNAL

Simply adding STOPSIGNAL to the Dockerfile is sufficient.
Let's configure SIGINT alongside server.js.

FROM node:22-slim
...
STOPSIGNAL SIGINT

CMD [“node”, “server.js”]

Once deployed successfully, run cdk deploy again as before to force the tasks to rebuild.

It caught the SIGINT!

Summary

This custom stop signal support isn't just an added parameter—it signifies that AWS Fargate now fully complies with the international standard for container runtimes (OCI).
This allows container operators to bring standard images used in other environments directly to Fargate, significantly improving deployment and operational quality (Graceful Shutdown).

Tips for exposing SPAs and APIs with Cloudfront + S3 + API Gateway

Akifumi Niida — Tue, 07 Jun 2022 13:30:10 +0000

This is a common pattern, but I would like to share some of the practices that I have arrived at.

Conclusion first

SPA: Change path with extension to /index.html using Cloudfront Functions
API: Point /api to API Gateway with Behavior

Background

Until now, when publishing SPA applications such as Vue or Angular with Cloudfront + S3
S3 returns 403 or 404 by making a request to a path that does not exist.
Therefore, we had to create a custom error page on the Cloudfront side and configure it to redirect to /index.html.

Common problem 1 (SPA path issue)

Even if there is a 404 that should be caught, it is redirected without care.
For example, it is hard to notice if there is a CSS or JS upload error.

Common Problem 2 (Cloudfront problem of assigning specific paths)

I want to assign /api paths to API Gateway.
This method seems to be good from a security point of view, as it eliminates the need to configure CORS.

SPA path problem

Using Cloudfront Functions, I can distribute requests like /users/xxx to /index.html in a good way without waiting for error pages.

Cloudfront Functions will be the ones you code to handle events at the edge.
There is a lambda@edge that is similar, but you can read more about the differences here for the difference.
It is better to understand that lambda@edge has limited functionality and can only perform simple processing.

Also, if you want to try Cloudfront Functions quickly, you can try this.
Cloudfront Functions tutorial

terraform example

Configuring Cloudfront Functions

Define resources like this. (Parts not directly related to AWS Provider settings, etc., are omitted.)
See documentation for a description of the parameters.

Write the actual process in code. Here, it is read from a separate file.
You can make it public by setting publish = true.

resource "aws_cloudfront_function" "spa_redirect" {
  name = "${var.product_name}-${var.env}-spa-redirect"
  runtime = "cloudfront-js-1.0"
  comment = "${var.product_name}-${var.env}-spa-redirect"
  publish = true
  code = file("cloudfront_functions/spa-redirect.js")
}

function part

It is written in a very old fashioned way, but for some reason it needs to be ECMAScript 5.1 compliant.
I would be very grateful if you could improve on this.

The presence or absence of extensions is determined by the presence or absence of a dot. if(request.uri.indexOf(".")) === -1)

var index = '/index.html';
function handler(event) {
    var request = event.request;
    // if extension not found (access not real file)
    if(request.uri.indexOf(".")) === -1) {
        request.uri = index;
    }
    return request;
}

Configuring Cloudfront Functions to adapt to Cloudfront

This is a long list, but all you need to focus on is the function_association at the bottom.

You can set it for each behavior!
The event_type is at the time of request, so we'll write viewer-request.
For more information on event_type, please refer to here for more information about event_type. Cloudfront Functions supports only viewer-request and viewer-response.

resource "aws_cloudfront_distribution" "front" {
  origin {
    domain_name = aws_s3_bucket.front.bucket_regional_domain_name
    origin_id = local.s3_origin_id

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.origin.cloudfront_access_identity_path
    }
  }
  enabled = true
  is_ipv6_enabled = true
  comment = "${var.product_name}-${var.env}"
  default_root_object = "index.html"
  aliases = ["test.${data.terraform_remote_state.network.outputs.domain}"]

  default_cache_behavior {
    allowed_methods = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]]
    cached_methods = ["GET", "HEAD"].
    target_origin_id = local.s3_origin_id

    forwarded_values {
      query_string = true

      cookies {
        forward = "none"
      }
    }

    function_association {
      event_type = "viewer-request"
      function_arn = aws_cloudfront_function.spa_redirect.arn
    }

    viewer_protocol_policy = "allow-all"
    min_ttl = 0
    default_ttl = 0
    max_ttl = 0

  }
... Omitted.
}

I want to separate /api

As mentioned above, you can use cloudfront's behavior to separate them.
Many people create API Gateways using the serverless framework or CDK.
It is convenient to get values from Cloudformation.

Point.

Get the endpoint and stage from the cloudformation in the domain_name of origin.
Specify api/* in path_pattern of behavior.
Since /api is not needed to access the APIGateway, it is removed by lambda@edge (this could also be replaced by cloudfront functions).

resource "aws_cloudfront_distribution" "front" {
...
  origin {
    custom_origin_config {
      http_port = "80"
      https_port = "443"
      origin_protocol_policy = "https-only"
      origin_ssl_protocols = ["TLSv1.2"]
    }
    # Extract xxx.execute-api.ap-northeast-1.amazonaws.com and v1 from ServiceEndpoint
    domain_name = split("/", data.aws_cloudformation_stack.api_test.outputs["ServiceEndpoint"])[2])
    origin_id = var.product_name
  }
...
  ordered_cache_behavior {
    path_pattern = "api/*"
    allowed_methods = ["HEAD", "DELETE", "POST", "GET", "OPTIONS", "PUT", "PATCH"].
    cached_methods = ["GET", "HEAD"].
    target_origin_id = var.product_name

    forwarded_values {
      headers = ["Authorization"].
      query_string = true
      cookies {
        forward = "none"
      }
    }

    min_ttl = 0
    default_ttl = 10
    max_ttl = 10
    viewer_protocol_policy = "https-only"
    lambda_function_association {
      event_type = "origin-request"
      lambda_arn = aws_lambda_function.redirect_trim_context.qualified_arn
    }

  }
...
}

That's all, obvious! Some of you may think it is, but we hope it will help you if you have any doubts when configuring your infrastructure.

Reference.
SPA routing process with AWS CloudFront Functions