TOMOAKI ishihara for AWS Community Builders

Posted on Jun 11

AWS CDK Logical ID Deep Dive: How Adding One CloudFront Origin Broke My Entire AWS CDK Deployment

#aws #cdk #cloudfront #infrastructure

👋 Introduction

Have you ever encountered a situation where simply changing the order of CloudFront origins in your AWS CDK code caused existing resources to be deleted and recreated? This seemingly innocent change can lead to significant infrastructure disruption, especially with resources like VPC Origins that have specific update constraints.

In this article, I'll dive deep into the root cause of this problem - CDK's logical ID generation mechanism - and provide practical solutions to prevent it.

📌 Key Points of This Article

Root cause: Changing the order of addBehavior calls changes the index, resulting in different logical IDs
Impact: VPC Origins cannot be updated while associated with distributions, causing deployment failures
Best practice: Maintain the order of addBehavior calls through defensive coding

🚨 The Problem Scenario

What Happened

Consider a CloudFront distribution with the following architecture:

Image suggestion: Diagram showing CloudFront distribution with multiple S3 origins and VPC origins connected to EC2 instances

This setup includes:

Multiple S3 origins for different content types
VPC origins pointing directly to EC2 instances (cost optimization for development)
A mix of public and private content delivery

The original CDK code had origins added in this order:

this.distribution = new cloudfront.Distribution(this, "Distribution", {
  defaultBehavior: {...} // Origin1 (S3)
});

// Original order
dist.addBehavior("/widget/*", widgetOrigin, {...});     // Origin2 (S3)
dist.addBehavior("/files/*", fileUploadOrigin, {...});  // Origin3 (S3)

const backendOrigin = origins.VpcOrigin.withEc2Instance(backendInstance, {
  httpPort: 8080,
  // other configuration...
});
dist.addBehavior("/api/*", backendOrigin, {...});       // Origin4 (VPC) - The problematic origin

const keycloakOrigin = origins.VpcOrigin.withEc2Instance(backendInstance, {
  httpPort: 8081,
  // other configuration...
});
dist.addBehavior("/keycloak/*", keycloakOrigin, {...}); // Origin5 (VPC)

Later, a new requirement came to add a private S3 origin for signed URLs. Since it was related to file uploads, the natural place seemed to be right after the existing fileUploadOrigin:

// After the change
dist.addBehavior("/files/*", fileUploadOrigin, {...});         // Origin3
dist.addBehavior("/private-files/*", privateFileUploadOrigin, {...}); // ← New addition
dist.addBehavior("/api/*", backendOrigin, {...});              // Origin4 → Origin5!

This seemingly innocent insertion caused all subsequent origins to shift their indices:

Origin1: Default behavior (S3)
Origin2: /widget/* (S3)
Origin3: /files/* (S3)
Origin4: /private-files/* (S3) → Added
Origin5: /api/* (VPC) → Shifted from Origin4
Origin6: /keycloak/* (VPC) → Shifted from Origin5

The Catastrophic Result

When running cdk diff, the output showed that the VPC origin would be deleted and recreated:

[-] AWS::CloudFront::VpcOrigin CloudFrontDistributionOrigin4VpcOrigin11F5FD9D destroy
[+] AWS::CloudFront::VpcOrigin CloudFront/Distribution/Origin6/VpcOrigin CloudFrontDistributionOrigin6VpcOrigin9403AA18
[~] AWS::CloudFront::Distribution CloudFront/Distribution CloudFrontDistributionEAB06B35
  │  [+]  "PathPattern": "/private-files/*",
  │  [+]  "TargetOriginId": "AppStackCloudFrontDistributionOrigin407629CA8",
  │  [~]  "PathPattern": "/api/*",
  │  [-]  "TargetOriginId": "AppStackCloudFrontDistributionOrigin407629CA8",
  │  [+]  "TargetOriginId": "AppStackCloudFrontDistributionOrigin5FCA7D57C",
  │  [~]  "PathPattern": "/keycloak/*",
  │  [-]  "TargetOriginId": "AppStackCloudFrontDistributionOrigin5FCA7D57C",
  │  [+]  "TargetOriginId": "AppStackCloudFrontDistributionOrigin698A0EC33",
[~] AWS::CloudFront::VpcOrigin CloudFront/Distribution/Origin5/VpcOrigin CloudFrontDistributionOrigin5VpcOrigin3A40350A
 └─ [~] VpcOriginEndpointConfig
     └─ [~] .HTTPPort:
         ├─ [-] 8081   // Keycloak port (8081)
         └─ [+] 8080   // Shifted to API port (8080)

This shows that CloudFormation would:

Delete the existing VPC origin (Origin4)
Create a new S3 origin (Origin4)
Update the existing VPC origin (Origin5) to change its port from 8081 to 8080
Create a new VPC origin (Origin6)

Why the Deployment Failed

The deployment failed during step 3 above with this error:

Resource handler returned message: "Invalid request provided: AWS::CloudFront::VpcOrigin: 
The specified VPC origin is currently associated with one or more distributions. 
Please disassociate the VPC origin from all distributions before updating or deleting"

This error occurs due to a fundamental constraint of VPC Origins: You cannot update or delete a VPC origin while it's still associated with any CloudFront distribution. Unlike other CloudFront resources, VPC origins require a two-step process:

First: Remove the VPC origin from all distributions
Then: Update or delete the VPC origin resource

This constraint is documented in the AWS CloudFormation documentation and has been reported in the AWS CDK community (Issue #34629). The AWS documentation for "Updating VPC Origins" explicitly describes this limitation.

The core problem: When logical IDs change, CloudFormation treats the VPC origin as a completely different resource that needs to be updated in-place, triggering this constraint violation.

🔍 Understanding CDK's Logical ID Generation

The Core Mechanism

AWS CDK generates unique logical IDs for CloudFormation resources based on:

Construct Path: The hierarchical position within the CDK construct tree
Construct ID: The identifier within the parent construct
Hash: A consistent hash generated from path components

The logical ID generation is handled by the makeUniqueId function in CDK's core library:

// aws-cdk-lib/core/lib/names.ts
// https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/core/lib/names.ts#L48
public static uniqueId(construct: IConstruct): string {
  const node = Node.of(construct);
  const components = node.scopes.slice(1).map(c => Node.of(c).id);
  return components.length > 0 ? makeUniqueId(components) : '';
}

// aws-cdk-lib/core/lib/private/uniqueid.ts
// https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/core/lib/private/uniqueid.ts#L33
export function makeUniqueId(components: string[]) {
  components = components.filter(x => x !== HIDDEN_ID);

  if (components.length === 0) {
    throw new UnscopedValidationError('Unable to calculate a unique id for an empty set of components');
  }

  const hash = pathHash(components); // Generate hash from path components
  const human = removeDupes(components) // Generate human-readable part
    .filter(x => x !== HIDDEN_FROM_HUMAN_ID)
    .map(removeNonAlphanumeric)
    .join('')
    .slice(0, MAX_HUMAN_LEN);

  return human + hash; // Final logical ID
}

CloudFront Origin ID Generation

The critical piece is in CloudFront's addOrigin method:

// aws-cdk-lib/aws-cloudfront/lib/distribution.ts
// https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-cloudfront/lib/distribution.ts#L625
public addBehavior(pathPattern: string, origin: IOrigin, behaviorOptions: AddBehaviorOptions = {}) {
  if (pathPattern === '*') {
    throw new ValidationError('Only the default behavior can have a path pattern of \'*\'', this);
  }
  this.validateGrpc(behaviorOptions);
  const originId = this.addOrigin(origin);
  this.additionalBehaviors.push(new CacheBehavior(originId, { pathPattern, ...behaviorOptions }));
}
// https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-cloudfront/lib/distribution.ts#L685
private addOrigin(origin: IOrigin, isFailoverOrigin: boolean = false): string {
  const existingOrigin = this.boundOrigins.find(boundOrigin => boundOrigin.origin === origin);
  if (existingOrigin) {
    return existingOrigin.originGroupId ?? existingOrigin.originId;
  } else {
    // Calculate new origin index based on existing origins
    const originIndex = this.boundOrigins.length + 1;      
    // Create new construct with this index
    const scope = new Construct(this, `Origin${originIndex}`);
    // Generate unique ID from this construct
    const generatedId = Names.uniqueId(scope).slice(-ORIGIN_ID_MAX_LENGTH);
    // ... rest of the binding logic
  }
}

Here's what happens when you insert an origin:

// Before insertion
addBehavior("/api/*", backendOrigin, {...})
  ↓
originIndex = 4  // boundOrigins.length + 1
  ↓
new Construct(this, "Origin4")
  ↓
Logical ID: "CloudFrontDistributionOrigin4..."

// After insertion
addBehavior("/api/*", backendOrigin, {...})
  ↓
originIndex = 5  // Now boundOrigins.length + 1 = 5
  ↓
new Construct(this, "Origin5")
  ↓
Logical ID: "CloudFrontDistributionOrigin5..."

The key insight: Origin ordering directly affects the construct path, which determines the logical ID. When you insert an origin in the middle, all subsequent origins get new logical IDs, causing CloudFormation to treat them as entirely new resources.

💡 Practical Solution Patterns

Pattern 1: Defensive Ordering (Recommended)

Always add new origins at the end of the existing sequence:

// Good: New origins added at the end
// Current order (DO NOT CHANGE):
dist.addBehavior("/widget/*", widgetOrigin, {...});
dist.addBehavior("/files/*", fileUploadOrigin, {...});
dist.addBehavior("/api/*", backendOrigin, {...});
dist.addBehavior("/keycloak/*", keycloakOrigin, {...});

// ✅ Safe to add new origins here:
dist.addBehavior("/private-files/*", privateFileUploadOrigin, {...}); // Safe addition

// Document the ordering constraint
// IMPORTANT: Origin order affects logical IDs. Always add new origins at the end.
// Current order:
// Origin1: default (S3)
// Origin2: widgetOrigin (S3)
// Origin3: fileUploadOrigin (S3)
// Origin4: backendOrigin (VPC)
// Origin5: keycloakOrigin (VPC)

Pattern 2: Configuration-Driven Origins

Define origins in a structured configuration object:

interface OriginConfig {
  index: number;
  id: string;
  type: 'S3_OAC' | 'VPC' | 'CUSTOM';
  pathPattern?: string;
  // ... other properties
}

export const ORIGIN_DEFINITIONS: OriginConfig[] = [
  {
    index: 0,
    id: 'DefaultS3Origin',
    type: 'S3_OAC',
    // default behavior configuration
  },
  {
    index: 1,
    id: 'WidgetS3Origin',
    type: 'S3_OAC',
    pathPattern: '/widget/*',
  },
  {
    index: 2,
    id: 'FileUploadS3Origin',
    type: 'S3_OAC',
    pathPattern: '/files/*',
  },
  {
    index: 3,
    id: 'BackendVpcOrigin',
    type: 'VPC',
    pathPattern: '/api/*',
  },
  // New origins can be inserted anywhere by adjusting indices
];

// Apply origins in index order
function applyOrigins(distribution: cloudfront.Distribution) {
  const sortedOrigins = [...ORIGIN_DEFINITIONS].sort((a, b) => a.index - b.index);

  sortedOrigins.forEach((config) => {
    if (config.index > 0) { // Skip default behavior
      const origin = createOriginFromConfig(config);
      distribution.addBehavior(config.pathPattern!, origin, config.behaviorOptions);
    }
  });
}

Pattern 3: Logical ID Override (Advanced)

For critical resources, you can override logical IDs directly:

// Create a custom aspect to manage VPC origin logical IDs
export class VpcOriginLogicalIdOverride implements cdk.IAspect {
  private overrides = new Map<string, string>();

  addOverrideByOriginIndex(originIndex: number, logicalId: string): void {
    this.overrides.set(`origin${originIndex}`, logicalId);
  }

  visit(node: IConstruct): void {
    if (node.node.defaultChild && 
        (node.node.defaultChild as cdk.CfnResource).cfnResourceType === 'AWS::CloudFront::VpcOrigin') {

      const cfnVpcOrigin = node.node.defaultChild as cdk.CfnResource;
      const nodePath = node.node.path.toLowerCase();

      for (const [pathPattern, logicalId] of this.overrides) {
        if (nodePath.includes(pathPattern)) {
          cfnVpcOrigin.overrideLogicalId(logicalId);
          break;
        }
      }
    }
  }
}

// Usage
const vpcOriginOverride = new VpcOriginLogicalIdOverride();
vpcOriginOverride.addOverrideByOriginIndex(4, 'BackendVpcOrigin');
vpcOriginOverride.addOverrideByOriginIndex(5, 'KeycloakVpcOrigin');

cdk.Aspects.of(this).add(vpcOriginOverride);

⚠️ Important Note: This example uses hardcoded origin indices (4, 5) for simplicity. In production environments, you should implement dynamic index detection to avoid maintenance issues when origins are added or removed. Consider using configuration-driven approaches (Pattern 2) for better maintainability.

📝 Conclusion

The CloudFront origin ordering problem reveals a fundamental aspect of AWS CDK's design: logical IDs are derived from construct hierarchy and naming, making them sensitive to code organization changes.

Understanding this mechanism is crucial for building robust infrastructure as code. The key takeaways are:

Order matters: In CDK, the order of resource creation affects logical IDs
Plan for growth: Design your origin architecture with future additions in mind
Use defensive patterns: Always add new resources at the end unless you have explicit migration strategies
Document constraints: Make the ordering dependency explicit in your code
Test stability: Include tests that verify logical ID stability

By applying these patterns and understanding CDK's internal mechanisms, you can:

Avoid costly infrastructure disruptions
Build more maintainable cloud applications
Prevent unexpected resource recreation
Maintain service availability during deployments

📚 References

Community Discussions

💭 Have you encountered similar logical ID issues in your CDK projects?

Share your experiences and solutions in the comments below! If this article helped you avoid a deployment disaster, give it a ❤️ and follow for more AWS CDK deep dives.

🔄 Found this useful? Don't forget to share it with your team!

Top comments (2)

Masaki Okuda • Jun 12

ohh…

Thank you for the interesting information!!

Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more