Forem: Christian Nunciato

CDKTF is deprecated: What's next for your team?

Christian Nunciato — Thu, 18 Dec 2025 20:36:56 +0000

In July, 2020, CDK for Terraform (CDKTF) was introduced, and last week, on December 10, it was officially deprecated. Support for CDKTF has stopped, the organization and repository have been archived, and HashiCorp/IBM will no longer be updating or maintaining it, leaving a lot of teams out there without a clear path forward.

For most teams, that means it's time to start looking for a replacement.

It's an unfortunate situation to suddenly find yourself in as a user of CDKTF, but you do have options, and Pulumi is one of them. In this post, we'll help you understand what those options are, how Pulumi fits into them, and what it'd look like to migrate your CDKTF projects to Pulumi.

What are the options?

Teams migrating away from CDKTF generally have three options:

Option 1: Fall back to HCL

HashiCorp's official recommendation is to export your projects to HashiCorp Configuration Language (HCL) and manage them with Terraform. CDKTF even has a command that makes this fairly simple:

cdktf synth --hcl

Of course, if you're using CDKTF, you probably chose it specifically to avoid HCL. So while possible, this probably isn't the choice most teams would make unless they had to.

Option 2: Migrate to AWS CDK

If your team is all-in on AWS, another option would be to migrate to AWS CDK. It's widely used, officially supported, the programming model is similar to CDKTF's, and both CDK and CDKTF transpile to an intermediate format (CloudFormation YAML and Terraform JSON, respectively) that gets passed on to their underlying tools for deployment.

But while their programming and deployment models are conceptually similar, their resource models and APIs are entirely different. Here's the code for an S3 bucket written in AWS CDK, for example:

import * as s3 from 'aws-cdk-lib/aws-s3';

const bucket = new s3.Bucket(this, 'my-bucket', {
    bucketName: 'my-example-bucket',
    versioned: true,
    publicReadAccess: false,
});

And here's the code for a similarly configured bucket in CDKTF:

import { S3Bucket } from '@cdktf/provider-aws/lib/s3-bucket';

const bucket = new S3Bucket(this, 'my-bucket', {
    bucket: 'my-example-bucket',
    versioning: {
        enabled: true,
    },
    acl: 'private',
});

Notice how different these APIs are — and this is just one simple resource with only a few properties; imagine having to rewrite dozens or hundreds of them. Beyond that, there's also the problem of state: How would you go about translating the contents of a Terraform state file containing hundreds of resources into the equivalent CloudFormation YAML or JSON?

Despite their surface similarities, CDKTF and AWS CDK have little in common. Migration would essentially mean a ground-up rewrite that'd also leave you without the multi-cloud support you already have with CDKTF. For most teams, that makes this option a practical non-starter.

Option 3: Migrate to Pulumi

This is where we should acknowledge our obvious bias — but we genuinely believe that for most users of CDKTF, Pulumi really is the simplest and most broadly compatible way forward.

Like CDKTF, Pulumi lets you build and manage your infrastructure with general-purpose languages like TypeScript, Python, Go, C#, and Java, and it supports organizing your code into higher-level abstractions called components, which you can think of like CDKTF constructs. Both organize cloud resources into stacks (think dev, prod), and both track deployment state similarly, with local, remote, and cloud-hosted options available.

Many of Pulumi's most popular providers (e.g., the AWS provider) are also built from open-source Terraform schemas, which means their resource models will be nearly identical to what you're used to with CDKTF. Here's what an S3 bucket looks like in Pulumi, for example:

import * as aws from '@pulumi/aws';

const bucket = new aws.s3.Bucket('my-bucket', {
    bucket: 'my-example-bucket',
    versioning: {
        enabled: true,
    },
    acl: 'private',
});

You can also use any Terraform provider with Pulumi, and you can even reference Terraform modules directly from within your Pulumi code.

Pulumi is also different from CDKTF in several ways. One is that rather than transpile your source code to a format like JSON as CDKTF does (and then deploying it separately later), Pulumi uses its own declarative deployment engine that resolves the resource graph at runtime and provisions cloud resources directly, which is much faster and more flexible. You can learn more about the deployment model in How Pulumi Works.

Given the API similarities, the support for all Terraform providers and modules, the ability to coexist alongside Terraform-managed projects, and the built-in support for conversion (which we'll cover next), we think Pulumi is the best option for most teams looking to migrate.

What migrating to Pulumi looks like

Migrating a CDKTF project to Pulumi generally happens in three steps:

Conversion, which translates your CDKTF code into a new Pulumi program
Import, which reads the contents of your CDKTF state into a new Pulumi stack
Refactoring, which brings the code in the new program into alignment with the stack's currently deployed resources

Conversion and import

Migration starts with exporting your CDKTF project to HCL with cdktf synth. From there, Pulumi's built-in convert and import commands handle creating the new program and importing your state:

# Export your project to HCL.
cdktf synth --hcl

# Convert the HCL into a new Pulumi project.
pulumi convert --from terraform --language typescript

# Create a new Pulumi stack.
pulumi stack init dev

# Import your CDKTF stack's resources into your new Pulumi stack.
pulumi import --from terraform ./terraform.dev.tfstate

The converter automatically translates Terraform input variables, data sources, resources, and outputs into their Pulumi equivalents. You can read more about how this works in Converting Terraform HCL to Pulumi.

Refactoring

Once you've imported your state, you'll often have to make some adjustments to the code to bring it in line with the new Pulumi stack. For instance, pulumi import marks new resources protected by default, to prevent them from being accidentally deleted — but since the code produced by pulumi convert doesn't include the protect resource option, you'll need to add it yourself. Fortunately the import step also emits code that you can copy into your program to make this process a little easier.

Refactoring can get a bit more complicated when custom logic and higher-level abstractions are involved, as fidelity to the original CDKTF code is often lost in the translation to HCL. In these situations, having the help of an LLM to recapture that original logic or translate your CDKTF constructs into Pulumi components can be a big time-saver.

An end-to-end example

The best way to get a feel for how this works, though, is to try it yourself.

The pulumi/cdktf-to-pulumi-example repository on GitHub contains a CDKTF project with multiple stacks written in TypeScript, along with a guide that walks you through the process of migrating that project to Pulumi. The guide covers everything we've discussed here so far, including:

Converting the CDKTF project into a new Pulumi project
Importing its actively running resources into Pulumi stacks
Modifying the generated code to align with imported state
Performing an initial deployment with Pulumi to complete the migration process

The walkthrough takes only a few minutes to complete, and it's a great way to stand up an example of your own to get more familiar with Pulumi.

What's next?

If you’re moving on from CDKTF, there are a few possible paths forward. For teams that want to keep using real languages and avoid a ground-up rewrite, Pulumi offers the clearest way forward.

To learn more about how Pulumi works, how it differs from CDKTF and from Terraform, how to handle additional conversion scenarios, and more, we recommend:

Diving into the Pulumi docs to get familiar with core concepts and features of the platform
Reading Migrating from Terraform or CDKTF to Pulumi for more detailed, Terraform-specific migration guidance
Joining us in the Pulumi Community Slack to ask questions and learn from others who've successfully made the leap from Terraform and CDKTF to Pulumi

And of course, feel free to reach out! We'd love to help in any way we can.

Agentic CI with Buildkite: Three practical examples

Christian Nunciato — Wed, 03 Dec 2025 01:45:09 +0000

If you read our previous post about how AI is reshaping CI, then you're aware of the new building blocks we recently added to the Buildkite platform. We're calling these building blocks our agentic workflow components — composable primitives designed to give platform teams the tools they need to bring AI-assisted processes into their CI/CD workflows.

In this post, we'll show you how to use those components, with three simple-but-practical examples that use AI agents to solve common real-world problems, including:

Reviewing GitHub pull requests
Automatically fixing broken PR builds
Generating first-draft PRs based on Linear issues

Each example is backed by a fully functioning GitHub repository containing a Buildkite pipeline template that you can fork, set up, run, and easily adapt to the needs of your team.

Recapping the components

Before diving into the examples themselves, let's quickly recap the workflow components we introduced in our previous post, as you'll be using them in the examples that follow:

The Buildkite MCP server, which gives your agents fine-grained access to the Buildkite REST API through our specialized MCP server tools
Buildkite model providers, which let you connect to popular frontier models like Anthropic's Claude Code and others directly through Buildkite, either by using your own API credentials or with a hosted, Buildkite-managed key
Buildkite pipeline triggers, inbound webhooks that you can use to invoke a Buildkite pipeline in response to any external event with an HTTP request, with first-class support for popular services like GitHub and Linear
The Buildkite SDK, which lets you compose and generate pipeline definitions dynamically at runtime using general-purpose programming languages like JavaScript, TypeScript, Python, Go, and Ruby
A collection of plugins powered by Claude, Codex, Amazon Bedrock, and others that make it easy to use LLMs to annotate your CI jobs with rich build summaries

Together, these building blocks, in combination with the broader Buildkite platform, let you build flexible and adaptive workflows that bring AI agents into your CI/CD process on your own terms. Let's see how.

Example 1: GitHub code-review bot

We all know how important code review is to the delivery process — but it also takes time, and with AI agents producing more code than ever for us humans to review, well, let's just say we need all the help we can get to keep up with it. Having a capable AI agent at hand to help out with the occasional first-pass code review (whether for yourself or someone else) can be a significant time-saver — not to mention a way to surface the kinds of non-obvious bugs that can easily sneak past human reviewers.

Get the code

The code for this example is available on GitHub at buildkite-agentic-examples/github-code-review-bot.

How it works

This first example configures a Buildkite pipeline to listen for GitHub PR events (specifically labeled events) using a Buildkite pipeline trigger. Triggers are essentially inbound webhooks — unique URLs tied to a specific pipeline — which means you can add them to any GitHub repository to have GitHub run the pipeline in response to certain events. This particular pipeline runs an AI agent (Claude Code by default) that evaluates a PR and submits a code review as a GitHub comment.

All three of the examples we're covering in this post follow the same general pattern:

A label gets applied, triggering a Buildkite pipeline.
The pipeline runs a script that parses and validates the webhook payload.
The script appends a step to the running Buildkite pipeline to spawn an AI agent.
The agent completes the task.

The most interesting work happens in the handler script, scripts/handler.ts, a Node.js program written in TypeScript that uses the Buildkite and GitHub SDKs to run Claude Code with a task-specific prompt:

import { execSync } from "child_process";
import { Pipeline } from "@buildkite/buildkite-sdk";
import { Octokit } from "octokit";

// ...

// Generate the pipeline with the Buildkite SDK.
function generateCodeReviewPipeline(webhookPullRequestUrl: string, agentBuildUrl: string): string {
  const pipeline = new Pipeline();
  const tokenArgs = [`PullRequestURL=${webhookPullRequestUrl}`, `AgentBuildURL=${agentBuildUrl}`];

  pipeline.addStep({
    label: ":buildkite: Reviewing the code",
    commands: [...runAgent(tokenArgs)],
    plugins: {
      docker: {
        "image": "buildkite-agentic-example-tools:latest",
        "mount-checkout": false,
        "mount-buildkite-agent": true,
        "environment": [
          //...
          "TRIGGER_ON_LABEL",
          "MODEL_PROVIDER",
        ],
      },
    },
    //...
  });

  return pipeline.toYAML();
}

async function main() {
  // Fetch the incoming payload from Buildkite.
  const event = buildkiteAgent("meta-data", "get", "buildkite:webhook").trim();
  const payload = JSON.parse(event);

  // ...

  // Exit unless the payload has a label matching the one we're listening for.
  const labelName = payload.label.name;
  if (labelName !== process.env.TRIGGER_ON_LABEL) {
    console.log(`Label is not '${process.env.TRIGGER_ON_LABEL}', exiting`);
    process.exit(0);
  }

  // ...

  // Generate and upload a new pipeline step to run the AI agent.
  const pipelineYaml = generateCodeReviewPipeline(pullRequestUrl);
  const uploadProcess = execSync("buildkite-agent pipeline upload", {
    input: pipelineYaml,
    encoding: "utf-8",
  });
}

main().catch(error => {
  console.error("Error:", error.message);
  process.exit(1);
});

The step runs Claude in a Docker container (using the Buildkite docker plugin) that includes only the tools Claude needs for this particular task: Node.js, the GitHub CLI, the local version of the Buildkite MCP server, the Claude Code CLI, and the necessary scripts, prompts, and environment variables. Running the agent in a container like this isn't technically necessary — it just adds some additional isolation and safety over running Claude Code directly on your filesystem. (The container itself is built and tagged locally at runtime using an agent lifecycle hook. See .buildkite/hooks/post-checkout for details.)

Inside the container, Claude clones the PR's GitHub repository, checks out the PR branch, analyzes the change, and posts a review back to the PR as a comment, annotating the Buildkite build (using the MCP server's annotation tooling) as it goes.

All settings — including the model provider and label — are configurable in .buildkite/pipeline.yml:

secrets:
  GITHUB_TOKEN: GITHUB_TOKEN
  BUILDKITE_API_TOKEN: API_TOKEN_BUILDKITE

env:
  GITHUB_CLI_VERSION: "2.83.0"
  BUILDKITE_MCP_SERVER_VERSION: "0.7.3"
  TRIGGER_ON_LABEL: "buildkite-review"
  MODEL_PROVIDER: "anthropic"

steps:
  - label: ":node: Generate the pipeline"
    command: |
      # Generate and upload the pipeline to handle the webhook.
      echo "--- :webhook: Run the webhook handler"
      npm install && npm run build
      node dist/handler

One last thing to point out (as it's easy to miss) is that the claude CLI isn't running in the usual way, communicating with the Anthropic API as it normally would. Instead, it's using a Buildkite-managed model-provider endpoint that proxies the Anthropic API for you. Open up scripts/claude.sh and you'll see that the two environment variables Claude uses to configure its backend are being set using Buildkite environment variables:

#!/bin/bash
# ...

# Set up Buildkite Hosted Models
export ANTHROPIC_BASE_URL="$BUILDKITE_AGENT_ENDPOINT/ai/anthropic"
export ANTHROPIC_API_KEY="$BUILDKITE_AGENT_ACCESS_TOKEN"

# ...

echo "--- :robot_face: Starting Claude Code"
echo "$prompt" | claude -p --mcp-config mcp.json

These two values are being set with pipeline environment variables applied automatically by Buildkite at runtime. Together, they give you a seamless way to use Claude — or any supported model provider — in your pipelines without having to provide (and then expose, and manage) your own Anthropic credentials. See the model providers documentation for details.

The following diagram shows the operative components and how they come together:

See the example's README for complete setup instructions and configuration details.

Once you're up and running (which takes only a few minutes — the README walks you through it), you'll have a fully functioning code-review bot that you can call up by adding a buildkite-review label to any pull request — and a reusable Buildkite pipeline and workflow that you can adapt as you like to the needs of your team.

Now let's have a look at the next example.

Example 2: GitHub PR build fixer

I don't know about you, but every now and then, I'll have a PR build fail, and I'll have no idea why. Often the failure has something to do with a linter — although the logs may or may not make this clear, as linters are famous for failing in totally obscure ways that make no sense at all.

☝️ Me, when this happens.

In situations like these (and in other, more complex ones), it can be nice to be able to reach for an AI agent for a hand in diagnosing and fixing the issue.

Get the code

The code for this example is available on GitHub at buildkite-agentic-examples/github-pr-build-fixer.

How it works

This example follows the same high-level pattern as before:

You create a new Buildkite pipeline and trigger with the example, adding the trigger as a GitHub webhook to whichever repositories you'd like Claude to fix for you.
When a PR build fails, you add the appropriate label to it — in this case, buildkite-fix (although as before, this is configurable in ./buildkite/pipeline.yml).
GitHub invokes the webhook, triggering the pipeline, which evaluates the webhook payload and adds a step that runs Claude in a Docker container.
Claude uses the Buildkite MCP server to query the logs, finds the root cause, clones the repo, implements a fix, and pushes a new branch containing the fix to GitHub.
Claude makes a new PR (on the original, still-broken PR), waits for the Buildkite build to pass (iterating if necessary), and posts a summary comment on the original PR explaining the fix.

If the PR looks good, you click the merge button, pull in the fix, and get on with your day.

Here's how this looks architecturally:

Full setup and configuration details are in the README.

I'd also encourage you to have a look at the prompt for this example, which you'll find at prompts/user.md. You may want to make some adjustments to it to align with whatever standards or guidelines you use on your team.

Now let's turn to our last example.

Example 3: Linear issue handler

Linear has become an incredibly popular tool for managing software projects. (We even use it ourselves here at Buildkite.) If you've used Linear, you know how it works, but if you haven't, it's essentially an issue tracker — much like Jira, Trello, GitHub Issues, or others you've likely used before.

Issue trackers are at the center of how most of us organize our work, and at any given moment, I might have dozens or hundreds of issues assigned to me personally. Some are simple and well-defined — dependency bumps, framework upgrades. Others are much more complex, requiring deeper analysis to come up with more substantive estimates.

Fortunately, both types can often be handled (in whole or in part) with a good LLM, given a good-enough issue description and prompt as a starting point.

Get the code

The code for this example is available on GitHub at buildkite-agentic-examples/linear-issue-handler.

How it works

Hopefully by now, the pattern is clear: You set up a pipeline, give it a trigger, wire up the trigger to be invoked in response to some external event, and your agent of choice takes care of the rest. Here, the goal is to have Linear kick off a Buildkite pipeline in response to an issue-label event.

Once the trigger is set up in your Linear project — see the README for how to do that — whenever you add the buildkite-analyze label to an issue, the pipeline runs scripts/handler.ts, extracts the issue details, discerns from the issue description which GitHub organization and repository the issue refers to, and generates a pipeline step that runs Claude as it does in the other examples.

In this one, Claude analyzes the codebase as well, and makes a judgment call (much as a human would) on the level of complexity involved. For simpler issues, Claude will go ahead and implement a fix, opening a PR and commenting back on the Linear issue with a summary and link to it. For more complex issues, it'll render its findings, suggest possible approaches, etc., and post an analysis back to the Linear issue instead.

As with the other examples, all of the work performed by the LLM is done within a Docker container, which includes the Linear CLI here to give Claude more controlled access to Linear functionality.

What makes these work

Stepping back, these simple yet practical examples all reflect the core principles and foundational tools we think teams need in order to bring agentic processes into their CI workflows:

Composable primitives that enable flexible, adaptive CI workflows
Built-in access to frontier models with minimal configuration
Pipeline triggers that let you extend your Buildkite pipelines to external services more easily
A highly configurable, performance-focused MCP server that gives your agents fine-grained access to Buildkite resources
A multi-language SDK that lets you build pipelines intelligently and dynamically at runtime

... and more. Each component is individually useful, but taken together they enable agentic workflows that can branch, adapt, fan out, etc., based on what the agent discovers at runtime — something statically written YAML-based pipelines can't really do.

Next steps

The best way to get a feel for these components is to try them yourself. All you need are a Buildkite account, a GitHub personal access token, a Buildkite API access token, and a Linear API key for the Linear example, and each one takes only a few minutes to set up.

Follow the instructions in their respective READMEs to get started:

To learn more about the each of these components, see their documentation:

Have fun! We can't wait to see what you build. 🙌

Automating Jenkins with Configuration as Code (JCasC)

Christian Nunciato — Fri, 20 Jun 2025 17:56:41 +0000

When you've spent as much time as I have in the world of infrastructure-as-code (IaC), you develop a bit of an allergy to anything that looks like click-ops.

There's nothing inherently wrong with click-ops, of course; sometimes it's just what you need. But in general, if I'm standing up infrastructure, especially on a team, I'm going to reach for something like Pulumi, Terraform, or Ansible to help. To me, that's just how it's done — there's no better way to get the reliable, repeatable results I need without being able to run an automated, code-driven process to produce them.

Which is why I was surprised to learn that in the Jenkins world, clicking around in the UI to configure the controller, plugins, agents, etc., isn't just common — it's still largely the norm.

Fortunately for IaC-heads like me, there's an answer. Jenkins Configuration as Code (JCasC) is a Jenkins plugin that lets you define your Jenkins configuration entirely in code, specifically YAML, that can be version-controlled, tested, and applied just as you would with any other IaC tool. And it's actually quite simple to use — once you understand the basics.

So in this hands-on post, we'll dive into JCasC with an example that sets up a Jenkins controller from scratch, complete with global settings, plugins, inbound agents, and a working pipeline wired up to a GitHub repository, all configured in code. To make it easy to spin up and work on, we'll also use Docker and Docker Compose, but the concepts apply no matter what you're deploying to — whether that's a Kubernetes cluster, a fleet of cloud-hosted VMs, or a stack of Raspberry Pis on your desktop.

What is JCasC and how does it work?

Like all things Jenkins, JCasC is a plugin. But it's a special one — it's a plugin for managing Jenkins itself, meant to allow you to spin up and configure a Jenkins cluster without having to sign into the Jenkins UI. Chances are, if you can configure it in the UI, you can accomplish the same thing with JCasC in YAML.

To make it work, you'll need a few things:

An instance of Jenkins. Specifically, you'll need to be able to start an instance of Jenkins somehow, as JCasC configuration is applied on startup. (Containers, system services, and CLI commands are all fine.)
A JCasC configuration file — a YAML file. You can name this file anything you like, but the convention is to name it jenkins.yaml. We'll walk through what goes into this file in this post.
To tell Jenkins where this file is located when you start it up. This is generally done with an environment variable. You'll see some examples of this later as well.

Technically that's all there is to it — but if you'd like to install plugins automatically as well (which you almost surely will), you'll also need:

A file containing a list of plugins to install on the Jenkins controller. You can name this file anything as well, but it's usually called plugins.txt. Installing plugins this way is immensely valuable in that it not only lets you specify the plugin versions you'd like to install (one of many Jenkins plugin best practices), but also track changes to your plugin versions over time, test them before you deploy, and keep them all configured consistently across your organization.
The Jenkins plugin CLI, which is how you'll install the plugins on that list. There are other ways to do this, but the CLI is the standard.

The beauty of this approach is that it lets you configure not just one instance of Jenkins, but as many as you like, and to keep your production Jenkins clusters stable until you're ready to upgrade them with well-tested replacements.

All right, enough talk. Let's take a look at the actual files to see how this works.

Diving into a JCasC example

Start by cloning the example repository, which you'll find on GitHub at cnunciato/jenkins-jcasc-example:

gh repo clone cnunciato/jenkins-jcasc-example
cd jenkins-jcasc-example

Once you've done that, you should see the following layout:

.
├── app # A simple Node.js app that we're using for testing.
│ ├── index.js
│ ├── index.test.js
│ ├── package-lock.json
│ └── Jenkinsfile # The Jenkinsfile that builds and tests the app.
├── Dockerfile.controller # The Dockerfile for the Jenkins controller.
├── Dockerfile.agent # The Dockerfile for the Jenkins agents.
├── docker-compose.yml # The service definitions for the controller and agents.
├── agent.sh # The setup script for the agents.
├── jenkins.yaml # 👈 The JCasC configuration file.
└── plugins.txt # 👈 The list of plugins to be installed on the controller.

There are several files here, but the two most important ones are jenkins.yaml and plugins.txt.

Configuring the controller with `jenkins.yaml`

The jenkins.yaml file is what'll define most of your Jenkins controller's configuration, so where you'll spend most of your time. Open that file in your editor of choice, and let's walk through each section to get a sense of what it does.

Basic build settings

jenkins:
  mode: NORMAL
  numExecutors: 0

These settings — mode: NORMAL and numExecutors: 0 — tell Jenkins to accept any build job assigned to it and to delegate those jobs to connected agents instead — i.e., to run zero jobs on the controller itself. It's a common best practice for several reasons:

Running jobs in on agents, as opposed to on the controller, reduces the risk of those jobs affecting the performance stability of the controller (and of exposing sensitive data).
Pushing work off to agents keeps the controller's CPU and memory focused on managing the build queue, handling front-end requests, and coordinating workloads rather than competing with them.
Agent-only execution makes horizontal scaling more straightforward. To handle more load, you can add more agents without having to upgrade the controller.

The default mode is NORMAL, so this particular setting is optional, but in the spirit of configuration as documentation, it's a good idea to be explicit about it so that others understand it's intentional.

Provisioning users

jenkins:
  # ...
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: ${JENKINS_ADMIN_USERNAME}
          password: ${JENKINS_ADMIN_PASSWORD}
        - id: ${JENKINS_AGENT_USERNAME}
          password: ${JENKINS_AGENT_PASSWORD}

Here, the securityRealm block tells Jenkins to configure two users on the controller:

An admin user, so you can sign into the Jenkins UI without having to copy and paste the generated password from the startup logs
An agent user to allow build agents to self-register (which you'll see later)

The allowsSignup: false setting keeps users from being able to create accounts on the controller by visiting ${JENKINS_URL}/signup. Like mode: NORMAL, it's also the default, so technically optional — but again, a good one to be explicit about as well.

Setting roles and permissions

jenkins:
  # ...
  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: admin
            permissions:
              - Overall/Administer
            entries:
              - user: ${JENKINS_ADMIN_USERNAME}
          - name: agent
            permissions:
              - Overall/Read
              - Agent/Connect
              - Agent/Disconnect
              - Agent/Build
            entries:
              - user: ${JENKINS_AGENT_USERNAME}

Surprisingly, Jenkins has no support for authorization out of the box; the authorizationStrategy block has just two modes: unsecured, meaning anyone with access to the controller can use it without restriction (i.e., everyone's a fully anonymous administrator), and loggedInUsersCanDoAnything, meaning anyone with an account can act as an administrator. Both are far too permissive to be useful in production, so to secure your controllers properly, you'll need to look to a plugin.

The role-strategy plugin is a popular choice — and of course, it's configurable with JCasC. Here, we're using that plugin to:

Define an admin role with full privileges and apply it to the admin user
Define an agent role with narrower privileges (allowing it to connect, self-register, and run builds) and apply it to the agent user

To learn more about the role-strategy plugin, see its documentation.

Configuring agents

jenkins:
  # ...
  nodes:
    - permanent:
        name: agent1
        launcher: inbound
        remoteFS: /home/jenkins/agent
    - permanent:
        name: agent2
        launcher: inbound
        remoteFS: /home/jenkins/agent

The nodes block configures the controller to support two permanently connected Jenkins agents using the inbound connection type. Here, agents connect to the controller, rather than the other way around, as they would with the ssh or command types. It's a flexible pattern that allows agents to spin up, connect, run builds, and terminate without Jenkins needing to be able to reach them directly.

Installing build tools

tool:
  nodejs:
    installations:
      - name: Node 22.x
        properties:
          - installSource:
              installers:
                - nodeJSInstaller:
                    id: 22.16.0

The tool block lets you configure the build tools that you use in your pipelines. This particular block — nodejs — automates the provisioning of Node.js on all build agents that need it. It tells Jenkins that when an agent connects, the agent should download and install Node version 22.16.0 from npm if it hasn't already.

Configuring a pipeline job and GitHub repo

jobs:
  - script: |
      multibranchPipelineJob('jenkins-jcasc-example') {
        branchSources {
          github {
            id('jenkins-jcasc-example')
            repoOwner('cnunciato')
            repository('jenkins-jcasc-example')
            scanCredentialsId('github-pat')
          }
        }
        factory {
          workflowBranchProjectFactory {
            scriptPath('app/Jenkinsfile')
          }
        }
        configure { node ->
          def traits = node / sources / data / 'jenkins.branch.BranchSource' / source / traits
          traits << 'org.jenkinsci.plugins.github__branch__source.BranchDiscoveryTrait' {
            strategyId(3)
          }
        }
      }

The jobs block uses the Job DSL, a Groovy-based API for creating Jenkins jobs declaratively. Here, we're defining a single multibranch pipeline job that:

Fetches this blog post's example repository from GitHub (using credentials we'll define in a moment)
Locates the Jenkinsfile of the app to be built in the repository (in the app folder)
Configures a branch-discovery trait that builds all branches of the repository — namely strategy 3, a magic number that means "regular and PR branches", where strategies 1 and 2 mean "exclude PR branches" and "only PR branches," respectively

Configuring shared GitHub credentials

credentials:
  system:
    domainCredentials:
      - credentials:
          - usernamePassword:
              id: github-pat
              username: "${GITHUB_USERNAME}"
              password: "${GITHUB_TOKEN}"
              description: "${GITHUB_USERNAME}'s personal access token"

Finally, while we don't necessarily have to configure GitHub credentials for this example, since the job we've configured pulls from a public repository (and we're only running it locally), in most cases, you'll want to do so, both to be able to access private repositories and to avoid running up against GitHub's fairly restrictive API limits.

The credentials block defines a shared, reusable Jenkins credential named github-pat (for personal access token) using environment variables that you specify at startup time. As you likely noticed above, the jenkins-jcasc-example job uses this credential to fetch the associated repository from GitHub.

Discovering more configuration settings

It's important to note that there are many more settings that you can configure with JCasC and jenkins.yaml than the few we've covered here — and figuring out what they are can be rather challenging (and require a good deal of trial and error). Two suggestions I'll offer to help you save time:

Consult the configuration-as-code-plugin repository. Most of the settings you'll be interested in are covered in the demos folder of that repository, including those of commonly used plugins like the ones we've used here. If you're stuck, have a look around, and there's a good chance you'll find what you're looking for there.
Cheat — with the Jenkins UI. In addition to letting you write the configuration of a Jenkins controller, the JCasC plugin also allows you to read it — an incredibly handy way to discover settings you didn't know existed. In Manage Jenkins > Configuration as Code, there's a View Configuration button that surfaces everything as one big YAML blob, plugin settings and all, that you can paste right into jenkins.yaml and apply. If you're not sure how to configure something, try setting it up first with the UI, then click that button to see what changed, and copy out what you need. Having a Docker-based setup like this one makes this approach especially easy.

Configuring plugins with `plugins.txt`

I won't spend too much time on this topic, since it's technically independent of JCasC (and there's not all that much to it anyway) — but automating the installation of plugins along with the configuration of the server itself is such a common practice alongside JCasC that it's worth covering here as well.

The plugins.txt file simply contains the names of the plugins you'd like to install on the controller:

ansicolor # Support for colorized build logs
configuration-as-code # 👈 The core JCasC functionality
github-branch-source # Support for multibranch GitHub
job-dsl # Declarative job definitions
nodejs # Support for Node.js
workflow-aggregator # Core pipelines functionality
workflow-multibranch # Multibranch pipeline features

The snippet above uses unpinned versions, which implicitly tells Jenkins to fetch and install the latest version of each plugin. This is a fairly risky approach, given how easy it'd be for one or more of these plugins to fall out of compatibility with the Jenkins controller (or with one another) and break your pipeline, so most of the time, you'll want to pin each plugin to a specific version for stability and repeatability.

You can do this easily by adding the version after the plugin name (which you can get from the Jenkins Plugins Index):

ansicolor:1.0.6
configuration-as-code:1971.vf9280461ea_89
github-branch-source:1822.v9eec8e5e69e3
job-dsl:1.93
nodejs:1.6.4
role-strategy:777.v4fe2599cb_f48
workflow-aggregator:608.v67378e9d3db_1
workflow-multibranch:806.vb_b_688f609ee9

Maintaining this list can be tedious, and there doesn't seem to be a way to specify an acceptable range (only concrete versions), but it's much safer than simply relying on the latest.

That covers the two files — jenkins.yaml and plugins.txt — that govern most of your ability to configure a Jenkins server from scratch.

You might still be wondering, though: How exactly do you instruct Jenkins to use these two files?

Spinning up the cluster with Docker Compose

The best way to see how this works is to spin it up and have a look under the hood.

Now open a terminal, and assuming you have Docker installed (and the Docker daemon running), you should be able to bring everything up with a single command:

docker-compose up

Now browse to http://localhost:8080 and sign in with the configured username and password — both of which are admin, as specified in docker-compose.yml:

You can even trigger a build and see the connected agents pick it up and run it:

Defining controller and agent services

In docker-compose.yml, you'll see three container definitions: jenkins, agent1, and agent2. There's a bunch of other YAML in there, but it's mostly environment variables and defaults, so I'll assume you know your way around that stuff and just focus on the relevant lines:

services:
  # The controller.
  jenkins:
    build:
      context: .
      dockerfile: Dockerfile.controller
    ports:
      - "8080:8080"

  # An agent.
  agent1:
    build:
      context: .
      dockerfile: Dockerfile.agent
    depends_on:
      - jenkins

  # Another agent.
  agent2:
    build:
      context: .
      dockerfile: Dockerfile.agent
    depends_on:
      - jenkins

In the jenkins service's referenced Dockerfile, you'll see that it pulls in jenkins.yaml and plugins.txt, then runs the jenkins-plugin-cli (which is included in the Jenkins container image) to install all plugins, baking them into the container image:

FROM jenkins/jenkins:jdk21

# Copy in jenkins.yaml and plugins.txt.
COPY jenkins.yaml /usr/share/jenkins/ref/jenkins.yaml
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt

# Install all plugins with jenkins-plugin-cli.
RUN jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt

# Disable the setup wizard and tell JCasC where the config file is located.
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"
ENV CASC_JENKINS_CONFIG="/usr/share/jenkins/ref/jenkins.yaml"

The last two environment variable settings are the key to making this work:

JAVA_OPTS tells Jenkins to skip running the setup wizard (you no longer need it — you're rolling with JCasC now)
CASC_JENKINS_CONFIG tells the JCasC plugin where to find jenkins.yaml

From there, Docker runs the default entrypoint to start the Jenkins service.

In a non-Docker-based environment, it's conceptually the same, just slightly different in that you'll need to fetch the Jenkins package and the Jenkins Plugin Manager individually. On Ubuntu, that'd look something like this:

#!/bin/bash
set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

# Install Jenkins dependencies.
apt-get update
apt-get install -y wget gnupg curl openjdk-21-jdk

# Install Jenkins.
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/" | tee /etc/apt/sources.list.d/jenkins.list > /dev/null
apt-get update
apt-get install -y jenkins

# Make sure jenkins.yaml and plugins.txt and exist.
ls -al /usr/share/jenkins/ref/jenkins.yaml
ls -al /usr/share/jenkins/ref/plugins.txt

# Download the Jenkins Plugin Installation Manager.
wget -O /tmp/jenkins-plugin-manager.jar https://github.com/jenkinsci/plugin-installation-manager-tool/releases/download/2.12.13/jenkins-plugin-manager-2.12.13.jar

# Run the plugin manager to install the plugins listed in plugins.txt.
java -jar /tmp/jenkins-plugin-manager.jar \
  --war /usr/share/java/jenkins.war \
  --plugin-file /usr/share/jenkins/ref/plugins.txt \
  --plugin-download-directory /var/lib/jenkins/plugins

# Set required environment variables.
export JENKINS_HOME=/var/lib/jenkins
export JAVA_OPTS="-Djenkins.install.runSetupWizard=false"
export CASC_JENKINS_CONFIG="/usr/share/jenkins/ref/jenkins.yaml"

# Start Jenkins.
# systemctl start jenkins # Or just `jenkins`, to start.

Either way, that's all there is to it: a jenkins.yaml file, a plugins.txt file, a CLI command to load your plugins, and two environment variables, and your Jenkins controller is fully configured — no need to sign into the Jenkins UI to configure a thing.

Now let's have a last look at how the agent setup and self-registration work.

Configuring agent self-registration

If you open the agents' Dockerfile, you'll see that it's built from jenkins/inbound-agent, but it's using a shell script, agent.sh, as the entrypoint, rather than the default:

FROM jenkins/inbound-agent:jdk21

USER root
RUN apt-get update && apt-get install -y curl
USER jenkins

COPY agent.sh /agent.sh
ENTRYPOINT ["/agent.sh"]

That's because in order to connect to the Jenkins controller, an agent needs to supply a secret — and those secrets are generated dynamically (and uniquely for each agent) by the controller. You can view them in the Jenkins UI, and you can copy from the UI into the terminal when you fire up an agent — but who wants to do that? The whole point of JCasC, after all, is to avoid having to click around in the Jenkins UI and paste things into a terminal.

Fortunately, there's a way to retrieve the generated secret directly from the controller, which is exactly what agent.sh does. The script calls the jenkins-agent.jnlp endpoint (using the credentials we specified for the agent account earlier) and parses the response with sed to pull out the secret value before setting it as the JENKINS_SECRET environment variable (along with a few others):

#!/bin/bash
set -e

echo "Waiting for Jenkins to start..."
sleep 15

echo "Registering '${JENKINS_AGENT_NAME}' with the controller..."

# Retrieve the agent secret from its metadata endpoint.
AGENT_ENDPOINT="${JENKINS_URL}/computer/${JENKINS_AGENT_NAME}/jenkins-agent.jnlp"
AGENT_METADATA="$(curl -s -X GET -u "${JENKINS_AGENT_USERNAME}:${JENKINS_AGENT_PASSWORD}" $AGENT_ENDPOINT)"
AGENT_SECRET="$(echo $AGENT_METADATA | sed "s/.*<application-desc><argument>\([a-z0-9]*\).*/\1\n/")"

echo "Starting '${JENKINS_AGENT_NAME}'..."

export JENKINS_SECRET="$AGENT_SECRET"
export JENKINS_AGENT_NAME="$JENKINS_AGENT_NAME"
export JENKINS_URL="$JENKINS_URL"

exec jenkins-agent

And with that, you're good to go: You now have a fully functioning example of configuring Jenkins completely from scratch, including plugins, self-registering agents, and a Docker-based setup to make development and testing a little less painful.

Wrapping up

We've covered a lot, and hopefully we've given you a solid understanding of how JCasC works and how to use it. There's much more to learn — we've just scratched the surface — but what's here should give you a decent foundation to build onto.

To keep the learning going, you might want to check out:

The Jenkins Configuration as Code sub-project
The Configuration as Code page of the Jenkins Handbook
The jenkinsci/configuration-as-code repository on GitHub — especially the demos folder

And of course, keep on tinkering! You'll find our example on GitHub at cnunciato/jenkins-jcasc-example.

Understanding the SLSA framework

Christian Nunciato — Fri, 21 Mar 2025 19:30:15 +0000

If writing secure software is difficult, securing software build pipelines and supply chains can be even more challenging. The Software Supply Chain Levels for Software Artifacts (SLSA) has emerged in response to recent large-scale public attacks on digital infrastructure. It provides an industry-wide structured approach to fortifying software supply chains against threats, and is designed to evolve to address new threats as they arise.

If you’re looking for assurances that the open-source software you’re using is secure, find yourself needing to assess the security of your software vendors, or want to protect your internal systems and software from supply chain attacks, you'll likely want to learn more about SLSA.

What is SLSA?

Pronounced “salsa”, SLSA is an open-source framework that helps you secure your software delivery pipelines to better protect yourself and your users from serious risks. SLSA is based on security practices developed internally at Google and has been developed in public since 2021. It was initially released as a response to the massive security breach now known as the SolarWinds attack (which we’ll cover below).

SLSA is stewarded by the Open Source Security Foundation (OpenSSF) and governed by a diverse industry-wide group of specialists. As a relative newcomer on the security scene, its evolving definition targets threats to software supply chains—something not covered well by existing efforts like NIST, SSDF, and PBOM.

How SLSA works

SLSA provides documentation, automated tooling, and in-depth guidelines to help teams evaluate and communicate the security level of software artifacts and their dependencies.

SLSA advocates for evaluating each software artifact independently; it does not endorse transitive trust. The SLSA level of your dependencies and build tooling helps inform the security of your software artifacts without guaranteeing it.

Practitioners can self-evaluate or audit others, and you should be able to use SLSA’s language to quickly and effectively communicate about how secure your software is, what you’re expecting from third parties (such as software vendors and OSS), and the security of the software you’re providing to your customers.

SLSA tracks and levels

SLSA evaluations are organized along tracks and levels. Each track in SLSA addresses a different aspect of the software supply chain, and levels in a track indicate an artifact’s maturity therein. The currently published version (release 1.0) focuses on the “Build” track, and details four levels, from 0 to 3. Future releases will introduce additional levels and tracks (under active consideration: "Source" and "Platform Operations").

The levels within the SLSA Build track are designed to classify artifacts based on how secure the build pipeline is, and provide requirements for build systems.

Level 0: No assurances. The baseline level for software artifacts with no security measures in place.
Level 1: Basic security and provenance (protects against accidents). This level focuses on documentation and consistency. Builds are consistent—artifacts only change when their source code changes. Artifacts are tagged with documentation about how they were built (i.e., provenance), and this documentation is accessible for audits.
Level 2: Enhanced security enforcement (protects against common attacks). This level focuses on authenticity and preventing post-build artifact tampering. Software artifact provenance should be signed, and builds use tamper-resistant platforms.
Level 3: Comprehensive security controls (protects against concerted attacks). This level focuses on tamper prevention during the build process. Artifact provenance and documentation should be made hard to forge. Build systems compliant with this level provide advanced protections, such as stringent access controls and a secured and hardened environment.

A proposed Level 4 will likely focus on aspects like the consistency and integrity of build environments and require adopting more complex protective measures such as hardware attestation and reproducible builds.

Core principles

The Build track levels above embody SLSA’s three core principles:

Trust platforms, verify artifacts

Validating platforms is time-consuming and difficult. SLSA encourages adopters to minimize the number of platforms used in a build system and to prefer shared and hosted platforms over self-managed ones. This makes auditing and securing platforms more efficient since platform users can coordinate and share related costs. For artifact verification, SLSA is biased toward automatic and rapid verification, and provides some standard tools and guidelines for the work.

Trust code, not individuals

Individuals are difficult to vet and audit—and even vetted and trusted individuals can make mistakes, be temporarily compromised, or unintentionally contribute to malicious updates in other ways. SLSA believes it's more efficient to audit, validate, statically analyze, and develop trust in written code. Code changes can be evaluated more efficiently—piecemeal, on a per-change or per-update basis.

Prefer attestations over inferences

Securing and trusting systems is difficult—systems may operate as expected but still produce compromised artifacts. SLSA is built around the idea that it is much more efficient and easy to check each artifact and audit its build process than to attempt to (automatically) infer whether an artifact is secure by auditing and analyzing the systems and people involved in building it.

How SLSA is used today

Companies may use SLSA to assess their internal infrastructure, highlight security status, and identify gaps. Or they might use it to audit software vendors, suppliers, and external dependencies. At a high level:

Participation and use of SLSA are voluntary.
There isn’t (yet) a formal certification process for SLSA. Companies may self-assess or run audits for each other.
OSS software, build platforms, and industry insiders are gradually documenting their compliance levels and other contributions to supply chain security to build trust—e.g., Buildkite, Chainguard Academy, Eclipse Temurin, Tekton, and ActiveState.

Key terms

SLSA uses some specialized language and terminology. The most important concepts and terms include:

Software artifact: A file or bundle resulting from a build process. Examples include binaries, container images, and software packages.
Software supply chain: All the steps and activities used to make and deliver software artifacts. This includes writing code and using libraries and tools produced by others, code repositories, build systems, package managers, and deployment mechanisms.
Auditability: The capacity to know about, inspect, and verify each step in a supply chain, usually to check compliance with security practices and standards.
Attestation: Machine-readable and cryptographically signed data associated with a specific software artifact. It usually provides security-related context, such as what the artifact contains, how it was produced, and how it might be used.
Provenance: An attestation that describes how, where, and by whom an artifact was produced, and which other artifacts were involved in its production.
Build process integrity: The extent to which a build process hasn't been tampered with.
Reproducibility: The ability to produce identical artifacts from the same source code and build environment.
Hermetic builds: Builds that are entirely self-contained, consistent, and 100% reproducible. They do not depend on external factors like network access or the state of the build environment.
Two-person review: A process where at least two individuals review—and then verifiably sign off on—change proposals to a system before they are incorporated.

What threats is SLSA designed to address?

SLSA aims to protect supply chains against a full range of pervasive and sophisticated threats. It can guide your defense from vulnerabilities introduced through dependencies, build system compromises, code injection, and various insider threats.

Real-world supply chain attack examples

Below are some examples of the kinds of attacks SLSA addresses today. (Over time, these will expand.)

Build process compromise

The now-canonical example of a supply chain attack, the SolarWinds hack from 2020, exposed the systems and data of more than 18,000 public, private, government, military, and other highly sensitive organizations. Attackers compromised the build processes at multiple companies, including SolarWinds, which allowed them to issue a malicious update to SolarWinds’s Orion IT software. The update was quickly installed worldwide and provided hackers with significant systems and data access. This hack prompted the creation of the SLSA framework, and its Level 2 and Level 3 build-process controls are designed to prevent and mitigate future similar attacks.

Malicious artifact injection

In an example from 2021, attackers gained access to Codecov’s software delivery systems and were able to replace one of Cocodev’s tool offerings with a malicious artifact. Most of Codecov’s 23,000 clients used the tool in automated scripts as part of their software build chains and were compromised in the attack.

The initial security breach would have been mitigated with the security controls SLSA recommends for build processes. The malicious artifact injection was ultimately detected through a basic provenance check (checking the SHA signature of downloaded software against the expected signature). SLSA formalizes provenance and attestation processes as part of Build Level 2.

Chained dependency and code injection into released artifacts

An attack from 2018 called the event-stream incident is an example of chained dependency and code injection through a compromised open-source software project.

Hackers tried to steal data and cryptocurrency from the users of an app called Copay. They modified the popular event-stream package to include a new dependency in its build. Most developers didn’t notice, because the dependency only introduced malware when included in the Copay application.

This complex attack would be detected and prevented with the optional recursive check included with regular SLSA 1.0 audits. It’s a check that explicitly catches chained dependency attacks by recursing through and validating all dependencies.

Malicious code injection and insider tampering

In a sophisticated attack from 2024 known as the XZ Backdoor attack, state actors created a fake persona and spent four years gaining trust and contributing to a popular open-source compression library called XZ Utils. They added thousands of legitimate contributions and code to many projects before making a camouflaged malicious update.

SLSA Build Level 3 protections and recursive SLSA audits provide some protection from these kinds of attacks. As a wider portion of the industry audits, reviews, and ensures the OSS tools they use are properly authenticated and attributed, this kind of attack gets harder to perform and to keep hidden. The proposed Build Level 4 would also increase protection from this type of attack, for example by encouraging hermetic builds. (Once code has been reviewed and audited, a hermetic build guarantees it hasn’t been tampered with, no matter who builds it.) Future SLSA versions will require auditable, authenticated, and verified two-person reviews for code changes, targeting attacks like this.

How to get started using SLSA

SLSA is designed with ease of adoption in mind. Moving up through Build track levels (and implementing future tracks once they’re released) should feel as minimally invasive as possible. Let’s look at how you might go about adopting SLSA on your own.

Educate and self-evaluate

To start adopting SLSA into your software supply chain, take inventory of your software artifacts and build chains. List all the components and artifacts and assign a SLSA level to each one based on its current state. Questions like "How critical is this artifact to our operations?" or "What level of exposure to risk does this artifact have?" can help guide your evaluations and prioritize compliance work to improve SLSA levels for artifacts.

Once you have an inventory for your systems, you can begin implementing automated verification tools across your codebase. SLSA provides some examples, like the SLSA-verifier on GitHub, that help check compliance with SLSA standards and identify improvements.

Having verified your artifacts to determine the baseline, you can now set a target SLSA level for each artifact. Your business needs will dictate priorities and naturally lead to a strategic plan with actions to move each artifact to its target level. The SLSA team provides example tools, references, and guidelines for how this work can be efficient, quick, automated, and scaled up.

Implementing SLSA will be a continual effort. Regularly review your progress and watch for developments in the specifications. Adapt your security strategy to match the standards as they change, and your software supply chain will remain resistant to emerging threats.

Implement the SLSA Build Track

Here’s an overview of the kind of work required to implement each level of SLSA’s Build track. For more information, refer to the how-to guides on the SLSA website.

Level 0

Do nothing. This level indicates there are no SLSA assurances in place.

Level 1

Identify your build artifacts, and automatically generate, log, and distribute standardized provenance metadata.
Standardize and document all build processes for auditability.
Monitor the supply chain to detect inconsistencies.
Use tools like GitHub Actions or Buildkite Package Registries, which has built-in support for SLSA Build Level 1.

Level 2

Sign provenance data, attestations, and artifacts with cryptographic signatures for integrity validation and continuous monitoring.
Secure and centralize the build process, ensuring auditability and tamper resistance.
Consider tools like Windows signed attestation, IPFS Signed Attestation plugin, and public attestation repos like Rektor.

Level 3

Harden build platforms with regular audits and updates, ensuring documentation authenticity.
Automate validations and checks to prevent tampering.
Continuously audit the build process, follow relevant hardening guidelines, and consider resources like SLSA attestations for toolsets (e.g., for flux).

Level 4 and beyond (future)

Implement hermetic builds, and add audits and attestations to your build frameworks.
Clearly define roles, responsibilities, and processes for SLSA enforcement and monitoring.
Resources include tools like Bazel (for hermetic builds) and the OWASP Kubernetes security cheat sheet.

Keep up to date with SLSA

SLSA is maintained and modified in the open, and its administration follows public OpenSSF standards. The team behind SLSA includes a steering committee, specific working groups, and community contributions. You can contribute directly, and there are many ways to keep up to date with their work:

Regularly look for changes posted to the authoritative location for SLSA, https://slsa.dev. Updates are available in both human-readable and machine-accessible formats. (As time of writing, the SLSA v1.1 draft is available for public review.)
Refer to the SLSA blog for regular updates, directions, and feedback requests.
For more granular updates, see the weekly meeting notes published.
Review the changes to the SLSA standard that are suggested, processed, and accepted via pull requests against https://github.com/slsa-framework.

Buildkite Package Registries: Built-in support for SLSA provenance

Buildkite complements SLSA and can help accelerate your adoption. Buildkite Package Registries includes native provenance support features that are easy to cryptographically sign, artifact signing (for SLSA Level 1 and 2 support), and dependency management (supporting audit efforts).

The SLSA provenance of a Python package built with Buildkite Package Registries.

If you're looking to boost your software's security framework, consider exploring how Buildkite can help enhance your SLSA efforts. Check out the docs to learn more and get started.

Summary

SLSA is a security framework designed to protect software-delivery supply chains from vulnerabilities and targeted attacks:

It has structured and standardized security levels (0→3) for its Build track.
Its levels are designed to be adopted incrementally, with each level extending lower levels.
Implementing its guidance and complying with the requirements of each level increases the security and stability of your builds.
SLSA provides a clear and optimized path toward securing build chains and their dependencies.
Buildkite Package Registries offers built-in support for SLSA provenance and attestation.

How to Create and Share a Pulumi Template

Christian Nunciato — Tue, 20 Dec 2022 18:23:05 +0000

Last month, we released our first set of architecture templates — configurable Pulumi projects designed to make it easy to bootstrap new stacks for common cloud architectures like static websites, containers, virtual machines, and Kubernetes clusters. Architecture templates are a great way to get a new project up and running quickly, and they've already grown quite popular with our users, several of whom have asked if whether it's possible to create templates of their own.

It's not only possible, it's surprisingly easy! So in this post, I'll show you how. It all starts with a Pulumi project — a template is really just a Pulumi project with a few extra lines of config — so we'll start with one of those, add a few resources, and turn the project into a template. When it's ready to go, we'll test it out locally, and then finally, we'll publish it so that anyone with Pulumi can use our template to kickstart their own projects.

We'll do this in five steps:

Create a new project
Design and build the program
Turn the project into a template
Test-drive the template locally
Publish the template to share it with the world

Let's get to it.

Step 1: Create a new project

If you haven't already, make sure you've installed Pulumi and configured your AWS credentials. For this example, we'll be using AWS, but the process is the same for any cloud provider, so if you aren't set up on AWS, you should still be able to follow along anyway.

Create a new project in the usual way using one of our starter templates. The YAML template is a good choice for this walkthrough, as YAML projects are simple and lightweight (no runtime dependencies!), and because they combine both project and program into one file, they're especially conducive to sharing:

$ mkdir my-template-project && cd my-template-project
$ pulumi new aws-yaml

Step through the prompts, choosing whichever AWS region works best for you. When the new-project wizard finishes, open the generated Pulumi.yaml project file in your editor of choice to start building.

Step 2: Design and build the program

I'm a static-website person, myself — I think pretty much everyone should have one. So in order to realize my grand vision of home page for every human, you and I are going to build a Pulumi project template that lets anyone with the Pulumi CLI deploy a super-simple website of their own on AWS.

As Pulumi template authors, our general goal is twofold:

Make it easy for our users to create deployable projects that address an infrastructural need.
Make it easy to customize and extend those projects after they've been created.

A good template, in other words, is one that not only takes you from pulumi new to pulumi up in as few steps as possible, but that also leaves you with an open, extensible program you can use as a foundation to build upon. For us, the goal is to create a template that can be used to provision the minimal set of cloud resources one needs to run a static website on AWS. And done well, our template will also lend itself easily to further development — custom domains, serverless functions, edge caching, and so on.

To that end, let's start by replacing the contents of Pulumi.yaml with the following program, which defines the core resources we need: an S3 bucket to hold the files of the website and an S3 bucket object (index.html) to serve as its home page. We'll also export the computed URL of the website as a Pulumi stack output to give us something to navigate to after deployment:

name: my-template-project
description: A simple static website on Amazon S3
runtime: yaml
resources:

  # Create an S3 bucket and configure it as a website.
  bucket:
    type: aws:s3:Bucket
    properties:
      acl: public-read
      website:
        indexDocument: index.html

  # Create a home page for the website.
  index.html:
    type: aws:s3:BucketObject
    properties:
      bucket: ${bucket.id}
      acl: public-read
      contentType: text/html
      content: |
        <html>
          <meta name="content" charset="utf-8">
          <title>My Awesome Website</title>
          <body>
            <h1>My Awesome Website</h1>
            <p>Hello, internet person! 👋</p>
          </body>
        </html>

# Export the URL of the website.
outputs:
  url: http://${bucket.websiteEndpoint}

Try running this program now, just to make sure everything works. In a few seconds, you should see that the new resources were created and be able to browse to the new website at the exported url:

$ pulumi up
...

Updating (dev)

     Type                    Name                        Status
 +   pulumi:pulumi:Stack     my-s3-website-template-dev  created (3s)
 +   ├─ aws:s3:Bucket        bucket                      created (2s)
 +   └─ aws:s3:BucketObject  index.html                  created (0.37s)

Outputs:
    url: "http://bucket-f1bb181.s3-website-us-west-2.amazonaws.com"

Resources:
    + 3 created

Duration: 5s

Good — we have a working program. And looking back at our design goals, you'll see that this program already meets them pretty well:

It works out of the box. Deployed now, it produces a running website on AWS.
It's open and extensible. To build onto it — add more pages, attach a custom domain, etc. — you'd simply add more resources alongside those already defined.

Now let's turn this program into a template.

Before moving on, though, let's quickly destroy and remove this stack, as we'll be making a few changes to the project's settings in the next step, and it'd be better to begin from a clean slate:

$ pulumi destroy --yes --remove

Step 3: Turn the project into a template

The next thing to do is decide which parts of the program should be configurable. Earlier, when you ran pulumi new aws-yaml, you probably noticed you were prompted for an optional AWS region:

$ pulumi new aws-yaml
...

This command will walk you through creating a new Pulumi project.
...

aws:region: The AWS region to deploy into: (us-west-2)

You got this prompt because the authors of the aws-yaml template knew that not every user would want to deploy into the same hard-coded AWS region, so they defined an aws:region setting to make it both configurable and optional, falling back to us-west-2 by default. Users of this template are free to change this value if they like or leave it alone and accept the default.

This is all made possible by the existence of the template block in Pulumi.yaml:

# The `template` block from the aws-yaml template.
template:
  description: A minimal AWS Pulumi YAML program
  config:
    aws:region:
      description: The AWS region to deploy into
      default: us-west-2

Indeed, this block is all that defines aws-yaml as a template; without it, the aws-yaml project would be a regular ol' Pulumi YAML program like any other.

The template block defines two properties:

An optional description property to give new projects created from the template
A config block that lists the names, descriptions, and default values of any settings that should be configurable for new projects

The template block is essentially where all the new-project magic happens. Any settings you define in this block will prompt users for their values and apply those settings to the project's initial stack (for example, the dev stack we created in Step 1). By default, these values are captured and applied as plain-text strings, but you can also capture them as encrypted Pulumi secrets by adding secret: true — an appropriate choice for prompting for sensitive data like passwords, API keys, and the like.

Our super-simple website template doesn't need much in the way of configurability — but let's define some anyway just to see how it's done. While we're at it, we'll make a few adjustments to make project names and descriptions configurable as well.

Make the changes below to Pulumi.yaml to use the project name and description supplied by the CLI, add a template section to let users configure a home-page title and greeting, and interpolate those values into the body of the home page itself. Here's the full content of the program for reference:

# Configure project names and descriptions with values obtained from `pulumi new`.
name: ${PROJECT}
description: ${DESCRIPTION}
runtime: yaml

# Define the template's configuration settings.
template:
  description: A simple static website on Amazon S3
  config:
    aws:region:
      description: The AWS region to deploy into
      default: us-west-2
    title:
      description: A title to use for the home page
      default: My Awesome Website
    greeting:
      description: A greeting to show on the home page
      default: Hello, internet person! 👋

resources:

  # Create an S3 bucket and configure it as a website.
  bucket:
    type: aws:s3:Bucket
    properties:
      acl: public-read
      website:
        indexDocument: index.html

  # Create a home page for the website.
  index.html:
    type: aws:s3:BucketObject
    properties:
      bucket: ${bucket.id}
      acl: public-read
      contentType: text/html
      content: |
        <html>
          <meta name="content" charset="utf-8">
          <title>${title}</title>
          <body>
            <h1>${title}</h1>
            <p>${greeting}</p>
          </body>
        </html>

# Export the URL of the website.
outputs:
  url: http://${bucket.websiteEndpoint}

That's all there is to it! The template's complete and ready to be published. However, before we do that, let's first give it a try locally just to make sure it delivers the new-project experience we're looking for.

Step 4: Test-drive the template locally

Back in your terminal (and assuming you're still in the my-template-project folder), create a new folder alongside my-template-project, then run pulumi new to create a new project, pointing Pulumi to the folder containing your template:

$ cd ..
$ mkdir my-test-website && cd my-test-website
$ pulumi new ../my-template-project

Step through the prompts, which should include each of the configurable settings we defined in the template block earlier, along with their default values:

$ pulumi new ./my-template-project

project name: (my-test-website)
project description: (A simple static website on Amazon S3)
...

aws:region: The AWS region to deploy into: (us-west-2)
greeting: A greeting to show on the home page: (Hello, internet person! 👋) Hi, world!
title: A title to use for the home page: (My Awesome Website) Hey look, it's my website!

Your new project is ready to go! ✨
To perform an initial deployment, run `pulumi up`

Go ahead and deploy the new project with pulumi up, then navigate to the newly deployed website in your favorite web browser:

$ pulumi up --yes
$ open $(pulumi stack output url)

Well done! You've written your first Pulumi template.

Let's finish things off by publishing your template so others can use it. As before, tidy up with pulumi destroy before moving on:

$ pulumi destroy --yes --remove

Step 5: Publish the template

A moment ago, when you tested your first template, you did so by pointing Pulumi to the template's path on your local filesystem. Before that, when you ran pulumi new aws-yaml, you instructed Pulumi (implicitly) to fetch the source of the template over HTTPS from the official pulumi/templates repository on GitHub. So to make your template available to others, all you have to do publish it to a file path or Git URL that's accessible by the Pulumi CLI.

You could create a new GitHub repository for this project, then pass the fully-qualified URL of the path containing Pulumi.yaml to pulumi new, like so:

$ pulumi new https://github.com/{your-org}/{your-repo}/tree/main

But since our project consists of just one file, you don't even need to do that much: you can just paste the contents of the file into a new GitHub gist (since a gist is also a Git repository), name the gist Pulumi.yaml, and point Pulumi to the gist's URL.

Go ahead and create a gist of your own, then use it to create a new project:

$ mkdir ../my-second-test-website && cd ../my-second-test-website
$ pulumi new https://gist.github.com/cnunciato/b331efae6a4740c237a0364d17fe220f
$ pulumi up

Be sure to tidy up as before with pulumi destroy when you're done.

Bonus step: Add a Deploy with Pulumi button

In addition to the CLI, your users can also create new projects in the Pulumi Service with the Deploy with Pulumi button. This is a great option for making your project installable from GitHub READMEs and other team docs. Here, for example, is a Deploy button that creates a new project using my version of the gist we created above:

Embedding these buttons yourself is easy — just use one of the snippets below, swapping the values of {template-url} for the URL of your template's gist or Git repository:

<a href="https://app.pulumi.com/new?template={template-url}">
    <img src"https://get.pulumi.com/new/button.svg" title="Deploy with Pulumi">
</a>

[![Deploy with Pulumi](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template={template-url})

Project creators who go down this path will be prompted in the browser for the same configuration values as they would with the CLI, and afterward, they'll be able to deploy the project either with the Pulumi CLI or with Pulumi Deployments.

Wrapping up

As you can see, creating a template is both simple and powerful, and I hope this post encourages you to experiment with a few of your own. Feel free to peruse the following links for inspiration:

And as always, be sure to stop by Pulumi Community Slack to let us know know how it goes.

Happy templating!

Deploying a Data Warehouse with Pulumi and Amazon Redshift

Christian Nunciato — Wed, 30 Nov 2022 23:14:36 +0000

It's fun to think about how much data there is swirling around in the global datasphere these days. However you choose to measure it (and there are various ways), it's a quantity so massive — hundreds of zettabytes, by some estimates — that it's kind of a hard thing to quite get your head around.

If you could convert all the world's data into droplets of water, for instance, at one megabyte per drop, you'd have enough 1MB drops to fill two more Lake Washingtons. If you could store all that data on 3.5" floppies, you'd need more than a hundred quadrillion floppies to capture it all — enough to cover the planet entirely (with much room for overlap) or to pave a nice bridge for yourself from your front porch well into interstellar space. If you could pull all that data into an HD movie, and you sat down to start watching that movie 2.5 million years ago (with your favorite saber-toothed friend, say), you'd still be watching the same movie today.

That's a lot of data — and there's more every day.

Of course, the volume of data you care about is probably a lot more modest. But even so, there's a good chance whatever system you're building can and will produce useful data of its own, and there's an equally good chance you'll want to learn something from it. That data will likely exist in many forms and in multiple places — flat file here, transactional database there, the REST API of some third-party service — so before you can really look at it, you'll need to figure out how to get it all into one place. And for that, you might want to consider a data warehouse.

A data warehouse is a specialized database that's purpose built for gathering and analyzing data. Unlike general-purpose databases like MySQL or PostgreSQL, which are designed to meet the real-time performance and transactional needs of applications, a data warehouse is designed to collect and process the data produced by those applications, collectively and over time, to help you gain insight from it. Examples of data-warehouse products include Snowflake, Google BigQuery, Azure Synapse Analytics, and Amazon Redshift — all of which, incidentally, are easily managed with Pulumi.

Today, though, we're going to focus on Amazon Redshift. Specifically, we're going to walk through the process of writing a Pulumi program that provisions a single-node Redshift cluster in an Amazon VPC, then we'll load some sample data into the warehouse from Amazon S3. We'll load this data manually at first, just to get a sense of how everything works when it's all wired up, and then later, in a follow-up post, we'll go a step further and weave in some automation to load the data on a schedule.

Let's get going!

We'll begin, as always, by creating a new Pulumi project. (And if you haven't already, make sure you've installed Pulumi and configured your AWS credentials in the usual way.)

$ mkdir my-data-warehouse && cd my-data-warehouse
$ pulumi new aws-python

Step through the new-project wizard, accepting the defaults if they'll work for you. For this project, we'll deploy both cluster and data into the us-west-2 region of AWS, but you can change the target to region you like, as both S3 and Redshift are supported in all AWS regions.

With the project and stack in place, it's time to start building.

Configure the stack

To launch a new Redshift cluster, you'll need to give AWS a few details:

A unique (in the scope of your AWS account) name for the Redshift cluster
A name for the cluster's initial database
A username and password for the cluster's administrative user
The node type to use for the Redshift cluster

The following configuration settings should work for this walkthrough (but again, feel free to adjust them if you'd like):

$ pulumi config set clusterIdentifier my-redshift-cluster
$ pulumi config set clusterNodeType ra3.xlplus
$ pulumi config set clusterDBName dev
$ pulumi config set clusterDBUsername admin
$ pulumi config set clusterDBPassword StrongPass1 --secret

Compose the program

With these settings in place, you can start writing the Pulumi program.

Open __main__.py in your editor of choice and replace its contents with the following code, which imports the Pulumi and AWS libraries, reads the configuration values you just set, and provisions an S3 bucket that we'll use to store some raw, unstructured data later:

import json
import pulumi
from pulumi_aws import ec2, iam, redshift, s3

# Import the stack's configuration settings.
config = pulumi.Config()
cluster_identifier = config.require("clusterIdentifier")
cluster_node_type = config.require("clusterNodeType")
cluster_db_name = config.require("clusterDBName")
cluster_db_username = config.require("clusterDBUsername")
cluster_db_password = config.require_secret("clusterDBPassword")

# Import the provider's configuration settings.
provider_config = pulumi.Config("aws")
aws_region = provider_config.require("region")

# Create an S3 bucket to store some raw data.
events_bucket = s3.Bucket("events", s3.BucketArgs(
    force_destroy=True,
))

Next, define a new VPC and the associated network resources for the Redshift cluster. Since the aim is to launch the cluster into a VPC (providing the network isolation we need to keep the cluster from being accessed over the internet), you'll define the VPC first, then define a private subnet within it, and then finally designate a Redshift subnet group to allow to tell AWS where to provision the cluster:

# ...

# Create a VPC.
vpc = ec2.Vpc("vpc", ec2.VpcArgs(
    cidr_block="10.0.0.0/16",
    enable_dns_hostnames=True,
))

# Create a private subnet within the VPC.
subnet = ec2.Subnet("subnet", ec2.SubnetArgs(
    vpc_id=vpc.id,
    cidr_block="10.0.1.0/24",
))

# Declare a Redshift subnet group with the subnet ID.
subnet_group = redshift.SubnetGroup("subnet-group", redshift.SubnetGroupArgs(
    subnet_ids=[
        subnet.id,
    ],
))

Now, on to the cluster itself.

The plan, you'll recall, is to pull data from an S3 bucket into Redshift (using the Redshift service itself, for now), so you'll need to give Redshift the appropriate permissions to read from Amazon S3. You can do that with an IAM role and AWS-managed policy granting Redshift read-only access to the S3 service:

# ...

# Create an IAM role granting Redshift read-only access to S3.
redshift_role = iam.Role("redshift-role", iam.RoleArgs(
    assume_role_policy=json.dumps({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Principal": {
                    "Service": "redshift.amazonaws.com",
                },
            },
        ],
    }),
    managed_policy_arns=[
        iam.ManagedPolicy.AMAZON_S3_READ_ONLY_ACCESS,
    ],
))

That takes care of the permissions part — but there's one last thing you'll need to do to make this work. Because the cluster will reside in a private subnet, and that subnet won't have access to the public internet, you'll need to give the cluster a way to communicate with S3 without having to leave the VPC network. You can do this by adding a VPC gateway endpoint:

# ...

# Create a VPC endpoint so the cluster can read from S3 over the private network.
vpc_endpoint = ec2.VpcEndpoint("s3-vpc-endpoint", ec2.VpcEndpointArgs(
    vpc_id=vpc.id,
    service_name=f"com.amazonaws.{aws_region}.s3",
))

Finally, add the cluster itself, using everything you've defined up to now — config settings, network settings, IAM role, and the VPC's default security group, which enables traffic to and from the cluster on all ports and subnet IPs. Finish things off by exporting the name of the S3 bucket and Redshift ARN, as you'll need both in order to post and then import the S3 data (both will be auto-named):

# ...

# Create a single-node Redshift cluster in the VPC.
cluster = redshift.Cluster("cluster", redshift.ClusterArgs(
    cluster_identifier=cluster_identifier,
    database_name=cluster_db_name,
    master_username=cluster_db_username,
    master_password=cluster_db_password,
    node_type=cluster_node_type,
    cluster_subnet_group_name=subnet_group.name,
    cluster_type="single-node",
    publicly_accessible=False,
    skip_final_snapshot=True,
    vpc_security_group_ids=[
        vpc.default_security_group_id,
    ],
    iam_roles=[
        redshift_role.arn,
    ],
))

# Export the bucket name and role ARN.
pulumi.export("dataBucketName", events_bucket.bucket)
pulumi.export("redshiftRoleArn", redshift_role.arn)

And with that, you're ready to deploy.

Deploy the stack

Back at the command line, deploy the stack in the usual way with pulumi up:

$ pulumi up
...
Updating (dev)

     Type                         Name                   Status
 +   pulumi:pulumi:Stack          my-data-warehouse-dev  created (3s)
 +   ├─ aws:s3:Bucket             events                 created (2s)
 +   ├─ aws:ec2:Vpc               vpc                    created (12s)
 +   ├─ aws:iam:Role              redshift-role          created (1s)
 +   ├─ aws:ec2:Subnet            subnet                 created (0.81s)
 +   ├─ aws:ec2:VpcEndpoint       s3-vpc-endpoint        created (6s)
 +   ├─ aws:redshift:SubnetGroup  subnet-group           created (0.80s)
 +   └─ aws:redshift:Cluster      cluster                created (270s)

Outputs:
    dataBucketName : "events-50d8c96"
    redshiftRoleArn: "arn:aws:iam::616138583583:role/redshift-role-47ad8f0"

Resources:
    + 8 created

Duration: 4m48s

After a few moments (deploying a new cluster takes about five minutes), you should see that the new cluster and all of its resources were created. Again, you won't be able to access the cluster over the internet, as it'll have been deployed into an isolated VPC network, but you should be able to browse to the cluster in the AWS Console and start running some queries with it.

Choose Query data > Query in query editor v2 to open the Redshift query editor:

Then, from the query editor, find your newly created cluster in the left-hand pane and connect using the database, username, and password you configured earlier:

Choose Create connection, expand my-redshift-cluster, and you should see the dev database there, ready to go (only without any tables — we'll get to that next):

Now let's load some data!

Getting data into Redshift

In a real-world situation, you'd probably already have some data to load — web server logs, for example, or some other sort of raw or unstructured data residing in an S3 bucket, a DynamoDB table, an RDS database, etc. But we're starting from scratch, so we'll create and publish that data manually.

Copy and paste the following command to generate a text file containing a few lines of JSON, each representing a fictitious event to be loaded into the data warehouse:

$ echo '{"id": 1, "name": "An interesting event"}
{"id": 2, "name": "Another interesting event"}
{"id": 3, "name": "An event of monumental importance"}' > events-1.txt

Notice there aren't any commas between these events, so it's not actually a valid JSON file -- it's just a text file containing some individually valid JSON events, one event per line. Adhering to this format allows Redshift to ingest and map the data to table we create next using one of its built-in parsers.

Upload the file to S3 with the AWS CLI, using Pulumi to supply the name of the destination bucket:

$ aws s3 cp events-1.txt "s3://$(pulumi stack output dataBucketName)"

At this point, you have all you need in terms of infrastructure: You've got a single-node Redshift cluster, an empty Redshift database, some unstructured S3 data, and the right set of permissions and network settings to allow the cluster to reach and import the data. All that's left is to run a couple of SQL queries to make that happen: one to create a new table to hold the data, another to load it.

Back in the Redshift query editor, create the new table by choosing the dev database, pasting the following query into the editor, and hitting Run:

CREATE TABLE events (
    id INT,
    name VARCHAR(255)
);

You should see that the query succeeded and that the table was created:

Now you're ready to import the data. Replace the contents of the query editor with the following Redshift COPY command, replacing the {bucket-name}, {iam-role}, and {aws-region} placeholders with the corresponding values from your Pulumi stack:

COPY events
FROM 's3://{bucket-name}/events-1.txt'
IAM_ROLE '{iam-role}'
FORMAT AS JSON 'auto'
REGION AS '{aws-region}';

Here's what I see, for example, after running that command:

After the COPY command completes (which may take a few seconds), you should be able to run a new query the events table and see the new data:

SELECT *
  FROM events
 ORDER BY id;

Huzzah! You've now got yourself a working data warehouse.

Taking stock, and looking ahead

If you're like me, you're probably happy to be up and running — but you've also got a few questions:

What would happen if you added a few more rows to events-1.txt, then ran that COPY command a second time? Would Redshift ignore the records it's already processed, and only import the new ones?
Running SQL commands in a browser like this seems less than ideal. Can I run this process automatically somehow — like once a day, or in response to some other event?
My source data's kind of a mess. Can I process the data somehow when I load it — like drop certain fields, rename columns, split records into multiple tables, and so on?
Can I pull data from other sources that aren't Amazon S3?

All good questions, and I'm glad that you asked them.

First, as you'll notice, adding lines to events.txt and running COPY a second time probably doesn't behave as you might hope:

Duplicate records — the bane of the data scientist.

As of now, your only real recourse is to leave events-1.txt alone, create new data files for each batch of events, and only COPY those files that haven't been processed already. For small projects that don't generate much data, this might be a viable option, but over time, it'd probably become pretty tedious, and it'd be all too easy to make a mistake that leaves you with messed-up data. Not ideal.

Instead, what you'll ultimately want is some process by which your data can be extracted (duplicate-free and from multiple sources), transformed according to rules you specify (keeping what you want, dropping what you don't), and loaded into the warehouse without your involvement. What you'll want, in other words, is an ETL pipeline.

The good news is that with the data warehouse up and running, you're well on your way already. The bad news is that Redshift alone won't finish the job for you. Redshift is just the destination — the target of your ETL pipeline-to-be. The pipeline itself still needs to be written.

So in the next post, we'll do that: We'll take what we've done here, add a few more components with Pulumi and AWS Glue, and wire it all up with a few magical lines of Python scripting.

Until then, happy coding — and stay tuned!

Shared configuration stacks with AWS Systems Manager

Christian Nunciato — Fri, 01 Jul 2022 01:58:05 +0000

One thing I love about Pulumi is how easy it is to configure a stack. As a builder mainly of web applications, I'm always thinking about how I'll configure my apps from one environment to the next, and being able to use Pulumi's built-in support for configuration and secrets to manage the API keys and database credentials for my dev, staging, and production stacks individually is incredibly convenient.

For larger teams and organizations, though, where multiple applications rely on a set of common configuration settings --- dozens of apps, say, depending on the the same API service or database --- having to keep all of those config settings in sync across all of those individually can become a bit of a pain. When this happens, you may find yourself looking for ways to extract those settings into some sort of a service to allow you to manage them easily in one place, and in a way that allows any application to inherit them automatically.

There are lots of ways of addressing this sort of problem. One that I like is AWS Systems Manager (SSM) --- specifically, SSM Parameter Store. With Systems Manager and Parameter Store, you can store plain-text and encrypted secrets and expose them easily to any other resource in your AWS infrastructure. Parameter Store is a powerful tool to have in your cloud-programming toolbox --- and combined with Pulumi, and in particular Pulumi stack references, you can use it to build specialized stacks make managing shared configuration simple.

In this post, I'll show you one way to do this. We'll start with a simple Pulumi stack that defines a few config settings with Parameter Store, then stand up a couple of other stacks to illustrate how to inherit and use those settings. We'll also use Pulumi YAML, one of the newest additions to the Pulumi language family, to give you an opportunity to kick the tires on that as well.

Let's get going.

Sketching it out

The plan is to begin with a Pulumi stack to act as the keeper of our hypothetical organization's shared configuration. This stack will define two config settings: a message of the day, which we'll store in plain text, and a secret message of the day (ssh!), which we'll store as an encrypted secret. Later, we'll build two stacks to act as downstream consumers of this shared-config stack --- one, a static website that'll use our config values to render some text in the browser, the other a serverless function that'll return those values as JSON. Both will use Pulumi stack references to obtain their config values from Parameter Store. Here's a visual:

Let's start Stack 1 --- the shared-config stack.

Building the shared config stack

First, make sure you've installed and configured Pulumi for AWS, and then, in your shell of choice, create a new folder to house our three stacks-to-be:

$ mkdir shared-config-with-ssm && cd shared-config-with-ssm
$ mkdir shared-config my-website my-service

This'll leave you with three folders to work with, one for each stack:

$ tree .
.
├── my-service
├── my-website
└── shared-config

Change to the shared-config folder and generate a new Pulumi YAML project, accepting the defaults to create a new dev stack:

$ cd shared-config
$ pulumi new aws-yaml

When the new-project wizard completes, add two new settings for the dev stack with pulumi config set:

$ pulumi config set motd 'Hello from Pulumi!'
$ pulumi config set motd_secret 'Ssh! This is a secret.' --secret

These are the two values we're going to make available to Stacks 2 and 3.

Now let's build out the Pulumi program to provision these items as parameters in AWS Systems Manager. Replace the contents of Pulumi.yaml with the following YAML program, which reads the config values we just set, declares two new aws.ssm.Parameter resources (using fully-qualified paths as a best practice), and exports their parameter names as Pulumi stack outputs:

name: shared-config
runtime: yaml

# The values we're reading from stack configuration.
configuration:
  motd:
    type: String
  motd_secret:
    type: String

# The AWS SSM Parameters managed by this stack.
resources:
  motd_param:
    type: aws:ssm:Parameter
    properties:
      name: /shared-config/dev/motd
      value: ${motd}
      type: String
  motd_secret_param:
    type: aws:ssm:Parameter
    properties:
      name: /shared-config/dev/motd_secret
      value: ${motd_secret}
      type: SecureString

# Outputs are the fully-qualified AWS SSM Parameter names
# that consuming stacks can use to retrieve associated values.
outputs:
  motd_param_name: ${motd_param.name}
  motd_secret_param_name: ${motd_secret_param.name}

At this point, you might be wondering why we're only exporting our parameters' names, and not their values.

We certainly could export their values, since we already have them at hand. But if we did, there wouldn't be much point to using Parameter Store at all, since Pulumi itself is perfectly capable of managing plain-text and encrypted values, and stack references give other stacks an easy way of accessing them programmatically. Why not export the values, then?

For one, it'd be less flexible. If we exposed these values as stack outputs, downstream stacks referencing them would only be able to read them at deploy-time --- i.e., during their own Pulumi updates. For shared-config values that change rarely, that might be okay, but for others, it'd be less than ideal in that all downstream stacks would have to be updated immediately whenever one of those values were changed.

Secondly, it keeps secrets a bit more secret. Pulumi is great at protecting the secrets it manages, even across stack boundaries like these, but applications still need a way to get access to the underlying plain-text values of secrets at runtime. Lambda functions, for example, need access tokens and database passwords to do the work we need them to do --- but figuring out how to provide them with those raw values safely can be tricky. By requiring client applications to fetch secrets from Parameter Store, we can keep their plain-text values out of the bodies of our Lambda functions and prevent them from surfacing inadvertently in the AWS Lambda Console.

Finally, it leaves room for the occasional out-of-band change. Now and then, you might be faced with having to update a value in Parameter Store directly --- e.g., with the AWS Console or CLI. Downstream stacks not reading from Parameter Store, however, wouldn't have access to these updated values until after the shared-config stack were refreshed and redeployed --- and would themselves have to be redeployed in order to use them.

Perhaps that was more explanation than necessary for what amounts to just a few lines of YAML, but hopefully it gives you a sense of the kinds of things to be thinking about as you explore solutions like this on your own. For now, let's forge ahead and get this first stack deployed.

Back in your terminal, do that with pulumi up. The deployment should take only a few seconds to complete:

$ pulumi up
...

Updating (dev)

     Type                  Name               Status
 +   pulumi:pulumi:Stack   shared-config-dev  created
 +   ├─ aws:ssm:Parameter  motd_param         created
 +   └─ aws:ssm:Parameter  motd_secret_param  created

Outputs:
    motd_param_name       : "/shared-config/dev/motd"
    motd_secret_param_name: "/shared-config/dev/motd_secret"

Resources:
    + 3 created

Duration: 4s

Notice we've named our outputs clearly, and that the secret message is safely encrypted in Pulumi.dev.yaml and in Parameter Store. You can confirm the latter by visiting the AWS Console:

Now let's put these two new values to practical use.

Fetching configuration at deploy-time

Assuming you're still in the shared-config folder, change to the my-website folder you created a few moments ago and generate another YAML project, this one for our humble little website:

$ cd ../my-website
$ pulumi new aws-yaml

Once again, follow the prompts to create a new dev stack, then replace the contents of Pulumi.yaml with the program below. Here, we're fetching the names of the parameters being managed by shared-config, pulling their values from Systems Manager, and creating a one-page website to render those values in the browser. (Be sure to adjust the Fn::StackReferences to point to your stack instead of mine.)

name: my-website
runtime: yaml

variables:

  # Get the names of the parameters we care about from the shared-config stack.
  motd_param_ref:
    Fn::StackReference:
      - cnunciato/shared-config/dev  # <-- Change this.
      - motd_param_name
  motd_secret_param_ref:
    Fn::StackReference:
      - cnunciato/shared-config/dev  # <-- And this.
      - motd_secret_param_name

  # Fetch (and decrypt) their values from Systems Manager.
  motd_param:
    Fn::Invoke:
      Function: aws:ssm:getParameter
      Arguments:
        name: ${motd_param_ref}
  motd_secret_param:
    Fn::Invoke:
      Function: aws:ssm:getParameter
      Arguments:
        name: ${motd_secret_param_ref}
        withDecryption: true

resources:

  # Create an S3 bucket and configure it as a website.
  my-bucket:
    type: aws:s3:Bucket
    properties:
      website:
        indexDocument: index.html

  # Create a homepage and render the values.
  index.html:
    type: aws:s3:BucketObject
    properties:
      bucket: ${my-bucket}
      acl: public-read
      contentType: text/html
      content: |
        <html>
          <head>
            <meta charset="utf-8">
            <style>
              span { display: none; }
              button:active + span { display: inline; }
            </style>
          </head>
          <body>
            <h1>${motd_param.value}</h1>
            <button>Reveal the secret message!</button>
            <span>${motd_secret_param.value}</span>
          </body>
        </html>

outputs:

  # Export the publicly accessible URL of the website.
  url: http://${my-bucket.websiteEndpoint}

Save the file and deploy the stack, which again should take only a few seconds to complete:

$ pulumi up
...

Updating (dev)

     Type                    Name            Status
 +   pulumi:pulumi:Stack     my-website-dev  created
 +   ├─ aws:s3:Bucket        my-bucket       created
 +   └─ aws:s3:BucketObject  index.html      created

Outputs:
    url: "http://my-bucket-269e711.s3-website-us-west-2.amazonaws.com"

Resources:
    + 3 created

Duration: 5s

With the exported URL, open the website in your favorite browser, and you should also see the message of the day on the homepage, along with a button you can click to reveal the secret message:

If you're using the Pulumi Service, which tracks dependencies between stacks, you should see that the my-website stack is now listed as a downstream consumer of shared-config:

Now, when the hypothetical team managing the shared-config stack decides it's time to update the message of the day, it can do so with a pulumi config set and pulumi up, and my-website will be able to pick up the new message automatically.

However, it's worth mentioning that that won't happen immediately, as the my-website stack is configured to pull the message from Systems Manager at deploy-time. If we want to reflect the new value on the homepage, we'll either need to deploy the my-website again or come up with a way to fetch the message at runtime --- which brings us to our third and final stack, in which you'll learn how to do just that.

Fetching configuration at runtime

The easiest way to see this in action is with AWS Lambda. With Lambda and Pulumi's support for function serialization, we can capture the parameter names with stack references, pass them into the Lambda at deploy-time (as plain-text strings, which is fine, because they're just names), and fetch their values from Systems Manager at runtime --- i.e., when the Lambda is invoked sometime later --- with the AWS SDK for Node.js. This way, any updates made by the shared-config team will be usable by our Lambda function immediately, no redeployment necessary.

So to finish things off, change to the my-service folder you created earlier and generats a third and final project, this one with TypeScript, again accepting the defaults to create a dev stack for it:

$ cd ../my-service
$ pulumi new aws-typescript

Replace the contents of index.ts with the following program. As before, we're using StackReferences to read our parameter names from the shared-config stack --- except here, because StackRerefences are provided as pulumi.Outputs, which can't be serialized into the Lambda, we'll need to use .get() to unwrap the parameter name into a string so we can use it at runtime with the AWS SDK:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Get the names of the parameters we care about from the shared-config stack.
const stack = new pulumi.StackReference("cnunciato/shared-config/dev"); // <-- Your name here.
const motdParamRef = stack.requireOutput("motd_param_name");
const motdSecretParamRef = stack.requireOutput("motd_secret_param_name");

const callback = new aws.lambda.CallbackFunction("lambda", {
    policies: [
        aws.iam.ManagedPolicy.LambdaFullAccess,

        // Give the Lambda read-only access to AWS Systems Manager.
        aws.iam.ManagedPolicy.AmazonSSMReadOnlyAccess
    ],
    callback: async () => {

        // Use the AWS SDK for JavaScript to fetch parameter values from Systems Manager.
        const ssm = new aws.sdk.SSM();

        // Fetch the message of the day.
        const motdParam = await ssm.getParameter({
            Name: motdParamRef.get(),  // <-- Use .get() for you runtime access to Output values.
        }).promise();

        // Fetch the secret message of the day.
        const motdSecretParam = await ssm.getParameter({
            Name: motdSecretParamRef.get(),
            WithDecryption: true,
        }).promise();

        return {
            statusCode: 200,
            contentType: "application/json",

            // Return the result.
            body: JSON.stringify({
                motd: motdParam.Parameter?.Value,
                motd_secret: motdSecretParam.Parameter?.Value
            }),
        };
    },
});

const lambdaUrl = new aws.lambda.FunctionUrl("lambda-url", {
    functionName: callback.name,
    authorizationType: "NONE",
});

// Export the publicly accessible URL of the service.
export const url = lambdaUrl.functionUrl;

When you're ready, deploy the my-service stack with pulumi up:

$ pulumi up
...

Updating (dev)

     Type                             Name             Status
 +   pulumi:pulumi:Stack              my-service-dev   created
 +   ├─ aws:iam:Role                  lambda           created
 +   ├─ aws:lambda:Function           lambda           created
 +   ├─ aws:iam:RolePolicyAttachment  lambda-b63d666c  created
 +   ├─ aws:iam:RolePolicyAttachment  lambda-b5aeb6b6  created
 +   └─ aws:lambda:FunctionUrl        lambda-url       created

Outputs:
    url: "https://6y3vkpktyybylhua266bu7yplu0dgqgv.lambda-url.us-west-2.on.aws/"

Resources:
    + 6 created

Duration: 19s

Verify the deployment with a request to the Lambda function's public endpoint:

$ curl $(pulumi stack output url)
{
  "motd": "Hello from Pulumi!",
  "motd_secret": "Ssh! This is a secret."
}

That tells you the runtime lookup is working, which is great.

Now try updating the message of the day. Change to the shared-config folder, use pulumi config set to update the stack's configured value, and finally, run pulumi up to write the new value to Systems Manager:

$ cd ../shared-config
$ pulumi config set motd 'Hello *again* from Pulumi!'
$ pulumi up
...

Updating (dev)

     Type                  Name               Plan       Info
     pulumi:pulumi:Stack   shared-config-dev
 ~   └─ aws:ssm:Parameter  motd_param         updated    [diff: ~value]

Resources:
    ~ 1 updated
    2 unchanged

If all goes well, you should be able to invoke the Lambda again and see the new value immediately:

$ cd ../my-service
$ curl $(pulumi stack output url)
{
  "motd": "Hello *again* from Pulumi!",
  "motd_secret": "Ssh! This is a secret."
}

And with that, we're done. You can tear everything down with pulumi destroy:

$ cd ..
$ pulumi -C my-service destroy
$ pulumi -C my-website destroy
$ pulumi -C shared-config destroy

Wrapping up, and next steps

Managing configuration at scale is a much bigger topic than we could possibly cover in a blog post, but hopefully you've learned enough here to fuel some experimentation of your own. You'll find the full source code for this walkthrough on GitHub as well.

Additionally, as an alternative to Parameter Store, you might also want to check out AWS Secrets Manager. It's built specifically for managing secrets, and gives you additional capabilities like rotation and auditing, so depending on your use case, it could be a better fit for certain scenarios.

Happy configuring!

Test-Driven Infrastructure Development with Pulumi and Jest

Christian Nunciato — Tue, 14 Jun 2022 23:30:09 +0000

When I was a kid growing up in Southern California, there was a phone number you could call to find out what time it was. It was a local number, 853-1212 (easy to remember as the arrangement of the numbers on the keypad made a capital T), and I used it all the time, to set my watch, adjust the alarm clock, fix the display on the VCR. I don't recall the last time I used it, probably sometime in the mid '90s, but I do remember clearly the sound of the voice at the other end of the line.

Services like these were called speaking clocks (although I never called them that; to me, it was just "calling Time"), and they date back to the early 1930s or so. In the U.S., most were powered by a machine called an Audichron that used mechanical drums to play back the time of day, often with a little advertisement or the current temperature to go along with it.

By the mid-2000s, though, most of these locally run services had been shut down --- by then, we had clocks baked into our phones --- and today, only a handful of Audichrons survive. Thanks to the National Institute of Standards and Technology, however, you can still call a phone number to get the time, and while the voice might not be the same, and long distance rates will apply, it's definitely there, and you can use it. So on the off chance you happen to find yourself with no idea what time it is and only an analog phone line in reach, fear not --- old-school telephone tech has your back. For now. Assuming you remember the number.

Recalling all this stuff did make me wonder, though, what a more modern version of a speaking clock might look like. So in this post, we're going to build one ourselves. We won't use an actual phone number, but we will use Pulumi and AWS --- and because we want to do it right, we'll take a test-driven approach to developing the infrastructure with Jest, the JavaScript testing framework. We'll use TypeScript and Node.js for everything, focus on unit tests, and when we're done, we'll have a single, serverless, browser-friendly HTTPS endpoint that returns an MP3 audio stream that speaks the current time.

Let's get started.

Sketching it out

The first thing we'll need is a runtime environment --- someplace to run some server-side JavaScript that can render and deliver an audio file. Until recently, the easiest way to get an HTTP endpoint up and running on AWS has generally been with AWS Lambda and API Gateway, using Lambda to run the requisite code and API Gateway to expose the Lambda to the internet. Pulumi Crosswalk actually makes this really easy, too --- but with the release of AWS Lambda Function URLs this April, we now have another option, one that doesn't need API Gateway at all.

A Lambda Function URL is just what it sounds like: a URL that exposes a Lambda function. Specifically, it's an AWS cloud resource that consists of a few properties that tell AWS whether to allow anonymous access to the function or to protect it with AWS IAM, and optionally, you can also provide a few cross-origin resource-sharing (CORS) rules to provide (or restrict) access by websites running on other domains.

For an app like this one, a Function URL is a good fit, as it's simple, quick to deploy (and update), and it gives us all the control we need to get the job done. Architecturally, then, our little app will look something like this:

That about covers the infrastructure --- but what about the sound?

As it happens, Google Translate offers a publicly accessible text-to-speech API that can, as long as you don't overuse it, render a string to MP3 audio. We can use that API to generate the time-of-day message, tack the sound of a beep onto the end of it (for maximum nostalgic effect), and voilà: instant, infinitely scalable, cloud-native speaking clock.

Plan in place, we can kick things off by creating a new Pulumi project.

Create a new TypeScript project

Start by creating a new AWS TypeScript project in the usual way:

$ mkdir audichron-2022 && cd audichron-2022
$ pulumi new aws-typescript

Step through the prompts to create a new stack (you'll only need one stack for this project), and when the new-project wizard completes, clear out the contents of index.ts entirely, as we'll be building this program entirely from the ground up.

Install and configure Jest

Since we're working with TypeScript, we can use ts-jest, which conveniently brings Jest along for the ride:

$ npm install ts-jest @types/jest --save-dev

Jest requires a bit of configuration, so add a new file at the root of the project, called jest.config.ts, with the following code:

import type { Config } from "@jest/types";

const config: Config.InitialOptions = {
    preset: "ts-jest",
};

export default config;

With Jest now installed and configured, and its helpers (like describe and expect) declared globally, you can create a new spec file to hold your tests. Add another file at the root of the project called index.spec.ts, and give it the following temporary content:

import "jest";

describe("My speaking clock", () => {
    it ("works", () => {
        expect(true).toBeTruthy;
    });
});

Lastly, in package.json, add a scripts block with a helper to make running the tests a little easier:

{
    "name": "audichron-2022",
    "devDependencies": {
        "@types/node": "^14",
        "ts-jest": "^28.0.3"
    },
    ...
    "scripts": {
        "test": "jest"
    }
}

Now run the tests, just to make sure you've got everything wired up correctly. If all goes well, you should see that the temporary spec we just added happily passes:

$ npm test

PASS  ./index.spec.ts
  My speaking clock
    ✓ works

With that, it's time to get to start writing some real tests.

Start with some failing tests

As devoted practitioners of test-driven development, we're going to start by writing some unit tests --- specifically, some failing unit tests that we can fix by writing the Pulumi code to make them pass.

Recall that our design requires just two cloud resources: a Lambda function and a Lambda function URL. For the function itself, we'll use the high-level aws.lambda.CallbackFunction resource, one of my favorites for managing for Lambdas because it requires only one property: an inline JavaScript function to handle the event that triggers the Lambda.

For the URL resource --- the eventual triggerer of that event --- you'll use an aws.lambda.FunctionURL configured to make the Lambda publicly accessible (i.e., available to anyone on the internet) and embeddable on any domain, making it easy, for example, to embed the URL as the src attribute of an HTML5 audio element.

So to get things going, we'll need to:

Configure Pulumi to mock AWS resources. We're writing unit tests, after all, and we want them to be fast, so we'll need to prevent those tests from provisioning any real cloud infrastructure.
Import your Pulumi resource declarations --- the function and the function URL --- into index.spec.ts so you can reference them in tests.

In that light, replace the contents of index.spec.ts with the following code, which, after some setup to switch Pulumi into unit-testing mode, contains just one test: a check to make sure that the function URL is configured with the right authorizationType. I've included some comments to help explain what each line is doing:

import * as pulumi from "@pulumi/pulumi";
import "jest";

describe("My speaking clock", () => {

    // Define the infra variable as a type whose shape matches that of the
    // to-be-defined resources module.
    // https://www.typescriptlang.org/docs/handbook/2/typeof-types.html
    let infra: typeof import("./resources");

    beforeAll(() => {

        // Put Pulumi in unit-test mode, mocking all calls to cloud-provider APIs.
        pulumi.runtime.setMocks({

            // Mock requests to provision cloud resources and return a canned response.
            newResource: (args: pulumi.runtime.MockResourceArgs): {id: string, state: any} => {

                // Here, we're returning a same-shaped object for all resource types.
                // We could, however, use the arguments passed into this function to
                // customize the mocked-out properties of a particular resource based
                // on its type. See the unit-testing docs for details:
                // https://www.pulumi.com/docs/guides/testing/unit
                return {
                    id: `${args.name}-id`,
                    state: args.inputs,
                };
            },

            // Mock function calls and return whatever input properties were provided.
            call: (args: pulumi.runtime.MockCallArgs) => {
                return args.inputs;
            },
        });
    });

    beforeEach(async () => {

        // Dynamically import the resources module.
        // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#dynamic_imports
        infra = await import("./resources");
    });

    describe("function URL", () => {

        it("is publicly accessible", (done) => {
            infra.timeURL.authorizationType.apply(authType => {
                try {
                    expect(authType).toBe("NONE");
                    done();
                } catch (err) {
                    done(err);
                }
            });
        });
    });
});

We can't quite run these tests just yet, though, as they rely on a module called resources that we haven't yet defined. Let's do that now by creating a new file, resources.ts, containing some stubs for the cloud resources we've elected to use:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {
    callback: () => {
        // Leave the callback implementation empty for now.
    },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
    functionName: timeFunction.name,
    authorizationType: "", // Leave this empty, too. (We want our first test to fail.)
});

Now run the tests, and you should see that the public-accessibility test fails --- which shouldn't be surprising, as we haven't yet supplied an acceptable value:

$ npm test

FAIL  ./index.spec.ts
  My speaking clock
    function URL
      ✕ is publicly accessible

  ● My speaking clock › function URL › is publicly accessible

    expect(received).toBe(expected) // Object.is equality

    Expected: "NONE"
    Received: ""

Now let's turn that sad-looking ❌ into a ✅.

Make the tests pass

Update the FunctionUrl resource to change its authorizationType input to NONE (meaning for anonymous access, no authorization required):

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
    functionName: timeFunction.name,
-   authorizationType: "",
+   authorizationType: "NONE",
});

Once you do that, you should be able to run the test suite again, and this time, see it pass:

$ npm test

PASS  ./index.spec.ts
  My speaking clock
    function URL
      ✓ is publicly accessible

Excellent: you're now well on your way to writing more tests with Jest --- and we'll do that in a moment. But first, let's digress momentarily to explore a few other ways of writing Jest specs with Pulumi.

Tidy up the specs

If you've written JavaScript tests before, particularly with tools like Jest, Jasmine, and Mocha, there's a good chance you've gotten used to writing those tests in a particular way. And if you're like me, you might've raised a bit of an eyebrow when you saw how that first test was written. Here it is again, this time with comments that capture the questions I might've had myself if I were seeing this code for the first time:

it("is publicly accessible", (done) => {
    infra.timeURL.authorizationType.apply(auth => { // Why do I have to use .apply() here?
        try {                                       // What happens if I don't try/catch?
            expect(auth).toBe("NONE");              // Is it possible to use expect() by itself?
            done();                                 // Do I have to call done()? What if I don't?
        } catch (err) {                             // How can I make these tests more readable?
            done(err);                              // Is all of this code necessary?
        }
    });
});

This is a perfectly valid test, of course; we know because we just ran it. Still, you might wonder, especially since we're only checking one property, whether this test might be written a bit more concisely --- something more like this, maybe:

it("is publicly accessible", () => {
    expect(infra.timeURL.authorizationType).toBe("NONE");  // Alas, nope.
});

The main reason is the asynchronous nature of the objects we're testing. An aws.lambda.FunctionUrl, after all, isn't just a plain ol' JavaScript object --- it's a Pulumi CustomResource, a representation of an eventual resource running in the cloud, with properties that may not be known until sometime after the resource has been deployed. While it's true we provided authorizationType as a regular JavaScript string, by the time we attempt to read it in the test, it's been transformed into a pulumi.Output<string> --- an asynchronous value, like a promise, that has to be unwrapped with a call to apply() before it can be read.

However, just adding apply and calling expect from within its callback doesn't quite work, either:

it("is publicly accessible", () => {
    infra.timeURL.authorizationType.apply(authType => {
        expect(authType).toBe("NONE");  // Also nope.
    });
});

Although that might not be obvious at first, since on the surface, the test appears to pass:

$ npm test

PASS  ./index.spec.ts
  My speaking clock
    function URL
      ✓ is publicly accessible

It isn't until you force the test to fail that it becomes clearer something's gone sideways. Try changing the expected authorizationType to something other than NONE and you'll see what I mean:

$ npm test

PASS  ./index.spec.ts
  My time app
    function URL
      ✓ is publicly accessible

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function
without a catch block, or by rejecting a promise which was not handled with .catch(). The promise
rejected with the reason "Error: expect(received).toBe(expected) // Object.is equality

Expected: "AWS_IAM"
Received: "NONE"".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

All is not lost, though. We do have other options, and that UnhandledPromiseRejection offers a bit of a clue.

Turn outputs into promises

Jest actually has built-in support for writing asynchronous tests with promises, and there are a couple of ways you can write specs that use them:

Have the function passed to it (or its alias, test) return a promise containing your expectation
Pass an async function to it and await the promise in the usual way

There's only one problem: pulumi.Outputs may indeed be promise-like, but they aren't actual promises.

They can, however, be wrapped in promises pretty easily. I like the way Pulumi community member @aviflax handled this with a little utility function that wraps the Output and preserves the type of its underlying value. Here's that function, followed by a couple of examples that use it:

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
    return new Promise(resolve => output.apply(resolve));
}

// Our test, rewritten to have it() return a Promise<string>.
it("is publicly accessible", () => {
    return promiseOf(infra.timeURL.authorizationType).then(authType => {
        expect(authType).toBe("NONE");
    });
});

// The same test, rewritten to use async/await.
it("is publicly accessible", async () => { // <-- Don't miss that async!
    const authType = await promiseOf(infra.timeURL.authorizationType);
    expect(authType).toBe("NONE");
});

Either approach works equally well, but I'm slightly more partial to the latter for its conciseness. So for now, update index.spec.ts to add the promiseOf helper and adjust the test accordingly, then run the suite one more time to make sure all's well:

// ...

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
    return new Promise(resolve => output.apply(resolve));
}

describe("My speaking clock", () => {
    // ...

    describe("function URL", () => {

        it("is publicly accessible", async () => {
            const authType = await promiseOf(infra.timeURL.authorizationType);
            expect(authType).toBe("NONE");
        });
     });
});

Assuming you're back in ✅ territory, let's move on so we can finish things off.

Finish the app

We're in good shape: we've got scaffolding for our Jest-powered unit tests and we're generally happy with how they look, so it's time to round out the rest of the app and its tests so we can deploy. We'll go through this last part briskly, as we've covered most of the important stuff at this point, and pause only briefly to point out what's important as we go.

Here's the plan:

We'll add one more test to make sure the function URL is CORS-friendly
We'll add another test to make sure the URL resource is bound to the right Lambda function
We'll add the Lambda code to generate and return the audio file
We'll deploy, and bask in the warm glow of nostalgia

Let's get to it.

Add the remaining tests

To index.spec.ts, add two more tests, one to ensure the function is accessible to browsers running on any domain, the other to verify the function URL resource is exposing the right function (if we didn't, it'd be easy to change that reference by mistake and accidentally expose the wrong function):

describe("My speaking clock", () => {
    // ...

    describe("function URL", () => {
        // ...

        it("is CORS-friendly", async () => {
            const authType = await promiseOf(infra.timeURL.cors);
            expect(authType).toEqual({
                allowOrigins: ["*"],
                allowMethods: ["GET"],
            });
        });

        it("is bound to the right Lambda function", async () => {
            const timeFuncName = await promiseOf(infra.timeFunction.name);
            const timeURLFunc = await promiseOf(infra.timeURL.functionName);
            expect(timeFuncName).toEqual(timeURLFunc);
        });
    });
});

Run the tests, and you should see one failure:

$ npm test

FAIL  ./index.spec.ts (14.256 s)
  My speaking clock
    function URL
      ✓ is publicly accessible (3022 ms)
      ✕ is CORS-friendly (3 ms)
      ✓ is bound to the right Lambda function (1 ms)

    Expected: {"allowMethods": ["GET"], "allowOrigins": ["*"]}
    Received: undefined

Tests:       1 failed, 2 passed, 3 total

A quick addition to the FunctionUrl in resources.ts should get you to green:

// ...
export const timeURL = new aws.lambda.FunctionUrl("time-url", {
    functionName: timeFunction.name,
    authorizationType: "NONE",
+   cors: {
+       allowOrigins: ["*"],
+       allowMethods: ["GET"],
+   },
});

$ npm test

PASS  ./index.spec.ts
  My speaking clock
    function URL
      ✓ is publicly accessible
      ✓ is CORS-friendly
      ✓ is bound to the right Lambda function

Tests:       3 passed, 3 total

That's it for the tests! Now for the final additions.

Produce the audio message

The last thing we'll need to do is render some audio. For this, we'll need to communicate with Google Translate, our text-to-speech provider of choice, and that means we'll need an HTTP client. I generally like node-fetch, so let's go with that:

npm install --save node-fetch@2

Now, finish the program by importing node-fetch into resources.ts, adding a function to produce a speech-friendly string indicating the current time and updating the CallbackFunction's callback property to send the string to Google Translate to produce the desired result. Here's the complete file, with comments, for reference:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const fetch = require("node-fetch");

// Convert the current time into a speech-friendly string.
export function getSpeechText() {
    const now = new Date();
    now.setSeconds(now.getSeconds() + 6);

    const local = new Date(now.toLocaleString("en-US", { timeZone: "America/Los_Angeles" }));
    const localHour = local.getHours();
    const localMinute = local.getMinutes();
    const h = localHour > 12 ? localHour - 12 : localHour;
    const m = localMinute < 10 ? `oh ${localMinute}` : localMinute; // So 2:03 -> "two-oh-three"
    const s = local.getSeconds();

    return `At the tone, the time will be ${h} ${m}. And ${s} seconds.`;
}

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {

    // Update the Lambda callback body to convert the current time into an MP3 file.
    callback: async () => {
        const text = getSpeechText();
        const speechURL = `https://translate.google.com/translate_tts?ie=UTF-8&q=${encodeURIComponent(text)}&textLen=${text.length}&tl=en&client=tw-ob`;
        const beepURL = "https://www.pulumi.com/uploads/beep.mp3";
        const ttsResponse = await fetch(speechURL);
        const beepResponse = await fetch(beepURL);
        const speech = await ttsResponse.arrayBuffer();
        const beep = await beepResponse.arrayBuffer();

        // Tack a beep onto the end of the audio returned from Google Translate, then
        // render the whole thing as a base-64 encoded string.
        const body = Buffer.concat([Buffer.from(speech), Buffer.from(beep)]).toString("base64");

        // Return an appropriately shaped HTTP response.
        return {
            body,
            headers: { "Content-Type": "audio/mpeg" },
            isBase64Encoded: true,
            statusCode: 200,
        };
    },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
    functionName: timeFunction.name,
    authorizationType: "NONE",
    cors: {
        allowOrigins: ["*"],
        allowMethods: ["GET"],
    },
});

// Export the public URL of our shiny new service.
export const audioURL = timeURL.functionUrl;

Finally, open index.ts (which should still be empty) and add a couple of lines to import the resources module and export the function URL as a Pulumi stack output:

import "./resources";

export { audioURL } from "./resources";

And that's it. We're ready to deploy.

Deploy the service

Doublecheck your AWS credentials are configured, then deploy the new service with a single pulumi up:

$ pulumi up

...
Updating (dev)

View Live: https://app.pulumi.com/cnunciato/audichron-2022/dev/updates/1

     Type                             Name                    Status
 +   pulumi:pulumi:Stack              audichron-2022-dev      created
 +   ├─ aws:iam:Role                  time-function           created
 +   ├─ aws:lambda:Function           time-function           created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-7cd09230  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-a1de8170  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-1b4caae3  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-019020e7  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-4aaabb8e  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-74d12784  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-e1a3786d  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-6c156834  created
 +   ├─ aws:iam:RolePolicyAttachment  time-function-b5aeb6b6  created
 +   └─ aws:lambda:FunctionUrl        time-url                created

Outputs:
    audioURL: "https://52rjmmybwfwcud3nxxhosznpum0cycbw.lambda-url.us-west-2.on.aws/"

Resources:
    + 13 created

Duration: 20s

When the update completes, open the URL in your browser of choice, and you should hear the message you've been waiting for --- one that might even make Pat Fleet proud:

$ open $(pulumi stack output audioURL)

When you're ready, you can tear everything down with a pulumi destroy.

Wrapping up

Beyond just having a nifty (and admittedly rather silly) new way to tell time, you should have a much better sense at this point of how you can use Pulumi with Jest to write better, safer infrastructure code.

From here, there's a bunch more you might think about next: writing more tests to cover the code we just added, exploring some additional flavors of testing in the docs, or having a look at a few examples. You'll find the full source for this walkthrough up on GitHub as well.

Happy testing!

Fullstack Pulumi: Deploying the MERN Stack on DigitalOcean

Christian Nunciato — Mon, 14 Mar 2022 17:33:32 +0000

As a developer, I get lots of ideas for web apps—little things, mostly: nifty ways to keep track of my kids' allowances, habit trackers, shopping lists. Most of them, however, never see the light of day, and not just because I'm lazy; I also tend to get hung up trying to decide what to use for the technology stack.

And as a JavaScript developer, I certainly have options—too many, in fact, and that's part of the problem. Having roughly a hundred and fifty million libraries and frameworks to choose from is definitely better than having none, but at the same time, all that choice can make the actual choosing rather difficult. Which is why, when I just want to get something done, I'll often reach for a combination of tools known as the MERN stack.

MERN-stack apps are three-tier web apps built with MongoDB, Express, React, and Node.js. You can read all about them in the MongoDB docs, but the gist is that they allow you use one language—JavaScript (or TypeScript, if you like)—to manage all three layers of the application stack: the front end as a single-page app built statically with React, the back end as a REST API managed with Express, and the database as a collection of JSON-like documents with MongoDB. MERN might not always the right tool for the job, but for the kinds of apps I tend to find myself building, it generally works out pretty well.

Still, once I'm finished building my app, I'm often faced with a whole other problem: figuring out how to get the app off of my laptop and onto the web.

The cloud hasn't made this an easy task for developers. Choosing a cloud provider, deciding which resources to use (and how to use them), setting up networking, debugging permissions, navigating billing, and all the rest, can be overwhelming—and that's before you've given a single thought to anything having to do with automation or infrastructure as code. What we want, I think, is to be able to focus on our apps, and we're ready to ship, push our code to a repository and wait patiently for a URL to emerge that we can paste into a browser and have everything just work.

Which is why I was so delighted when I discovered DigitalOcean's App Platform.

If you've used DigitalOcean before, you know it's all about making infrastructure more accessible to developers. What you may not know, though, or at least I didn't myself until recently, is that you can do a lot more with DigitalOcean than just virtual machines. A fairly new, fully-managed platform service (think Heroku), App Platform gives you a set of high-level abstractions built to align with the tiers of a typical web application, which means you can focus on what you care about most—your app—and leave the infrastructure and its management to someone else. It's a compelling option for anyone looking to deploy and manage any web application (MERN or otherwise), and as you'll see, with Pulumi and a little bit of code, you can easily do so without ever having to leave the comfort of your IDE.

So let's build ourselves a MERN app and deploy it on DigitalOcean with Pulumi. Given the goal is to focus primarily on the infrastructure and how to code it, we'll start with a pre-baked web application (a simple grocery list), and we'll map its tiers to App Platform constructs and wire everything up with Pulumi.

First steps: setting up

The code for this walkthrough is available as a template repository on GitHub, so if you want to follow along (and you should!), you should grab a copy of your own to work with by forking the repository or creating a new one from the template. Once you've done that, you should also:

Clone the repository to your local machine.
Install Pulumi and Node.js.
Sign into DigitalOcean and obtain a personal access token with read-write permissions.
Grant DigitalOcean access to your GitHub repository by visiting the Apps page, choosing Create App, and following the steps to install DigitalOcean's GitHub app.
Optionally, if you'd like to develop the application locally as well, install and configure MongoDB Community Edition.

One thing to note: Since we'll be provisioning real DigitalOcean resources, there's a chance you could incur a slight cost for what you use. However, as we'll be using the least expensive plan settings available, and tearing everything down when we're through, that cost shouldn't amount to more than a few pennies or so.

Let's get started.

Cloning and inspecting the repository

Once you've cloned your copy of the template repository and navigated to the root, you'll see a couple of files and folders that look something like this:

├── frontend
├── backend
└── package.json

The frontend folder contains the React application, and its job is to render the list of groceries and give you something to interact with (to add items, check them off, delete them, and so on). The scaffolding for the app was generated with a tool called Vite, and all of its logic—form fields, click handlers, API calls, etc.—is contained in src/App.tsx.

The backend folder contains the Express application that defines the REST API. It sets up four API routes to handle the CRUD operations you'd expect in an app like this one:

GET /api/items fetches all items from the database and returns them as a JSON array.
POST /api/items accepts a new item and writes it to the database.
PUT /api/items/:id updates an existing item (to toggle its checked/unchecked status).
DELETE /api/items/:id deletes an item from the database.

The back end supports three configurable properties as well, all of which are exposed as optional environment variables:

BACKEND_SERVICE_PORT, which defaults to 8000
BACKEND_ROUTE_PREFIX, which defaults to /api
DATABASE_URL, which defaults to mongodb://127.0.0.1 for local development

Running the application locally (optional)

You don't have to do this, but if you'd like to, here's how. After installing MongoDB and starting the service (which should be listening by default on port 27107), you can install all front-end and back-end dependencies and start the development server:

$ npm install
$ npm start

With the development server running, you can browse to http://localhost:3000 and see the app:

The front-end and back-end dev servers are set up to compile your TypeScript to JavaScript automatically, and the front-end server is configured to proxy the back-end service (which runs at http://localhost:8000) at a root-relative path of /api. Proxying the API in this way lets you avoid having to wrestle with CORS-related issues, and as you'll see when we deploy to DigitalOcean later, App Platform conveniently supports the same configuration out of the box.

Try adding a few items and marking them off, just to make sure everything's working as expected. If you've got a MongoDB client installed as well—I generally use MongoDB Compass—you should be able to find the grocery-list database and see the items collection filling up with delicious foods:

Now let's have a look at how to go about deploying this stuff.

Charting a course

Earlier I mentioned that every cloud provider handles application deployment a little differently, sometimes in multiple ways, and that's true for DigitalOcean as well. You could deploy the front end as a DigitalOcean Space, or both the front end and back end (and even the database) as a DigitalOcean Droplet. But given the shape of this application, the best fit is really is App Platform, for several reasons.

One is that because App Platform apps are comprised these high-level components—abstractions like static site, service, and database—it's pretty much purpose-built for an application like this one, and DigitalOcean customizes the deployment of each component based according to its type. Static websites are distributed and cached on DigitalOcean's CDN, services are packaged and delivered as containers (with its Kubernetes platform), and databases are deployed as configurable managed services. All of this means you're not only able to stay focused on the application itself, but you're able to scale each one of these components up or down however you like, and even delegate your front-end and back-end build processes to DigitalOcean to be handled in response to commits on one or more external Git repositories.

App Platform apps can be configured in one of two ways: manually, by configuring their components individually in the DigitalOcean web console, or programmatically, in the form of an App Platform spec, a JSON document submitted over DigitalOcean's REST API. In our case, we'll indirectly go the latter route, using Pulumi with the DigitalOcean provider package to define an app spec comprised of three components:

A staticSite component mapped to the frontend folder
A service component mapped to the backend folder
A database component mapped to a managed MongoDB cluster (which we'll also configure to be accessible only by the service component)

And once deployed, it'll all be available at a single DigitalOcean-provided URL.

Let's begin by creating new Pulumi project.

Creating the project

In the root of the repository, make a new folder called infra, change to it, then run pulumi new using the digitalocean project template:

$ mkdir infra && cd infra
$ pulumi new digitalocean-typescript

At the prompts, use the following values:

For project name, use grocery-list
For description, use Deploying a MERN-stack app on DigitalOcean
For stack name, use dev, the default

When the command completes, you'll have a new Pulumi stack, but you'll still have a few things to configure. For one, the Pulumi DigitalOcean provider needs to be configured to communicate with DigitalOcean on your behalf (to provision your app and its various resources). For this, you can use the access token you obtained earlier from the DigitalOcean console, and you can apply it by setting a single environment variable:

$ export DIGITALOCEAN_TOKEN="your-access-token"

Also, since one of our goals is to have App Platform deploy automatically on every GitHub commit, you'll need to tell DigitalOcean where to find the source code for your front- and back-end components. You could bake these settings right into the Pulumi program itself, but it'd be better to apply them as stack-specific configuration settings, as that'd let you deploy to different stacks later (say, in CI) based on the branch of the commit. Everyone does this a little differently, so for now, let's configure the currently selected stack (which should be dev) to use the default branch of your GitHub repository:

$ pulumi config set repo "your-github-org/your-github-repo" # e.g., cnunciato/fullstack-pulumi-mern-digitalocean
$ pulumi config set branch "your-main-branch"               # e.g., main

With these values in place, you're ready to start writing the program.

App Platform also supports GitLab and other Git-based repositories as well. See the App Specification docs for details.

Writing the program

In your IDE of choice, open index.ts and replace the sample code with the following lines to import the Pulumi and DigitalOcean SDKs and the configuration values you just set, and add a line to specify the DigitalOcean region to deploy into:

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";

// Our stack-specific configuration.
const config = new pulumi.Config();
const repo = config.require("repo");
const branch = config.require("branch");

// The DigitalOcean region to deploy into.
const region = digitalocean.Region.SFO3;

Next, add a few lines to declare the managed MongoDB cluster. We'll use just one node for now—additional replica nodes can easily be added later by increasing the nodeCount value—and go with the least expensive performance settings:

// ...

// Our MongoDB cluster (currently just one node).
const cluster = new digitalocean.DatabaseCluster("cluster", {
    engine: "mongodb",
    version: "4",
    region,
    size: digitalocean.DatabaseSlug.DB_1VPCU1GB,
    nodeCount: 1,
});

// The database we'll use for our grocery list.
const db = new digitalocean.DatabaseDb("db", {
    name: "grocery-list",
    clusterId: cluster.id,
});

Now for the App Platform spec itself. Notice the digitalocean.App resource takes just one argument, spec, which defines all three of the components of the app: static site, service, and database. Both the static site and the service are configured to use the same GitHub repository (the sourceDir properties indicate their folders within the repository), and both are configured (via the deployOnPush flag) to be rebuilt and redeployed by DigitalOcean on every commit.

The service has a few additional settings that you can use to manage its runtime behavior and deployment topology as well. As in development, we'll configure the service to listen on port 8000 and be available at /api—the entire app will ultimately be proxied transparently by an App Platform load balancer—and it'll be powered by just one container instance, again using the least expensive performance settings:

// ...

// The App Platform spec that defines our grocery list.
const app = new digitalocean.App("app", {
    spec: {
        name: "grocery-list",
        region: region,

        // The React front end.
        staticSites: [
            {
                name: "frontend",
                github: {
                    repo,
                    branch,
                    deployOnPush: true,
                },
                sourceDir: "/frontend",
                buildCommand: "npm install && npm run build",
                outputDir: "/dist",
            }
        ],

        // The Express back end.
        services: [
            {
                name: "backend",
                github: {
                    repo,
                    branch,
                    deployOnPush: true,
                },
                sourceDir: "/backend",
                buildCommand: "npm install && npm run build",
                runCommand: "npm start",
                httpPort: 8000,
                routes: [
                    {
                        path: "/api",
                        preservePathPrefix: true,
                    },
                ],
                instanceSizeSlug: "basic-xxs",
                instanceCount: 1,

                // To connect to MongoDB, the service needs a DATABASE_URL, which
                // is conveniently exposed as an environment variable thanks to its
                // membership in this app spec (below). The CA_CERT value enables
                // a secure connection between API service and database.
                envs: [
                    {
                        key: "DATABASE_URL",
                        scope: "RUN_AND_BUILD_TIME",
                        value: "${db.DATABASE_URL}",
                    },
                    {
                        key: "CA_CERT",
                        scope: "RUN_AND_BUILD_TIME",
                        value: "${db.CA_CERT}",
                    },
                ],
            },
        ],

        // Include the MongoDB cluster as an integrated App Platform component.
        databases: [
            {
                // The name `db` defines the prefix of the tokens used (above) to
                // read the environment variables exposed by the database cluster.
                name: "db",

                // MongoDB clusters are only available in "production" mode.
                // https://docs.digitalocean.com/products/app-platform/concepts/database/
                production: true,

                // A reference to the `DatabaseCluster` we declared above.
                clusterName: cluster.name,

                // The engine value must be uppercase, so we transform it with JS.
                engine: cluster.engine.apply(engine => engine.toUpperCase()),
            }
        ]
    },
});

Technically that's all we need to configure the application—but it wouldn't be a bad idea to add one last thing.

By default, managed MongoDB clusters are configured to be publicly accessible—which is great if you need to be able to connect one yourself, but not so great as a strategy for preventing internet miscreants from doing the same. You can fix this easily by adding a DatabaseFirewall resource to declare the app as a trusted source, thereby rejecting all inbound traffic originating from elsewhere:

// ...

// Adding a database firewall setting grants access solely to our app.
const trustedSource = new digitalocean.DatabaseFirewall("trusted-source", {
    clusterId: cluster.id,
    rules: [
        {
            type: "app",
            value: app.id,
        },
    ],
});

And finally, add one last line to export the app URL, to be generated by DigitalOcean, as a Pulumi stack output:

// ...

// The DigitalOcean-assigned URL for our app.
export const { liveUrl } = app;

With that, you're ready to deploy.

Deploying

Quickly, to recap, here's what we've done so far:

We took a pre-baked MERN app configured to run on localhost.
We mapped the tiers of that app to their corresponding App Platform components.
We wrote a Pulumi program to codify that mapping as an App Platform spec and a managed MongoDB cluster to go along with it.

When you deploy this app in a moment with pulumi up, Pulumi will provision a new MongoDB cluster (which usually takes a few minutes), and then once that's available, DigitalOcean will take our spec and use it to fetch the components of the app from GitHub at the specified branch and build them. From that point forward, any commit you make to that branch will trigger DigitalOcean to fetch, rebuild, and redeploy the app automatically.

Make sure you've installed the DigitalOcean GitHub app as described above—you should see it listed at https://github.com/settings/installations:

Now return to the command line and run pulumi up:

$ pulumi up

Previewing update (dev)

View Live: https://app.pulumi.com/cnunciato/grocery-list/dev/previews/605bf32a-95b1-4221-bc35-0e667b30f38a

     Type                                    Name              Plan
 +   pulumi:pulumi:Stack                     grocery-list-dev  create
 +   ├─ digitalocean:index:DatabaseCluster   cluster           create
 +   ├─ digitalocean:index:DatabaseDb        db                create
 +   ├─ digitalocean:index:App               app               create
 +   └─ digitalocean:index:DatabaseFirewall  trusted-source    create

Resources:
    + 5 to create

Do you want to perform this update?
> yes

Updating (dev)

View Live: https://app.pulumi.com/cnunciato/grocery-list/dev/updates/1

     Type                                    Name              Status
     pulumi:pulumi:Stack                     grocery-list-dev
 +   ├─ digitalocean:index:App               app               created
 +   └─ digitalocean:index:DatabaseFirewall  trusted-source    created

Outputs:
  + liveUrl: "https://grocery-list-fb9bd.ondigitalocean.app"

Resources:
    + 2 created
    3 unchanged

Duration: 2m30s

Again, it'll probably take a few minutes to get everything spun up for the first time, but when the process completes, you'll have a working app at the URL provided by DigitalOcean, emitted as a Pulumi stack output:

$ open $(pulumi stack output liveUrl)

You should also be able to explore your shiny new grocery-list app in the DigitalOcean Console, with all three of its components (and their build-time and runtime logs!) now represented:

Now try making a commit to your repository (any commit will do, but ideally one to the frontend or backend folder), and watch as the app redeploys automatically:

And finally, try scaling the service by bumping the instanceCount from 1 to 2 in the code—or better, if you're up for it, making that value configurable by stack:

  const config = new pulumi.Config();
  const repo = config.require("repo");
  const branch = config.require("branch");
+ const serviceInstanceCount = config.requireNumber("service_instance_count");
  ...
        services: [
            digitalocean.AppSpecServiceArgs(
                ...
+               instanceCount: serviceInstanceCount,

$ pulumi config set service_instance_count 2

$ pulumi up

Updating (dev)

     Type                       Name              Status      Info
     pulumi:pulumi:Stack        grocery-list-dev
 ~   └─ digitalocean:index:App  app               updated     [diff: ~spec]

Outputs:
    liveUrl: "https://grocery-list-fb9bd.ondigitalocean.app"

Resources:
    ~ 1 updated
    4 unchanged

Duration: 1m27s

And when you're finished experimenting, you can tear everything down in just a few seconds with a }}">pulumi destroy:

$ pulumi destroy

Destroying (dev)

     Type                                    Name              Status
 -   pulumi:pulumi:Stack                     grocery-list-dev  deleted
 -   ├─ digitalocean:index:DatabaseFirewall  trusted-source    deleted
 -   ├─ digitalocean:index:DatabaseDb        db                deleted
 -   ├─ digitalocean:index:App               app               deleted
 -   └─ digitalocean:index:DatabaseCluster   cluster           deleted

Resources:
    - 5 deleted

Duration: 19s

Wrapping up, and next steps

Hopefully this gives you a sense of the kinds of things you can do with Pulumi and DigitalOcean—and I definitely encourage you to spend a little time with the App Platform docs to dig a bit deeper into some of these concepts and explore a few others we weren't able to cover. You'll find the full source for this walkthrough on GitHub, of course, with finished branch containing the completed Pulumi program for reference.

From here, you might think about:

Adding a digitalocean.DnsRecord to give your app a custom domain name.
Creating a second stack with pulumi stack init and adjusting the program to make the source branch configurable—a production stack, say, designed to deploy in response to commits to a release branch.
Using Pulumi's GitHub Action to run previews and updates as part of a pull-request based workflow.

Happy coding!

Run Your Own RSS Server on AWS with Pulumi

Christian Nunciato — Tue, 25 Jan 2022 21:27:42 +0000

It's been quite a few years since Google shut down Google Reader, and while a number of nice commercial alternatives have sprung in its wake, none of them has ever been quite the right fit for me personally.

So a while back, after far too much time spent wandering the blogsphere manually, typing URLs into address bars by hand, I decided to go looking to see whether the universe had produced an open-source solution to this problem --- and to my surprise and delight, it had! Miniflux is an excellent little open-source RSS server and reader, written in Go and backed by PostgreSQL, that also happens to be packaged as a Docker container. So in this post, I'll show how easy it is to deploy a Miniflux server of your own on AWS, using only Pulumi and a few lines of TypeScript.

If you're already comfortable with Pulumi, and you just want to get up and running, I've set up a GitHub repo (complete with a Deploy with Pulumi button!) that should have all you need to get going. Just click the button, set a few configs (like your RSS server's administrative password, which will be stored as an encrypted Pulumi secret), and follow the prompts. Your shiny new server should be up and running within minutes.

Sketching it Out

First, let's have a look at the app we'll be building. It'll consist mainly of two parts:

The Miniflux service, which runs as a web application and API service, and
The Miniflux database, a PostgreSQL instance whose schema and data are fully managed by the Miniflux service.

As I mentioned, the service is packaged as a Docker container that runs on port 8080 by default, and exposes a number of environment variables we can use to configure it, including:

DATABASE_URL, a PostgreSQL connection string used by the service to communicate with the PostgreSQL database,
ADMIN_USERNAME and ADMIN_PASSWORD, which we'll use to sign into the service for the first time, and
flags for CREATE_ADMIN and RUN_MIGRATIONS, which denote whether to create the administrative user and run database migrations on startup.

We'll set these properties in the course of developing our program. The service itself, as a container, can easily be run on AWS Fargate, so we can use the @pulumi/awsx package to declare it, and for the database, we'll use @pulumi/aws to provision a small RDS instance of PostgreSQL. And finally, to make the Miniflux service publicly accessible (initially over HTTP; we'll switch to HTTPS later), we'll use an AWS Network Load Balancer, since we'll only need to be able to route traffic by ports and protocols.

Our architecture is therefore shaping up to look something like this:

Let's get started!

Setting Up

We'll start by creating a new Pulumi project. (If you're completely new to Pulumi, it might be good to begin with our Getting Started guide, which walks you through installing Pulumi and configuring it for your cloud provider.) We'll use the built-in aws-typescript project template:

$ mkdir miniflux && cd miniflux
$ pulumi new aws-typescript

Once you've stepped through the prompts, you should be left with a minimal index.ts file that defines and exports an S3 bucket --- but since we won't be needing an S3 bucket for this project, we can remove that code, and keep only the imports we'll need:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

Now let's define a few configuration values for the project.

Configuring the Stack

While we could certainly hard-code all of these values into our program, it'd be better to use Pulumi to set them, since doing so give us the option to vary them by stack (say, if we wanted to run this particular app in multiple environments), but more importantly, to set some passwords for the database user and service administrator. So let's do that first, so we'll have them all ready as we develop our program:

$ pulumi config set db_name miniflux
$ pulumi config set db_username miniflux
$ pulumi config set db_password somesupersecretpassword --secret
$ pulumi config set admin_username admin
$ pulumi config set admin_password anothersupersecretpassword --secret

You can change any of these values if you like (and you should definitely change the two passwords, of course), but note that the passwords are set with the --secret flag, which encrypts their values using Pulumi's built-in support for secrets.

Okay. With our configuration set, we're ready to start building.

Writing the Code

The Pulumi program we're writing will:

import our newly created Pulumi configuration, so we can use its values in our program;
create a new DB Subnet Group using the default VPC for your region;
create a new instance of PostgreSQL, with minimal settings, placing it into the subnet group and giving it access to your default ECS Cluster;
create a new Network Listener to define a publicly accessible URL for the Miniflux service;
create a new Fargate service for the Miniflux app, in your default ECS Cluster, passing it the newly created DB connection and Pulumi config settings; and finally,
export the URL as a Pulumi stack Output so we can navigate to the service.

Let's get coding!

Replace the contents of index.ts with the following code, which comprises the whole program:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Import our Pulumi configuration.
const config = new pulumi.Config();
const dbName = config.require("db_name");
const dbUsername = config.require("db_username");
const dbPassword = config.require("db_password");
const adminUsername = config.require("admin_username");
const adminPassword = config.require("admin_password");

// Get the default VPC and ECS Cluster for your account.
const vpc = awsx.ec2.Vpc.getDefault();
const cluster = awsx.ecs.Cluster.getDefault();

// Create a new subnet group for the database.
const subnetGroup = new aws.rds.SubnetGroup("dbsubnets", {
    subnetIds: vpc.publicSubnetIds,
});

// Create a new database, using the subnet and cluster groups.
const db = new aws.rds.Instance("db", {
    engine: "postgres",
    instanceClass: aws.rds.InstanceTypes.T3_Micro,
    allocatedStorage: 5,
    dbSubnetGroupName: subnetGroup.id,
    vpcSecurityGroupIds: cluster.securityGroups.map(g => g.id),
    name: dbName,
    username: dbUsername,
    password: dbPassword,
    skipFinalSnapshot: true,
});

// Assemble a connection string for the Miniflux service.
const connectionString = pulumi.interpolate `postgres://${dbUsername}:${dbPassword}@${db.endpoint}/miniflux?sslmode=disable`;

// Create an NetworkListener to forward HTTP traffic on port 8080.
const listener = new awsx.lb.NetworkListener("lb", { port: 8080 });

// Create a Fargate service consisting of just one container instance (since that's all we
// really need), passing it the cluster, DB connection and Pulumi config settings.
const service = new awsx.ecs.FargateService("service", {
    cluster,
    desiredCount: 1,
    taskDefinitionArgs: {
        containers: {
            service: {
                image: "miniflux/miniflux:latest",
                portMappings: [
                    listener
                ],
                environment: [
                    { name: "DATABASE_URL", value: connectionString },
                    { name: "RUN_MIGRATIONS", value: "1" },
                    { name: "CREATE_ADMIN", value: "1" },
                    { name: "ADMIN_USERNAME", value: adminUsername },
                    { name: "ADMIN_PASSWORD", value: adminPassword },
                ]
            }
        }
    }
});

// Export the publicly accessible URL.
export const url = pulumi.interpolate `http://${listener.endpoint.hostname}:${listener.endpoint.port}`;

With the program in place, we can bring it to life with pulumi up:

$ pulumi up

In a few minutes, you should see that Pulumi has created 32 new resources:

And if you examine the logs for this stack using pulumi logs -f, you'll see that the database was created, migrations were run, administrative user was created, and the service is now listening:

Finally, looking back at the output of pulumi up, you'll see the URL we exported, to which you can navigate, sign in with the administrative user created by our program, and start RSSing:

Excellent! You've now got your very own fully functioning RSS service, created with only a few dozen lines of code.

Technically, we're done --- but there are a couple of things we might like to do to make this project a bit more complete.

Running on Port 80

Right now, our service's load balancer is configured to listen and forward on the same port, 8080. If we wanted to have the load balancer listen on port 80 (or some other port) instead, we'd need to change its declaration a bit, to create the load balancer and target group more explicitly. Let's do that:

// Create a load balancer, target group and network listener explicitly.
const listener = new awsx.lb.NetworkLoadBalancer("lb")
    .createTargetGroup("group", { port: 8080 })
    .createListener("listener", { port: 80 });

Run pulumi up, and you'll see that the service is now accessible at a new host and port (though the database was of course left unchanged):

Great! But what if we wanted to run over HTTPS, instead?

Running over HTTPS

In order to run the Miniflux service over HTTPS, you'll need to have obtained an SSL certificate from AWS Certificate Manager that corresponds with the domain you plan to use for the service.

Provided you've obtained that certificate, it's defined in the same AWS region as the one you've configured for your Pulumi stack, and you're able to make changes to the DNS records for the domain associated with the certificate, then updating the Pulumi program is easy --- just change the listener declaration to use TLS and port 443, and add a certificateArn property to apply the certificate:

// Run the service over HTTPS, terminating SSL at the load balancer and forwarding to port 8080 on the container.
const listener = new awsx.lb.NetworkLoadBalancer("lb")
    .createTargetGroup("group", { port: 8080, protocol: "TCP" })
    .createListener("listener", { port: 443, protocol: "TLS", certificateArn: "arn:aws:acm:..." });

Run pulumi up one last time, and you should see your service now running on port 443 --- but you won't be able to access it until you add CNAME record to the DNS settings for your domain (I happen to use Google for mine) mapping a new subdomain to your load balancer's hostname:

And with that, you should now be able to browse your RSS server securely:

Finishing Up

In this post, we've seen how easy it is to run a container as a service connected to an RDS database with Pulumi, and to expose that container securely on the web. If we wanted, we could go even farther --- we could refactor the program into Pulumi Components, perhaps (one for the service, one for the database), package it up for sharing on npm, and so on.

But we'll leave those improvements for another day. For now, let's enjoy what we've created! And start catching up on all that reading we've missed.

API Gateway to EventBridge with Pulumi

Christian Nunciato — Thu, 20 Jan 2022 05:28:17 +0000

If you're familiar with Amazon API Gateway, you know it's all about making it easier to provision and manage a web API. Maybe you've used it, as I have, with Crosswalk, our AWS extension library, to stand up a REST API and handle requests with AWS Lambda functions:

import * as awsx from "@pulumi/awsx";

// Create a new API Gateway instance.
const api = new awsx.apigateway.API("my-api", {
    routes: [
        {
            // Define an HTTP endpoint.
            method: "GET",
            path: "/things",

            // Handle requests with an AWS Lambda function.
            eventHandler: async (apiGatewayEvent) => {
                return {
                    statusCode: 200,
                    body: JSON.stringify([
                        "thingOne",
                        "thingTwo",
                    ]),
                };
            },
        },
    ],
});

// Export the API's public URL. 🎉
export const apiUrl = api.url;

I love this abstraction, and I use it all the time; it's an incredibly convenient way to solve a really common problem. But if you examine the code, you'll notice that it's making a fairly strong assumption about how you'll be handling HTTP requests — namely, that you'll do so with a single Lambda function, and that that function will always return a JavaScript object of a particular shape.

Indeed, this arrangement is the API contract of a Lambda proxy integration — API Gateway integrations come in many shapes and sizes; Lambda integrations just happen to be one of the more popular — and much of the time, an approach like this one works out just fine. But depending on the needs of the application, it might not always be the best fit.

Imagine you were building a print-on-demand service, for example, and you wanted to expose an API to let your customers upload documents and have them converted into orders. On AWS, you might reach for API Gateway, as above, to define an HTTP method and route (POST /uploads, say), wire it up to an AWS Lambda, and have the Lambda parse the upload, write the order to a database, and return a response. Visually, such a design might look something like this:

It'd definitely work, and again, it's quite common. But at some point, you might find this tight coupling between API Gateway and Lambda too limiting. Say you wanted to be notified whenever a new order was received, with a Slack message, maybe, in one of your team's shared workspace channels. Under the current design, you'd probably add a few lines of code to the Lambda function to import an HTTP library and make a call to the Slack API to post the message:

That'd work, too — but it'd be less than ideal for a number of reasons. For one, that Lambda would now have two jobs: taking orders and sending Slack notifications. That might be fine for today (it’s only a few lines of code, after all), but over time, those two jobs could easily become three, and then four, and soon enough, that poor Lambda could wind up becoming a lot more difficult to maintain. And given the importance of its main job — capturing orders — it's not something you'd want to risk failing at runtime because of a random Slack outage or other transient internet mishap. Moreover, with every bit of extra work you tack on to that function, you take a tiny step closer to hitting API Gateway's 30-second limit.

What you really need, then, is to be able to take multiple independent, possibly long-running actions based on a single API Gateway request. And one way to do that is with Amazon EventBridge.

Hello, EventBridge

Amazon EventBridge (formerly CloudWatch Events) is as a serverless event bus whose job is to receive structured event data — from your own applications, from other AWS Services — and use that data to notify other applications or services using event-handling rules that you specify. With EventBridge, you can build loosely coupled, event-driven systems that take advantage of AWS's rich support for serverless architectures and that scale gracefully as the needs of those systems change over time.

For this particular application, EventBridge would let you address the multiple-handler problem in a more scalable and easily maintainable way. Rather than have API Gateway invoke Lambda directly, leaving one Lambda responsible for handling multiple tasks, you could have API Gateway publish to EventBridge instead, and let EventBridge invoke as many Lambdas (or other AWS services) as you like, all serverlessly and in parallel — and of course, all easily managed with Pulumi.

Let's see how. Here's a revised architecture diagram showing how you might approach building an application like this one with EventBridge positioned between API Gateway and Lambda:

Now let's have a look at what it'd be like to build it with Pulumi. We won't build everything in this diagram — things like writing to the database or messaging Slack are left for you to explore — but we will build enough to give you clear picture and a working example of how to connect all of these parts into a working application. Specifically:

an API Gateway instance to act as a container for your public API, along with a stage and a route to handle inbound HTTP requests;
an EventBridge integration (comprised of an event bus and event rule) to handle notifications from API Gateway; and finally,
one or more Lambda functions to be invoked in response to event-rule matches.

Let's get started.

Create a new project and stack

As always, it’s good practice to begin with a new project and stack:

$ pulumi new aws-typescript

Make sure you configure your AWS credentials as well, and when prompted, choose whatever stack name and AWS region work best for you.

In this post, we'll be using API Gateway V2 as it's newer and has a few features (like auto-deploy) that make it easier to work with in this context. The same ideas apply to both V1 and V2, though, so while V2 still lacks full feature parity with V1, the underlying resources we'll be using — methods, routes, integrations — are the same. You'll find links to examples of both at the end of this post.

Create the API gateway and stage

Start by replacing the contents of index.ts with the following code to declare a new API Gateway API:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an HTTP API.
const api = new aws.apigatewayv2.Api("api", {
    protocolType: "HTTP",
});

Next, add a stage (feel free to name it whatever you like; I usually use the current stack name for convenience), and set it to deploy automatically whenever a change is made to the API:

// ...

// Create a stage and set it to deploy automatically.
const stage = new aws.apigatewayv2.Stage("stage", {
    apiId: api.id,
    name: pulumi.getStack(),
    autoDeploy: true,
});

The next thing to do is register a route on the gateway to give your users a publicly accessible endpoint to upload to. But in order to do that, you'll need to tell API Gateway what to do when an upload happens. Since the plan of record is to notify EventBridge (using API Gateway’s built-in support for it), you’ll need to declare a few EventBridge things first.

Add an event bus and event rule

Every AWS account gets an event bus by default (one that's aptly named default), but given how easy it is to create one, we might as well do that for this application. You'll also need to define an event rule — a resource that watches a specific event bus for events that conform to a particular pattern or shape, and then routes those events to one or more targets (e.g., Lambda functions). Add the following lines to your program for both:

// ...

// Create an event bus.
const bus = new aws.cloudwatch.EventBus("bus");

// Create an event rule to watch for events.
const rule = new aws.cloudwatch.EventRule("rule", {
    eventBusName: bus.name,

    // Specify the event pattern to watch for.
    eventPattern: JSON.stringify({
        source: ["my-event-source"],
    }),
});

The most noteworthy property of the EventRule resource is probably the eventPattern. EventBridge events all conform to a certain schema, and in this case, we’re expressing that this particular event rule should take action on any event that originates from my-event-source. (The source property is just a free-form string that by convention identifies the application or service responsible for the event.)

With the event bus and event rule in place, you’re ready to define the integration itself — the resource responsible for connecting the gateway route (which we’ll get back to shortly) to your newly created event bus. As I mentioned earlier, there are several types of API Gateway integration to choose from, each one suited to a particular purpose. For this example, the AWS_PROXY type is good fit, as it’s simple and requires very little code; it doesn't give you quite as much control over the API Gateway response as you might like — as a proxy, it just returns to the caller whatever is returned by the backend, in this case EventBridge — but it's more than enough for the task at hand.

Add the following lines for the integration and route. The comments should explain what each block is doing:

// ...

// Define a policy granting API Gateway permission to publish to EventBridge.
const apiGatewayRole = new aws.iam.Role("api-gateway-role",
    {
        assumeRolePolicy: {
            Version: "2012-10-17",
            Statement: [
                {
                    Action: "sts:AssumeRole",
                    Effect: "Allow",
                    Principal: {
                        Service: "apigateway.amazonaws.com",
                    },
                },
            ],
        },
        managedPolicyArns: [
            "arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess",
        ],
    },
);

// Create an API Gateway integration to forward requests to EventBridge.
const integration = new aws.apigatewayv2.Integration("integration", {
    apiId: api.id,

    // The integration type and subtype.
    integrationType: "AWS_PROXY",
    integrationSubtype: "EventBridge-PutEvents",
    credentialsArn: apiGatewayRole.arn,

    // The body of the request to be sent to EventBridge. Note the
    // event source matches pattern defined on the EventRule, and the
    // Detail expression, which just forwards the body of the original
    // API Gateway request (i.e., the uploaded document).
    requestParameters: {
        EventBusName: bus.name,
        Source: "my-event-source",
        DetailType: "my-detail-type",
        Detail: "$request.body",
    },
});

// Finally, define the route.
const route = new aws.apigatewayv2.Route("route", {
    apiId: api.id,
    routeKey: "POST /uploads",
    target: pulumi.interpolate`integrations/${integration.id}`,
});

With that, you’re ready to configure the Lambda.

Add a Lambda function handler

The hard part’s done: you’ve declared an API and route, mapped that route to an integration, configured the integration to publish events to an event bus, and defined an event rule to respond to those events. All that's left now is to tell the rule how to respond.

So to finish things off, you need:

a Lambda function to handle uploads,
an EventBridge target to bind your event rule to that function, and
an IAM policy granting EventBridge permission to invoke the function.

Add the following lines to your program to complete it:

// ...

// Create a Lambda function handler with permission to log to CloudWatch.
const lambda = new aws.lambda.CallbackFunction("lambda", {
    policies: [aws.iam.ManagedPolicies.CloudWatchLogsFullAccess],
    callback: async (event: any) => {

        // For now, just log the event, including the uploaded document.
        // That'll be enough to verify everything's working.
        console.log({ source: event.source, detail: event.detail });
    },
});

// Create an EventBridge target associating the event rule with the function.
const lambdaTarget = new aws.cloudwatch.EventTarget("lambda-target", {
    arn: lambda.arn,
    rule: rule.name,
    eventBusName: bus.name,
});

// Give EventBridge permission to invoke the function.
const lambdaPermission = new aws.lambda.Permission("lambda-permission", {
    action: "lambda:InvokeFunction",
    principal: "events.amazonaws.com",
    function: lambda.arn,
    sourceArn: rule.arn,
});

// Export the API Gateway URL to give us something to POST to.
export const url = pulumi.interpolate`${api.apiEndpoint}/${stage.name}`;

All Together, Now

Now that the program's complete, you can run Pulumi to bring it to life:

$ pulumi up
...

Updating (dev)
...

     Type                             Name                      Status
 +   pulumi:pulumi:Stack              eventbridge-v2-dev        created
 +   ├─ aws:apigatewayv2:Api          api                       created
 +   ├─ aws:apigatewayv2:Stage        stage                     created
 +   ├─ aws:cloudwatch:EventBus       bus                       created
 ...

Outputs:
    apiURL: "https://geqfietbcl.execute-api.us-west-2.amazonaws.com/dev"

Resources:
    + 15 created

Duration: 31s

When the update finishes, you'll have a fully functioning API Gateway-EventBridge integration that you can verify with curl:

$ curl --data '{"some-key": "some-value"}' --header "Content-Type: application/json" \
     "$(pulumi stack output url)/uploads"

{"Entries":[{"EventId":"cdc44763-6976-286c-9378-7cce674dff81"}],"FailedEntryCount":0}

Note the response, which comes directly from EventBridge (courtesy of the AWS_PROXY integration), confirming the event was received and written to the event bus.

The final step is to confirm that the request made it all the way to Lambda, which you can easily do by tailing its output with pulumi logs:

$ pulumi logs --follow

Collecting logs for stack dev since 2022-01-06T16:18:48.000-08:00.
...

{
    source: 'my-event-source',
    detail: { 'some-key': 'some-value' }
}

When you're happy, be sure to tidy up with a pulumi destroy.

What's next?

There's a lot more you can do with integrations like this that we didn't cover: add more Lambda function handlers, have EventBridge target other AWS services (Step Functions might be a good one to try next), validate HTTP request bodies (with API Gateway models, to keep bad data from ever reaching EventBridge), and more. Hopefully this gives you sense of what's possible, though. And as promised, you'll find examples that use both versions of API Gateway in our examples repository on GitHub:

API Gateway V2 to EventBridge in TypeScript
API Gateway V2 to EventBridge in Python
API Gateway V1 to EventBridge in TypeScript, with request validation and custom HTTP response mapping

Imperatively Declarative: How (and why) Pulumi is different

Christian Nunciato — Sun, 17 Oct 2021 20:44:49 +0000

In conversations about infrastructure as code, the debate over imperative versus declarative tools still comes up from time to time. Actually, there's not much left to debate: declarative's pretty much won. But somehow, the subject still manages to get people going, probably because what "declarative" means isn't quite as clear as it used to be --- and that's partly because of tools like Pulumi.

When Pulumi comes up in one of these conversations, it usually gets placed on the imperative end of the spectrum; it's an easy mistake to make, considering Pulumi programs are written in imperative languages like JavaScript. But it's a mistake nonetheless. Here's an example of such an exchange from a couple of weeks ago, for instance:

I will be very specific using my previous example. JS is imperative. JSON is declarative.

--- Brian LeRoux (@brianleroux) July 14, 2021

It's worth mentioning that Brian is the creator of arc.codes, a command-line tool that lets you write blocks of JSON or YAML to deploy serverless functions and other things on AWS. Arc is a perfect example of simple, declarative infrastructure as code that's focused on making the easy things easy. Take a look at this terse little Arc file, for example:

app: "hello-world"
http:
  - get: "/thing1"
  - get: "/thing2"

In Arc, this bit of YAML states that at the end of an Arc run, two publicly accessible HTTP endpoints should exist in AWS Lambda (at a URL dynamically assigned by AWS) at the paths /thing1 and /thing2, and that both endpoints should be wired up to respond to HTTP GETs. When you run this file with the Arc CLI --- assuming you've stashed your AWS credentials in the right place, and put your JavaScript functions in a nearby subfolder --- that'll indeed be the case: a minute or so later, those endpoints will exist, and all will be right with the world. Easy.

Moreover, if you were to run that code a second time (having made no changes to the YAML or JavaScript), nothing would happen, because the "desired state" you'd expressed in the arc.yaml file would already have been achieved: with those two endpoints deployed and running in the AWS cloud, Arc (by way of CloudFormation) would have nothing more to do for you. That's declarative infrastructure-as-code (IaC) at work: you describe what you want --- two HTTP endpoints --- and the IaC tool determines the how, computing the work to be done and then making it happen for you.

Imperative IaC, on the other hand, is different. In imperative programming (e.g., in most JavaScript), the code that you write is all about control --- do this, then that; if this, then that. A good example of the difference between declarative and imperative programming would be to compare the experience of building a web page statically with hand-crafted HTML (which is about as declarative as you can get):

...
<section id="things">
    <ol>
        <li>Thing 1</li>
        <li>Thing 2</li>
        <li>Thing 3</li>
    </ol>
</section>
...

... to building one dynamically by scripting the DOM:

let ul = document.createElement("ol");

for (let i = 0; i < 3; i++>) {
    let li = document.createElement("li");
    li.textContent = `Thing ${i + 1}`;
    ul.appendChild(li)
}

document.querySelector("#things").appendChild(ul);

Both yield the same result --- a three-item list --- but in fundamentally different ways. In HTML, the author says what they want, up front, and lets the browser handle the rest. In JavaScript, however, the author tells the browser how to create that list, algorithmically, one element at a time before attaching it programmatically to the page at some point later.

IaC tools vary similarly. Classically declarative tools like Arc, CloudFormation, Terraform, and others have you type out what you want, usually in some sort of structured configuration, and handle the work of provisioning and updating for you. Imperative tools don't do nearly as much; instead, they give you the APIs to tell them what to do and how to do it.

As an example, imagine you wanted to create a couple of storage buckets on Amazon S3. To do that imperatively, you might reach for Amazon's SDK for JavaScript and tap out a small imperative program like this one:

const { S3Client, CreateBucketCommand, ListBucketsCommand } = require("@aws-sdk/client-s3");
const client = new S3Client({ region: "us-west-2" });

(async () => {
    // Name a couple of buckets.
    const desiredBuckets = ["bucket-1", "bucket-2"]
        .map(bucket => `some-interestingly-named-${bucket}`);

    // Imperatively create them, by calling the AWS S3 API directly.
    desiredBuckets
        .forEach(async bucket => {
            await client.send(
                new CreateBucketCommand({ Bucket: bucket })
            );
        });

    // Finally, list all buckets, including the two you just created.
    console.log(
        (await client.send(new ListBucketsCommand({}))).Buckets
    );
})();

You could run this program with Node.js (again, assuming your AWS creds were stashed in their proper locations), and in a few moments, produce the following result:

$ node index.js
[
  {
    Name: 'some-interestingly-named-bucket-1',
    CreationDate: 2021-03-08T18:00:04.000Z
  },
  {
    Name: 'some-interestingly-named-bucket-2',
    CreationDate: 2021-03-08T18:00:04.000Z
  },
]

Nice, right? And easy enough --- assuming you're comfortable with JavaScript.

However, unlike the Arc example I shared earlier, running the program a second time would fail:

$ node index.js
UnhandledPromiseRejectionWarning: BucketAlreadyOwnedByYou

... which is unfortunate, but makes sense, considering the buckets would already have been created. To keep repeated runs of the program from failing --- an important consideration, say, if the program were running as a part of an automated deployment process --- you'd have to write a bit more code to check for the existence of each bucket before attempting to create it:

// ...

(async () => {
    const desiredBuckets = ["bucket-1", "bucket-2"]
        .map(bucket => `some-interestingly-named-${bucket}`);

    // First, fetch a list of all buckets.
    const allBuckets = await client.send(new ListBucketsCommand({}));
    const allBucketNames = allBuckets.Buckets.map(b => b.Name);

    // Create the new buckets...
    desiredBuckets

        // ...but only if they haven't been created already.
        .filter(name => !allBucketNames.includes(name))

        .forEach(async bucket => {
            await client.send(
                new CreateBucketCommand({ Bucket: bucket })
            );
        });
    // ...
})();

And that'd certainly work.

But at the same time, all you really need is a couple of S3 buckets, here, and already you've begun to accumulate a good bit of code --- code that has to be debugged, tested, maintained, and all the rest. If you wanted to assemble something a little more complicated --- a couple of serverless endpoints, maybe, or the virtual infrastructure to run a typical web application --- you'd be looking at writing a lot more code, and this pattern of checking whether to do something before actually doing it (or doing something slightly different, maybe, under certain conditions) would continue to the point that it'd be hard for someone else (or even a future version of yourself) to look at the code and understand what was really going on --- certainly much harder than looking at a few lines of declarative YAML. Sometimes, of course, imperative code is just what you need. But for lots of reasons, declarative tools are usually the right way to go --- which is why, as I said, the debate's pretty much over.

Where does that leave Pulumi, though? If Pulumi programs really are written in imperative languages like JavaScript, doesn't that make Pulumi itself an imperative tool, too, by extension?

In a word, no --- but understanding why the answer is no takes a bit more explanation.

Breakfast as code

I haven't always been a big breakfast person, but these days, I am, and for me, breakfast usually means an egg, some toast, and a bit of orange juice, with an occasional bunch of leafy-green things thrown in for good measure. Represented as JSON, my usual breakfast looks something like this:

{
    "breakfast": {
        "eggs": {
            "count": 1,
            "kind": "scrambled"
        },
        "toast": {
            "count": 1,
            "kind": "multi-grain"
        },
        "juice": {
            "count": 1,
            "kind": "orange"
        }
    }
}

It's a fairly common choice, as breakfasts go --- so common that I could probably walk into any café, hand someone this snippet of JSON, and wait patiently for the result to show up on the table in front of me. In a way, this is declarative breakfast-as-code: I say what I want --- egg, toast, juice --- and a bunch of skilled humans conspire to make that happen for me.

And while I certainly know there's an order in which these things tend to happen --- the eggs need scrambling, so the chef may prep them first; the toast goes quicker, so that'll probably happen later, etc. --- that order isn't important to me as a customer. In the end, all I care about is that when breakfast is ready, it's hot, and on my plate. The JSON document just describes my desired breakfast; it doesn't tell the chef or anyone else how to make it. That's what makes it declarative.

Static text like JSON and YAML aren't the only ways to declare a desired breakfast, though. Here's a little JavaScript program that allocates a similar set of breakfast objects and relationships. Again, notice there isn't any how going on, here --- we're still firmly in what territory:

import { Breakfast, Eggs, Toast, Juice } from "some-menu-or-something";

const breakfast = new Breakfast({
    eggs: new Eggs(1, "scrambled"),
    toast: new Toast(1, "multi-grain"),
    juice: new Juice(1, "orange")
});

Here, breakfast still consists of three things --- object instances of Eggs, Toast, and Juice --- just as it did in the JSON representation. Assuming the constructors of these objects weren't doing anything fancy under the hood (just allocating local instance properties of their own, say), you'd expect that running this program with Node.js would produce, for a moment, a breakfast variable referring to an instance of the Breakfast class, and that the breakfast instance would itself contain references to instances of each of its ingredients before the program finally exited. Without a doubt, this is imperative JavaScript code --- but this particular expression is totally declarative; we've simply stated that breakfast depends on three ingredients, and left it up to the JavaScript engine to handle the dependent allocations and the order in which to perform them.

As it happens, this a lot like how Pulumi works, too. A call to a Pulumi resource constructor (like new aws.s3.Bucket(), for example) is just an object declaration like any other, an expression of your desire to have an S3 bucket exist --- not to create the S3 bucket in that moment, but to have it exist when the program completes. At runtime, the Pulumi SDK and engine conspire to gather up all of the object allocations in your program, figure out their relationships (which objects depend on which, what values they need from each other, and so on), assemble a JSON-serializable object graph representing the full picture, and then use that graph to call on the cloud provider directly to produce the appropriate result. Just like with Arc and other statically declarative tools, the code you write with Pulumi still says what, not how, and Pulumi takes care of delivering the outcome for you.

Here's what it looks like to make a couple of S3 buckets with Pulumi and JavaScript, for example:

const aws = require("@pulumi/aws");

const bucket1 = new aws.s3.Bucket("bucket1");
const bucket2 = new aws.s3.Bucket("bucket2");

If you wanted, since you're working with JavaScript, you could even get a bit fancier by declaring the buckets with Array#map:

[1, 2].map(i => new aws.s3.Bucket(`bucket${i}`));

Run the program once, and you get two buckets (along with a "stack," if you didn't already have one):

$ pulumi up

Updating (dev)

     Type                 Name         Status
 +   pulumi:pulumi:Stack  buckets-dev  created
 +   ├─ aws:s3:Bucket     bucket1      created
 +   └─ aws:s3:Bucket     bucket2      created

Resources:
    + 3 created

Run it again, you get nothing, because the buckets you declared already exist:

$ pulumi up

Updating (dev)

     Type                 Name
     pulumi:pulumi:Stack  buckets-dev

Resources:
    3 unchanged

You could even reverse the sort order and still get the same result (since ultimately, it's up to Pulumi to determine what needs to be done, and how):

[1, 2].map(i => new aws.s3.Bucket(`bucket${i}`)).reverse();

$ pulumi up

Updating (dev)

     Type                 Name
     pulumi:pulumi:Stack  buckets-dev

Resources:
    3 unchanged

Again, that's declarative (and idempotent!) infrastructure as code --- it just happens to have been written with an imperative programming language. You could modify this program to add a third bucket, remove a bucket, declare a JavaScript function to be invoked in response to a bucket event, whatever you want, it's always the same: Pulumi launches your chosen language runtime, listens for object allocations (by way of the @pulumi/aws SDK, for example), registers those allocations with the engine, computes an in-memory graph of resources and relationships, and then calls on your cloud provider directly to issue the appropriate set of changes, in the right order.

Great --- so now you know how Pulumi works.

But it's still worth asking: is all of this really necessary? What kinds of problems does Pulumi actually solve? What makes this "imperatively declarative" approach to infrastructure worth the additional layers of indirection --- the language, runtime, dependencies, and the rest? Wouldn't it be easier just to write a few lines of YAML and be done than to have to contend with all of this extra stuff?

Sure --- for simple things, maybe. But software has a funny way of starting out simple and suddenly becoming annoyingly complex --- often a lot sooner than you think.

When breakfast gets complicated

For me, thanks to my basic breakfast needs, getting what I want is usually no big deal. That's because most cafés are going to have eggs, bread, and orange juice on hand and ready to make --- and also because I'm not all that fussy about the details.

But for my family, it's more complicated. I have three kids, for example, all of whom have mild food sensitivities, and a wife who rarely eats out because of how hard it is to find something she likes. None of them could walk into a diner with an order like mine, because they'd need to be able to ask certain questions first: Are the eggs made with milk? Are the waffles gluten-free? Each of these questions needs to be answered, for real and important reasons, before our collective order can be submitted and fulfilled.

It'd be impossible, in other words, to walk into a restaurant with a handwritten order for a family like ours expecting to have it accepted verbatim without some kind of interaction first. Oh, the waffles aren't gluten-free? Okay --- we'll take an omelet instead. It's always something, and I imagine it's probably like that for most of us: we know what we want, and we're usually able to get it, but not without a little negotiation during the process. At a high level, we know want "breakfast", which is easy. But in practice, we almost always end up having to apply some sort of algorithm, however simple, during that process.

In fact, that's kind of how everything works, software included --- and infrastructure (especially the cloud-based kind) is nothing not fundamentally software. If all you need is a couple of storage buckets or Lambdas or VMs, sure, you can kick out that stuff with a few lines of YAML and get on with your day --- and that's awesome, to be sure. But more often, what you'll find is that you'll eventually need something more, some tiny bit of customization or other that the simple tool can't quite give you out of the box --- and that's when the trouble begins.

When the problem is straightforward and well bounded, in other words, simple tools are great, and often more than enough to get the job done. But when the problem is even a little bit complicated, or when the problem space expands beyond what those simple tools were originally designed for, the tools themselves will tend to bend and crack in the places that weren't really made with complexity in mind.

Take our two buckets, for example. If you knew how many buckets you wanted to create and how you wanted to name them, you could do that fairly easily with HCL, the config language of Terraform:

provider "aws" {
  region = "us-west-2"
}

variable "buckets" {
  type = list(string)
  default = ["1", "2", "3"]
}

resource "aws_s3_bucket" "bucket" {
  count = length(var.buckets)
  bucket = "some-interestingly-named-bucket-${var.buckets[count.index]}"
}

If you're not familiar with HCL, you might need to squint to figure out what's going on here, but it's a lot like our first bucket-provisioning example from earlier: we just loop through a list of strings ("1", "2", and "3"), creating a bucket for each one:

$ terraform apply

aws_s3_bucket.bucket[1]: Creating...
aws_s3_bucket.bucket[2]: Creating...
aws_s3_bucket.bucket[0]: Creating...
aws_s3_bucket.bucket[0]: Creation complete after 3s [id=some-interestingly-named-bucket-1]
aws_s3_bucket.bucket[1]: Creation complete after 3s [id=some-interestingly-named-bucket-2]
aws_s3_bucket.bucket[2]: Creation complete after 3s [id=some-interestingly-named-bucket-3]

Again, this totally works --- assuming the names you've chosen are globally unique.

Now imagine you had to name those buckets in a slightly more complicated way --- using stringified date, perhaps. Naming a bucket dynamically with a format string like YYYY-MM-DD is maybe possible with Terraform (or if not, maybe using a bit of shell scripting with and an HCL variable), but you'd definitely be running into the limits of what HCL is able to do on its own. That's not a knock against HCL, either: every special-purpose language runs the risk of hitting these kinds of limitations eventually.

With general-purpose languages like JavaScript, though, this kind of thing is trivially easy, either with the language alone or with the help of a third-party package to make things even easier --- one like Day.js, for example:

import * as aws from "@pulumi/aws";
import * as dayjs from "dayjs";

// Keep a bucket for each of the last 7 days.
for (let i = 0; i < 7; i++) {
    new aws.s3.Bucket(dayjs().subtract(i, "day").format("YYYY-MM-DD"));
}

$ pulumi up

Updating (dev)

     Type                 Name         Status
 +   pulumi:pulumi:Stack  buckets-dev  created
 +   ├─ aws:s3:Bucket     2021-03-24   created
 +   ├─ aws:s3:Bucket     2021-03-29   created
 +   ├─ aws:s3:Bucket     2021-03-28   created
 +   ├─ aws:s3:Bucket     2021-03-27   created
 +   ├─ aws:s3:Bucket     2021-03-25   created
 +   ├─ aws:s3:Bucket     2021-03-23   created
 +   └─ aws:s3:Bucket     2021-03-26   created

Resources:
    + 8 created

Duration: 9s

When you carve away the language, Pulumi and Terraform are doing a lot of the same things: both work to assemble graphs of resources and dependencies, both use those graphs to communicate with cloud providers directly, and both manage state in conceptually similar ways. It's at the language layer --- and up --- that they really start to diverge.

Again, how much that matters is for you to decide. But as a developer, I'll take a full programming language (especially one I know well) any day of the week, because it means I can do anything the language and its ecosystem can do, and that I probably won't end up in tears in six months when I'm faced with a problem that my tools can't handle. Just yesterday, for example, I found myself wrestling with Bash trying to move a few files between Git repositories. After a frustrating couple of hours of hacking and Googling, I realized I could just use Node.js instead --- and when I did, I was done in a matter of minutes. An expert shell programmer might've made light work of what I was trying to do --- but I'm not an expert shell programmer, and Bash isn't JavaScript. All it took was a couple of Node.js built-ins and libraries:

$ yarn add glob micromatch

... and eight lines of JavaScript later, I was done.

For me, language --- and all that comes with it --- is ultimately what it's all about.

Like React for infrastructure

All of this reminds me of the progression we've seen over the last two decades in web development.

Think of React. Why do we have it? Because HTML alone isn't enough, and imperative DOM scripting leads to reams of unmaintainable code. We got React because we, as developers, wanted to think about, and compose, our front-end applications in declarative ways --- but we needed to retain the flexibility of the JavaScript language. So we got React --- and with it, an imperatively declarative programming model for the web:

// Imperative code...
const offices = [
    "Akron",
    "Nashua",
    "Rochester",
    "Scranton",
    "Syracuse",
    "Utica",
];

export default function DunderMifflinBranchOffices() {
    // ... declaratively rendered...
    return <ul>
        {
            offices.map(office => <li>
                <span>{ office }</span>
                { office === "Scranton" && <span>← The best one</span> }
            </li>)
        }
    </ul>
}

...
<html>
<body>
    <aside>
        <nav>
            <!-- ... and composed. -->
            <DunderMifflinBranchOffices />
        </nav>
    </aside>
    <main>
    ...

It's the same thing with infrastructure: we want a declarative mental model, but we need the control and composabilty of general-purpose languages. Hence tools like Pulumi.

It'll be interesting to see where things go from here; I'm certainly biased, but also a fascinated observer. The trajectory is what interests me most, though --- that, and being able to manage my own infrastructure in ways that feel comfortable to me as a developer.

Forem: Christian Nunciato

CDKTF is deprecated: What's next for your team?

What are the options?

Option 1: Fall back to HCL

Option 2: Migrate to AWS CDK

Option 3: Migrate to Pulumi

What migrating to Pulumi looks like

Conversion and import

Refactoring

An end-to-end example

What's next?

Agentic CI with Buildkite: Three practical examples

Recapping the components

Example 1: GitHub code-review bot

Get the code

How it works

Example 2: GitHub PR build fixer

Get the code

How it works

Example 3: Linear issue handler

Get the code

How it works

What makes these work

Next steps

Automating Jenkins with Configuration as Code (JCasC)

What is JCasC and how does it work?

Diving into a JCasC example

Configuring the controller with jenkins.yaml

Basic build settings

Provisioning users

Setting roles and permissions

Configuring agents

Installing build tools

Configuring a pipeline job and GitHub repo

Configuring shared GitHub credentials

Discovering more configuration settings

Configuring plugins with plugins.txt

Spinning up the cluster with Docker Compose

Defining controller and agent services

Configuring agent self-registration

Wrapping up

Understanding the SLSA framework

What is SLSA?

How SLSA works

SLSA tracks and levels

Core principles

How SLSA is used today

Key terms

What threats is SLSA designed to address?

Real-world supply chain attack examples

Build process compromise

Malicious artifact injection

Chained dependency and code injection into released artifacts

Malicious code injection and insider tampering

How to get started using SLSA

Educate and self-evaluate

Implement the SLSA Build Track

Level 0

Level 1

Level 2

Level 3

Level 4 and beyond (future)

Keep up to date with SLSA

Buildkite Package Registries: Built-in support for SLSA provenance

Summary

How to Create and Share a Pulumi Template

Step 1: Create a new project

Step 2: Design and build the program

Step 3: Turn the project into a template

Step 4: Test-drive the template locally

Step 5: Publish the template

Bonus step: Add a Deploy with Pulumi button

Wrapping up

Deploying a Data Warehouse with Pulumi and Amazon Redshift

Configure the stack

Compose the program

Deploy the stack

Getting data into Redshift

Taking stock, and looking ahead

Shared configuration stacks with AWS Systems Manager

Configuring the controller with `jenkins.yaml`

Configuring plugins with `plugins.txt`