Forem: Lee Briggs

Structuring your Infrastructure as Code

Lee Briggs — Fri, 18 Aug 2023 14:02:41 +0000

If you're thinking of migrating to another infrastructure as code tool (and why would you, everything is great in the IaC world now, right?!) you might find yourself asking yourself a fundamental question when you get started: how do I structure things in a way that scales well and stands the test of time?

There's no canonical answer. Everyone does things slightly different, and different tools have different ideas on the best way.

In my day to day role as a Solutions Engineer at Pulumi I get to answer this question a lot. Customers are migrating from other IaC tools and they want to take this opportunity to think about the way they'd like to structure things.

This blog post is designed to detail my high(ish) level thoughts on the concepts and principals I like to use, and why. As we explore these concepts, I'll talk about some of the lessons I learned from my time in configuration management and the myriad IaC tools I've used before today.

A lot of the concepts in this post are focused on Pulumi, but lots are broadly applicable to other tools.

Layers

I'm sure my system administrator background is showing, but I like to think about infrastructure through the concept of layers similar to the OSI Model. Most of the layers I'll outline here closely mirror the OSI model, but what you'll likely want to do before you create your Git repo or write a single line of code, is group your cloud infrastructure into layers. The reason why will become apparent later.

Layer 0: Billing

The billing layer is where you sign up or input your credit card. Each cloud provider does this differently

AWS: Organization
Azure: Account
Google Cloud: Account

In my experience, while there's API for this stuff you likely don't want to manage this layer with IaC, so do yourself a favour and do it manually.

Layer 1: Privilege

The privilege layer is how you fundementally separate access in the cloud provider. Again, each provider does this a little differently.

Example Resources

AWS: Account
Azure: Subscription
Google Cloud: Project

You might want to manage this layer with IaC, but you need to decide how that'd work. Personally, I find that the API level support for this layer and the rarity of needing to perform this operation means it's often easier to manage this layer manually.

Layer 2: Network

Now we're getting to the layers that'll should definitely be managed by IaC. The network layer is foundational to how everything will work in your infrastructure, and includes things like a VPC, subnets, NAT Gateways, VPNs, and anything else that facilitates network communication.

Example Resources

AWS: VPCs, Subnets, Route Tables, Internet Gateways, NAT Gateways, VPNs
Azure: Virtual Networks, Subnets, Route Tables, Internet Gateways, NAT Gateways, VPNs
Google Cloud: VPCs, Subnets, Route Tables, Internet Gateways, Cloud Nat, VPNs

Layer 3: Permissions

Now we've laid down a network layer, we need to allow other people or applications to talk to the cloud provider API. IAM roles, or service principals live in this layer.

Example Resources

AWS: IAM Roles, IAM Users, IAM Groups
Azure: Service Principals, Managed Identities
Google Cloud: Service Accounts

Layer 4: Data

The data layer is where the resources you're managing really start to open up. This is where you'll find things like databases, object stores, message queues, and anything else that's used to store or transfer data.

Example Resources

AWS: RDS, DynamoDB, S3, SQS, SNS, Kinesis, Redshift, DocumentDB, ElastiCache, DynamoDB
Azure: SQL, CosmosDB, Blob Storage, Queue Storage, Event Grid, Event Hubs, Service Bus, Redis Cache
Google Cloud: Cloud SQL, Cloud Spanner, Cloud Storage, Cloud Pub/Sub, Cloud Datastore, Cloud Bigtable, Cloud Memorystore

Layer 5: Compute

The compute layer is where your applications actually run - this is where you'll find things like virtual machines, containers, and serverless functions.

Example Resources

AWS: EC2, ECS, EKS, Fargate
Azure: Virtual Machines, Container Instances, AKS
Google Cloud: Compute Engine, GKE

Layer 6: Ingress

Layer 6 is where you'll find the resources that allow your applications to be accessed by the outside world.

Example Resources

AWS: Application Load Balancers, Network Load Balancers, Classic Load Balancers, API Gateways
Azure: Application Gateways, Load Balancers, API Management
Google Cloud: Load Balancers, API Gateways

Layer 7: Application

Once we've provisioned all the supporting infrastructure, we now need to actually deploy the application itself. This is where things really get a little tricky and depend entirely on your application's deployment model, technology and architecture.

You might choose not to use IaC for application at all, but if you do..

Example Resources

AWS: Lambda, ECS Tasks, Kubernetes Manifests, EC2 User Data
Azure: Azure Functions, Kubernetes Manifests
Google Cloud: Cloud Functions, Kubernetes Manifests

Visualization

If you're a visual learner like me, you might find this visualization helpful:

Layer	Name	Example Resources
0	Billing	AWS Organization/Azure Account/Google Cloud Account
1	Privilege	AWS Account/Azure Subscription/Google Cloud Project
2	Network	AWS VPC/Google Cloud VPC/Azure Virtual Network
3	Permissions	AWS IAM/Azure Managed Identity/Google Cloud Service Account
4	Data	AWS RDS/Azure Cosmos DB/Google Cloud SQL
5	Compute	AWS EC2/Azure Container Instances/GKE
6	Ingress	AWS ELB/Azure Load Balancer/Google Cloud Load Balancer
7	Application	Kubernetes Manifests/Azure Functions/ECS Taks/Google Cloud Functions

Principal 1: The Rate of Change

One of the difficult concepts to quantify when thinking of these layers is the rate of change that these resources undergo when you're managing them. The lower layers generally will change less frequently than your higher layers, and in addition to this these layers are generally most fraught with risk when changing them.

You might be wondering why this matter - the answer to this is because when structuring your IaC, you'll want to consider how you group resources together in your chosen IaC's encapsulation mechanism. For example, Pulumi uses the concept of projects to group resources together.

When creating and defining resources in a Pulumi project, the fundamental consideration you need think of when adding a resource is "which layer does this resource live in?". You generally shouldn't have resources from different layers in the same project, because the rate of change of those resources will be different and the risk of changing them is different.

Principal 2: Resource Lifecycle

As with all principals in life, there are situations where principal 1 doesn't broadly apply.

There are resources within the above layers where you might think "ah! this is a network resources so I'll put it in my network project" but the lifecycle of the resource doesn't necessarily fit as a shared resource. A great example of this is an AWS security group.

Security groups are generally specific to another resource - perhaps an application you're deploying, a loadbalancer that's shared or maybe a database in the data layer. With these resources, it's generally best to consider the overall lifecycle of the dependent resources when deciding where to put it.

My rule of thumb here is this - if I wanted to provision this resource in a different environment, or better yet, destroy it - what other resources do I want to destroy at the same time?

Another great considering for this is the permissions layer. I already mentioned when discussing permissions that you'll need to think about that layer as shared permissions, application specific permissions are entirely different - they really want to go directly with your application deployment code.

The summary here is: don't be afraid to break the first principal, but make sure when you're doing it you're thinking about the resource lifecycle.

Principal 3: Repositories

The mono-repo vs multi-repo debate is one that's will rage long after we're all done with cloud computing and have migrated back to physical infrastructure, and I'm not going to try and solve it here. What I will say is that I've seen both work well, and both work poorly.

When it comes to IaC repositories, I again come back to our layering system and make differing decisions for where the code for deploying the repo should live is based on the layer.

The Control Repo

For the foundational, shared aspects of the infrastructure, I generally like to include those projects in a single mono-repo which back in my days of using Puppet we called a control repo. I still like to use this nomenclature.

If we refer back to our layers, I generally like to ensure layers 1 and 2 in this shared repo. Things get a little trickier once we get to layer 3, the permissions layer. At this stage, we need to decide if the resource itself is shared or not. A good example of this is an IAM role that might be used for human users instead of application user. This is generally going to be shared across multiple humans and teams, so it's a good candidate for the control repo.

Layer 4 really depends on your application architecture. If you have a message bus spanning multiple applications, putting it in your control repo probably makes sense, but if you have a database that is only used by a single application, you likely don't want it in the control repo for a variety of reasons.

Layer 5 again depends on your organisation's permission model and cloud architecture. It's not uncommon to share shared compute like an ECS cluster or Kubernetes cluster which spans many applications, so including it in an application repo probably isn't going to make much sense. However if you're isolating compute on a per-application basis, you're almost certainly going to want to make this application specific.

Layer 6: As it likely becoming an obvious trend, you'll need to take your application architecture and permission model into account. If you're using a shared load balancer and routing traffic that way, you'll likely want to include it in the control repo, but if you're using a per-application load balancer you'll want to include it in the application repo.

Application Repositories

At the very least, if you're using IaC to deploy your application, having a deploy/ directory in your application repo is a great starting point. If you use Pulumi and want to use the same language as your application to do your deployments, you might consider having all of your dependencies in a single package.json or requirements.txt depending on your chosen language.

You'll need to think about the rate of change here when you're defining projects to group resources together. Do you perhaps need to separate your database layer and your application layer resources? I'd argue that you do, because the rate of change of your application layer is likely going to be much higher than your database layer, but you'll need to make a decision that makes sense for your organisation and project.

Why do this?

The primary reason for making the decision to use both mono-repos and keeping deployment code with applications is built from a perspective of ownership and orchestration.

Foundational infrastructure at layers 1, 2 and possibly up to layer 5 is an order of operations problem and a workflow orchestration problem. In most circumstances, you'll be creating resources that depend on other resources while building the IaC graph.

By deciding to break the resources into different projects, you can create a workflow that allows you to deploy the resources in the correct order. You'll be able to utilize Pulumi stack references to share resources between stacks and projects, but you'll need to ensure that a resource in a project in layer 2 that depends on a project in layer 1 has been created and resolved first.

In a mono-repo, this is as simple as ensuring that the workflow or CI/CD tool runs the projects in the correct order, but in a multi-repo implementation, it becomes a complex orchestration problem that likely involves multi repo webhooks and a lot of duct tape.

Application repos are far enough down the layering system that all of the infrastructure required to run your application will be in place. Placing application deployment infrastructure code in the application repo allows you to give the application developers full ownership of their code from writing and features to getting them into production.

Principal 3: Encapsulation

Once you've made the foundational decisions above, you'll be well on the way to structuring a well defined set of infrastructure as code patterns, but the final thing you'll need to consider is how you'll share resource patterns across your control repo and application repos.

Every IaC tool has a different way of managing this. In Pulumi you can create a Component Resource for a single language or if you want to support multiple language, you might want to create a Pulumi Package, but the reason for doing this is the same: you want to encapsulate a set of best practices that you can share across multiple projects.

A good consideration for for when to start encapsulating resources is to think about your organisational structure and application architecture. If you're only one team deploying a single application, you might not need to go down the path of encapsulating anything, but if you're a platform team that's likely to support dozens of teams to deploy to a shared layer 5 compute resource, creating a Pulumi package that encapsulates the best practices for deploying your application or creating a package for a best practice object storage bucket which has the required permissions is going to save the teams you're supporting a lot of time.

These encapsulations should be in their own, distinct repository. You'll want to version these encapsulations in the same way you version and release your applications - follow semver and make sure you create an API that your downstream users can use.

As your downstream users start to depend on these encapsulations, you can introduce concepts like unit testing to make sure you don't break userspace with your infrastructure.

Pitfalls

A common mistake I see at the encapsulation layer when adopting Pulumi is trying to avoid object orientated principles and using a what I like to call the "function based approach".

As an example of this, you might try and encapsulate some resources into a function. In TypeScript it'd look like this:

export function createBucket(name: string) {
    return new aws.s3.Bucket(name);
}

and in Python like so:

def create_bucket(name: str) -> aws.s3.Bucket:
    return aws.s3.Bucket(name)

The problem with this implementation of an abstraction is that it creates a nested mechanism that is difficult to manage successfully.

If you use a component, you get you get an abstraction mechanism that is much more native to the way the language works. In TypeScript, it looks like this:

export class Bucket extends pulumi.ComponentResource {
    public readonly bucket: aws.s3.Bucket;

    constructor(name: string, args: BucketArgs, opts?: pulumi.ComponentResourceOptions) {
        super("lbrlabs:index:Bucket", name, {}, opts);

        this.bucket = new aws.s3.Bucket(name, args, { parent: this });
    }
}

and in Python:

class Bucket(pulumi.ComponentResource):
    def __init__(self, name: str, args: BucketArgs, opts: Optional[pulumi.ResourceOptions] = None):
        super().__init__("lbrlabs:index:Bucket", name, {}, opts)

        self.bucket = aws.s3.Bucket(name, args, parent=self)

While the instantiation of the resource is more complex, the manageability of this over time is exponentially easier. Trust me, I've untangled this mess before.

Putting it together

An example is worth a thousand words, so let's take a look at a hypothetical control repo and application repo.

Control Repo

Let's say we're going to be super original and call our repo infrastructure. Here's how that might look:

├── certs
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── cluster
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── README.md
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── shared_database
│   ├── Pulumi.development.yaml
|   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── components
│   ├── requirements.txt
│   └── venv
├── domains
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── cache
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── shared_example_app
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── README.md
│   ├── __main__.py
│   ├── productionapp.py
│   ├── requirements.txt
│   └── venv
├── shared_bucket
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── vpc
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
└── vpn
    ├── Pulumi.development.yaml
    ├── Pulumi.production.yaml
    ├── Pulumi.yaml
    ├── __main__.py
    ├── requirements.txt
    └── venv

You can see here that we're using Pulumi stacks to target differing environments (in this case, development and production), and creating a new project for different layers and resources.

You'll likely also notice that I've been quite liberal with my use of directories for each set of services. I'm not grouping all of the network/layer 2 resources into a single project, however I'm following the layering principal by not grouping any resources from different layers into the same project.

You can definitely reduce the number of projects here (for example, you might choose to groups the VPC and VPN projects together in a network project) but I generally find that projects/directories are "free" and reducing the blast radius of changes makes people feel comfortable about contributing to these shared elements.

Application Repo

Once we get to our application repo, it's a lot harder to be prescriptive, but let's say we have a simple Go application called example-app. Here's how that might look:

.
├── Dockerfile
├── Makefile
├── docker-compose.yml
├── deploy
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   └── main.go
├── go.mod
├── go.sum
├── main.go
├── readme.md
└── README.md

Hopefully this is fairly self explanatory, you've got your application and mechanisms for local development with a Dockerfile and Makefile, and we can put our Pulumi code in a deploy/ directory.

Encapsulation Repo

Finally, let's take a look at an example encapsulation repo. These repos can be quite complex, so as an example, take a look at this Pulumi package which encapsulates some level compute here.

Conclusion

We've covered a lot of ground here, but hopefully this has given you some ideas on how you might want to structure your infrastructure as code. If you're using Pulumi, as with all my content, always open to hearing better ideas!

Stop using static cloud credentials in GitHub Actions

Lee Briggs — Sun, 23 Jan 2022 23:20:02 +0000

Picture the scene. You're configuring your automation pipelines, whether it's deploying infrastructure, applications or any other piece of your CI/CD pipeline that needs to access a cloud provider. You want to do things properly, so you define a well scoped role with a minimum set of permissions that you need for the pipeline to be successful. Then you assign those permissions to your cloud provider's authentication mechanism. If you're lucky, your CI/CD pipeline runs in the cloud too, so you never need to define a set of static credentials.

If you're not using self hosted runners for your CI/CD pipeline, you might pause for a second. "I need to remember to rotate these credentials" you think. Maybe you'll set a reminder to rotate them in a month's time, or perhaps you'll set up some elaborate mechanism to rotate them. More likely than not, you'll forget about it completely until your wonderful InfoSec team bug you about them, hopefully it'll be for a compliance reasons and not because someone got hold of them.

Until recently, these hard coded credentials have been not only dangeorus, but unavoidable. Mechanisms for accessing cloud provider from outside the cloud provider itself have been almost non-existent. You defined an IAM user/service principal/service account/insert other mechanism here and you just...hoped.

In addition to these credentials being static and hard to rotate, often the credentials stored in CI/CD services can have extremely broad and permissive privileges. If you're running infrastructure automation, for example, you might need to scope the credentials to basically admin permissions, which is an extremely worrying prospect.

The good news is, this is starting to change, and a well defined protocol is in the middle of these changes.

If you're using GitHub Actions to as your CI/CD tool of choice, you can now use OIDC with the 3 major cloud providers to securely authenticate to that provider. You can find a long, well defined document in the GitHub documentation here. This document clearly states the benefit of using OIDC, but for posterity, we'll repeat them here:

No cloud secrets: You won't need to duplicate your cloud credentials as long-lived GitHub secrets. Instead, you can configure the OIDC trust on your cloud provider, and then update your workflows to request a short-lived access token from the cloud provider through OIDC.

Authentication and authorization management: You have more granular control over how workflows can use credentials, using your cloud provider's authentication (authN) and authorization (authZ) tools to control access to cloud resources.

Rotating credentials: With OIDC, your cloud provider issues a short-lived access token that is only valid for a single workflow run, and then automatically expires.

This all sounds pretty amazing right? No cloud credentials?! How do I set this up?

In the rest of this blog post, we'll look at how you can use Pulumi's TypeScript SDKs to quickly an easy set up GitHub actions, so you don't have to manually configure the access!

It's of course quite possible to use Pulumi's other SDKs, as well as other infrastructure as code tools to do the setup, but we'll use Pulumi in this walkthrough. If you don't want to read a whole blog post, you can go directly to the code here with example actions running to show you this does really work, honest.

To jump directly to your cloud provider, use these links: AWS | Azure | Google Cloud

AWS

The complete code for this example can be found here"

AWS has leaned into OIDC as an authentication mechanism since they introduced IAM roles for service accounts back in 2019. The ability to use OIDC as an authentication mechanism has also been extended to other services, and GitHub actions is one of them.

We'll need the Pulumi AWS provider in order to interact with AWS, as well as the GitHub provider, so make sure you've got those installed in your Pulumi program like so:

npm install @pulumi/aws @pulumi/github

Create an OIDC Provider

The first step in being able to use OIDC in GitHub actions is to define an OIDC provider.

const oidcProvider = new aws.iam.OpenIdConnectProvider("example", {
  thumbprintLists: ["6938fd4d98bab03faadb97b34396831e3780aea1"],
  clientIdLists: ["https://github.com/jaxxstorm", "sts.amazonaws.com"],
  url: "https://token.actions.githubusercontent.com",
});

The URL is important here, as it the thumprint. You can essentially copy and paste these static values. The clientIDList needs to be updated to use your GitHub organization, and this can be used across repositories within your GitHub Org.

Define an IAM Role

Next up, we'll need to define an IAM role. We'll set a condition on this IAM role to scope the role to repository in the Condition field. You can scope the access to anything that exists in the OIDC token - in this example we're allowing all, because YOLO.

const role = new aws.iam.Role("secure-cloud-access", {
  description: "Access for github.com/jaxxstorm/secure-cloud-access",
  assumeRolePolicy: {
    Version: "2012-10-17",
    Statement: [
      {
        Action: ["sts:AssumeRoleWithWebIdentity"],
        Effect: "Allow",
        Condition: {
          StringLike: {
            "token.actions.githubusercontent.com:sub":
              "repo:jaxxstorm/secure-cloud-access:*", // replace with your repo
          },
        },
        Principal: {
          Federated: [oidcProvider.arn],
        },
      },
    ],
  } as aws.iam.PolicyDocument,
});

Next up, we'll need to attach a policy to this role, to define what this repository will be able to do in AWS. In this example I'm going to just add `ReadOnly permissions, but we'll need to be considerate about what this repo is going to do.

`typescript
// get our AWS account ID
const partition = aws.getPartition();

// Attack the readonlyaccess policy
partition.then((p) => {
new aws.iam.PolicyAttachment("readOnly", {
policyArn: arn:${p.partition}:iam::aws:policy/ReadOnlyAccess,
roles: [role.name],
});
});
`

Our final step is to use Pulumi's GitHub provider to store the role name in a GitHub actions secret, so we can quickly and easy access it from a workflow:

typescript new github.ActionsSecret("roleArn", { repository: "secure-cloud-access", secretName: "ROLE_ARN", plaintextValue: role.arn, });

Your final Pulumi program will look a bit like this:

`typescript
import * as aws from "@pulumi/aws";
import * as github from "@pulumi/github"

const oidcProvider = new aws.iam.OpenIdConnectProvider("secure-cloud-access", {
thumbprintLists: ["6938fd4d98bab03faadb97b34396831e3780aea1"],
clientIdLists: ["https://github.com/jaxxstorm", "sts.amazonaws.com"],
url: "https://token.actions.githubusercontent.com",
});

const role = new aws.iam.Role("secure-cloud-access", {
description: "Access for github.com/jaxxstorm/secure-cloud-access",
assumeRolePolicy: {
Version: "2012-10-17",
Statement: [
{
Action: ["sts:AssumeRoleWithWebIdentity"],
Effect: "Allow",
Condition: {
StringLike: {
"token.actions.githubusercontent.com:sub":
"repo:jaxxstorm/secure-cloud-access:*",
},
},
Principal: {
Federated: [oidcProvider.arn],
},
},
],
} as aws.iam.PolicyDocument,
});

const partition = aws.getPartition();

partition.then((p) => {
new aws.iam.PolicyAttachment("readOnly", {
policyArn: arn:${p.partition}:iam::aws:policy/ReadOnlyAccess,
roles: [role.name],
});
});

new github.ActionsSecret("roleArn", {
repository: "secure-cloud-access",
secretName: "ROLE_ARN",
plaintextValue: role.arn,
});

export const roleArn = role.arn;
`

Define your GitHub Actions workflow

Now, let's define a workflow to verify what credentials we got:

`yaml

The workflow Creates static website using aws s3

name: AWS Workflow
on:
push
permissions:
id-token: write
contents: read
jobs:
CheckAccess:
runs-on: ubuntu-latest
steps:
- name: Git clone the repository
uses: actions/checkout@v2
- name: configure aws credentials
uses: aws-actions/configure-aws-credentials@master
with:
role-to-assume: ${{ secrets.ROLE_ARN }}
role-session-name: githubactions
aws-region: us-west-2
- name: Check permissions
run: |
aws sts get-caller-identity
`

You'll notice we're passing the ROLE_ARN from the repository secret directly, so we don't have to hardcode anything. Now, run your Pulumi program to create all the AWS resources needed, and check everything in. You should have access to AWS with ReadOnly access, without having to specify any AWS credentials!

Azure

The complete code for this example can be found here

To configure "credentialless access" in Azure, we can follow a similar pattern. We'll need to use Pulumi's Azure AD provider, the GitHub as well as the Azure Native provider. We're also going to use the Azure SDK to make our life a little easier, so make sure you've run the following in your Pulumi program before you proceed:

yaml npm install @pulumi/azure-native @pulumi/azuread @pulumi/github @azure/arm-authorization @azure/ms-rest-js

Create an Azure AD Application and Service Principal

Our first step is to define the user that GitHub actions will use to get its permissions. We create an Azure AD Application, a Service Principal and a random password, like so:

`typescript
// create an azure AD application
const adApp = new azuread.Application("gha", {
displayName: "githubActions",
});

// create a service principal
const adSp = new azuread.ServicePrincipal(
"ghaSp",
{ applicationId: adApp.applicationId },
{ parent: adApp }
);

// mandatory SP password
const adSpPassword = new azuread.ServicePrincipalPassword(
"aksSpPassword",
{
servicePrincipalId: adSp.id,
endDate: "2099-01-01T00:00:00Z",
},
{ parent: adSp }
);
`

Create a Federated Identity Credential

Now here comes the magic. We're going to create a Federated Identity credential, which has a subject for our repository in it. Azure is stricter about the subject than AWS, so we'll need to define what part of the OIDC token we want to allow access. In this example, I'm allowing the main branch access, but you can use any part of the OIDC token like the environment.

`typescript
/*

This is the magic. We set the subject to the repo we're running from
Also need to ensure your AD Application is the one where access is defined */ new azuread.ApplicationFederatedIdentityCredential( "gha", { audiences: ["api://AzureADTokenExchange"], subject: "repo:jaxxstorm/secure-cloud-access:ref:refs/heads/main", // this can be any ref issuer: "https://token.actions.githubusercontent.com", applicationObjectId: adApp.objectId, displayName: "github-actions", }, { parent: adApp } ); `

Define a permissions this application gets

Now we've got the federated identity credential, we need to create a role assignment. There's going to be a lot to unpack here, so let's look at the code first, then walk through it.

`typescript

import { AuthorizationManagementClient } from "@azure/arm-authorization";
import { TokenCredentials } from "@azure/ms-rest-js";
import * as authorization from "@pulumi/azure-native/authorization";

async function getAuthorizationManagementClient(): Promise {
const config = await authorization.getClientConfig();
const token = await authorization.getClientToken();
const credentials = new TokenCredentials(token.token);
// Note: reuse the credentials and/or the client in case your scenario needs
// multiple calls to Azure SDKs.
return new AuthorizationManagementClient(credentials, config.subscriptionId);
}

async function getRoleIdByName(
roleName: string,
scope?: string
): Promise {
const client = await getAuthorizationManagementClient();
const roles = await client.roleDefinitions.list(scope || "", {
filter: roleName eq '${roleName}',
});
if (roles.length === 0) {
throw new Error(role "${roleName}" not found at scope "${scope}");
}
if (roles.length > 1) {
throw new Error(
too many roles "${roleName}" found at scope "${scope}". Found: ${roles.length}
);
}
const role = roles[0];
return role.id!;
}

const subInfo = authorization.getClientConfig();

subInfo.then((info) => {
new authorization.RoleAssignment("readOnly", {
principalId: adSp.id,
principalType: authorization.PrincipalType.ServicePrincipal,
scope: pulumi.interpolate/subscriptions/${info.subscriptionId},
roleDefinitionId: getRoleIdByName("Reader"),
});
});
`

Huge thanks to Mikhail Shilkov here for his prior art on retrieving the roleDefinitionId

Why do we need all this? Well, we could hard code the roleDefinitionId but looking it up is cleaner. Let's step through it:

First we define an auth client to talk to Azure using the Azure TypeScript SDK

typescript async function getAuthorizationManagementClient(): Promise<AuthorizationManagementClient> { const config = await authorization.getClientConfig(); const token = await authorization.getClientToken(); const credentials = new TokenCredentials(token.token); // Note: reuse the credentials and/or the client in case your scenario needs // multiple calls to Azure SDKs. return new AuthorizationManagementClient(credentials, config.subscriptionId); }

Then, we define a function which can look up an Azure role by its name, rather than the long string you define it by:

typescript async function getRoleIdByName( roleName: string, scope?: string ): Promise<string> { const client = await getAuthorizationManagementClient(); const roles = await client.roleDefinitions.list(scope || "", { filter:roleName eq '${roleName}', }); if (roles.length === 0) { throw new Error(role "${roleName}" not found at scope "${scope}"); } if (roles.length > 1) { throw new Error(too many roles "${roleName}" found at scope "${scope}". Found: ${roles.length}); } const role = roles[0]; return role.id!; }

Now, we use azure-native's authorization package to get the current client information:

typescript const subInfo = authorization.getClientConfig();

Now, we can create a role assignment for our service principal that defines the ReadOnly permission:

typescript subInfo.then((info) => { new authorization.RoleAssignment("readOnly", { principalId: adSp.id, principalType: authorization.PrincipalType.ServicePrincipal, scope: pulumi.interpolate/subscriptions/${info.subscriptionId}, roleDefinitionId: getRoleIdByName("Reader"), }); });

At this stage, our service principal has read only permissions on the subscription we're using. Now, let's allow GitHub actions to use it.

Define GitHub secrets

Our final step is to define the GitHub secrets that we use in our workflow:

`typescript
subInfo.then((info) => {

// define some github actions secrets so your AZ login is correct
new github.ActionsSecret("tenantId", {
repository: "secure-cloud-access",
secretName: "AZURE_TENANT_ID",
plaintextValue: info.tenantId,
});

new github.ActionsSecret("subscriptionId", {
repository: "secure-cloud-access",
secretName: "AZURE_SUBSCRIPTION_ID",
plaintextValue: info.subscriptionId,
});
});

// finally, we set the client id to be the application we created
new github.ActionsSecret("clientId", {
repository: "secure-cloud-access",
secretName: "AZURE_CLIENT_ID",
plaintextValue: adApp.applicationId,
});
`

Note, we need to define the tenantId and subcriptionId inside the promise returned by the subInfo call. The clientId is set to our Azure AD application client id.

Our complete Pulumi program looks like this:

`typescript
import * as pulumi from "@pulumi/pulumi";
import * as authorization from "@pulumi/azure-native/authorization";
import * as azuread from "@pulumi/azuread";
import * as github from "@pulumi/github";

import { AuthorizationManagementClient } from "@azure/arm-authorization";
import { TokenCredentials } from "@azure/ms-rest-js";

// create an azure AD application
const adApp = new azuread.Application("gha", {
displayName: "githubActions",
});

// create a service principal
const adSp = new azuread.ServicePrincipal(
"ghaSp",
{ applicationId: adApp.applicationId },
{ parent: adApp }
);

// mandatory SP password
const adSpPassword = new azuread.ServicePrincipalPassword(
"aksSpPassword",
{
servicePrincipalId: adSp.id,
endDate: "2099-01-01T00:00:00Z",
},
{ parent: adSp }
);

This is the magic. We set the subject to the repo we're running from
Also need to ensure your AD Application is the one where access is defined */ new azuread.ApplicationFederatedIdentityCredential( "gha", { audiences: ["api://AzureADTokenExchange"], subject: "repo:jaxxstorm/secure-cloud-access:ref:refs/heads/main", // this can be any ref issuer: "https://token.actions.githubusercontent.com", applicationObjectId: adApp.objectId, displayName: "github-actions", }, { parent: adApp } );

// retrieve the current tenant and subscription
const subInfo = authorization.getClientConfig();

subInfo.then((info) => {

new github.ActionsSecret("subscriptionId", {
repository: "secure-cloud-access",
secretName: "AZURE_SUBSCRIPTION_ID",
plaintextValue: info.subscriptionId,
});

/* define a role assignment so we have permissions on the subscription

We use the helper to get the role by name, but you can of course define it explicitly */ new authorization.RoleAssignment("readOnly", { principalId: adSp.id, principalType: authorization.PrincipalType.ServicePrincipal, scope: pulumi.interpolate/subscriptions/${info.subscriptionId}, roleDefinitionId: getRoleIdByName("Reader"), }); });

Run your Pulumi program and define all your infrastructure, then we can define our workflow.

Define the GitHub Actions workflow

Our GitHub actions workflow will use the secrets we defined to know how to authenticate. It looks a little bit like this:

`yaml
name: Run Azure Login with OpenID Connect
on: [push]

permissions:
id-token: write
contents: read

jobs:
CheckAccess:
runs-on: ubuntu-latest
steps:
- name: 'Az CLI login'
uses: azure/login@v1
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

- name: 'Run Azure CLI commands'
  run: |
      az account show
      az group list
      pwd

Check all this in, and watch the magic as GitHub Actions is now authenticated against Azure!

Google Cloud

The complete code for this example can be found here

Google Cloud supports OIDC authentication using workflow providers. We'll use the @pulumi/gcp and @pulumi/google-native to achieve our goals here, so make sure the following packages are installed in your Pulumi program:

yaml npm install @pulumi/gcp @pulumi/google-native

Define the Service Account

We'll need a GCP service account to for GitHub actions to use. We'll assign the service account the viewer role:

`typescript
const serviceAccount = new google.iam.v1.ServiceAccount(name, {
accountId: "github-actions"
})

new gcp.projects.IAMMember("github-actions", {
role: "roles/viewer",
member: pulumi.interpolateserviceAccount:${serviceAccount.email}
})
`

Create a WorkloadIdentityPool

Now we'll define a workload identity pool for GitHub actions to use

typescript const identityPool = new gcp.iam.WorkloadIdentityPool("github-actions", { disabled: false, workloadIdentityPoolId:github-actions, });

Create a WorkloadIdentityPoolProvider

We'll now need to define a provider for this workload identity pool. The mappings section is important, here is where we map Google OIDC subjects to the OIDC token objects. The following works pretty well:

typescript const identityPoolProvider = new gcp.iam.WorkloadIdentityPoolProvider( "github-actions", { workloadIdentityPoolId: identityPool.workloadIdentityPoolId, workloadIdentityPoolProviderId: "github-actions", oidc: { issuerUri: "https://token.actions.githubusercontent.com", }, attributeMapping: { "google.subject": "assertion.sub", "attribute.actor": "assertion.actor", "attribute.repository": "assertion.repository", }, } );

Assign the workload identity permission

Now we've defined the workload identity and provider, we need to allow our earlier defined service account to use these new resources:

typescript new gcp.serviceaccount.IAMMember("repository", { serviceAccountId: serviceAccount.name, role: "roles/iam.workloadIdentityUser", member: pulumi.interpolateprincipalSet://iam.googleapis.com/${identityPool.name}/attribute.repository/jaxxstorm/secure-cloud-access})

Notice here that we interpolate the name of the identity pool, and also the name of the repository that we want to access.

Create the GitHub actions secrets

Now we'll store some important information in GitHub secrets so we don't have to hardcode them in our workflow:

`typescript
new github.ActionsSecret("identityProvider", {
repository: "secure-cloud-access",
secretName: "WORKLOAD_IDENTITY_PROVIDER",
plaintextValue: identityPoolProvider.name,
});

new github.ActionsSecret("subscriptionId", {
repository: "secure-cloud-access",
secretName: "SERVICE_ACCOUNT_EMAIL",
plaintextValue: serviceAccount.email,
});
`

We're storing the identity pool provider name, and the service account we created's email address as actions.

Your complete Pulumi program should look like this:

`typescript
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as google from "@pulumi/google-native";
import * as github from "@pulumi/github";

const name = "github-actions";

const serviceAccount = new google.iam.v1.ServiceAccount(name, {
accountId: "github-actions",
});

new gcp.projects.IAMMember("github-actions", {
role: "roles/viewer",
member: pulumi.interpolateserviceAccount:${serviceAccount.email},
});

const identityPool = new gcp.iam.WorkloadIdentityPool("github-actions", {
disabled: false,
workloadIdentityPoolId: ${name}-4,
});

const identityPoolProvider = new gcp.iam.WorkloadIdentityPoolProvider(
"github-actions",
{
workloadIdentityPoolId: identityPool.workloadIdentityPoolId,
workloadIdentityPoolProviderId: ${name},
oidc: {
issuerUri: "https://token.actions.githubusercontent.com",
},
attributeMapping: {
"google.subject": "assertion.sub",
"attribute.actor": "assertion.actor",
"attribute.repository": "assertion.repository",
},
}
);

new gcp.serviceaccount.IAMMember("repository", {
serviceAccountId: serviceAccount.name,
role: "roles/iam.workloadIdentityUser",
member: pulumi.interpolateprincipalSet://iam.googleapis.com/${identityPool.name}/attribute.repository/jaxxstorm/secure-cloud-access,
});

new github.ActionsSecret("identityProvider", {
repository: "secure-cloud-access",
secretName: "WORKLOAD_IDENTITY_PROVIDER",
plaintextValue: identityPoolProvider.name,
});

new github.ActionsSecret("subscriptionId", {
repository: "secure-cloud-access",
secretName: "SERVICE_ACCOUNT_EMAIL",
plaintextValue: serviceAccount.email,
});

export const workloadIdentityProviderUrl = identityPoolProvider.name;
export const serviceAccountEmail = serviceAccount.email;
`

Run your Pulumi program, created the needed resources and then we can define our workflow.

Define the workflow

Now we've configured all the access we need, we can define a workflow to check our access:

`yaml
name: List services in GCP
on:
push

permissions:
id-token: write

jobs:
Get_OIDC_ID_token:
runs-on: ubuntu-latest
steps:
- id: 'auth'
name: 'Authenticate to GCP'
uses: 'google-github-actions/auth@v0.3.1'
with:
create_credentials_file: 'true'
workload_identity_provider: ${{ secrets.WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.SERVICE_ACCOUNT_EMAIL }}
- id: 'gcloud'
name: 'gcloud'
run: |-
gcloud auth login --brief --cred-file="${{ steps.auth.outputs.credentials_file_path }}" --project briggs-237615
gcloud auth list
`

We're using the auth action, creating a credentials file and then verifying we're authenticated.

Check all this in, and watch in awe as your GitHub action runs with GCP access without any hardcoded credentials!

Wrap Up

This blog post guides you through accessing the 3 major cloud providers with GitHub Actions without specifying hardcoded credentials. It's my hope that more CI/CD providers will offer this support soon, as well as other awesome cloud providers.

VB.NET - The Future of Infrastructure as Cloud

Lee Briggs — Thu, 17 Dec 2020 06:35:27 +0000

I can remember the exact moment I first realized I didn't want to be a software developer. It was May 2009, and I was desperately trying to finish my final University project for the module "object-orientated Programming."

My module tutor had tasked us with creating an application of our choosing in VB.NET. I can unapologetically say I didn't enjoy the process. I preferred focusing on systems, understanding how things fit together. I was spending my days getting stuck in the minutiae of fixing bugs in my code.

My career progression from there followed a fairly traditional "System Administrator" path. I managed some Linux servers, the number grew, and I went headfirst into configuration management. As the DevOps movement came around, I embraced Kubernetes early and spent several years using cutting edge technologies in exciting ways. Despite telling myself in 2009 I didn't want to be a software engineer, in 2020 I got my first "software engineer" title here at Pulumi, and now I'm lucky enough to spend my time writing code to program the cloud.

Revisiting the Past

Pulumi is an exciting product, but what's incredible to see working here is how the platform scales. All of the Pulumi provider SDKs get programmatically generated, which means that once support for a language gets added, all of the available providers will get that SDK generated. Adding a new language has significant engineering work up front, of course, but ongoing maintenance burden drops over time rather than increases.

In November 2019, Pulumi added support for the .NET core languages. We've seen significant adoption from the .NET community, and if you take a look at our examples repo, you'll see dozens of examples of creating cloud infrastructure with C# and F#, the flagship languages of the .NET framework.

Of course, .NET core languages also include my trusty old friend VB.NET. I haven't written a single line of VB.Net since I handed in my university project in 2009, and I have a vague recollection that I told myself that I never would again.

However, this is 2020. So I asked myself, what is the very last thing I would want to create with Pulumi in VB.NET. The answer is obvious, of course - Kubernetes resources.

The old and the new

In many ways, VB.NET and Kubernetes couldn't be further apart. VB.NET, of old Microsoft - a language of "code-behind", written for the era of monolithic desktop apps and windows forms and Kubernetes, a complex distributed system written to be cloud-native.

With nostalgia in mind, I asked in our internal slack just how good our VB.NET support was. I quickly got a helpful answer from one of our resident .NET experts which can when summarized is along the lines of:

It should work, but...why?

The answer to that question in my mind was of course, because it's 2020.
Our Pulumi templates (used to bootstrap new Pulumi projects) helpfully already had VisualBasic examples. I imagine they're lonely, because I couldn't find a single record of them ever getting used.

Bootstrapping my new cloud VB.NET project, I could feel the old familiarity of hating being a software developer return to me. Shaking off the innate feeling, I set about doing the Pulumi equivalent of "Hello, world" - creating an S3 bucket in AWS.

Imports Pulumi
Imports Pulumi.Aws.S3

Class MyStack
    Inherits Stack

    Public Sub New()
        ' Create an AWS resource (S3 Bucket)
        Dim bucket = New Bucket("my-bucket")

        ' Export the name of the bucket
        Me.BucketName = bucket.Id
    End Sub

    <Output>
    Public Property BucketName As Output(Of String)
End Class

I sort of hoped it would fail. VB.Net is 19 years old. Surely it has no place provisioning AWS resources?

I had some difficulty familiarizing myself with the syntax, but after yelling at my IDE a couple times, I finally managed to get my S3 bucket created.

At this point, I decided to antagonize the internet and rehash a meme from earlier in the year:

The responses I received weren't surprising:

Undeterred, I went for broke. Adding the Kubernetes provider and building out the required resources to deploy to an already provisioned EKS cluster was easier than I expected. I actually started to feel like the syntax of VB.net felt oddly suited to the task. It looks.... good?

Imports Pulumi.Kubernetes.Apps.V1
Imports Pulumi.Kubernetes.Core.V1
Imports Pulumi.Kubernetes.Types.Inputs.Apps.V1
Imports Pulumi.Kubernetes.Types.Inputs.Core.V1
Imports Pulumi
Imports Pulumi.Kubernetes.Types.Inputs.Meta.V1

Class NginxStack 
  Inherits Stack

  Public Sub New()    

    ' an input map of labels
    Dim labels = New InputMap(Of String) From {{"app", "nginx"}}

    ' define the deployment spec
    Dim containerPortArgs = New ContainerPortArgs With { .ContainerPortValue = 80 }
    Dim containerArgs = New ContainerArgs With { 
          .Image = "nginx",
          .Name = "nginx",
          .Ports = containerPortArgs }
    Dim podSpecArgs = New PodSpecArgs With { .Containers = containerArgs }
    Dim template = New PodTemplateSpecArgs With { .Spec = podSpecArgs, .Metadata = New ObjectMetaArgs With { .Labels = labels } }
    Dim labelSelectorArgs = New LabelSelectorArgs With { .MatchLabels = labels }
    Dim deploymentSpecArgs = New DeploymentSpecArgs With { 
          .Template = Template,
          .Replicas = 2,
          .Selector = labelSelectorArgs}

    ' create the actual deployment
    Dim deployment = New Pulumi.Kubernetes.Apps.V1.Deployment("nginx", New DeploymentArgs With { .Spec = deploymentSpecArgs })

    ' define the service spec
    Dim servicePortArgs = New ServicePortArgs With { .Port = 80, .TargetPort = 80 }
    Dim serviceSpecArgs = New ServiceSpecArgs With { .Ports = servicePortArgs, .Type = "LoadBalancer", .Selector = labels }
    Dim objectMetaArgs = New ObjectMetaArgs With { .Name = "nginx", .Labels = labels }

    ' create a service
    Dim svc = New Pulumi.Kubernetes.Core.V1.Service("nginx", New ServiceArgs With { .Metadata = objectMetaArgs, .Spec = serviceSpecArgs } )

    Address = svc.Status.Apply(Function(status) If(status.LoadBalancer.Ingress(0).Ip, status.LoadBalancer.Ingress(0).Hostname))

  End Sub

  <Output>
  Public Property Address As Output(Of String)
End Class

I shared what I'd done with some of the Slack communities I frequent:

I knew what I was doing was a little controversial, but I wasn't ready for this:

I felt defensive all of a sudden. Was I starting to feel defensive of VB.NET?

Luckily, the CEO of Pulumi, a man who knows a thing or two about programming languages, saw my example code and saw the beauty of what I'd done:

Not everyone saw the inner-beauty:

You can see this incredible collaboration of old and new here:

The full code, if you'd like to see it, is here

If you'd ask me what I learned from this experience, I'd be honest and say - nothing. Do I really believe VB.NET is the future of cloud engineering? No. It's not. I'm unlikely to recommend to Pulumi customers that they break out Visual Studio and write some VB.NET, but we know the options is there if they ever want to use it.

I've gone on record before to talk about how much I loathe templating YAML and I love the expressability Pulumi gives me with a familiar programming languages. Even though I hated VB.NET in university, I would still prefer to express my cloud resources with a 19 year old programming language over a configuration language with template replacement.