Forem: Magnus Markling

Uiua: Weekly challenge 242

Magnus Markling — Sat, 11 Nov 2023 13:56:20 +0000

Introduction
Task 2
Task 1

Introduction

Uiua is an interesting new language. Strongly influenced by APL and BQN, it's array-oriented and stack-based. To explore it briefly, I will walk through my solutions to this week's Perl weekly challenge (242).

Uiua has an excellent language tour as well as a tutorial on its own site, which I'd recommend you go through if you're just starting out. I'm going to assume you have at least some syntactic familiarity with the language already.

Task 2

We'll actually start with Task 2 this week, since it's way easier.

Here is the problem description:

You are given `n x n` binary matrix.

Write a script to flip the given matrix as below.

1 1 0
0 1 1
0 0 1

a) Reverse each row

0 1 1
1 1 0
1 0 0

b) Invert each member

1 0 0
0 0 1
0 1 1

We'll start by putting the example matrix on the stack, in Uiua notation

[[1 1 0] [0 1 1] [0 0 1]]

giving

Reversing each row can by done by a combination of the ≡ rows and ⇌ reverse functions. Like this:

[[1 1 0] [0 1 1] [0 0 1]]
≡⇌

giving

That takes care of part a). What we need to do now is just reverse each element. Since most simple functions in Uiua are pervasive, this is as simple as calling the ¬ not function on our previous result. Like this:

[[1 1 0] [0 1 1] [0 0 1]]
¬ ≡⇌

giving

which is the expected answer. In the task there were two additional examples:

Example 1

Input: @matrix = ([1, 1, 0], [1, 0, 1], [0, 0, 0])
Output: ([1, 0, 0], [0, 1, 0], [1, 1, 1])

Example 2

Input: @matrix = ([1, 1, 0, 0], [1, 0, 0, 1], [0, 1, 1, 1], [1, 0, 1, 0])
Output: ([1, 1, 0, 0], [0, 1, 1, 0], [0, 0, 0, 1], [1, 0, 1, 0])

Let's put our logic in a function:

FlipMatrix ← ¬≡⇌

and add these examples as unit tests by combining ⍤ assert and ≍ match in a --- test scope:

---
FlipMatrix [[1 1 0] [1 0 1] [0 0 0]]
⍤.        ≍[[1 0 0] [0 1 0] [1 1 1]]

FlipMatrix [[1 1 0 0] [1 0 0 1] [0 1 1 1] [1 0 1 0]]
⍤.        ≍[[1 1 0 0] [0 1 1 0] [0 0 0 1] [1 0 1 0]]
---

Both unit tests pass, so we are done with Task 2.

Task 1

Let's move on to the more challenging task of the two. Here is the problem description:

You are given two arrays of integers.

Write a script to find out the missing members in each other arrays.

Let's start with the first example.

Example 1

Input: @arr1 = (1, 2, 3)
       @arr2 = (2, 4, 6)
Output: ([1, 3], [4, 6])

(1, 2, 3) has 2 members (1, 3) missing in the array (2, 4, 6).
(2, 4, 6) has 2 members (4, 6) missing in the array (1, 2, 3).

Since we'll be working with these example arrays a lot, let's start by putting them into variables a and b:

a ← [1 2 3]
b ← [2 4 6]

To get what members are common between the two arrays we can use ∊ member like this:

∊ a b

which gives us a filter mask with 1s corresponding to the common elements:

[0 1 0]

We want the missing ones (i.e. the not common elements), so we invert this array with ¬ not:

¬∊ a b

giving

[1 0 1]

We can now apply this inverted mask to our original array using ▽ keep to get the missing members

¬∊ a b
▽: a

giving

[1 3]

which is correct. Note that this could also be put on a single line like this:

▽: a ¬∊ a b

We also need to do the opposite by reversing the arrays in our input

▽: b ¬∊ b a

giving

[4 6]

which is also correct. Since the output should be in the form of a single 2d array

Output: ([1, 3], [4, 6])

we could do it like this (variable names are case sensitive)

A = ▽: a ¬∊ a b
B = ▽: b ¬∊ b a
[A B]

or by using ⊟ couple like this

A = ▽: a ¬∊ a b
B = ▽: b ¬∊ b a
⊟ A B

both giving the expected 2d array

Let's try to do away with some repetition. We'll start by creating a single function that gets us the missing members in one direction between two arrays:

F ← ▽¬∊,

And combining them with ⊟ couple

⊟  F b a  F a b

Note that this is equal to

⊟  F : a b  F a b

Now things start to get interesting. Notice the symmetry, where we do two different operations to the same set of values. Uiua has a couple interesting functions for doing that, in this case we'll use ⊃ fork:

⊟ ⊃(F∶)(F) a b

still giving us what we want

So far so good.

Example 2

Input: @arr1 = (1, 2, 3, 3)
       @arr2 = (1, 1, 2, 2)
Output: ([3])

(1, 2, 3, 3) has 2 members (3, 3) missing in the array (1, 1, 2, 2). Since they are same, keep just one.
(1, 1, 2, 2) has 0 member missing in the array (1, 2, 3, 3).

If we try our last function on this new data

a ← [1 2 3 3]
b ← [1 1 2 2]
F ← ▽¬∊,
⊟⊃(F∶)(F) a b

we get an error

Error: Cannot couple arrays with shapes [2] and [0]  
at 4:1  
4 | ⊟ ⊃(F∶)(F) a b  
    ─

This is because arrays in Uiua must have a uniform shape, i.e. the same number of elements/rows/subarrays at each level. The solution is to □ box the result of F:

⊟⊃(□F∶)(□F) a b

giving

[⟦3 3⟧ ⟦⟧]

Notice the ⟦⟧ meaning it's a boxed array. There are two problems left to solve here though. First we want to get rid of the duplicate 3 in the first box. We'll use ⊝ deduplicate for that:

⊟⊃(□⊝F∶)(□⊝F) a b

which works as expected

[⟦3⟧ ⟦⟧]

We also want to get rid of the empty second box altogether. We can do that by taking the ⧻ length of each row and then ▽ keep the ones >0:

▽>0≡⧻. ⊟ ⊃(□⊝F∶)(□⊝F) a b

giving us

[⟦3⟧]

which is correct! Putting it altogether, we can inline F for brevity and add our two examples as unit tests:

MissingMembers ← ▽>0≡⧻. ⊟⊃(□⊝▽¬∊,∶)(□⊝▽¬∊,)

---
⍤. ≍{[1 3][4 6]} MissingMembers [1 2 3]   [2 4 6]
⍤. ≍{[3]}        MissingMembers [1 2 3 3] [1 1 2 2]
---

The unit tests pass and we're happy once again!

Was it hard or easy to follow along? Do you want more content like this? Leave a comment!

Autoprovisioning NFS volumes in EKS with CDK

Magnus Markling — Sun, 17 Jul 2022 18:44:01 +0000

Introduction
Setting up CDK
Setting up an EKS cluster
Binding a PVC to EBS
Setting up EFS autoprovision
Binding a PVC to EFS
Cleaning up
Summary

Introduction

I recently set up an EKS cluster for a client. From the cluster we wanted to autoprovision EFS volumes based on persistent volume claims. We wanted to do all of this using CDK.

I assume you're already somewhat familiar with CDK, Kubernetes, EKS and have your ~/.aws/credentials configured.

Setting up CDK

Let's get started!

Create an empty working directory:

mkdir eks-efs && cd eks-efs

Then create a basic CDK TypeScript app:

npx cdk@^2.32 init app --language typescript

If this is the first time you use CDK on this AWS account, you need to bootstrap it (= creating some S3 buckets etc) by running:

npx cdk bootstrap

Next let's add a variable to keep the naming prefix used with all our resources. (If you already have similarly named resources, change it to something else.) Add this to the top of the file, after the imports:

// lib/eks-efs-stack.ts
...
const prefix = "my";
...

Setting up an EKS cluster

Add this (and all code snippets following this one) to the end of the constructor:

// lib/eks-efs-stack.ts
...
    const vpc = new aws_ec2.Vpc(this, `${prefix}-vpc`, {
      vpcName: `${prefix}-vpc`,
    });

    const cluster = new aws_eks.Cluster(this, `${prefix}-eks`, {
      clusterName: `${prefix}-eks`,
      version: aws_eks.KubernetesVersion.V1_21,
      vpc,
    });
...

The explicit VPC is strictly not needed, but we will use it later.

Now deploy the cluster by running:

npx cdk@^2.32 deploy

Creating all the required resources usually takes around 20-25 minutes.

The result should look something like this (I have removed sensitive information, such as my AWS account number):

 ✅  my-stack

✨  Deployment time: 1373.08s

Outputs:
my-stack.myeksConfigCommand76382EC1 = aws eks update-kubeconfig --name my-eks --region eu-west-1 --role-arn arn:aws:iam::...:role/my-stack-myeksMastersRole...-...
my-stack.myeksGetTokenCommand0DD2F5A8 = aws eks get-token --cluster-name my-eks --region eu-west-1 --role-arn arn:aws:iam::...:role/my-stack-myeksMastersRole...-...
Stack ARN:
arn:aws:cloudformation:eu-west-1:...:stack/my-stack/bbb03590-05cf-11ed-808a-02bd3ce8aca3

Configuring kubectl

Now that we have our cluster up and running, we can administer it in the usual way using kubectl. First we need to add our cluster to our kubeconfig by copying the ConfigCommand output value from the previous step. I also recommend adding an --alias flag, as the context name will otherwise be the not-so-memorable cluster ARN.

In my case, I run:

aws eks update-kubeconfig --name my-eks --region eu-west-1 --role-arn arn:aws:iam::...:role/my-stack-myeksMastersRoleD2A59038-47WE3AI9OHS3 --alias my-eks

Followed by some simple verification, e.g.:

kubectl get pod -A

And get:

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   aws-node-nmwzs             1/1     Running   0          36m
kube-system   aws-node-r2hjk             1/1     Running   0          36m
kube-system   coredns-7cc879f8db-8ntfk   1/1     Running   0          42m
kube-system   coredns-7cc879f8db-dsrs9   1/1     Running   0          42m
kube-system   kube-proxy-247kc           1/1     Running   0          36m
kube-system   kube-proxy-6g45p           1/1     Running   0          36m

Supported Kubernetes versions

You might have noticed that I chose to use Kubernetes 1.21 here, which is pretty old. At the time of writing this is unfortunately the newest version of Kubernetes available with CDK. (When using the console, you can go up to 1.22.) Even though 1.21 is no longer supported by the Kubernetes project, I recently learned that it's still supported by AWS themselves, meaning they do backports of security fixes.

Binding a PVC to EBS

EBS autoprovisioning support is already built into EKS, via a storage class called gp2.

If we create a persistent volume claim (PVC):

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-rwo
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp2
  resources:
    requests:
      storage: 1Gi
EOF

And a pod that uses the PVC:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: ebs-rwo
spec:
  containers:
    - name: nginx
      image: nginx
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-rwo
EOF

Wait a few seconds, then check the status of the new PVC:

kubectl get pvc

NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
ebs-rwo   Bound    pvc-71606839-b2d9-4f36-8e1c-942b8d7e38f1   1Gi        RWO            gp2

And the associated newly created persistent volume (PV):

kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS
pvc-71606839-b2d9-4f36-8e1c-942b8d7e38f1   1Gi        RWO            Delete           Bound    default/ebs-rwo   gp2

However, there are some reasons why we won't want to use EBS for persistent storage. One of them is that EBS doesn't support access mode ReadWriteMany. EFS to the rescue!

Setting up EFS autoprovision

However, EFS is not supported out-of-the-box as EBS is, and needs some setting up to work as smoothly. The are complete instructions on the AWS website, but they use imperative CLI commands that doesn't rhyme very well with a modern Infrastructure-as-Code (IaC) approach.

Adding an EFS filesystem

First we need to create a network security group:

// lib/eks-efs-stack.ts
...
    const efsInboundSecurityGroup = new aws_ec2.SecurityGroup(
      this,
      `${prefix}-efs-inbound-security-group`,
      {
        securityGroupName: `${prefix}-efs-inbound`,
        vpc,
      },
    );
...

With an ingress rule that allows for incoming EFS traffic:

// lib/eks-efs-stack.ts
...
    new aws_ec2.CfnSecurityGroupIngress(
      this,
      `${prefix}-efs-inbound-security-group-ingress`,
      {
        ipProtocol: "tcp",
        cidrIp: vpc.vpcCidrBlock,
        groupId: efsInboundSecurityGroup.securityGroupId,
        description: "Inbound EFS",
        fromPort: 2049,
        toPort: 2049,
      },
    );
...

Port 2049 is the standard EFS port. (fromPort and toPort simply defines the port range to be from 2049 to 2049, not to be confused with source and destination ports in the world of TCP.)

Then we create the EFS file system:

// lib/eks-efs-stack.ts
...
    const efs_fs = new aws_efs.FileSystem(this, `${prefix}-efs`, {
      fileSystemName: `${prefix}`,
      vpc,
      securityGroup: efsInboundSecurityGroup,
      removalPolicy: RemovalPolicy.DESTROY,
    });
...

We specify RemovalPolicy.DESTROY to make sure the filesystem is deleted together with the rest of the CDK stack.

Adding an IAM role

First we create a policy that gives the necessary permissions required to view and create EFS access points:

// lib/eks-efs-stack.ts
...
    const efsAllowPolicyDocument = aws_iam.PolicyDocument.fromJson({
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "elasticfilesystem:DescribeAccessPoints",
            "elasticfilesystem:DescribeFileSystems",
          ],
          "Resource": "*",
        },
        {
          "Effect": "Allow",
          "Action": [
            "elasticfilesystem:CreateAccessPoint",
          ],
          "Resource": "*",
          "Condition": {
            "StringLike": {
              "aws:RequestTag/efs.csi.aws.com/cluster": "true",
            },
          },
        },
        {
          "Effect": "Allow",
          "Action": "elasticfilesystem:DeleteAccessPoint",
          "Resource": "*",
          "Condition": {
            "StringEquals": {
              "aws:ResourceTag/efs.csi.aws.com/cluster": "true",
            },
          },
        },
      ],
    });
...

Next we specify who can assume this role (what's called Trust relationships in the AWS console):

// lib/eks-efs-stack.ts
...
    const efsAssumeRolePolicyDocument = aws_iam.PolicyDocument.fromJson({
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": cluster.openIdConnectProvider.openIdConnectProviderArn,
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": new CfnJson(
              this,
              `${prefix}-assume-role-policy-document-string-equals-value`,
              {
                value: {
                  [`${cluster.openIdConnectProvider.openIdConnectProviderIssuer}:aud`]:
                    "sts.amazonaws.com",
                  [`${cluster.openIdConnectProvider.openIdConnectProviderIssuer}:sub`]:
                    "system:serviceaccount:kube-system:efs-csi-controller-sa",
                },
              },
            ),
          },
        },
      ],
    });
...

We reference the IODC provider created together with the cluster. We also use a special magic CfnJson construct to be able to use dynamic keys in the JSON. (This took quite a while to figure out.)

Finally we tie them all together by creating the IAM role:

// lib/eks-efs-stack.ts
...
    const efsRole = new aws_iam.CfnRole(
      this,
      `${prefix}-efs-csi-controller-sa-role`,
      {
        roleName: `${prefix}-efs-csi-controller-sa-role`,
        assumeRolePolicyDocument: efsAssumeRolePolicyDocument,
        policies: [
          {
            policyName: `${prefix}-efs-csi-controller-sa-policy`,
            policyDocument: efsAllowPolicyDocument,
          },
        ],
      },
    );
...

Installing the EFS CSI driver

Now we need to install the EFS CSI driver. This is a Kubernetes controller that checks for unbound PVC, and creates PV and EFS access points. The easiest way is via its Helm chart:

// lib/eks-efs-stack.ts
...
    cluster.addHelmChart(`${prefix}-aws-efs-csi-driver`, {
      repository: "https://kubernetes-sigs.github.io/aws-efs-csi-driver/",
      chart: "aws-efs-csi-driver",
      version: "2.2.7",
      release: "aws-efs-csi-driver",
      namespace: "kube-system",
      values: {
        controller: {
          serviceAccount: {
            annotations: {
              "eks.amazonaws.com/role-arn": efsRole.attrArn,
            },
          },
        },
      },
    });
...

eks.amazonaws.com/role-arn is a special annotation recognized by EKS, that associates that service account with the specified IAM role. This means all pods/containers running under that service account can call AWS APIs as that role. (Pretty much the same way you would associate an IAM role with an EC2 instance.)

Adding the storage class

The last thing we need to do is add a Kubernetes storage class (SC) for our PVCs to use:

// lib/eks-efs-stack.ts
...
    cluster.addManifest(`${prefix}-k8s-efs-storageclass`, {
      apiVersion: "storage.k8s.io/v1",
      kind: "StorageClass",
      metadata: {
        name: "efs",
      },
      provisioner: "efs.csi.aws.com",
      parameters: {
        directoryPerms: "700",
        provisioningMode: "efs-ap",
        fileSystemId: efs_fs.fileSystemId,
        uid: "1001",
        gid: "1001",
      },
    });
...

Notice how the SC references our newly set-up EFS filesystem, as well as specifies some Linux related parameters.

Binding a PVC to EFS

Let's add a PVC which uses the SC:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-rwx
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs
  resources:
    requests:
      storage: 1Gi
`EOF`

And a pod that uses the PVC:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: efs-rwx
spec:
  containers:
    - name: nginx
      image: nginx
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: efs-rwx
`EOF`

Now check the status of the new PVC:

kubectl get pvc

NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
ebs-rwo   Bound    pvc-71606839-b2d9-4f36-8e1c-942b8d7e38f1   1Gi        RWO            gp2
efs-rwx   Bound    pvc-5743b62a-1e1a-4b0b-b930-921465fd9d9b   1Gi        RWX            efs

And the associated newly created PV:

kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS
pvc-71606839-b2d9-4f36-8e1c-942b8d7e38f1   1Gi        RWO            Delete           Bound    default/ebs-rwo   gp2
pvc-5743b62a-1e1a-4b0b-b930-921465fd9d9b   1Gi        RWX            Delete           Bound    default/efs-rwx   efs

Notice that the new one is bound, uses SC efs and has access mode RWX (ReadWriteMany).

We can also view the EFS access point that was created:

aws efs describe-access-points

AccessPoints:
- AccessPointArn: arn:aws:elasticfilesystem:eu-west-1:...:access-point/fsap-082fd6323a789739d
  AccessPointId: fsap-082fd6323a789739d
  ClientToken: pvc-5743b62a-1e1a-4b0b-b930-921465fd9d9b
  FileSystemId: fs-0a724fe641cbd9da7
  LifeCycleState: available
  OwnerId: '...'
  PosixUser:
    Gid: 1001
    Uid: 1001
  RootDirectory:
    CreationInfo:
      OwnerGid: 1001
      OwnerUid: 1001
      Permissions: '700'
    Path: /pvc-5743b62a-1e1a-4b0b-b930-921465fd9d9b
  Tags:
  - Key: efs.csi.aws.com/cluster
    Value: 'true'

Cleaning up

In order to avoid further costs, let's clean up the resources we just created. This is as simple as:

npx cdk destroy

Again, this will take around 20-25 minutes.

Summary

That's it! We set up autoprovisioning of NFS volumes in EKS using CDK. Full working code will soon be available on GitHub.

Any comments or questions are welcome!

Running a GraphQL API in .NET 6 on AWS Lambda

Magnus Markling — Wed, 12 Jan 2022 18:07:39 +0000

Introduction
Why GraphQL
Why Lambda
Initial application setup
Running locally
Lambda runtimes
Bootstrapping
Creating the Lambda package
Deploying to AWS
Calling the Lambda
Cleaning up
Bonus: Running on ARM
Summary

Introduction

I recently set up a brand new API for a client. AWS and .NET were givens, the remaining choices were up to me. This article is my way of writing down all the things I wish I knew when I started that work.

I assume you already know your way around .NET 6, C# 10, GraphQL and have your ~/.aws/credentials configured.

Why GraphQL

GraphQL has quickly become my primary choice when it comes to building most kinds of APIs for a number of reasons:

Great frameworks available for a variety of programming languages
Type safety and validation for both input and output is built-in (including client-side if using codegen)
There are different interactive "swaggers" available, only much better

Something often mentioned about GraphQL is that the client can request only whatever fields it needs. In practice I find that a less convincing argument because most of us are usually developing our API for a single client anyway.

For the .NET platform my framework of choice is Hot Chocolate. It has great docs and can generate a GraphQL schema in runtime based on existing .NET types.

Why Lambda

Serverless is all the hype now. What attracts me most is the ease of deployment and the ability to dynamically scale based on load.

AWS Lambda is usually marketed (and used) as a way to run small isolated functions. Usually with 10 line Node.js examples. But it is so much more! I would argue it is the quickest and most flexible way to run any kind of API.

The only real serverless alternative on AWS is ECS on Fargate, but that comes with a ton of configuration and also requires you to run your code in Docker.

Initial application setup

We start by creating a new dotnet project:

dotnet new web -o MyApi && cd MyApi

Add AspNetCore and HotChocolate:

dotnet add package DotNetCore.AspNetCore --version "16.*"
dotnet add package HotChocolate.AspNetCore --version "12.*"

Add a single GraphQL field:

// Query.cs
using static System.Runtime.InteropServices.RuntimeInformation;

public class Query {
  public string SysInfo =>
    $"{FrameworkDescription} running on {RuntimeIdentifier}";
}

Set up our AspNetCore application (using the new minimal API):

// Program.cs
var builder = WebApplication.CreateBuilder(args);

builder.Services
  .AddGraphQLServer()
  .AddQueryType<Query>();

var app = builder.Build();

app.UseRouting();

app.UseEndpoints(endpoints =>
  endpoints.MapGraphQL());

await app.RunAsync();

Running locally

Let's verify that our GraphQL API works locally.

Start the API:
dotnet run

Verify using curl:
curl "http://localhost:<YourPort>/graphql?query=%7B+sysInfo+%7D"

You should see a response similar to:

{ "data": { "sysInfo":".NET 6.0.1 running on osx.12-x64" } }

Lambda runtimes

AWS offers a number of different managed runtimes for Lambda, including .NET Core, Node, Python, Ruby, Java and Go. For .NET the latest supported version is .NET Core 3.1, which I think is too old to base new applications on.

.NET 6 was released a few months ago, so that's what we'll be using. There are two main alternatives for running on a newer runtime than what AWS provides out of the box:

Running your Lambda in Docker
Using a custom runtime

Running your Lambda in Docker was up until recently the easiest way for custom runtimes. The Dockerfile was only two or three lines and easy to understand. But I still feel it adds a complexity that isn't always justified.

Therefore we will be using a custom runtime.

Using a custom runtime

There is a hidden gem available from AWS, and that is the Amazon.Lambda.AspNetCoreServer.Hosting nuget package. It's hardly mentioned anywhere except in a few GitHub issues, and has a whopping 425 (!) downloads as I write this. But it's in version 1.0.0 and should be stable.

Add it to the project:
dotnet add package Amazon.Lambda.AspNetCoreServer.Hosting --version "1.*"

Then add this:

// Program.cs
...
builder.Services
  .AddAWSLambdaHosting(LambdaEventSource.HttpApi);
...

The great thing about this (except it being a one-liner!) is that if the application is not running in Lambda, that method will do nothing! So we can continue and run our API locally as before.

Bootstrapping

There are two main ways of bootstrapping our Lambda function:

Changing the assembly name to bootstrap
Adding a shell script named bootstrap

Changing the assembly name to bootstrap could be done in our .csproj. Although it's a seemingly harmless change, it tends to confuse developers and others when the "main" dll goes missing from the build output and an extensionless bootstrap file is present instead.

Therefore my preferred way is adding a shell script named bootstrap:

// bootstrap
#!/bin/bash

${LAMBDA_TASK_ROOT}/MyApi

LAMBDA_TASK_ROOT is an environment variable available when the Lambda is run on AWS.

We also need to reference this file in our .csproj to make sure it's always published along with the rest of our code:

// MyApi.csproj
...
<ItemGroup>
  <Content Include="bootstrap">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </Content>
</ItemGroup>
...

Creating the Lambda package

We will be using the dotnet lambda cli tool to package our application. (I find it has some advantages over a plain dotnet publish followed by zip.)

dotnet new tool-manifest
dotnet tool install amazon.lambda.tools --version "5.*"

I prefer to install tools like this locally. I believe global tools will eventually cause you to run into version conflicts.

We also add a default parameter to msbuild, so we don't have to specify it on the command line.

// aws-lambda-tools-defaults.json
{
  "msbuild-parameters": "--self-contained true"
}

Building and packaging the application is done by
dotnet lambda package -o dist/MyApi.zip

Deploying to AWS

The way I prefer to deploy simple Lambdas is by using the Serverless framework.

(For an excellent comparison between different tools of this kind for serverless deployments on AWS, check out this post by Sebastian Bille.)

You might argue that Terraform has emerged as the de facto standard for IaC. I would tend to agree, but it comes with a cost in terms of complexity and state management. For simple setups like this, I still prefer the Serverless framework.

We add some basic configuration to our serverless.yml file:

// serverless.yml
service: myservice

provider:
  name: aws
  region: eu-west-2
  httpApi:
    payload: "2.0"
  lambdaHashingVersion: 20201221

functions:
  api:
    runtime: provided.al2
    package:
      artifact: dist/MyApi.zip
      individually: true
    handler: required-but-ignored
    events:
      - httpApi: "*"

Even though we are using AspNetCore, a Lambda is really just a function. AWS therefore requires an API Gateway in front of it. Serverless takes care of this for us. The combination of httpApi and 2.0 here means that we will use the new HTTP trigger of the API Gateway. This would be my preferred choice, as long as we don't need some of the functionality still only present in the older REST trigger.

runtime: provided.al2 means we will use the custom runtime based on Amazon Linux 2.

Now we are finally ready to deploy our Lambda!

npx serverless@^2.70 deploy

The output should look something like this:

...
endpoints:
  ANY - https://ynop5r4gx2.execute-api.eu-west-2.amazonaws.com
...

Here you'll find the URL where our Lambda can be reached. Let's call this <YourUrl>.

Calling the Lambda

Using curl:
curl "https://<YourUrl>/graphql?query=%7B+sysInfo+%7D"

You should see a response similar to:

{ "data": { "sysInfo":".NET 6.0.1 running on amzn.2-x64" } }

Cleaning up

Unless you want to keep our Lambda running, you can remove all deployed AWS resources with:
npx serverless@^2.70 remove

Take me to the summary!

Bonus: Running on ARM

AWS recently announced the possibility to run Lambda on the new ARM-based Graviton2 CPU. It's marketed as faster and cheaper. Note that ARM-based Lambdas are not yet available in all AWS regions and that they might not work with pre-compiled x86/x64 dependencies.

If we want to run on Graviton2 a few small changes are necessary:

Compiling for ARM
Configuring Lambda for ARM
Add additional packages for ARM

Compiling for ARM

Here we need to add our runtime target for the dotnet lambda tool to pick up:

// aws-lambda-tools-defaults.json
{
  "msbuild-parameters":
    "--self-contained true --runtime linux-arm64"
}

Configure Lambda for ARM

We need to specify the architecture of the Lambda function:

// serverless.yml
functions:
  api:
    ...
    architecture: arm64
    ...

Adding additional packages for ARM

According to this GitHub issue we need to add and configure an additional package when running a custom runtime on ARM:

// MyApi.csproj
...
<ItemGroup>
  <RuntimeHostConfigurationOption
    Include="System.Globalization.AppLocalIcu"
    Value="68.2.0.9"/>
  <PackageReference
    Include="Microsoft.ICU.ICU4C.Runtime"
    Version="68.2.0.9"/>
</ItemGroup>
...

When adding this the API stops working on non-ARM platforms though. A more portable solution is to use a condition on the ItemGroup, like this:

// MyApi.csproj
...
<ItemGroup Condition="'$(RuntimeIdentifier)' == 'linux-arm64'">
  <RuntimeHostConfigurationOption
    Include="System.Globalization.AppLocalIcu"
    Value="68.2.0.9"/>
  <PackageReference
    Include="Microsoft.ICU.ICU4C.Runtime"
    Version="68.2.0.9"/>
</ItemGroup>
...

Building, deploying, and calling it once more

Build and deploy as before.

Call the Lambda as before.

You should see a response similar to:

{ "data": { "sysInfo":".NET 6.0.1 running on amzn.2-arm64" } }

confirming that we are now running on ARM!

Clean up as before.

Summary

That's it! We have now deployed a minimal serverless GraphQL API in .NET 6 on AWS Lambda. Full working code is available at GitHub.

Opinionated take aways:

Use GraphQL for most APIs
Use Hot Chocolate for GraphQL on .NET
Use Lambda for entire APIs, not just simple functions
Use dotnet lambda cli tool for packaging
Use Amazon.Lambda.AspNetCoreServer.Hosting for custom runtimes
Use a simple bootstrap script to start the API
Use Serverless framework for deployment
Use ARM if you can

Any comments or questions are welcome!

Forem: Magnus Markling

Uiua: Weekly challenge 242

Table of contents

Introduction

Task 2

Example 1

Example 2

Task 1

Example 1

Example 2

Autoprovisioning NFS volumes in EKS with CDK

Table of contents

Introduction

Setting up CDK

Setting up an EKS cluster

Configuring kubectl

Supported Kubernetes versions

Binding a PVC to EBS

Setting up EFS autoprovision

Adding an EFS filesystem

Adding an IAM role

Installing the EFS CSI driver

Adding the storage class

Binding a PVC to EFS

Cleaning up

Summary

Running a GraphQL API in .NET 6 on AWS Lambda

Table of contents

Introduction

Why GraphQL

Why Lambda

Initial application setup

Running locally

Lambda runtimes

Using a custom runtime

Bootstrapping

Creating the Lambda package

Deploying to AWS

Calling the Lambda

Cleaning up

Bonus: Running on ARM

Compiling for ARM

Configure Lambda for ARM

Adding additional packages for ARM

Building, deploying, and calling it once more

Summary