Forem: Tuan Anh Tran

aws-cli v2: how much smaller can it get? Answer: a lot smaller :)

Tuan Anh Tran — Sun, 19 Mar 2023 16:54:26 +0000

Pulling the official aws-cli image, I got a huge 374MB image.



docker images | grep amazon
amazon/aws-cli                                              latest            abb3d3272080   3 days ago      374MB

It's rather big for a CLI and so I thought: "Maybe I can make it smaller".

Entering Wolfi

Wolfi is a project sponsored by Chainguard. Wolfi is described as:

Wolfi is a stripped-down distro designed for the cloud-native era.

Well, you can think of it like Google's distroless.

The reason I want to select Wolfi is because of a bunch of reasons:

Better SBOM support. Well, not all the SBOMs are created equally and what's better than building everything from sources and create SBOM from there.
Multi-arch (amd64, arm64 are must but maybe armv7 as well?)
Smaller image size.
Signed and verifiable :)
Secure. Let's aim for that sweet 0 CVE.

aws-cli v2

aws-cli v2 is powered by a bunch of C project like aws-c-http, aws-c-auth, etc... and so I need to package those first.

What I don't quite understand is AWS's decision to still use Python as frontend for the CLI. And packaging Python packages is like going down the rabbit hole.

I'm going to use melange for packaging. I write melange package's manifest in YAML and melange spits out APK file for me.

Here's what a package build file looks like with melange



package:
  name: aws-c-http
  version: 0.7.5
  epoch: 0
  description: AWS C99 implementation of the HTTP/1.1 and HTTP/2 specifications
  copyright:
    - license: Apache-2.0
environment:
  contents:
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    repositories:
      - https://packages.wolfi.dev/os
      - '@local ./packages'
    packages:
      - aws-c-cal-dev@local
      - aws-c-common-dev@local
      - aws-c-compression-dev@local
      - aws-c-io-dev@local
      - build-base
      - busybox
      - ca-certificates-bundle
      - cmake
      - s2n-tls-dev@local
      - samurai
pipeline:
  - uses: fetch
    with:
      expected-sha512: 90bfd0abfce8727bd4ef6e9a016e6183c2c4349c2d6e4cc02b932f5109ceb3476d79b853e74ca4888b768f44eba9ae953ca5e9b900704a2a3370a200c0168c02
      uri: https://github.com/awslabs/aws-c-http/archive/refs/tags/v${{package.version}}.tar.gz
  - runs: |
      if [ "$CBUILD" != "$CHOST" ]; then
        CMAKE_CROSSOPTS="-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_HOST_SYSTEM_NAME=Linux"
      fi
      CFLAGS="$CFLAGS -flto=auto" \
      CXXFLAGS="$CXXFLAGS -flto=auto" \
      cmake -B build -G Ninja \
        -DCMAKE_INSTALL_PREFIX=/usr \
        -DCMAKE_INSTALL_LIBDIR=/usr/lib \
        -DBUILD_SHARED_LIBS=True \
        -DCMAKE_BUILD_TYPE=None \
        -DBUILD_TESTING="$(want_check && echo ON || echo OFF)" \
        $CMAKE_CROSSOPTS
      cmake --build build
  - runs: |
      DESTDIR="${{targets.destdir}}" cmake --install build
  - uses: strip
subpackages:
  - name: aws-c-http-dev
    pipeline:
      - uses: split/dev
      - runs: |
          mkdir -p "${{targets.subpkgdir}}"/usr/lib
          mv "${{targets.destdir}}"/usr/lib/aws-c-http "${{targets.subpkgdir}}"/usr/lib/
    dependencies:
      runtime:
        - aws-c-http
    description: aws-c-http dev

I also need to package a bunch of Python libs like py3-certifi, py3-distro, etc... because they are not available on Wolfi yet.

Once those are done, I just need to build aws-cli package, put those APK files in a final image with Chainguard's apko.

Preliminary result

The early result looks very promising. The final image is much smaller. 50% smaller to be precised. And I mean without any optimizations at all. I haven't even split the docs into their own packages yet.

Let's look at it.

186MB vs the original 374MB. A whooping 188MB stripped off from the official aws-cli image. That's 50% size reduction.

Are you sure it's even working :D

Final

I haven't quite done with it yet but you can try it by pulling ghcr.io/tuananh/aws-cli.

For now, it's still very much work-in-progress so expect things may break here and there :)

AWS - The YAML Way

Tuan Anh Tran — Tue, 19 Oct 2021 06:47:08 +0000

Originally posted here on GitHub

AWS - The YAML way

Motivation

I want to manage AWS infrastructure with YAML
I want to use Kubernetes RBAC for authorization
I want to be able to defines rules to govern my cloud resources

But why?

Why not :)

How?

For (1), there's AWS Controllers for Kubernetes (ACK) or maybe Crossplane
For (2), it's quite straight forward. If we can express AWS resources using Kubernetes CRDs, then it's done.
For (3), I'm thinking Kyverno or OpenPolicyAgent.

Let's do it

Setup the env

I will skip the part where you setup AWS CLI and an EKS cluster Suppose that is all set and done. If not, follow the brief instructions below to setup a new EKS cluster. The easiest way IMO is to use eksctl from Weaveworks.

aws ec2 create-key-pair --region ap-southeast-1 --key-name my-yaml-eks-key

eksctl create cluster \
    --name my-yaml-eks \
    --region ap-southeast-1 \
    --with-oidc \
    --ssh-access \
    --ssh-public-key my-yaml-eks-key \
    --managed

Wait for a bit for the cluster to be provisioned.

There will be 2 CloudFormation stacks being provisioned so it might take awhile. If something goes wrong (disconnection, etc..), you can check with this command below

eksctl utils describe-stacks --region=ap-southeast-1 --cluster=my-yaml-eks

2021-10-18 22:35:28 [ℹ]  eksctl version 0.70.0
2021-10-18 22:35:28 [ℹ]  using region ap-southeast-1
2021-10-18 22:35:29 [ℹ]  setting availability zones to [ap-southeast-1a ap-southeast-1b ap-southeast-1c]
2021-10-18 22:35:29 [ℹ]  subnets for ap-southeast-1a - public:192.168.0.0/19 private:192.168.96.0/19
2021-10-18 22:35:29 [ℹ]  subnets for ap-southeast-1b - public:192.168.32.0/19 private:192.168.128.0/19
2021-10-18 22:35:29 [ℹ]  subnets for ap-southeast-1c - public:192.168.64.0/19 private:192.168.160.0/19
2021-10-18 22:35:29 [ℹ]  nodegroup "ng-5c441b7d" will use "" [AmazonLinux2/1.20]
2021-10-18 22:35:29 [ℹ]  using EC2 key pair %!q(*string=<nil>)
2021-10-18 22:35:29 [ℹ]  using Kubernetes version 1.20
2021-10-18 22:35:29 [ℹ]  creating EKS cluster "my-yaml-eks" in "ap-southeast-1" region with managed nodes
2021-10-18 22:35:29 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2021-10-18 22:35:29 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-southeast-1 --cluster=my-yaml-eks'
2021-10-18 22:35:29 [ℹ]  CloudWatch logging will not be enabled for cluster "my-yaml-eks" in "ap-southeast-1"
2021-10-18 22:35:29 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-southeast-1 --cluster=my-yaml-eks'
2021-10-18 22:35:29 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "my-yaml-eks" in "ap-southeast-1"
2021-10-18 22:35:29 [ℹ]
2 sequential tasks: { create cluster control plane "my-yaml-eks",
    2 sequential sub-tasks: {
        4 sequential sub-tasks: {
            wait for control plane to become ready,
            associate IAM OIDC provider,
            2 sequential sub-tasks: {
                create IAM role for serviceaccount "kube-system/aws-node",
                create serviceaccount "kube-system/aws-node",
            },
            restart daemonset "kube-system/aws-node",
        },
        create managed nodegroup "ng-5c441b7d",
    }
}
2021-10-18 22:35:29 [ℹ]  building cluster stack "eksctl-my-yaml-eks-cluster"
2021-10-18 22:35:30 [ℹ]  deploying stack "eksctl-my-yaml-eks-cluster"

Once it's done. Make sure the cluster is accessible

Setup Crossplane

At first, I plan to use ACK but then I remember a friend of mine talked about Crossplane the other day so I want to give it a try. Bonus point, it works with multiple cloud providers :)

I'm going to use Helm so make sure you have it installed. Simply follow the official installation instructions on Crossplane website here.

As of this post, 1.4.1 is the latest chart version.

kubectl create namespace crossplane-system

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane --version 1.4.1

NAME: crossplane
LAST DEPLOYED: Mon Oct 18 23:05:25 2021
NAMESPACE: crossplane-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Release: crossplane

Chart Name: crossplane
Chart Description: Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume.
Chart Version: 1.4.1
Chart Application Version: 1.4.1

Kube Version: v1.20.7-eks-d88609

Check the status

helm list -n crossplane-system
kubectl get all -n crossplane-system

Also, make sure to install the Crossplane CLI.

curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh

Setup Crossplane provider for AWS

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: crossplane/provider-aws:alpha

After the provider is ready, apply the following next. You need to do this in 2 steps because otherwise, ProviderConfig kind is unknown.

apiVersion: v1
stringData:
  config: |
    [default]
    aws_access_key_id = <your-aws-access-key-id>
    aws_secret_access_key = <your-aws-secret-access-key>
kind: Secret
metadata:
  creationTimestamp: null
  name: aws-credentials
---
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws-provider-config
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-credentials
      key: config

Check the status of the provider

kubectl get providers

Let's try to create some resources on AWS

Let's do a S3 bucket first. Make sure you do a random bucket name so that it won't conflict with an existing bucket.

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: test-bucket-1923981293819238
  namespace: default
spec:
  providerConfigRef:
    name: aws-provider-config
  forProvider:
    locationConstraint: ap-southeast-1
    acl: private

After a few seconds, the resource status will change to Ready and Synced.

So that's cool. What's next? Let's trying to use OPA or Kyverno to set some rules for our newly created resources.

Setup Kyverno

I'm gonna go with Kyverno here. No particular reason. I just feel OPA is too mainstream. The concept with OPA is similar. You can take it as homework for your lab session.

Let's install Kyverno with Helm

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno-crds kyverno/kyverno-crds --namespace kyverno --create-namespace
helm install kyverno kyverno/kyverno --namespace kyverno

Now, let's write a simple policy. Say, we don't want to allow creating S3 bucket anywhere except in ap-southeast-1 region.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: allow-s3-ap-southeast-1-only
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
    policies.kyverno.io/title: Allow creating S3 bucket in ap-southeast-1 only
    policies.kyverno.io/subject: Bucket
    policies.kyverno.io/description: >-
      Allow creating S3 bucket in ap-southeast-1 only
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: s3-in-ap-southeast-1-only
    match:
      resources:
        kinds:
        - Bucket
    validate:
      message: "Using any location other than `ap-southeast-1` is not allowed"
      pattern:
        spec:
          forProvider:
            locationConstraint: "ap-southeast-1"

Now, let's try to create a S3 bucket in us-east-1 region. You will be blocked by the admission controller.

# bad-s3.yaml
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: test-bucket-091289310923801928309123
  namespace: default
spec:
  providerConfigRef:
    name: aws-provider-config
  forProvider:
    locationConstraint: us-east-1
    acl: private

kubectl create -f bad-s3.yaml

Now, the policy we have here is very simple but you can do all kind of stuff with Kyverno cluster policy. It would be a cool weekend hack to convert all rules you have currently to Kyverno cluster policy. That sounds like tons of fun :)

Conclusion

I'm not going to tell you should start doing this but it's a feasible way of managing infrastructure at small scale.

Also with the release of AWS Cloud Control API, the resources model is very close with the Kubernetes resources model now. I'm expecting ACK to get much much better.

Have fun hacking AWS :)