Forem: Sebastian Korfmann

Skymood - Watch Bluesky's heartbeat through emojis in real-time 🌟

Sebastian Korfmann — Mon, 18 Nov 2024 08:56:07 +0000

This is a brief background story to how I built Skymood over the course of two evenings.

Since becoming more active on bluesky again, I started to look into how bluesky is built behind the scenes. The backbone which ties everything together is the Firehose, which is a full stream of events (posts, likes, follows, handle changes, etc). While that's one of the coolest aspects of Bluesky and ATProto, it's also a ton of data (in the realm of ~ 50 GB / day as of today) which would have to be transferred and processed, since it's an all or nothing approach with no filtering option.

However, there's another option which got released a few weeks ago: Jetstream. In contrast to the Firehose, this allows filtering by Collection NSIDs and Repositories. This means, we can filter for e.g. all posts, either globally or scope it to a bunch of given user ids. That sounds pretty intriguing, doesn't it?

As one does, I started to play around with just getting bunch of posts and a way to demonstrate what's possible. When looking at the stream of posts flying by I was like, it would be nice to get some kind of moodboard based on the emojis. One prompt later:

Well, that's promising, so let's evolve that into a website we can put on the Internet.

Literally Serverless

The Jetstream Websocket endpoint is open and can be connected to from any browser. So technically, there's no reason why a server would be required. And sure enough, consuming the Jetstream filtered for posts is totally doable.

Just give it a try in your terminal

$ websocat wss://jetstream2.us-east.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post

So I went ahead and threw together a quick React SPA using Waku which was subscribing straight to the Jetstream Websocket from the client side. The prototype was done pretty quickly. The only noticeable hurdle was, that the Jetstream SDK depends on node:events. Rather than trying to workaround that, it seemed a lot simpler to just go with a plain Websocket implementation.

  const ws = new WebSocket('wss://jetstream2.us-west.bsky.network/subscribe?wantedCollections=app.bsky.feed.post');

  ws.addEventListener('message', async (event) => {
    console.log(event)
  })

I was kind of expecting some performance issues, but it was rather smooth. The only small optimization was to debounce rendering, so that not each and every new emoji would re-render.

Deployed this to Cloudflare and pretty much done. Well, not so fast. Let's look at the consumed bandwidth.

$ websocat wss://jetstream2.us-east.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post | pv > /dev/null

Depending on the time of the day, this is currently using between ~ 60 KB/s and 150 KB/s for the global posts stream - and it's gonna increase by the day. Not really an issue on a fast, unmetered fibre connection, but on mobile it might be another story. Last but not least, it feels wrong to let the client do all the expensive work while potentially increasing the bandwidth & resources bill for the Bluesky team. Back to the drawing board.

Cloudflare Durable Objects

Cloudflare's Durable Objects were on my list to try for a long time, handling Websockets are one of major use-cases. Rewriting the React client side handling was just a few prompts away and I was good to go.

The main changes were to subscribe to Jetstream as an upstream connection, match for emojis and publish the emojis to subscribed clients.

// Track connected WebSocket clients and their emoji filters
const clients = new Map<WebSocket, Set<string>>();

// When receiving a message from the data source
ws.addEventListener('message', async (event) => {
  try {
    const data = JSON.parse(event.data as string);
    const postText = data.text.toLowerCase();

    // Extract unique emojis from the text
    const emojiRegex = /[\p{Emoji_Presentation}\p{Extended_Pictographic}]/gu;
    const emojis = [...new Set(postText.match(emojiRegex) || [])];

    // Skip if no emojis found
    if (emojis.length === 0) return;

    // Notify all connected clients
    for (const [client, filters] of clients) {
      // Send the list of emojis found
      client.send(JSON.stringify({
        type: 'emojis',
        emojis
      }));

      // If client has emoji filters and post contains matching emoji,
      // send the full post details
      if (filters.size > 0 && emojis.some(emoji => filters.has(emoji))) {
        client.send(JSON.stringify({
          type: 'post',
          text: data.text,
          emojis,
          timestamp: Date.now()
        }));
      }
    }
  } catch (error) {
    console.error('Error processing message:', error);
  }
});

And with that, we were down to somewhere between 0.5 - 3 KB/s for a client connection while the server is doing the heavylifting only once. That's a lot better!

However, while the Durable Object was doing ok it seemed to be a bit slow (gut feeling, no data) and close the maximum memory of 128 MB (see limits). What to do? So far it's a single Durable Object. Ideas from the forum / Discord of Cloudflare were along the lines of sharding, aka introduce a few durable objects for client handling while a single one does the upstream processing. That's a lot of complexity right there. I'm sure there are better ways, if anyone at Cloudflare reads this: I'd be more than happy to iterate on my approach. But for now I just wanted to ship it. So off to the next chapter.

Pivot: Bun.js

Bun.js is another thing I wanted to look into for a while. In the back of my head it's categorized as Nodejs, but more performant. It turned out that Bun has a custom websocket server baked in, which claims 7x more throughput compare to Nodejs and ws. Haven't verified those claims, but that's enough of an excuse to use it in this case.

Again, a few prompts later the new version was up and running and deployed to fly on a rather small machine in Frankfurt, Germany. Let's see:

$ websocat wss://skymood-bun.fly.dev
{"type":"clientCount","count":1}
{"type":"emojis","emojis":["😘"]}
{"type":"emojis","emojis":["🧐"]}
{"type":"emojis","emojis":["🙏"]}
{"type":"emojis","emojis":["😀"]}
{"type":"emojis","emojis":["😂"]}
...

and the filtered version

$ (echo '{"type":"filter","emoji":"🥰"}'; cat) | websocat wss://skymood-bun.fly.dev
{"type":"emojis","emojis":["😊"]}
{"type":"emojis","emojis":["‼","✨"]}
{"type":"emojis","emojis":["🥰"]}
{"type":"post","text":"Good morning have a lovely day 🥰","url":"https://bsky.app/profile/did:plc:q4st6fcbn5jdi7xdf4o7a2jy/post/3lb7jrngi7c2m","timestamp":1731919511338,"emojis":["🥰"]}
{"type":"emojis","emojis":["🌻","🔪"]}
{"type":"emojis","emojis":["❤"]}

Beautiful :)

Quite a journey, but we now have a rather robust Websocket relay, which connects to the Bluesky Jetstream once, and republishes only a small subset of only relevant messages. Find the current server code over at Github

Make sure to check out the end result https://skymood.skorfmann.com/ and don't forget to share this post, the website or both :) Also, post comments either here or on this Bluesky post

The Journey of CDK.dev: From Static Site to Bluesky

Sebastian Korfmann — Wed, 13 Nov 2024 13:48:43 +0000

When we launched cdk.dev in 2020, we created a simple landing page for the Cloud Development Kit ecosystem. This launch happened right after HashiCorp released CDK for Terraform. Our main goals were to connect people through our Slack community and share content from various sources.

In the beginning, we built the website using NextJS, with all content stored directly in our git repository. Adding new content meant creating new files and submitting a pull request. This process wasn't very user-friendly, and people often made mistakes. Despite these challenges, community members continued to contribute new links.

Earlier this year, we made changes to improve the website. We moved all our links and content to AWS using Amplify, storing everything in DynamoDB and S3. We thought this would make things easier for users, but it actually created new problems. Now we had to manage a state and deal with complex content submission processes. As a result, we noticed that people stopped adding new content over the past few months.

Recently, many people started using Bluesky, a new social platform. I had created an account there last year but hadn't used it much. When I looked at it again, I became very interested in how it works, especially its AT Protocol. I realized we could use Bluesky's open data structure in many interesting ways.

After successfully rebuilding my personal website using my Bluesky data, I became excited about the possibilities. Within a few hours, I created a new version of cdk.dev that gets its information from our Bluesky profile. I wrote a small script to move our existing links to the Bluesky account. When I shared this idea with others, people really liked it.

Now, cdk.dev pulls all its data from Bluesky. The website doesn't need to store any data itself, and it gets information from Bluesky through public requests that don't require login. We can still make many improvements to this system. For example, we could automate content publishing, create a special content feed, or analyze the shared content in new ways.

For now, we're keeping it simple: if you want to share CDK-related content, just tag @cdk.dev on Bluesky, and we'll help spread the word.

Exploring TypeScript Type Generation for JSON Paths: An AI-Assisted Journey

Sebastian Korfmann — Thu, 27 Jun 2024 09:26:58 +0000

As developers, we often find ourselves pushing the boundaries of what's possible with our tools. Recently, I conducted an intriguing experiment: creating a TypeScript type generator for multiple, arbitrary JSON path inputs. While the experiment isn't complete, the progress made with AI assistance is noteworthy.

The Challenge

The goal was to create a TypeScript type that represents the structure of an object based on multiple JSON paths. For example, given paths like $.dynamodb.NewImage.id.S and $.dynamodb.NewImage.url.S, we wanted to generate a type that accurately represents the nested structure.

The Approach

Here's a simplified version of the code that attempts to solve this problem:

type GenerateTypeFromJsonPaths<Paths extends JsonPath[]> = Paths extends [infer Path, ...infer Rest]
  ? Path extends JsonPath
    ? MergeTypes<ParseJsonPath<Path>, GenerateTypeFromJsonPaths<Rest>>
    : {}
  : {};

type ParseJsonPath<Path extends string> = // ... implementation

type MergeTypes<T, U> = // ... implementation

const foo: ResultType = {
  dynamodb: {
    NewImage: {
      id: { S: "foo" },
      url: { S: "foo" },
      comment: { S: "foo" }
    }
  }
};

This code uses recursive type definitions to parse JSON paths and attempt to merge the resulting types into a single object type.

You can view and experiment with the full code in the TypeScript Playground

The AI Contribution

Interestingly, most of this code was generated with the help of AI models like GPT-4 and Claude 3.5 Sonnet. The experiment also drew inspiration from an existing GitHub repository, jsonpath-ts, which provides type inference for JSONPath queries.

Current Limitations

While the structure for each path is correctly generated, the code doesn't yet perfectly merge different branches into a single object type. This highlights an important point about AI-assisted coding: it can get you very far, but deep understanding is still crucial for solving complex problems.

Reflections on AI and Programming

This experiment brings to mind recent discussions about the changing landscape of software development, such as those outlined in the article "The Death of the Junior Developer". AI is indeed amplifying our capabilities, but it's a double-edged sword. It can accelerate both progress and dead ends.

As we navigate this new era of AI-assisted development, it's clear that a solid understanding of fundamental concepts remains invaluable. AI can generate impressive code snippets, but knowing how to piece them together, debug issues, and push beyond the AI's limitations is where human expertise shines.

Looking Ahead

While the current implementation isn't perfect, it's remarkable that we can generate such complex types programmatically. As AI tools continue to evolve, experiments like this will likely become more feasible, opening up new possibilities in type-safe programming.

For now, this experiment serves as a fascinating example of the potential - and current limitations - of AI-assisted coding in tackling advanced TypeScript challenges.

For those interested in the original prompt that led to this experiment, you can find it here.

A Cloud Development Troubleshooting Treasure Hunt

Sebastian Korfmann — Tue, 25 Jul 2023 09:03:55 +0000

In the process of constructing a small example project utilizing the Website SDK resource of Winglang, I unintentionally found myself engaged in a troubleshooting quest due to some unforeseen behavior.

In summary, the application I've created is depicted as a Wing diagram, and you can interact with the same version on the Wing playground.

The specifics of what this application does are not our focus today. However, it's crucial to note that the Website resource is composed of a Cloudfront Distribution that serves as a proxy to a public S3 Bucket, which has been configured for hosting. Today, these will be our main subjects of discussion.

A Brief Introduction to Wing

Wing integrates both infrastructure and runtime code into a single language. This integration empowers developers to remain in their creative flow, enabling the delivery of superior, faster, and more secure software.

Wing is designed to compile down to various targets. This feature provides the flexibility to execute tests against each of these targets. You have the option to perform swift tests against the Wing simulator by using the command wing test main.w. Alternatively, you can run a fully integrated test against the cloud provider of your choice with wing test --progress -t tf-aws main.w. This command will render the application down to Terraform and JavaScript, allowing you to deploy the entire application and run the test against it, all without modifying a single line of code.

Understanding the Wing Examples Repository

The examples repository leverages GitHub Actions to test each of the examples against both the simulator and the tf-aws target.

The setup employs a GitHub/AWS OIDC configuration, utilizing a finely scoped AWS IAM role to manage the resources of all the examples within the repository. This aspect is vital, not just for security purposes, but also to trace the thought process that will be explained later.

First Act - Building & Testing Locally

One of the great features of Wing is that it allows you to build and test your entire application locally on your own computer. Since the application itself is rather straightforward, I quickly assembled some resources and gave it a go with wing it.

The impressive web interface for my small application is also functioning perfectly ;)

After adding a test, I was pleased by the speed and simplicity of the whole process.

Naturally, running the same test against AWS took a bit more time because it involved a Cloudfront Distribution. However, the deployment and tests also executed flawlessly on my machine.

Now, let's move on to the next step.

Second Act - Creating the Pull Request & So Begins the Treasure Hunt

I created a pull request to initiate the previously mentioned Github Actions workflow. This would run all the tests in the CI setup. In previous new examples, I encountered some permission issues for the narrowly scoped CI role, so I wasn't too surprised when I saw some permission issues arise.

✖ terraform apply
Command failed: terraform apply -auto-approve
╷
│ Error: creating CloudFront Distribution: AccessDenied: User: arn:aws:sts::***:assumed-role/wing-examples/gh-actions-winglang-examples is not authorized to perform: cloudfront:CreateDistribution on resource: arn:aws:cloudfront::***:distribution/* because no identity-based policy allows the cloudfront:CreateDistribution action
- terraform destroy
│     status code: 403, request id: d1dfd897-05fa-4c0c-a69f-f082f9c1af28
│
│   with aws_cloudfront_distribution.env0_cloudWebsite_Distribution_5CC04CFB,
│   on main.tf.json line 133, in resource.aws_cloudfront_distribution.env0_cloudWebsite_Distribution_5CC04CFB:
│  133:       }

To fix this, I proceeded to add the necessary permissions to manage Cloudfront distributions for the CI role. I clicked on re-run, only to encounter another strange error related to S3.

Command failed: terraform apply -auto-approve
╷
│ Error: Error putting S3 policy: AccessDenied: Access Denied
│     status code: 403, request id: V1A0DQ4Y85ZJQQ6K, host id: wR65P8tjzKvu+1f7RZMhBJ65aFNWYCBVS90ck43Ji+YNDyHJw140V0dz5j/6XV+AHiDKG7ktrfM=
│
│   with aws_s3_bucket_policy.env0_cloudWebsite_PublicPolicy_67A62A0C,
│   on main.tf.json line 394, in resource.aws_s3_bucket_policy.env0_cloudWebsite_PublicPolicy_67A62A0C:
│  394:       }

Hmm, an Error putting S3 policy: AccessDenied: Access Denied for aws_s3_bucket_policy seemed unusual. It pointed to the s3:PutBucketPolicy action.

Trap Door Number 1 - IAM Role Permissions?

So, what's up with that s3:PutBucketPolicy permission which's pretty obviously missing. It's working locally with admin rights for the very same AWS account, after all.

I double-checked the permissions for the CI role and even set s3:* and eventually *, but to no success. Puzzling.

I decided to do some research on Google to see if I could uncover any helpful information. I found several discussions related to recent changes in AWS concerning public buckets. The threads suggested that one now has to explicitly configure the public settings for an S3 bucket. This seemed promising, so I checked the implementation of the Wing SDK bucket, which the website was using. It looked reasonable.

  if (isPublic) {
    const publicAccessBlock = new S3BucketPublicAccessBlock(
      scope,
      "PublicAccessBlock",
      {
        bucket: bucket.bucket,
        blockPublicAcls: false,
        blockPublicPolicy: false,
        ignorePublicAcls: false,
        restrictPublicBuckets: false,
      }
    );
    const policy = {
      Version: "2012-10-17",
      Statement: [
        {
          Effect: "Allow",
          Principal: "*",
          Action: ["s3:GetObject"],
          Resource: [`${bucket.arn}/*`],
        },
      ],
    };
    new S3BucketPolicy(scope, "PublicPolicy", {
      bucket: bucket.bucket,
      policy: JSON.stringify(policy)
    });
  }

However, this path seemed to be a dead end. It was time to go back to the drawing board.

Trap Door Number 2 - What Else Could Cause This Behavior?

To recap, we're dealing with a situation where a Terraform application can be deployed from my local machine, where I have admin rights to the AWS account. However, when deploying the same application to the same AWS account from Github Actions, the deployment fails due to permission issues. The Github runner uses OIDC for authentication and authorization. Could it be related to this configuration? Or could organization policies disallowing public buckets be at play?

To get more insight, I decided to examine the CloudTrail logs to see if anything unusual might catch my eye. The most conspicuous difference was the presence of webIdFederationData in the sessionContext for the CI role.

"webIdFederationData": {
    "federatedProvider": "arn:aws:iam::<accountId>:oidc-provider/token.actions.githubusercontent.com",
    "attributes": {}
},

It seems unlikely that this should make a difference, but let's confirm this. I assumed the CI role on my local machine and ran the AWS test again. Unsurprisingly, the operation was just as successful as before.

I briefly considered the potential influence of Organizational policies or session policies, only to conclude that it's quite certain we're not dealing with a permissions issue after all. Time to move on.

Trap Door Number 3 - Is It a Terraform Bug?

The next step was to delve deeper into the actual Github Actions workflow.

Could the answer lie within the Terraform trace logs? Before proceeding with this, I needed to ensure that this wouldn't expose any sensitive data given that we're working on a public repository. An alternative approach could be to SSH into the Github runner instead. Or perhaps use act to simulate the situation locally? However, neither seemed feasible as getting OIDC to work or obtaining a quick shell via the command line wasn't possible. Back to the logging idea then.

The examination of the logs seemed safe enough for public visibility. I enlisted the help of ChatGPT to modify the workflow to print out logs when a failure occurs.

- name: Execute wing test in matrix directory
  env:
    TF_LOG: info
    TF_LOG_PATH: ${{ runner.workspace }}/terraform.log
  run: cd ${{ matrix.example.directory }} && wing test --debug -t tf-aws main.w
- name: Output Terraform log
  if: failure()
  run: cat ${{ runner.workspace }}/terraform.log

Alright, that should do it. Now let's analyze what we've got. As a precaution, I decided to double-check the versions of Terraform and AWS Provider. Both terraform 1.5.x and AWS Provider 4.65 looked correct for both local and CI environments – nothing suspicious there.

I then proceeded to comb through the vast log output - Terraform trace logs are indeed very detailed. In doing so, I came across a particularly intriguing find:

2023-07-19T15:49:49.105Z [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value for aws_s3_bucket_public_access_block.env0_cloudWebsite_PublicAccessBlock_E7BC7F4B, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .block_public_acls: was cty.False, but now cty.True
      - .block_public_policy: was cty.False, but now cty.True
      - .ignore_public_acls: was cty.False, but now cty.True
      - .restrict_public_buckets: was cty.False, but now cty.True

Keen observers may notice that this isn't just a TRACE log; it's actually a WARN statement. Hmm, what's going on here? It indeed concerns the aws_s3_bucket_public_access_block resource, which we previously verified was correctly configured in the Wing SDK implementation. If this is throwing an error, and the aws_s3_bucket_public_access_block values must be false to define a public S3 Bucket, it's no surprise things are going awry. But wait a minute - why is it working locally again? Could it be... perhaps... a bug in Terraform?

Third Act - Unraveling the Mystery

Well, at least we now have a promising lead. Some diligent googling and browsing through Github issues in the AWS provider project yielded no directly related findings. However, I did come across a few recent bug reports about the recent change AWS made regarding the treatment of public buckets. And interestingly, they described precisely the behavior I was encountering.

Error putting S3 policy: AccessDenied: Access Denied

What's more, within these very issues, I stumbled upon a potential workaround.

resource "aws_s3_bucket_policy" "b" {
  ...
  depends_on = [aws_s3_bucket_public_access_block.b]
}

Thus, the good news: it appears this isn't truly a Terraform bug – although the 'ERROR' log from earlier remains mysterious, perhaps it's just an AWS hiccup. Furthermore, it seems like this might simply be an issue with dependencies. However, what still baffles me is why this problem only surfaces in the CI environment and not locally.

Fourth Act - Hooray - We've found the treasure

It's time for the grand finale - attempting to replicate the CI behaviour locally and identify the simplest example that demonstrates the issue.

Given that it seems to work on MacOS, it might be related to some specific behaviours in Linux or Docker. I've decided to use the Docker image I created for the Wing Github Action - ghcr.io/winglang/wing-github-action:v0.1.0 to start my investigation.

Again, I'll double-check the Terraform versions just to be sure:

Terraform v1.5.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.65.0

Terraform v1.5.2
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.65.0

With the setup ready, it was time to initiate a test run within the Docker container.

Success! We were able to reproduce the error. The next step was to trim down the Terraform configuration. After several iterations, it was narrowed down to these three critical components:

aws_s3_bucket
aws_s3_bucket_policy
aws_s3_bucket_public_access_block

Interestingly, as the number of resources decreased, so did the error rate. The full Terraform application saw failure in about half of the attempts when run in the Docker container on my machine. In contrast, the slimmed-down version only failed in about one-third of the attempts. Strikingly, all builds failed when run on the Github Actions runner.

To quantify this, an informal test was performed, applying and destroying the minimal setup ten times in succession:

Linux: 3/10 runs failed
Mac: 0/10 runs failed

Applying a fix – an explicit depends_on relation between the aws_s3_bucket_policy and aws_s3_bucket_public_access_block resources – successfully resolved the error on Linux. So, it appears we've found our culprit: a race condition among resources that sometimes allows for success, and other times results in failure.

Failed run

aws_s3_bucket.cloudWebsite_WebsiteBucket_EB03D355: Creating...
aws_s3_bucket.cloudWebsite_WebsiteBucket_EB03D355: Creation complete after 2s [id=cloud-website-c8e58765-20230719204937669500000001]
aws_s3_bucket_policy.cloudWebsite_PublicPolicy_44BB71F3: Creating...
aws_s3_bucket_public_access_block.cloudWebsite_PublicAccessBlock_18A70311: Creating...
aws_s3_bucket_public_access_block.cloudWebsite_PublicAccessBlock_18A70311: Creation complete after 1s [id=cloud-website-c8e58765-20230719204937669500000001]
╷
│ Error: Error putting S3 policy: AccessDenied: Access Denied

Successful run

aws_s3_bucket.cloudWebsite_WebsiteBucket_EB03D355: Creating...
aws_s3_bucket.cloudWebsite_WebsiteBucket_EB03D355: Creation complete after 2s [id=cloud-website-c8e58765-20230719205053652900000001]
aws_s3_bucket_public_access_block.cloudWebsite_PublicAccessBlock_18A70311: Creating...
aws_s3_bucket_policy.cloudWebsite_PublicPolicy_44BB71F3: Creating...
aws_s3_bucket_policy.cloudWebsite_PublicPolicy_44BB71F3: Creation complete after 1s [id=cloud-website-c8e58765-20230719205053652900000001]
aws_s3_bucket_public_access_block.cloudWebsite_PublicAccessBlock_18A70311: Creation complete after 1s [id=cloud-website-c8e58765-20230719205053652900000001]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

The order of resource creation notably varied. For a deeper technical explanation of the fix, feel free to explore the corresponding pull request.

In a nutshell

The fix boils down to establishing the dependency outlined above, and it was achieved through a mere 2-line change. As for the discrepancies between operating systems, my guess is that it's tied to the availability of computing resources, although this is purely speculative. It might be illuminating to run the tests hundreds or even thousands of times to see if the error would appear on my local MacOS machine. But considering the problem is resolved now, I'm content with leaving the matter as is. You are welcome, though, to offer your speculations in the comments!

Despite the relatively simple resolution, this bug proved to be particularly elusive. It's difficult to attribute it definitively in this instance. The Wing resource was likely developed before the changes to S3 permissions and probably on a Mac. Meanwhile, the integration tests are still in their infancy. As for Terraform, it's unclear whether the dependency was mandatory before AWS altered S3 bucket permissions. Nevertheless, helpful pointers in the resource documentation would have been appreciated (there is indeed an issue open for this). Alternatively, a specific "public bucket policy" could be beneficial, though that's maybe a little too far fetched. All in all, a great example for moving targets in cloud engineering.

Key Lessons

This episode embodied the perfect storm of challenges, with cryptic error messages, slight variations in behavior across different operating systems, and a dollop of my own confirmation bias. Yet, it serves as a striking example of the hurdles one might encounter in cloud development. I have no doubt that this particular issue alone consumed countless hours of troubleshooting time, not to mention the time spent grappling with IAM issues in general. The AWS Terraform provider repository hosts numerous examples of similar dilemmas, and AWS CDK is no stranger to such issues either.

Developing in the cloud (like AWS) can be an absolute delight when everything slots seamlessly into place, freeing you from substantial operational burdens. However, when things go awry, it can feel like an insurmountable barrier. Frequently, we're left with limited error context, inadequate or non-existent logs, ever-changing targets, and outdated documentation or blog posts.

Projects like Wing are making strides towards simplifying the complex task of developing and operating applications in the cloud. The cloud resource SDKs being developed encapsulate the collective hard-earned knowledge and experience, distilling it into reusable components. While the system might not be perfect or bug-free at this stage, there's an incredible team and an active, albeit small, community at the ready to offer assistance.

Finally, if I could make one request of AWS, it would be to offer greater error context surrounding permission issues. If not directly in the API call error message, then at least within CloudTrail. I'm convinced that such a feature would save an unfathomable amount of engineering hours daily.

Perhaps such a feature already exists, and I'm just not aware of it? Please let me know in the comments! I'm also eager to hear about your experiences with similar challenges.

Cloud Driven Development - Episode #001

Sebastian Korfmann — Mon, 11 Oct 2021 20:24:30 +0000

Last week, I launched Cloud Driven Development (CDD) as a Youtube channel - a somewhat regular live coding series covering the topic of cloud development from various point of views. The sessions are usually in a pair programming setting, with changing guests & topics for each episode.

With CDD I want to explore developer focused tools, services and workflows beyond "Cloud Native" (which always translates to Kubernetes in my head). So, let's embrace that vendor lock-in, forget about containers for the most part and dive into a world without a local dev or testing environment.

If you're up for some casual talking / live coding about a topic which fits this category, I'd be more than happy to hear from you. I explicitly want to learn about new things outside of the CDK bubble as well and have some super exciting topics in the pipeline already.

Async Testing with AWS CDK

For the first episode I was super happy to have Matt Morgan as a guest, who wrote a very interesting blog post about "Async Testing with AWS CDK".

As the title suggests, Matt has been exploring cloud testing strategies. The approach we're talking about in this episode, leverages Cloudformation and Custom Resources. The best part, it's not just an academic excercise but it's actively being used at his current employer PowerSchool.

Technical Notes

Custom Resource

Next Episode

The next episode is scheduled already, where Ben Force and I will build a really simple Alexa Skill with the AWS CDK. In this context particularly interesting: How to keep iteration cycles small, get fast feedback and can tests help there?

CDK for Terraform 0.3

Sebastian Korfmann — Thu, 22 Apr 2021 13:29:50 +0000

The CDK (Cloud Development Kit) for Terraform allows developers to use familiar programming languages to define cloud infrastructure and provision it through HashiCorp Terraform.

Since the 0.2 release roughly a month ago, by far the biggest change was the onboarding of a newly hired full-time engineer. I'm super happy that Ansgar Mertens joined our team and am delighted to see the positive impact he already had on the project.

CDK for Terraform 0.3

We released CDK for Terraform 0.3 recently. Besides from various smaller and bigger improvements, there are two major new features: Remote templates and multiple stacks.

Remote Templates

The cdktf-cli ships with a basic template for each target language. These templates are the blueprint for the project which gets created by running cdktf init, e.g. cdktf init --template typescript.

With remote templates, users can now bring their own template. This enables customized starting points for cdktf projects tailored to the requirements of individual users and organizations.

Multiple Stacks

Up until CDK for Terraform version 0.2 only a single stack per application was supported. Starting with version 0.3, we're now enabling users to model their infrastructure in multiple stacks.

One stack is a collection of infrastructure which will be synthesized as a dedicated Terraform configuration. In comparison to the Terraform CLI, a Stack is equal to a dedicated working directory. Stacks are therefore a boundary to separate state within your infrastructure setup. This can be leveraged to separate a cdktf app in several layers (e.g. Network, Data, Compute), or to manage different versions of the same stack like in the following example.

import { Construct } from "constructs";
import { App, TerraformStack } from "cdktf";
import { AwsProvider, Instance } from "@cdktf/provider-aws";

interface MyStackConfig {
  environment: string;
  region?: string;
}

class MyStack extends TerraformStack {
  constructor(scope: Construct, id: string, config: MyStackConfig) {
    super(scope, id);

    const { region = 'us-east-1' } = config

    new AwsProvider(this, "aws", {
      region,
    });

    new Instance(this, "Hello", {
      ami: "ami-2757f631",
      instanceType: "t2.micro",
      tags: {
        environment: config.environment
      }
    });
  }
}

const app = new App();
new MyStack(app, "multiple-stacks-dev", { environment: 'dev' });
new MyStack(app, "multiple-stacks-staging", { environment: 'staging' });
new MyStack(app, "multiple-stacks-production-us", { environment: 'production', region: 'us-east-1' });
new MyStack(app, "multiple-stacks-production-eu", { environment: 'production', region: 'eu-central-1' });
app.synth();

There are two limitations to using multiple stacks at the moment:

Operations such as diff, deploy and destroy are limited to one stack at a time for now - See this issue
Referencing a resource across stacks can't be done automatically, but has to be built intentionally by the user - See this issue

Both limitations will be addressed in future releases.

What's next

Speaking of future releases, here's a rough plan what we'll be working on next. This includes the highest voted issue - Support for Go.

In the meantime, please checkout what we've built so far and make yourself familiar with CDK for Terraform :)

hashicorp / terraform-cdk

Define infrastructure resources using programming constructs and provision them using HashiCorp Terraform

CDK for Terraform

Cloud Development Kit for Terraform (CDKTF) allows you to use familiar programming languages to define cloud infrastructure and provision it through HashiCorp Terraform. This gives you access to the entire Terraform ecosystem without learning HashiCorp Configuration Language (HCL) and lets you leverage the power of your existing toolchain for testing, dependency management, etc.

We currently support TypeScript, Python, Java, C#, and Go.

CDKTF includes two packages:

cdktf-cli - A CLI that allows users to run commands to initialize, import, and synthesize CDK for Terraform applications.
cdktf - A library for defining Terraform resources using programming constructs.

Get Started

Choose a language:

Hands-on: Try the tutorials in the CDK for Terraform collection on HashiCorp Learn.

Documentation

Refer to the CDKTF documentation for more detail about how to build and manage CDKTF applications, including:

Application Architecture: Learn the tools and processes that CDKTF…

View on GitHub

CDK for Terraform 0.2

Sebastian Korfmann — Wed, 17 Mar 2021 09:45:14 +0000

The CDK (Cloud Development Kit) for Terraform allows developers to use familiar programming languages to define cloud infrastructure and provision it through HashiCorp Terraform.

We released CDK for Terraform 0.2 recently. Besides from various smaller and bigger improvements, the focus was on iterating on the support for Terraform modules.

Up until now, we did support generating code bindings for Terraform Modules which are published on the Terraform Registry only. The Terraform Registry API provides all the metadata we needed to generate code straight from the API response without having to process the Terraform module sources.

With the new 0.2 release, we changed the internal implementation for fetching modules to support all sources which Terraform supports. This includes Modules from Github, HTTP and local disk. Since we're not relying on the Terraform Registry API anymore, we had to come up with our own solution to extract the relevant metadata.

The main challenge here was: CDK for Terraform is primarily written in Typescript, while most of the Terraform ecosystem relies on Go. To bridge this gap, WebAssembly came to the rescue. It enabled us to leverage existing tools from the Go community by building a thin wrapper in Typescript. More details about the learnings from that journey in one of the next posts :)

In the meantime, please go ahead and checkout the latest release of CDK for Terraform. We're always looking for feedback and ideas to drive the product forward.

AWS AppSync Direct Lambda vs DynamoDB Resolver

Sebastian Korfmann — Tue, 17 Nov 2020 14:45:07 +0000

I've been experimenting with AWS AppSync and I'm intrigued by the idea of ditching VTL entirely in favour of Direct Lambda Resolvers. Ignoring the monetary cost aspect which Lambda adds, my biggest question was the performance impact this might have. So, I ended up doing some basic real world tests.

Setup

A direct lambda resolver for the field Query.getNote and a VTL DynamoDB resolver for Query.ddbGetNote. Both versions are performing a GetItem operation on the same DynamoDB table.

1st Attempt

The first try with a Lambda function mostly in AWS CDK default configuration (i.e. 128 MB Ram) was quite disappointing.

The (warm) Lambda resolver took ~ 200 ms compared to 14 ms of the VTL DynamoDB call. Interestingly enough, most time consuming in the Lambda function appears to be the DynamoDB call. This seems odd.

Back to the drawing board?

A bit of research suggested, that this is an issue for underpowered Lambda functions. I've found this slide deck with a nice chart for balancing performance / costs for DynamoDB calls in Lambda functions.

The sweet spot seems to be around 512 MB memory for a Lambda function.

2nd Attempt

I changed the Lambda function to 512 MB and left everything else as it was. And surprise, surprise - This looks way better now.

Now the (warm) Lambda resolver takes ~ 30 ms compared to still 14 ms of the VTL DynamoDB call.

Conclusion

Skipping VTL in favour of direct Lambda resolvers seems to be reasonable for the gained flexibility, even though there are a few drawbacks. There's a bit of an overhead to call the Lambda function, which gets drastically worse on cold starts (+ ~ 600 ms).

In terms of costs, 1 million invocations at 100 ms with 512 MB would clock in at roughly 1 USD. That's on top of the 4 USD per million operations in AppSync. All in all, not exactly cheap in scale but nothing I'd be concerned about for now. That's something to optimize when costs would become an issue.

I'll continue to explore Direct Lambda Resolvers for all cases and report how it goes :)

cdk.dev - Call for Contributors

Sebastian Korfmann — Thu, 13 Aug 2020 14:03:32 +0000

Matthew Bonig (Open Construct Foundation), Mike Bild and I are joining forces to build out cdk.dev as a content hub around the emerging Cloud Development Kit (CDK) ecosystem. With AWS CDK, cdk8s and Terraform CDK the CDK family has now three members, and I'm sure it'll continue to grow. Not only in regards to core products, but ancillary products, tools and services as well.

Our Mission

We set out to aggregate content and events, provide a home for little tools and last but not least, become a hub for ideas and questions around the CDK ecosystem.

Our short term goals include:

Build a simple content aggregator for blog posts, tweets and events
Build a dedicated section for educational content
Build a project show case listing for inspiration

While we think our mission is aligned with the goals of AWS (AWS-CDK / cdk8s) and HashiCorp (cdktf), this project is not affiliated with those companies. It's a private effort coming from us and whoever joins along the way.

Current State

We created a new Github Organization with one dedicated repository for organizing ourselves. A simple landing page was setup and is live already.

Other than that, I opened up my private Slack channel - which I used as a feedback channel for early adaptors of Terraform CDK. Feel free to grab an invite and join. The Slack chat can be used for ad-hoc discussions, whereas more thorough and detailed planning will happen in the Github repository.

cdk-dev / base

Repo for organizing things around cdk.dev

cdk.dev - The Cloud Development Kit (CDK) Ecosystem Hub

This is a community driven project for the Cloud Development Kit (CDK) and will cover the entire CDK ecosystem:

It's a place to bring up and discuss ideas, plan the next steps and provide feedback.

Mission

We build a dedicated hub for people interested in the CDK ecosystem.

Goals

Create a content hub for CDK topics
List upcoming events
Provide examples and howtos
Provide a space to discuss and exchange ideas
Provide helpful tools

Website Repo

https://github.com/cdk-dev/website

View on GitHub

Join Us

We're committing to build and steer this project in public and would love to get more people involved early on. If you're keen on joining us, grab a Slack invite and get in touch :)

A Terraform CDK Construct which doubles as native Terraform Module

Sebastian Korfmann — Mon, 27 Jul 2020 11:12:31 +0000

Lou raised the question of Terraform Module interoperability in the Terraform CDK:

Module Interoperability — It seems the CDK will support regular Terraform modules, for code sharing. But it does remain to be seen if module sharing can be reversed. Can a CDK project publish a module which is then consumed by HCL? If you’d have to learn HCL to write and share a module, that mostly defeats the point of using the CDK in the first place.

That's a great question, and while I was confident this is something which could be done today already, I thought this would be a good example to actually build out. So, that's what I did and I'd like to share with you.

General Concept

The Terraform CDK is synthesizing from code to HCL compatible JSON. This makes it less a question of if it's possible to, but more like a question of how to organize the code and how to distribute it.

Resource vs TerraformResource

The concept I came up with includes a custom construct, which is sort of the equivalent of a native Terraform module in the Terraform CDK. That's just a Typescript class, which wraps other Terraform constructs (an EC2 Instance in this case). It could contain more constructs of course, they could be nested, whatever you can imagine - it's all just Typescript in the end.

import { Construct } from 'constructs';
import { Resource } from 'cdktf';
import { Instance } from '../imports/providers/aws'

export interface CustomInstanceProps {
  instanceType?: string;
  tags?: {[key: string]: string};
}

export class CustomInstance extends Resource {
  public readonly instance: Instance;

  constructor(scope: Construct, name: string, props?: CustomInstanceProps) {
    super(scope, name);

    const { tags, instanceType = "t3.nano" } = props || {};

    this.instance = new Instance(this, 'ubuntu2', {
      ami: "ami-0ff8a91507f77f867",
      availabilityZone: "us-east-1a",
      instanceType,
      tags
    })
  }
}

In contrast to the generated Instance class, the CustomInstance class extends Resource and not TerraformResource. While CustomInstance is still a node in the Constructs tree, the CustomInstance class itself will be skipped during synth. Only the actual TerraformResource classes will be rendered down to HCL compatible JSON. It's really just a container for actual Terraform resources.

Distribution

NPM Package

Since our CustomInstance is a Typescript class, distributing this as a NPM package is straightforward. Make sure to setup the following keys in your package.json and off you go:

  "main": "path/to/your/construct.js",
  "types": "path/to/your/construct.d.ts",
  "peerDependencies": {
    "cdktf": "^0.0.12",
    "constructs": "^3.0.0"
  },

Declaring peerDependencies and not dependencies is the important point here. Otherwise, you'd likely end up in dependency hell.

Terraform Module

In order to distribute this as a Terraform module, this should be synthesized to JSON somehow. Let's build a Stack without a provider and that'll pretty much be synthesized to the equivalent of a Terraform module.

import { Construct } from 'constructs';
import { App, TerraformStack, TerraformOutput, Token } from 'cdktf';
import { CustomInstance } from './construct'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name);

    const custom = new CustomInstance(this, 'Custom')

    new TerraformOutput(this, 'arn', {
      value: custom.instance.arn
    })
  }
}

Rather than ignoring the the synthesized output in Git, let's commit this and configure cdktf to use a nicer folder name (module in this case) for its output:

{
  "language": "typescript",
  "app": "npm run --silent compile && node lib/module.js",
  "terraformProviders": [
    "aws@~> 2.0"
  ],
  "codeMakerOutput": "imports",
  "output": "module"
}

For the native Terraform module use-case it would be nice to omit the stack traces in the generated JSON, since that'll change depending on where it's build. A little bit of jq could be helpful here

cat ./cdktf.out/cdk.tf.json | jq 'walk(if type == "object" then with_entries(select(.key | test("\/\/") | not)) else . end)'

Mid / long term, native support would be better though. Check out this open issue here.

Input Variables

Terraform Input Variables are not natively supported by the Terraform CDK yet. There's an open issue to change this. However, that's a case where escape hatches come in handy:

const stack = new MyStack(app, 'cdktf-hybrid-module');

// See issue linked above, this will be natively supported
stack.addOverride('variable', {
  tags: {
    description: "Tags for the instance",
    type: "map(string)"
  },
  instance_type: {
    description: "Instance type",
    type: "string"
  }
})

From here on, it's really useable as any other Terraform module:

module "instance" {
  source = "github.com/skorfmann/cdktf-hybrid-module//packages/cdktf-hybrid-module/module"

  instance_type = "t3.nano"

  tags = {
    "CDKTF" = "IS AWESOME"
  }
}

Bonus: Terraform Module via NPM

One thing I really like about NPM: It doesn't make any assumptions about what you're intending to ship. Let's leverage this, to use the native Terraform module via NPM :)

npm init -y
npm install cdktf-hybrid-module

And then just reference it from node_modules.

module "instance" {
  source = "./node_modules/cdktf-hybrid-module/module"

  instance_type = "t3.nano"

  tags = {
    "CDKTF" = "IS AWESOME"
  }
}

That could be a way, to get dependency management for native Terraform modules via NPM. I'm certainly not the first one who had this idea, I'm pretty sure I saw this in other blog posts as well.

I think this could make sense in complex scenarios, where dependency management is important and manually managing this becomes a burden.

Current limitations

As mentioned a few times, there are a few usability gaps at the moment:

Outputs are already fully supported by cdktf but due to the random naming a bit hard to use. There' an open issue to address this
Variables aren't natively supported in cdktf yet, but can still be done with escape hatches. There's an open issue
Since there aren't official prebuilt provider packages at the moment, this has to inline the generated constructs.

The last point is the biggest drawback at the moment from my point of view, but the work to improve this is underway - see this open issue.

Conclusion

I think this demonstrates that hybrid CDK packages / Terraform modules are totally possible and have lots of future potential. Would love to hear what you think about it!

Check out the entire example project as well to see all of this in full context.

skorfmann / cdktf-hybrid-module

A Terraform CDK Construct which is also usable as Terraform Module

In the next post we'll build a Python package for our little CustomInstance, stay tuned!

The Journey from Terrastack to Terraform CDK

Sebastian Korfmann — Thu, 23 Jul 2020 16:58:42 +0000

The initial version of Terrastack was built over the course of a few days in March 2020. It was built around the same concepts as the AWS CDK and shared a similar vision: Leveraging imperative programming languages such as Typescript, Python, Java and C# to build out infrastructure with the Terraform engine.

// Detect dark theme var iframe = document.getElementById('tweet-1237848600354254849-746'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1237848600354254849&theme=dark" }

While my intention with the prototype was to raise awareness and get support for further development, I didn’t anticipate pushing at an open door.

Fast forward four months, the successor of Terrastack - the Terraform CDK - got launched as a joint effort by Hashicorp, the AWS CDK team and me :) It turned out that Hashicorp and the AWS CDK team started to work together on the exact same idea long before I published Terrastack. They reached out to me and I’m still very grateful for the opportunity to join the ongoing project as an independent contributor.

hashicorp / terraform-cdk

Define infrastructure resources using programming constructs and provision them using HashiCorp Terraform

CDK for Terraform

We currently support TypeScript, Python, Java, C#, and Go.

CDKTF includes two packages:

cdktf-cli - A CLI that allows users to run commands to initialize, import, and synthesize CDK for Terraform applications.
cdktf - A library for defining Terraform resources using programming constructs.

Get Started

Choose a language:

Hands-on: Try the tutorials in the CDK for Terraform collection on HashiCorp Learn.

Documentation

Refer to the CDKTF documentation for more detail about how to build and manage CDKTF applications, including:

Application Architecture: Learn the tools and processes that CDKTF…

View on GitHub

The community preview of the Terraform CDK is considered an alpha version. There are lots of rough edges which we’ll iron out over the next few months. Nevertheless, the sentiment of the feedback has been quite positive so far. There are more than 600 stars on Github, a bunch of contributions to the Github repository and roughly 1500 downloads via NPM within one week after the launch. That’s fantastic and makes me excited about the future of the Terraform CDK.

// Detect dark theme var iframe = document.getElementById('tweet-1284820925452615686-841'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1284820925452615686&theme=dark" }

Here’s a little real world example, which I migrated from Terrastack to Terraform CDK.

skorfmann / cdkweekly

Terraform CDK SSL Proxy for cdkweekly.com

The CDK Weekly newsletter is served by revue.co, which doesn't support SSL for custom domains.

This little workaround was initially built on top of Terrastack and got migrated to its successor Terraform CDK. It's the first real world example for the Terraform CDK and is running in production for a while now.

Setup

npm install
cdktf get

Deploy

cdktf deploy

That's where one of the Terraform CDK examples is stemming from.

Next Up:

Redirect naked domain -> www (probably including SSL via Cloudfront distribution)
Inject Google Analytics or something similar
Migrate state to Terraform Cloud

View on GitHub

I’m currently working on yet another example, which will be a bit more elaborate and demonstrate a more complex use case.

When you’re interested to learn more about the Terraform CDK, here are some resources to get you started:

If you have feedback or feature requests, the Github repository is a good starting point. We try to follow up on newly created issues quickly. I’d love to hear from you.

ps: The actual first version of Terrastack dates back to the end of 2018 and was sort of Terragrunt on steroids powered by Typescript. I had lots of fun building that prototype back then together with Andreas Litt.

pps: If you're interested to keep up with what's happening in the CDK ecosystem, you should consider signing up to cdkweekly

Introducing Terrastack: Polyglot Terraform supercharged by the CDK

Sebastian Korfmann — Thu, 12 Mar 2020 12:35:21 +0000

Note: This was published on my personal blog as well

Terrastack enables you to keep using Terraform as engine, while defining your resources in actual programming languages such as Typescript, Python, Java or C# - with more to come (perhaps Ruby?).

This is made possible by the Cloud Development Kit (CDK) and jsii for generating the polyglot libraries. While the major use-case for the CDK is generating Cloudformation configuration as YAML, it's capable of generating pretty much any configuration.

The actual idea of going forward with Terrastack was inspired by yet another project which leverages the CDK / jsii combo to generate Kubernetes configuration: cdk8s. When I saw that project, I eventually realized that the CDK could be useful for more scenarios than just generating Cloudformation YAML.

A first quick prototype was up and running in roughly two evenings. While not all types were properly generated, it was able to generate a simple but working Terraform configuration.

// Detect dark theme var iframe = document.getElementById('tweet-1235298514646773760-414'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1235298514646773760&theme=dark" }

The general workflow is:

Generate a JSON schema for a given Terraform provider (e.g. AWS or Google Cloud) with builtin Terraform command
Use this schema as input for a Typescript generator, to generate classes for each Resource and Data type of that provider
Compile polyglot libraries from these generated Typescript classes via jsii
Define your Cloud resources in your prefered language
Generate Terraform compatible JSON and use it as you would any other HCL based Terraform project

When looking at these steps, that's pretty much what the CDK team does for Cloudformation. Since Terraform and Cloudformation are both declarative, it's conceptually pretty close. This makes Terraform a perfect contender for getting "CDKified".

Similar to the CDK, that's how a simple Terrastack looks like right now

Background

Finishing up an on premise -> AWS migration utilising Terraform, here are the pain points we've hit on a regular basis over the last two years:

Refactoring is hard - Unless you love digging around in JSON state files
Lack of good dependency management
Most of the product teams, don't actually want to learn a new syntax like HCL.
Terraform Modules are a primitive way to share code. Want to dynamically adapt its definition? Good luck with that.
Code Distribution becomes complex in larger organisations
Ensuring certain configuration standards (think of Tagging, Object Storage / VM configurations)
Git Ops with small components to separate state is more complex than it should be
Established Software Engineering practices like unit / integration tests are hard to implement
HCL / Terraform integration in code editors such as VS Code is poor and mostly broke for us with Terraform 0.12

Terrastack tackles all of the above by leveraging existing technologies:

The Javascript ecosystem
Cloud Development Kit (CDK)
jsii for polyglot libraries

The engine of Terraform and its providers are quite good overall. From my point of view, most of the issues are stemming from the rather static and declarative nature of Terraform.

With Terrastack we can start to treat "raw" Terraform configuration as an "assembly" artifact which we can generate in a predictable way. This enables us to tackle most of the pain points above:

Use existing, well established dependency management like NPM or Yarn
Leverage mono repo / build tooling in the respective programming language
Build upon existing testing frameworks for unit and integration tests
Enjoy intellisense of your favourite language in whatever editor you're using right now
Create commonly shared packages of cloud solutions for internal, or even public usage.
Share code which is aimed at the AWS CDK (e.g. Tags, IAM Policies, ECS Task Definitions, Kubernetes config?)

I'm still thinking about the refactoring part. My current conclusion is, that there are mainly two things are getting in the way of proper refactorings:

Static naming of resources
Terraform modules aka namespaces

I strongly believe, that we can come up with a better concept in Terrastack. But that's for another blog post.

Summary

From the initial very rough prototype, it took about a week to release a bit more polished prototype on Github.

// Detect dark theme var iframe = document.getElementById('tweet-1237848600354254849-936'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1237848600354254849&theme=dark" }

Please check it out and give it a try. I would love to get feedback!

TerraStackIO / terrastack

This project is archived, but the idea of Terrastack lives on in the Terraform CDK. - https://github.com/hashicorp/terraform-cdk

This is the first post of a series regarding Terrastack. In the next post, I'll dive a bit deeper into the internals of the CDK itself and what I've learned about its mechanics so far.