Forem: Carlo van Overbeek

UX designers, hear my prayer...

Carlo van Overbeek — Fri, 14 Feb 2025 21:14:17 +0000

If I could get one wish granted right now on the condition I had to be 100% sure it would benefit everyone in the world, I would ask for this:

(Sorry for it being ugly, I'm not a UX designer after all...)

You might ask "what even is that?" or "how does that benefit anyone?" Very valid questions, but let's first take a step back.

Why do I want this?

Most webpages and an increasing amount of applications these days employ some kind of background component loading. Now that's all nice and good. It becomes a nuisance when I want to click something and just at that moment the UI changes. Something in the background fully loaded, shifts or overlays the screen and causes me to miss, or worse: click something else. I literally accepted meetings in MS Outlook I wanted to decline and vice versa because of this!

Now this post is not simply a rant. I understand the benefits of loading sub-components in the background and would never advise against it. Moreover, I have a possible solution for UX designers aspiring to tackle this problem.

The solution

Bring back a good-old loading bar!

When all data for a sub-component to render has fully loaded in the background, display a loading bar at a nice non-interfering place in the UI. It signals to the user "if you wanted to click anything, please do so now." When the time runs out, only then does the UI change. (Changing the UI immediately because of a user click would also still be acceptable, of course.)

You might argue that when a (web-)app starts up, so many things have to load in the background, this would never be practical. You would just have a stream of loading bar after loading bar, annoying the user. Also fine. In that case, just display a red bar for a while to signal: "we are still busy, you may look, but click at your own risk." I would be totally okay with that. Despite of you want might think of me after reading this, I'm actually a patient person. I just don't like misclicking when it's not my fault.

Wrapping up

Too many apps start to feel like a first-person shooter these days... Speaking of which, video games usually communicate very well when user input is or isn't desired, for example with frosting on the edges of the screen. Once again, I'm not a UX person, maybe a loading bar isn't the best solution to this problem, but some kind of signalling would be fantastic.

Straight to the Money 💰 minimalistic yet all-inclusive Python project template

Carlo van Overbeek — Mon, 02 Dec 2024 21:49:25 +0000

Stop wasting time in analysis paralysis over Python tooling choices 💸
Pick the defaults 💱
Go straight to the money 💹

tl;dr: I created a minimalistic yet all-inclusive Python project template.
Check it out.

The Background Story

When it comes to helping others with developing software, too many templates/guides/peoples focus on features, or even worse, tools. While my guided template extremely spotlights some tools, they are approached from a very different focus: needs. Every software development project has almost identical needs:

version control
dependency management
linting
testing
compiling
documentation
publishing

Of course, this list is not exhaustive. Software development these days also needs CI/CD, SBOM exports, ergonomic chairs etc. However, if anything from the list above is missing in your project, something is truly lacking. With this template I'm trying to fill in those needs as simple yet effective as possible.

Some fanatics will look at this template and might say something like: "Isn't pytest better than unittest?" Or: "Isn't tox also super useful?" Well, are those extra dependencies that will only really be useful to people who already know how to add them to this template? Guess what, all three yes!

Similarly silly discussions can be had about Hatch, MkDocs or Sphinx. I've had it with all these well-meaning 'developer advocates' forcing some tool on you at first chance, while no-one out there seems to be able to provide you a comprehensive end-to-end developer journey. Also, by keeping the template simple, it's open to extension with your own favorite tools without carrying the bloat from my choices.

Happy coding 🐍🤗

Input Variables File for Multiple Terraform Workspaces

Carlo van Overbeek — Wed, 04 May 2022 13:59:40 +0000

We've all been there: you have written a Terraform configuration that is so awesome, you have to apply it multiple times... Or at least to dev, test and prod. However amazing you configuration might be, it's very rare to have all details exactly the same over every environment. You usually end up with something like this, and that's perfectly fine (for readability, I've condensed the code a bit, see this gist for a more complete example):

resource "aws_instance" "this" {
  count = var.instance_count

  ami           = var.ami_id
  instance_type = var.instance_type

  tags = {
    Name = "project-xyz-${terraform.workspace}-${count.index}"
  }
}

Where this starts to hurt is during apply time, because you will have for example:

terraform workspace select prod
terraform apply -var='instance_count=5' -var='ami_id=ami-1a2b3c' -var='instance_type=t2.large'

This line of variables becomes unclear very fast. Also, remembering all those variables or making your command line remember the statement (per environment) is rather cumbersome. Moreover, making this work in automation (again, per environment...) will most likely require some env variables magic. This separates the definition of your infrastructure from the standard ways of calling it, which is bad. (By the way, if you have never heard of connascence, stop reading this article and read this one instead.)

Terraform Cloud offers some remediation to this, but also here you create a separation between template and env variable sets. It would be much better if we could keep the default sets of variables in version control next to our configuration. After some tinkering, I found the following solution:

variable "workspace_variables" {
  type = map(object({
    instance_count = number
    ami_id         = string
    instance_type  = string
  }))
  default = {}
}

locals {
  default_vars = lookup(var.workspace_variables, terraform.workspace, {
    instance_count = 2
    ami_id         = "ami-abc123"
    instance_type  = "t2.micro"
  })
  instance_count = var.instance_count != null ? var.instance_count : local.default_vars["instance_count"]
  ami_id         = var.ami_id != null ? var.ami_id : local.default_vars["ami_id"]
  instance_type  = var.instance_type != null ? var.instance_type : local.default_vars["instance_type"]
}

This code snippet reads one an optional variable to rule them all: workspace_variables. If none is provided, it will use the default sub-variable set. Also, you can still override any set of variables with the Terraform cli. Next to this snippet, you will need to make the default for all variables null and you will have to replace var. with local. when using them. Then you can create a Terraform vars file with something like this:

workspace_variables = {
  dev = {
    instance_count = 2
    ami_id         = "ami-abc123"
    instance_type  = "t2.micro"
  }
  test = {
    instance_count = 3
    ami_id         = "ami-a1b2c3"
    instance_type  = "t2.medium"
  }
  prod = {
    instance_count = 5
    ami_id         = "ami-1a2b3c"
    instance_type  = "t2.large"
  }
}

Much clearer and all kept in one findable place that can be put in version control! And you will get the configuration you want with a simple Terraform flow of selecting your workspace and hitting apply. Also, you can still override the variables with the command line for hot fixes, tests etc.:

terraform workspace select prod
terraform apply -var='ami_id=ami-x1337x'

It may look like too much boilerplate for simple parameter input, but I've found that it weighs up to setting variables the 'normal' way if there are a lot of them. See if it works for you!

Final remark: it would require some more tinkering and boilerplate to implement proper Terraform variable validation and forward compatibility of the vars file, but that might be fixed when optional variable attributes comes out of public beta.

(The code above is condensed for readability, for a more complete example see this gist.)

'Code injection' in AWS CodePipeline

Carlo van Overbeek — Wed, 16 Feb 2022 12:56:06 +0000

Now before you think I am casually going to write up here how I hacked AWS, I think CodePipeline works as intended. Still, I would like to bring a quirk to your attention in the way it's set up. This quirk can make it run different code, than the code that was put into the pipeline.

In short: overwrite the source artifact in S3 in the time between CodePipeline putting it there and follow-up actions downloading it for their business. This is not mentioned in their docs. (I did mention it as feedback to them some months ago, but nothing changed and I haven't heard from them.)

I could leave you with this, but it's of course more fun and insightful to give some more details.

Steps to reproduce

First of, you will need to have a CodePipeline pipeline deployed. If you don't know how to set it up, see AWS' documentation. I will use CodeCommit as repository source and CodeDeploy as execution environment in my example below, like in the provided AWS example. Any other combination of services that uses S3 to temporarily store the source artifact should work fine as well. (I verified that it also works with GitHub source / Lambda execution.)

Normal flow

For the demo, I created a minimal CodeCommit repository with only the following buildspec.yaml file in a repository called inject:

This should make a CodeBuild server echo hello world. (Normally, you would do something more interesting here, such as compiling code or running a Terraform configuration.)

Then I start a pipeline that uses this repository as its source and pass it to CodeBuild:

As expected, this causes the following to happen in CodeBuild:

The string hello world was correctly outputted.

Now to the more interesting part.

With injected code

Some general advice before you start: speed is key, so sharpen your AWS GUI navigation skills ;) (You could of course put a manual approval action in the pipeline to give yourself all the time you need, but I wanted to keep this demo as simple as possible.)

Again start the pipeline:

Now quickly go to the S3 bucket where the source artifacts are stored and check the name of the latest source zip file:

Now quickly rename your code package accordingly:

Note the different string to be echoed.

Upload that file to the bucket used as source, overwriting the original source file (to save time you could already setup this window, don't forget to use the correct settings for KMS etc.):

Now watch how CodeBuild executes the code just uploaded instead of what's in the repository:

Closing remarks

Being able to make a pipeline execute different code than that what was put into it is of course already undesirable by itself. What I think that makes it a more serious problem in this case is that it can be done via a completely different service (S3) which is usually used for all sorts of stuff. Editing the code to be executed in S3 can easily go unnoticed in an account hosting a score of different microservices or one with a lot of users. Also, in such a case the chance of a single service or user being hacked that is privileged enough to edit S3 is way higher than the chance that someone hacks their way up to a privilege to edit a pipeline directly.

Again, I would like to emphasize that this is not really a hack into CodePipeline. Still, I think CodePipeline is trying its best to hide all its shifting around with roles and artifacts from you. That is of course great for simplicity, but it might also lead to a security hole in your architecture when you are not careful or simply overlook it. It would be nice if CodePipeline would support pointing to a specific object version in the future, those have a stable unique id. For now, all you can do is protect the source artifact bucket at least as good as your pipeline. If you would like some inspiration on protecting your S3 buckets to the most paranoid level, I wrote a blog about that as well: Read-only buckets in shared AWS accounts.

I wrote this blog and performed the work described within it at Simacan, my employer at the time of writing.

Read-only buckets in shared AWS accounts

Carlo van Overbeek — Wed, 16 Feb 2022 10:49:29 +0000

You probably know that these days AWS advises you to create a separate account for every team/env/component/etc. I am certain you know that best practices and reality often differ. Maybe two teams need access to a shared component in one AWS account. Maybe the account still stems from the time AWS advised against too many accounts and refactoring to separate accounts is too cumbersome. Even if your account contains a single component managed by a single team, the component might encompass multiple services and you want to improve on defense in depth.

Whatever the reason, if you are in a situation where an AWS account is shared, it's very likely that everyone uses S3. Users may have data there that others are allowed to read, but not manipulate or even not touch at all. There are multiple ways to enforce rules like these and IAM seems like the go-to option in these cases. By the end of this post I hope you'll see things can be restricted more tightly than with IAM.

Best option: triple denial bucket policies

I am not going to drone about all the possible options before finally arriving at what I think works best. I have listed other viable options with their drawbacks below.

Bucket policies are your best option. Before you think "trivial answer", the exact statements are tricky and partly undocumented. Otherwise, I probably wouldn't have bothered to write this blog in the first place. The best way to start explaining the solution is by first showing the actual policy:

{
    "Version": "2012-10-17",
    "Id": "Triple-Denial-Policy",
    "Statement": [
        {
            "Sid": "AllowOnlySpecificPrincipals",
            "Effect": "Deny",
            "Principal": "*",
            "NotAction": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::a-team-specific-bucket/*",
                "arn:aws:s3:::a-team-specific-bucket"
            ],
            "Condition": {
                "StringNotLike": {
                    "aws:userid": [
                        "AIDAXXXXXXXXXX",
                        "AROAXXXXXXXXXX:*"
                    ]
                }
            }
        }
    ]
}

The meaning of a triple denial might be hard to grasp at first, so let me try to put into words what is happening. I think it works easiest if you read the policy from the bottom up: if you are not one of the users with a specified userid and you want to do something else with the specified resources than get/list, then you are not allowed.

Interestingly, the NotAction is specified in the IAM documentation, but not in the S3 documentation. But it works, go try it out :) Note that you still need to actually grant people read-only or more rights to this bucket and its objects, but there are already more than enough examples on how to do that, so I'll leave that out here. This statement only prevents others from ever manipulating that bucket and its data. Save the AWS root account of course, but I think that's fair.

A bit about the userid's. AWS actually has a nicely concise write-up about those. In short, they can prevent that reusing a particular name for an IAM entity will grant unwanted access. So keep in mind you have to update this policy each time the team's composition changes! If a team locks itself out, only the root account can save them.

Although it's not documented, I found that IAM roles (the userid's starting with AROA) need the :* appendix to work. (Maybe they are appended by a session key each time used or something?) Anyway, I haven't seen this behavior documented anywhere and it's there for both CodeBuild and CodePipeline. Maybe it's different for other services… At any rate, be warned that referencing to a userid alone in an S3 bucket policy might not be enough to actually grant access to roles. By the way, be careful with granting access to the bucket to too many users and roles, you may end up with the same issues I will be addressing below when discussing IAM.

Note, that the policy is easily convertible to fully deny access to all others than specified by changing the "NotAction":[...] to "Action": "s3:*". Finally, I haven't tried it, but a policy like this probably also works on other services with resource policies, such as KMS.

Bonus: Terraform example

So how to get this into your infrastructure-as-code. Suppose team-a wants a bucket with objects which only they and their pipeline can possibly manipulate and other teams are present in the account. Then this could be the way to go:

data "aws_iam_group" "team_a" {
  group_name = "team-a"
}

locals {
  team_a_and_pipeline = concat(
    data.aws_iam_group.team_a.users[*].user_id,
    ["${aws_iam_role.team_a_pipeline.unique_id}:*"]
  )
}

resource "aws_s3_bucket" "team_a_stuff" {
  bucket = "team-a-stuff"
}

data "aws_iam_policy_document" "team_a_stuff_bucket" {
  policy_id = "Triple-Denial-Policy"

  # No less than 3 denials in 1 statement so a little explanation:
  # This statement should prevent write operations by all users
  # except the ones specifically allowed
  statement {
    sid    = "AllowOnlySpecificPrincipals"
    effect = "Deny"
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    not_actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = [
      local.bucket_arn,
      "${local.bucket_arn}/*"
    ]
    condition {
      test     = "StringNotLike"
      variable = "aws:userid"
      values   = local.team_a_and_pipeline
    }
  }
}

resource "aws_s3_bucket_policy" "team_a_stuff" {
  bucket = aws_s3_bucket.team_a_stuff.id
  policy = data.aws_iam_policy_document.team_a_stuff_bucket.json
}

I would suggest leaving some comments near the triple-denial statement to keep the code more maintainable. Also, as already implied above when discussing the userid's, someone from team-a will have to run this code snippet when the team composition changes.

Other options (and their drawbacks)

IAM

You may rightly ask "Why all the complicated denials when a simple IAM deny on the specific resource would also suffice?" This is indeed possible, but in a shared account I would advise against it, because you are not really solving but rather shifting the problem. Who governs the IAM policies in a shared account? If you have a dedicated IAM team in that account you just created an extra burden for them. If IAM is a shared responsibility, then the IAM deny statement can be overwritten by anyone from the other team with sufficient IAM permissions, while the resource policy can only be overwritten by the accounts listed when it was created.

Furthermore, IAM usually suffers from privilege creep. Also, in shared accounts privilege escalation is always lurking because of all the roles spawned by multiple teams, which when used in the correct order can probably get you to any IAM rights you want. (Even when using triple denial bucket policies, you could still override someone else's password this way to gain access, but I think getting and using those rights unnoticed is the least likely form of privilege escalation in a well-designed AWS account.)

Lastly, you will have to enforce in some way that all users and roles spawned from unauthorized users also have this deny statement. Good luck. (It is possible to do that by the way, but the implementation is beyond the scope of this article and such a solution is not easier let alone clearer than a single resource policy.)

KMS

You could put one single simple statement in the bucket policy enforcing that all objects put to the bucket need to be encrypted with a specific KMS key, which only certain people have access to. This works, but again you are only shifting the problem here like with IAM. If you go for this strategy, you will have to protect the bucket policy and restrict usage of the key to some specific users… And so you are back at IAM problems.

Conclusion

If you want a bucket and its objects to be kept absolutely private to only some users/roles/root in an AWS account or at least keep others from write actions, then a tightly written bucket policy as shown above is your best option. IAM and KMS can perform the same task seemingly simpler, but keeping others from coming through the cracks will be a lot more complicated in the long run.

I wrote this blog and performed the work described within it at Simacan, my employer at the time of writing.