Forem: Jacob Martin

Introduction to Open Policy Agent (OPA) Rego Language

Jacob Martin — Fri, 16 Sep 2022 12:04:54 +0000

Policy as Code has been a very hot topic recently. It allows you to codify your rules and decision-making to execute them in an automated way. This lets products expose a programmatic, composable language for policies instead of having to build very complex purpose-specific UIs with all options available. One of the most – if not the most – popular Policy as Code engines is Open Policy Agent, used in many projects, like Kubernetes and Envoy, and also extensively used in Spacelift. It’s open source and uses the Rego language for policy authoring.

Rego, however, is a language that works very differently than most and can be quite unintuitive at first glance. It’s actually more similar to SQL than to common imperative languages like Python. This means that the learning curve can be quite steep. Moreover, copy-paste development will very often not help you understand Rego – and authoring complicated policies – better.

This is precisely why I wrote this article. It’s meant to guide you through some of the fundamental constructs and mechanisms of Rego, especially those that we’ve seen used a lot in the wild so that you can get a better intuitive understanding of how it all fits together and how to author larger, more advanced, policies. This is not aiming to be a complete (or even sizeable) reference, nor is it a Spacelift-specific guide. If you want to go along and play with the examples, the Open Policy Agent playground is the best place to do so.

Decisions

When talking about Rego, I think it’s best to start with decisions. Decisions in Rego are used as the output of policies but also as temporary variables all over the place. They don’t have to be true/false either – which is one of the common misconceptions! Decisions can be strings, arrays, objects, sets, etc. You can have as many decisions in your policy as you want.

For example, you can have a simple boolean decision with a constant,

allow = true

or reference a different variable.

allow = accessible_by_admin and is_admin

So basically normal variable (decision) assignment.

Rego also allows you to use block notation for assignments:

allow {
    accessible_by_admin := startswith(path, “/admin/”)
    accessible_by_admin
    is_admin
}

where each line can be an assignment or true/false expression. The result of the whole block will be true only if the expression in each line evaluates to true, letting you build complex rules that use many other decisions and variables inside of them. We can check a bunch of predicates, and only if all checks succeed does the decision come out to be true.

You can also have a decision that’s an OR of two other decisions, like here:

is_admin {
    input.user.admin
}

is_endpoint_public {
    startswith(path, “/public/”)
}

allow {
    is_admin
}
allow {
    is_endpoint_public
}

As you can see, a block for a decision can be repeated multiple times, and the decision will be true if any of the blocks evaluate to true.

Some

The above block is very simple and linear, but we can also do something slightly more complicated. Let’s say we have a request path and an array allowed_path_prefixes and want to check if the path matches any prefix. In that case, we can specify an additional variable for the index:

allow {
    some i
    startswith(request.path, allowed_path_prefixes[i])
}

Now, the way this will intuitively work is that rego will choose an i, move to the second line, and if it fails, go back, try another i and so on until it finds an i that leads to all lines being true. If no such i is found, then allow will be null (technically speaking, it will be undefined).

In human terms, you can read this as “set allow to true if, for some value of i, the request path starts with the i’th allowed path prefix,” or, in even more human terms, “set allow to true if the request path starts with any of the allowed path prefixes.”

If you have more such variables, you can add them separated by commas next to the i:

some i, j, k, l, m, n, o, p

Additionally, if this i is only needed in a single place, like in the above example, you can substitute it with an underscore instead:

allow {
    startswith(request.path, allowed_path_prefixes[_])
}

We can also take a look at an example that’s closer to Spacelift, where we might want to decide if a commit is worth a Terraform execution or whether it should just be ignored. We get a list of files changed by the commit, and we also have a couple of paths that are of interest to Terraform. Here we’ll check if any of the paths changed starts with one of the interesting paths:

interesting_path_prefixes := [
  “/src/terraform”,
  “/src/modules”
]
track {
    startswith(input.affected_files[_], interesting_path_prefixes[_])
}

We’re using two underscores in a single line to mean “is there any pair of (affected file, interesting path prefix) such that the file starts with the prefix.”

But wait, there’s no allow rule here? Is that a valid policy? Yes, it is! Policies can have arbitrary sets of decisions, with those decision being of arbitrary types. It’s just that allow based policies are one of the most obvious use cases, but the power of the Rego language extends much further and can be used for all kinds of decisions, as exemplified by the rich selection of policies available in Spacelift.

Sets

Decisions can also be specified as sets:

allowed_users := [“papaya”, “potato”]
allow[“papaya”] {
    “papaya” == allowed_users[_]
}

which means “Papaya should be in the allow set if it belongs to the allowed_users list.”

Moreover, with this block notation, you can specify the element as a variable reference from the block itself:

allow[user] {
    user := input.user
    user == allowed_users[_]
}

Using just a single block, you can even specify multiple elements of the set, by having multiple evaluation paths that successfully reach the end of the block and each path having a different user variable.

allow[user] {
    user := input.users[_]
    user == allowed_users[_]
}

The value in the square brackets can actually be an arbitrary expression referencing the block’s variables, so if we have a policy whose decisions are warnings based on resources changed, we could do the following:

forbidden := {“expensive_resource”, “expensive_resource2”}
warn[sprintf(“You shall not use %s”, resource_name)] {
    resource_name := input.resources_changed[_].name
    forbidden[resource_name]
}

which checks for forbidden resources and displays a pretty warning message if one of them is changed. In this example, you can also see the usage of a set containment check in the last line of the block.

Functions

A more advanced feature of rego is that it lets you define custom functions you can use as helpers in your policy. Writing functions is very similar to writing block decisions, but with some minor differences:

plus_custom(a, b) := c {
    c := a + b
}
out := plus_custom(42, 43)

You can see we’re specifying a list of arguments, the variable that should be used as the output, and then have a normal block body. The output of the function will be c as long as the function successfully reaches the end of its body.

However, instead of that output variable, we could also, again, have an arbitrary expression, a constant, for instance:

bucket_is_secure(bucket) := true {
    not bucket.public
    bucket.encrypted
}
bucket := {"public": false, "encrypted": false}
out := bucket_is_secure(bucket)

In this case, the function will evaluate to true if the bucket is not public and is encrypted.

Summary

All of this has just been a quick overview of the parts of Rego we’ve seen most commonly used. With these building blocks, you should be much better equipped to begin authoring your own policies, whichever project you’re using them with. If you want to learn more, the whole Rego policy language is much bigger, and the best place to dive in is the Official Open Policy Agent Policy Language documentation.

If you think Rego is cool and would like to play with a product where it’s put front-and-center, we’d love you to take Spacelift for a spin!

Adventures in AWS Lambda Land – a Migration Gone Well

Jacob Martin — Tue, 14 Jun 2022 11:40:48 +0000

As many others did before and regretted later, we’ve decided to move a big part of our workload from long-running processes to Lambda. That said, we haven’t regretted it a bit. It’s been a breeze so far, even if there are a few shortcomings of Lambda, which are just annoying.

In this blog post, I’ll describe the positive sides, the negatives, and the approach we’ve taken that allowed us to radically simplify our setup, as well as clarify common misconceptions you might have about AWS Lambda development.

But first of all, let’s take a look at…

Our Architecture

Our backend is based on two main constructs:

The server, which exposes GraphQL and REST APIs to the outside world.
The drain, which consumes a bunch of SQS queues to handle asynchronous jobs. Each SQS queue transports several event types, based on a logical separation (webhooks, async jobs, cronjobs, etc.)

We have a very asynchronous architecture. The server often emits async jobs instead of doing something by itself, and the drain often emits new events when handling existing ones.

Because we’re a CI/CD system, our traffic patterns are very unevenly distributed. Sometimes there are few jobs, at other times, somebody creates an avalanche of them.

Until recently, the drain was a long-running process on ECS, reading from multiple SQS queues. Autoscaling in such a situation is fairly hard. The simplest option is to have a separate process per SQS queue, with each process scaled independently. Even doing this would have resulted in paying too much for periods of low traffic, along with handling spikes in traffic too slowly. The diagram below describes that architecture visually:

Also, everything on the backend is written in Go, no exceptions.

Changing Our Codebase for Lambda

Many people think that moving to Lambda requires huge codebase changes and has to result in having a lot of binaries, one for each Lambda Function. That’s actually not necessary. Yes, Lambda does only call the binary without any arguments (and there’s no way to change that to my knowledge – I tried), but you can provide arbitrary environment variables.

In our case, we have a single spacelift binary that will run the server process if called as spacelift backend server, but if called as plain spacelift, then it assumes it’s being run as a Lambda handler and will take a look into the HANDLER environment variable to deduce which handler to actually run. This way, we can keep using a single binary for everything, and operational simplicity ensues. The following diagram shows the new setup:

Local Development

Lambda can also be quite annoying for local development. The way we solve this is by having all of our handlers – as you’ve seen above – fronted by SQS queues. This way we can run the handlers locally, and have a shim that polls the SQS queue (a copy the given developer has), translates it into the Lambda-native SQS message format, and passes that to the handler. Very similar to the way the original setup worked, but just for local development.

In practice, this means we have a spacelift backend local-drain command that just runs all handlers with their respective SQS queue pollers. That gets recompiled on each code change, so the feedback loop from code change to locally deployed new binary is around 5 seconds and you can easily attach a debugger to it.

Handling Secrets

On ECS, the system we moved from, you can just specify a list of environment variables and mark their sources as “to be taken from SecretsManager”. With Lambda, that’s not possible. However, we wanted to keep dependency injection uniform across the codebase, so we wanted to have the same flow in Lambda as before.

Thus, we just pass a plain environment variable with the list of objects to get from SecretsManager. The Lambda binary will get those objects from SecretsManager, iterate over all key values in each, and set them as environment variables on the current process. This happens first-thing after starting, so the rest of our initialization process is oblivious to it. This might sound like a hack, but it’s simple, and it does the job.

for _, secret := range strings.Split(os.Getenv("SECRETS_MANAGER_ENVIRONMENT_VARIABLE_KEYS"), ",") {
  value, err := sm.GetSecretValue(&secretsmanager.GetSecretValueInput{
     SecretId: aws.String(secret),
  })
  if err != nil {
     log.Fatal(err)
  }
  var data []byte
  if value.SecretBinary != nil {
     data = value.SecretBinary
  } else {
     data = []byte(*value.SecretString)
  }
  var pairs map[string]string
  if err := json.Unmarshal(data, &pairs); err != nil {
     log.Fatal(err)
  }
  for k, v := range pairs {
     if err := os.Setenv(k, v); err != nil {
        log.Fatal(err)
     }
  }
}

Container Images vs. Zip Archives

Another misconception about Lambda is that Container Image support means it can take arbitrary Docker images and run them as-is. Unfortunately, that’s not true.

Container Images in Lambda basically work as a glorified file archive, nothing more, nothing less. Your binary still has to adhere to the Lambda communication protocol, Although you can still use Docker to build your image. The main difference is that thanks to the improved caching story, they let you use 10 GB images instead of just 50 MB for ZIP archives.

Another difference is that with Container Images you can’t configure Lambda layers to be used through the AWS console, as you can with ZIP archives. That said, you can easily copy layers into your Container Image:

COPY --from=public.ecr.aws/datadog/lambda-extension:21 /opt/extensions/ /opt/extensions

Function Lifecycle and Asynchronous Goroutines

The way your Lambda Function works is that it gets spun up, launches your binary, which can do any setup it wants and then starts polling for jobs from the Lambda runtime. Then, when the Lambda is not needed anymore (scaling down) it gets terminated.

So, you can use goroutines to do some async tasks in your Lambdas while they’re running, right? Well, no.

The Lambda Runtime will actually freeze your process between invocations. This means that if you have an async process doing work between invocations, it might be frozen right after it tries to do an IO operation and be woken up 30 seconds later, which makes the IO operation time out immediately.

In other words, you can’t safely use async processes in your Lambda binaries – at least not outside a single invocation’s timespan. Also, make sure that your database doesn’t terminate connections if it didn’t get a keepalive in 30 seconds, in case you’re connecting to it directly (which btw. was not an issue for us).

The Lambda SQS Connector

It’s not ideal.

With Lambdas, there is a concept of throttling. It happens when you try to invoke a lot of Lambdas at once, but they can’t scale fast enough. Lambda will then return a 429 error.

So, if you suddenly have lots of SQS messages, the SQS connector will scale up and send lots of invocations which might fail with the 429 error. So, the SQS connector will slow down and retry them in a second, right?

Well, no. Instead of retrying with those most recent messages a moment later, it will actually fail the message handling and put it back to SQS to be handled again after the visibility timeout passes (which might take several minutes). That is not great and can introduce additional delays in times of high traffic.

That said, we haven’t encountered this issue again since setting up high reserved concurrent execution numbers on the Lambda functions. Based on the last few months of usage, this has been a non-issue to us.

Resetting a Lambda

With most systems, often “turn it off and back on again” is a quite good incident recovery mechanism. Your code gets into an unexpected state, and this way you reset it.

With Lambda, however, there’s no obvious way to invalidate all the current warm Lambda runtimes. So, if you need this, the way to go is actually to add a useless environment variable, so that you have a change, and then deploy it. Deploying will invalidate old Lambda runtime versions, and effectively “turn it off and back on again”.

Cost and Scalability

As described in the beginning, we often encounter load spikes, which required us to have many machines on standby before moving to Lambda. With Lambda, this works beautifully out of the box. Whenever we get a huge amount of events at a moment’s notice, Lambda is able to scale up to 10x the capacity, quickly handle all the events, and then scale down.

Moreover, we achieve all of this while having drastically reduced the cost of the whole setup. Lambdas for asynchronous event processing are very cost-effective. I’m not sure I’d recommend them for synchronous HTTP services, but here they definitely do shine.

Operational Simplicity

Lambdas deploy instantaneously. They have perfect isolation – we don’t have to worry about one Function’s execution influencing any other Function, by starving resources or something similar. Cold starts are very short with Go, warm starts are basically instant. So as far as these considerations are concerned, we’re very happy.

After migrating to Lambdas, they required very little upkeep, and haven’t been a source of incidents.

Summary

Overall, we’re very satisfied with our new setup. I hope the above lessons learned can be useful to somebody else considering the same switch, or planning their greenfield project on Lambda.

Tricking Postgres into using an insane – but 200x faster – query plan

Jacob Martin — Mon, 07 Feb 2022 14:25:21 +0000

At Spacelift, we use PostgreSQL (specifically Aurora Serverless) for most of our primary database needs. Our application query patterns are mostly small transactions that touch a few rows – a typical scenario that PostgreSQL handles without breaking a sweat.

We also use Datadog as a monitoring solution. Their agent has a handy feature to create metrics based on periodically executed SQL queries, allowing us to build smart and highly contextual monitors and alerts. Occasionally, such queries might be slow and dominate the processing time of the database. After hooking up Datadog Database monitoring, we identified a lot of slow queries and moved most of them to a separate Redshift cluster. There they execute once per day on a periodic snapshot of the database.

However, a number of these queries need to happen at a more frequent interval as they can detect alertable conditions. We couldn’t move these queries to Redshift because of the staleness of the data there. Aurora Serverless also doesn’t support read replicas to offload analytical queries to them, so no luck there either. Thus, we had to optimize, add indices, optimize, and again optimize until we cut the database usage of these metric-creating queries by 1-2 orders of magnitude.

This article will show you one such optimization story, which is interesting because both the before and after query plans will look somewhat ridiculous (in terms of cost). Moreover, it shows how PostgreSQL row count estimation can go very wrong. In this case, domain knowledge will help us trick PostgreSQL into a different query plan which will be waaay faster, without adding any additional indices.

Concepts

Spacelift is a CI/CD platform specializing in Infrastructure as Code. Users create Runs (executions of, i.e., Terraform) which will be executed on Worker Pools. There is a public shared Worker Pool managed by Spacelift and private Worker Pools that users can host for themselves.

Each Run belongs to a Stack, which can be thought of as a single environment, managing a bunch of infrastructure resources. If you know Terraform or CloudFormation, a single Stack maps to a single Terraform state file or a CloudFormation root stack, respectively.

A Run can have one of many types: Tracked, Proposed, and others. That type determines what the overall workflow will look like.

The Query

The query is supposed to notify us when the public Worker Pool cannot handle demand – user Runs are waiting too long to be handled by a Worker.

SELECT COUNT(*)                                                                  as "count",
       COALESCE(MAX(EXTRACT(EPOCH FROM age(now(), runs.created_at)))::bigint, 0) AS "max_age"
FROM runs
  JOIN stacks ON runs.stack_id = stacks.id
  JOIN worker_pools ON worker_pools.id = stacks.worker_pool_id
  JOIN accounts ON stacks.account_id = accounts.id
WHERE worker_pools.is_public = true
  AND runs.type IN (1, 4)
  AND runs.state = 1
  AND runs.worker_id IS NULL
  AND accounts.max_public_parallelism / 2 > (SELECT COUNT(*)
                                             FROM accounts accounts_other
                                               JOIN stacks stacks_other ON accounts_other.id = stacks_other.account_id
                                               JOIN runs runs_other ON stacks_other.id = runs_other.stack_id
                                             WHERE accounts_other.id = accounts.id
                                               AND (stacks_other.worker_pool_id IS NULL OR
                                                    stacks_other.worker_pool_id = worker_pools.id)
                                               AND runs_other.worker_id IS NOT NULL)

That’s a big query! What’s happening here?

We’re interested in the number of Runs (as well as the age of the oldest one) which are of the Proposed or Testing type (can be scheduled right away – other Run types require exclusiveness), are in the Queued state (which means they’re not yet being processed) and don’t have a Worker attached to them (are still waiting for one).

We only want Runs on Accounts that are far from their parallelism limit to reduce the number of false positives caused by users being correctly limited. So for each Run, we count the number of active Runs on its account and check if it’s under 50% of the parallelism limit, and only then do we count this Run as properly pending.

Unfortunately, that query lasts 20 seconds and consumes 50% of our database’s processing time. That’s pretty bad.

Let’s optimize!

Let’s take a look at the query plan. I’ll be using https://explain.dalibo.com to visualize query plans. To get the query plan, I’m executing it prefixed with EXPLAIN (FORMAT JSON, ANALYZE) … so we can also see information about the query’s actual execution (ANALYZE).

What can we see here?

On the left side, we’re joining Worker Pools with Stacks. We also calculate the number of active Runs for each Account using the SubPlan. We do a three-way hash join on the (Worker Pool, Stack) tuples, the Accounts, and the SubPlan results based on the Account ID. Then we loop over all those triples and get the pending Runs for each, finally counting all the pending Runs found.

The SubPlan goes over all Stacks in an Account, and for each Stack gets the active Runs for it (the ones where worker_id is not NULL).

Let’s look at the time distribution of the plan nodes.

We can see that the most expensive part of this query is the Bitmap Index Scan which looks for Runs whose worker_id is not NULL, so let’s take a closer look at that part of the query plan.

For each Stack in the SubPlan, we look for active Runs on it. How do we do that?

We scan the runs_worker_id index for storage pages containing Runs with a non-empty worker_id and the run_stack_id index for the pages containing Runs for the current Stack. Then we AND these together – we only want pages present in both of these scans. Finally, we scan the actual Runs from the pages and only take the ones that satisfy the predicates (pages found by the index may contain both relevant and irrelevant Runs).

In a different tab of the left Index Scan, we can see the root of the problem. We made 531358 loops over this, and we expected to scan 120 mil Runs over the whole processing of the query here, but we scanned 800 mils. That’s much more.

But later in the Heap Scan of the _AND_ed pages, we see a gross overestimation

We expected to scan 2 mil rows but scanned 42 overall, which means no Runs were executing on most Stacks at this instant.

Overall, it’s mostly this subquery being very expensive:

SELECT COUNT(*)
FROM accounts accounts_other
  JOIN stacks stacks_other ON accounts_other.id = stacks_other.account_id
  JOIN runs runs_other ON stacks_other.id = runs_other.stack_id
WHERE accounts_other.id = accounts.id
  AND (stacks_other.worker_pool_id IS NULL OR
       stacks_other.worker_pool_id = worker_pools.id)
  AND runs_other.worker_id IS NOT NULL

Now comes the domain knowledge. Is there something we know the optimizer seems not to…? Yes!

Only a minuscule part of the Runs in the database are active at any given time. Most Stacks stay dormant most of the time. Whenever a user makes a commit, only the affected Stacks will execute Runs. There will obviously be Stacks that are constantly in use, but most won’t. Moreover, intensive users will usually use private Worker Pools. The public shared Worker Pool is predominantly utilized by smaller users, so that’s another reason for not seeing many Runs here.

Right now, we’re iterating over all existing Stacks attached to the public worker pool and getting relevant Runs for each. How about we flipped that? We know there are at most as many active Runs as there are public Workers – orders of magnitude fewer than there are Stacks. Overall active Runs (not just the ones on the public Worker Pool) are also way less numerous, at most as many as there are Workers overall (and since the number of private Workers is the main dimension we currently base our Enterprise tier pricing on, query processing time being linearly correlated with the number of Workers is OK). Thus, we could get all active Runs and filter to just those whose Stack uses the public Worker Pool.

The only problem is, we’re working with a fairly opaque query optimizer, and PostgreSQL doesn’t have any way to give hints to it – like forcing a join strategy. So we’ll have to do that differently. We’ll create a query that scans the active Runs and then uses another subquery to filter them to just the ones related to a relevant Stack. Then we’ll have to hope the query is opaque enough for Postgres not to get too clever with it.

SELECT COUNT(*)
FROM runs runs_other
WHERE (SELECT COUNT(*)
      FROM stacks
      WHERE stacks.id = runs_other.stack_id
        AND stacks.account_id = accounts.id
        AND stacks.worker_pool_id = worker_pools.id) > 0
 AND runs_other.worker_id IS NOT NULL

This query does exactly what we described. Scan active Runs, filter to just those with more than 0 Stacks in the current account and connected to the public Worker Pool. There would be either 1 or 0 such Stacks for each Run.

Executing this query, we get down to 100ms instead of the original 20s. That’s a vast improvement. What does the query plan say? (apologies for the long vertical picture)

Exactly what we wanted! It scans active Runs, and for each of them, checks (using a primary key index lookup – fast!) whether the Stack it belongs to is relevant.

That’s it! The whole optimized query now looks like this:

SELECT COUNT(*)                                                                  as "count",
       COALESCE(MAX(EXTRACT(EPOCH FROM age(now(), runs.created_at)))::bigint, 0) AS "max_age"
FROM runs
  JOIN stacks ON runs.stack_id = stacks.id
  JOIN worker_pools ON worker_pools.id = stacks.worker_pool_id
  JOIN accounts ON stacks.account_id = accounts.id
WHERE worker_pools.is_public = true
  AND runs.type IN (1, 4)
  AND runs.state = 1
  AND runs.worker_id IS NULL
  AND accounts.max_public_parallelism / 2 > (SELECT COUNT(*)
                                             FROM runs runs_other
                                             WHERE (SELECT COUNT(*)
                                                    FROM stacks
                                                    WHERE stacks.id = runs_other.stack_id
                                                      AND stacks.account_id = accounts.id
                                                      AND stacks.worker_pool_id = worker_pools.id) > 0
                                               AND runs_other.worker_id IS NOT NULL)

Summary

Thanks for reading! The moral of this story is that query plans aren’t that hard to interpret with the right tools. Using your domain knowledge, you can rewrite your queries to achieve substantial performance boosts without adding unnecessary indices or denormalizing the data model.

(The original post was published at Spacelift)

How to Rename an AWS S3 Bucket in Terraform

Jacob Martin — Tue, 16 Nov 2021 23:00:05 +0000

Occasionally you might want to rename an AWS S3 bucket you are managing with Terraform. However, names of S3 buckets are immutable, which means you can’t change them directly. If you tried, Terraform would destroy the old one and then create a new one, resulting in data loss.

To avoid this, you need to create a new bucket with the desired name, move the data over to it, make the relevant Terraform state replacements, and finally delete the old bucket.

In this post you will learn how to rename an AWS S3 bucket in Terraform. First things first – let’s say you have a bucket definition in your Terraform code:

resource “aws_s3_bucket” “my_bucket” {
    bucket = “old-name”
}

and you want to change the name of the bucket to new-name.

Step 1 - Create the new bucket

Firstly, we’ll need to create a new bucket. You can do this using the AWS CLI or the AWS console. Just make sure to properly replicate the old settings, especially the ACL (so that your data doesn’t accidentally become public).

Now, we can copy all the files from the old to the new bucket:

aws s3 sync s3://old-name s3://new-name

Step 2 - Modify the State

Now that we have our new bucket, we need to remove the old one from our Terraform state and import the new one in its place:

terraform state rm aws_s3_bucket.my_bucket
terraform import aws_s3_bucket.my_bucket new-name

If you tried to run Terraform now, it would show you that there’s drift—and yes, there would be! We’ve just imported a bucket into a resource which still has old-name in the config.

Step 3 - Change the Code

That’s why we now have to finally change the name of the bucket in our Terraform config:

resource “aws_s3_bucket” “my_bucket” {
    bucket = “new-name”
}

If you run Terraform now, you’ll see that there are no changes to be made.

Step 4 - Do a Cleanup

If you want to, you can now delete your old bucket using:

aws s3 rm s3://old-name --recursive
aws s3 rb s3://old-name

Make sure all the data has successfully been copied over to the new bucket.

Step 5 - Run the Above Arbitrary AWS CLI Commands if You’re Using Spacelift

If you’re using Spacelift then you can use tasks to run the above arbitrary AWS CLI commands. The default runner image already contains the AWS CLI, so no changes are necessary there.

One additional thing you can do is lock the Stack while you’re running the migration, this way nobody will accidentally run Terraform or make any other changes while you’re moving the data.

And that’s it! If you have any questions about how to rename an AWS S3 bucket, drop me a line in the comments and I’ll get back to you straight away.

You will find more Terraform Tutorials on our website:

Creating a flexible Backoffice Tool in a Technical Company using Slack

Jacob Martin — Mon, 01 Nov 2021 22:13:29 +0000

Working on a SaaS product you’ll occasionally have to take administrative actions to assist a user or help them debug an issue. Maybe something is misconfigured and they need a hand, or possibly they just want another week in their trial. If you’re using something like Django, then you get a nice built-in admin UI based on your model definition, and that’s it – you have a great tool at your disposal by default. Even though you still need to carefully design the access flow, as they otherwise commonly become attack vectors.

If however you’re building a project in a custom way akin to “let’s combine many libraries and build exactly what we need” instead, then all you’ll have is the raw query language interface of your database – and that only if you’re lucky enough to have chosen a database which supports non-trivial querying.

In the long run though, as most developers know, running SQL queries on prod isn’t really a scalable or secure approach and requires significant coordination, communication and supervision in order not to accidentally corrupt your data. You also have to limit access to a few select individuals and have to manage access to sensitive data.

Moreover, you might have other dependencies than the SQL database. Then administering all the third-party services would require you to jump around various admin UIs, making everything even more complicated and error prone…

The Obvious™ Solution

There’s a whole class of tools tailored specifically to solving this use case – WYSIWYG back office tools. Using one of these tools you can easily create dialogs with SQL queries underneath and connect them directly to your database. Depending on the tool, it might even be usable by non-technical people.

Unfortunately, these tools also bring a whole class of problems with them, mainly access management – both for people, and for the tool itself. You can use a cloud-based, managed tool, but then you have to give it arbitrary querying access to your internal production database. This usually makes sense, but is a security consideration that you have to take into account. You can also use self-hosted open source tools, but these come with their own set of trade-offs:

You have to self-host them, costing you precious time.
They need to have access to your production database, and people in your organization need to have access to them. So you either make them publicly accessible – and trust their security – or you make them internally accessible, and set up limited internal network access for everybody who needs to use them.
Access-management in the tool itself.

Other than that, one common disadvantage is that they’re yet another tool you’ll have to use in your day-to-day work. They are also usually optimized for working with a SQL database, not additional third-party services.

After comparing various solutions on both sides of the cloud/open-source divide, we’ve decided we didn’t like either set of trade-offs, which put a hold on the whole initiative for a while.

Then we had an idea… What’s the most popular DevOps administration and management tool? Slack of course! I’m slightly joking, but the ChatOps trend’s significance cannot be denied. In this case, it did indeed look like it could solve all our problems.

Why Slack?

Slack is great for a few reasons:

It’s a textual interface – almost a terminal really – so you can transfer a lot of UX knowledge from your terminal experiences. Developers are always happy to use a text interface.
You already manage access to Slack channels, so you can piggyback on top of that and only make your tool available in select channels. This way you don’t have to build additional access management for your tooling.
If you design the UX well, it’s very easy to use for non-technical users.
You automatically have a public audit log of all executed commands – the channel message history.
You already use Slack.

It does have its disadvantages too:

When Slack is dead (which hasn’t been a rare occurrence in recent times), you can’t access your backoffice tool. But that’s ok, as long as you keep your most critical maintenance commands available through alternative access channels.

Now for some Slack technicalities. For a project like this you can use slash commands or a slackbot. The gist of it is that slash commands are more structured, but also more limited. Slackbots on the other hand just interact with your channels like a normal user would, so the flexibility is limitless, but it requires more work on your side. In order to provide an experience that’s as user-friendly and magical as possible, we went with the slackbot approach.

And thus the Backoffice Bot was born…

Building the Slackbot

Internally, we already had an event handling system in place for application events – i.e. GitHub push notifications. We actually even have a Slack integration, so there was ample opportunity for copy-paste driven development.

In practice though, it’s pretty simple. There’s an AWS API Gateway endpoint to handle Slack webhooks, it puts events on an SQS queue and then a handler which has access to all relevant production systems takes care of handling each message and potentially responding to it.

One more advantage of the slackbot using our existing event handling framework, is that we not only have access to our SQL database, we also have access to any third-party products we use, like various AWS offerings.

The Router Structure

All commands are registered with a message pattern, which is then converted into a regular expression with capture groups for the arguments. Here is an example code block specifying a command with its handler:

{
  Command:     "set <subdomain> trial remaining to <number> days",
  Description: "Set account to be on an Enterprise trial for a specified number of days from now.",
  Channels:    []string{s.BackofficeChannelID},
  Handler: func(ctx *Context, event slackevents.EventsAPIInnerEvent, params map[string]string) error {
     ctx.sendSimpleResponse("Setting %s trial remaining to %s days.", params[“subdomain”], params[“number”])
     // ...
  },
},

Initially we planned to make it a sophisticated trie-based router, but that’s just immensely more complex than going over a list of regular expressions and trying to match each. With this amount of traffic and number of commands there was simply no point in optimizing further.

This way of adding commands is really simple and there’s a bonus – an automatically generated @Backoffice Bot help message, which dynamically lists and describes all commands available in the current channel.

Approval Flow

With processes like these, oftentime you want certain commands to require approval. We do this by using Slack reactions.

The Slackbot will first detail what it would like to do (dry-run so to say), and then ask for approval. Approval is required from the caller, as well as a predefined number of other people, which depends on the “destructiveness” of the command at hand.

In order not to double-execute commands, we’re marking handled messages using a Slack reaction – this way we don’t have to use an external database. You can see this in the above picture with the Spacelift logo reaction on the root message.

Access Control

One additional requirement was that different commands are allowed to be executed by different people. In our case there are commands which are built for non-technical people, for example extending account trials; as well as commands built for technical people, for example displaying diagnostic information or cycling a worker pool’s workers.

We achieve this by limiting the available commands based on the channel. We have a #backoffice channel for non-technical people and a #backoffice-developers channel for technical people. They are private, so we can limit access levels by inviting select people to relevant channels.

The help message takes this into account, and only shows the commands available to you in the current channel.

How does it work in practice

Everybody using our new Backoffice Bot has quickly fallen in love with it. Adding new commands follows the same process as getting standard product changes into production, which is familiar to all developers out of the box. There are no additional tools, workflows, or processes to maintain. We definitely recommend this approach if you ever find yourself having similar challenges to solve.

(The original post was published at Spacelift)

Terraform Data Sources – How They Are Utilised

Jacob Martin — Mon, 25 Oct 2021 16:50:20 +0000

What are Terraform data sources?

Data sources in Terraform are used to get information about resources external to Terraform, and use them to set up your Terraform resources. For example, a list of IP addresses a cloud provider exposes.

Example usage

We can find excellent examples of data source usage in the AWS provider docs:

variable "vpc_id" {}

data "aws_vpc" "selected" {
  id = var.vpc_id
}

resource "aws_subnet" "example" {
  vpc_id            = data.aws_vpc.selected.id
  availability_zone = "us-west-2a"
  cidr_block        = cidrsubnet(data.aws_vpc.selected.cidr_block, 4, 1)
}

In this scenario, we may have created our aws_vpc manually, because it also contains other manually created resources, and now we want to start adding Terraform managed resources to it.

In order to do this, we use the aws_vpc data source to find the manually created aws_vpc, and then use its properties to configure our aws_subnet.

Another fancy use case would be to use the GitHub provider Pull Requests data source to list all Pull Requests, and then provision an on-demand preview environment for each Pull Request. It would go along the lines of:

data "github_repository_pull_requests" "pull_requests" {
  base_repository = "example-repository"
  base_ref        = "main"
  state           = "open"
}

module “preview-environment” {
  for_each        = data.github_repository_pull_requests.pull_requests.results
  name            = each.value.title
  commit_sha      = each.value.head_sha
  // ...
}

Refreshing data sources

By default, Terraform will refresh all data sources before creating a plan. You can also explicitly refresh all data sources by running terraform refresh.

Occasionally you’ll have data sources that change very often and would like to keep your resources in sync with those changes. The easiest way to achieve this is to just run Terraform every few minutes or so, do a refresh, and apply all resulting changes.

You could also use something akin to Spacelift’s Drift Detection to automate this process and make sure it doesn’t interfere with your manual Terraform executions.

Check more Terraform Tutorials:

How to Provision an AWS EKS Kubernetes Cluster with Terraform

Jacob Martin — Wed, 20 Oct 2021 17:49:26 +0000

In this guide, you will learn how to provision an AWS EKS Kubernetes cluster with Terraform. Let’s start with the basics.

What is AWS EKS?

AWS EKS provides managed Kubernetes clusters as a service. You’re on AWS and want to avoid getting into the details of setting up a Kubernetes cluster from scratch? EKS is the way to go!

Before we get started

You’ll need to have Terraform installed locally:

brew install terraform

as well as the AWS CLI:

brew install awscli

as well as kubectl:

brew install kubernetes-cli

If you’re on a different operating system, please find the respective installation instructions here:

Terraform: https://learn.hashicorp.com/tutorials/terraform/install-cli
awscli: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
kubectl: https://kubernetes.io/docs/tasks/tools/

Step 1 - Configuring the AWS CLI

You’ll need to configure your AWS CLI with access credentials to your AWS account. You can do this by running:

aws configure

and providing your Access Key ID and Secret Access Key. You will also need to provide the region. For the purposes of this guide we will use us-east-2. Terraform will later use these credentials to provision your AWS resources.

Step 2 - Getting the code

You can now clone a repository which contains everything you need to set up EKS:

git clone https://github.com/hashicorp/learn-terraform-provision-eks-cluster/

Inside you’ll see a few files, the main one being eks-cluster.tf:

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = local.cluster_name
  cluster_version = "1.20"
  subnets         = module.vpc.private_subnets

  tags = {
    Environment = "training"
    GithubRepo  = "terraform-aws-eks"
    GithubOrg   = "terraform-aws-modules"
  }

  vpc_id = module.vpc.vpc_id

  workers_group_defaults = {
    root_volume_type = "gp2"
  }

  worker_groups = [
    {
      name                          = "worker-group-1"
      instance_type                 = "t2.small"
      additional_userdata           = "echo foo bar"
      asg_desired_capacity          = 2
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
    },
    {
      name                          = "worker-group-2"
      instance_type                 = "t2.medium"
      additional_userdata           = "echo foo bar"
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id]
      asg_desired_capacity          = 1
    },
  ]
}

It uses the EKS Terraform module to set up an EKS cluster with 2 worker groups (the actual nodes running your workloads): one with a single medium machine, and one with two small machines.

Step 3 - Running Terraform

You can now create all of those resources using Terraform. First, run:

terraform init -upgrade

to initialize the Terraform workspace and download any modules and providers which are used.

In order to do a dry run of the changes to be made, run:

terraform plan -out terraform.plan

This will show you that 51 resources will be added, as well as the relevant details of them. You can then run terraform apply with the resulting plan, in order to actually provision the resources:

terraform apply terraform.plan

This might take a few minutes to finish. You might get a “timed out” error. In that case, just repeat both the terraform plan and terraform apply steps.

In the end you will get a list of outputs with their respective values printed out. Make note of your cluster_name.

Step 4 - Connecting with kubectl

In order to use kubectl, which is the main tool to interact with a Kubernetes cluster, you have to give it credentials to your EKS Kubernetes cluster. You can do that by running:

aws eks --region us-east-2 update-kubeconfig --name <output.cluster_name>

Just make sure to replace with the relevant value from your Terraform apply outputs.

Step 5 - Interacting with your cluster

You can now see the nodes of your cluster by running:

> kubectl get nodes -o custom-columns=Name:.metadata.name,nCPU:.status.capacity.cpu,Memory:.status.capacity.memory
Name                                       nCPU   Memory
ip-10-0-1-23.us-east-2.compute.internal    2      4026680Ki
ip-10-0-2-8.us-east-2.compute.internal     1      2031268Ki
ip-10-0-3-128.us-east-2.compute.internal   1      2031268Ki

This command is so long because it displays custom columns, and thanks to those we can indeed see that there are 2 smaller nodes, and 1 bigger node.

Let’s deploy an Nginx instance to see if the cluster is working correctly by running:

kubectl run --port 80 --image nginx nginx

You can see the status of it by running:

> kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          2m46s

And finally set up a tunnel from your computer to this pod by running:

kubectl port-forward nginx 3000:80

If you open http://localhost:3000 in your browser, you should see the web server greet you:

Step 6 - Cleaning up

In order to destroy the resources we’ve created in this session, you can run:

terraform destroy

This may again take up to a few minutes.

Conclusion

I hope this guide helped you on your Kubernetes journey on AWS! If you want more help managing your Terraform state file, building more complex workflows based on Terraform, and managing AWS credentials per run, instead of using a static pair on your local machine, check out Spacelift. We’d love to have you!

You will find more Terraform Tutorials on our website:

Practical Golang: Building a simple, distributed one-value database with Hashicorp Serf

Jacob Martin — Fri, 03 Feb 2017 17:44:15 +0000

This is a cross-post from my blog at jacobmartins.com

Introduction

With the advent of distributed applications, we see new storage solutions emerging constantly.
They include, but are not limited to, Cassandra, Redis, CockroachDB, Consul or RethinkDB.
Most of you probably use one, or more, of them.

They seem to be really complex systems, because they actually are. This can't be denied.
But it's pretty easy to write a simple, one value database, featuring high availability.
You probably wouldn't use anything near this in production, but it should be a fruitful learning experience for you nevertheless.
If you're interested, read on!

Dependencies

You'll need to

go get github.com/hashicorp/serf/serf

as a key dependency.

We'll also use those for convenience's sake:

"github.com/gorilla/mux"
"github.com/pkg/errors"
"golang.org/x/sync/errgroup"

Small overview

What will we build? We'll build a one-value clustered database. Which means, numerous instances of our application will be able to work together.
You'll be able to set or get the value using a REST interface. The value will then shortly be spread across the cluster using the Gossip protocol.
Which means, every node tells a part of the cluster about the current state of the variable in set intervals. But because later each of those also tells a part of the cluster about the state, the whole cluster ends up having been informed shortly.

It'll use Serf for easy cluster membership, which uses SWIM under the hood. SWIM is a more advanced Gossip-like algorithm, which you can read on about here.

Now let's get to the implementation...

Getting started

First, we'll of course have to put in all our imports:

import (
    "context"
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "sync"
    "time"
    "github.com/gorilla/mux"
    "github.com/hashicorp/serf/serf"
    "github.com/pkg/errors"
    "golang.org/x/sync/errgroup"
)

Following this, it's time to write a simple thread-safe, one-value store.
An important thing is, the database will also hold the generation of the variable. This way, when one instance gets notified about a new value, it can check if the incoming notification actually has a higher generation count. Only then, will it change the current local value.
So our database structure will hold exactly this: the number, generation and a mutex.

type oneAndOnlyNumber struct {
    num        int
    generation int
    numMutex   sync.RWMutex
}

func InitTheNumber(val int) *oneAndOnlyNumber {
    return &oneAndOnlyNumber{
        num: val,
    }
}

We'll also need a way to set and get the value.
Setting the value will also advance the generation count, so when we notify the rest of this cluster, we will overwrite their values and generation counts.

func (n *oneAndOnlyNumber) setValue(newVal int) {
    n.numMutex.Lock()
    defer n.numMutex.Unlock()
    n.num = newVal
    n.generation = n.generation + 1
}

func (n *oneAndOnlyNumber) getValue() (int, int) {
    n.numMutex.RLock()
    defer n.numMutex.RUnlock()
    return n.num, n.generation
}

Finally, we will need a way to notify the database of changes that happened elsewhere, if they have a higher generation count.
For that we'll have a small notify method, which will return true, if anything has been changed:

func (n *oneAndOnlyNumber) notifyValue(curVal int, curGeneration int) bool {
    if curGeneration > n.generation {
        n.numMutex.Lock()
        defer n.numMutex.Unlock()
        n.generation = curGeneration
        n.num = curVal
        return true
    }
    return false
}

We'll also create a const describing how many nodes we will notify about the new value every time.

const MembersToNotify = 2

Now let's get to the actual functioning of the application. First we'll have to start an instance of serf, using two variables. The address of our instance in the network and the -optional- address of the cluster to join.

func main() {
    cluster, err := setupCluster(
        os.Getenv("ADVERTISE_ADDR"),
        os.Getenv("CLUSTER_ADDR"))
    if err != nil {
        log.Fatal(err)
    }
    defer cluster.Leave()

How does the setupCluster function work, you may ask? Here it is:

func setupCluster(advertiseAddr string, clusterAddr string) (*serf.Serf, error) {
    conf := serf.DefaultConfig()
    conf.Init()
    conf.MemberlistConfig.AdvertiseAddr = advertiseAddr

    cluster, err := serf.Create(conf)
    if err != nil {
        return nil, errors.Wrap(err, "Couldn't create cluster")
    }

    _, err = cluster.Join([]string{clusterAddr}, true)
    if err != nil {
        log.Printf("Couldn't join cluster, starting own: %v\n", err)
    }

    return cluster, nil
}

As we can see, we are creating the cluster, only changing the advertise address.

If the creation fails, we of course return the error.
If the joining fails though, it means that we either didn't get a cluster address,
or the cluster doesn't exist (omitting network failures), which means we can safely ignore that and just log it.

To continue with, we initialize the database and the REST API:
(I've really chosen the number at random... really!)

    theOneAndOnlyNumber := InitTheNumber(42)
    launchHTTPAPI(theOneAndOnlyNumber)

And this is what the API creation looks like:

func launchHTTPAPI(db *oneAndOnlyNumber) {
    go func() {
        m := mux.NewRouter()

We first asynchronously start our server. Then we declare our getter:

        m.HandleFunc("/get", func(w http.ResponseWriter, r *http.Request) {
            val, _ := db.getValue()
            fmt.Fprintf(w, "%v", val)
        })

our setter:

m.HandleFunc("/set/{newVal}", func(w http.ResponseWriter, r *http.Request) {
            vars := mux.Vars(r)
            newVal, err := strconv.Atoi(vars["newVal"])
            if err != nil {
                w.WriteHeader(http.StatusBadRequest)
                fmt.Fprintf(w, "%v", err)
                return
            }

            db.setValue(newVal)

            fmt.Fprintf(w, "%v", newVal)
        })

and finally the API endpoint which allows other nodes to notify this instance of changes:

        m.HandleFunc("/notify/{curVal}/{curGeneration}", func(w http.ResponseWriter, r *http.Request) {
            vars := mux.Vars(r)
            curVal, err := strconv.Atoi(vars["curVal"])
            if err != nil {
                w.WriteHeader(http.StatusBadRequest)
                fmt.Fprintf(w, "%v", err)
                return
            }
            curGeneration, err := strconv.Atoi(vars["curGeneration"])
            if err != nil {
                w.WriteHeader(http.StatusBadRequest)
                fmt.Fprintf(w, "%v", err)
                return
            }

            if changed := db.notifyValue(curVal, curGeneration); changed {
                log.Printf(
                    "NewVal: %v Gen: %v Notifier: %v",
                    curVal,
                    curGeneration,
                    r.URL.Query().Get("notifier"))
            }
            w.WriteHeader(http.StatusOK)
        })
        log.Fatal(http.ListenAndServe(":8080", m))
    }()
}

It's also here where we start our server and print some debug info when getting notified of new values by other members of our cluster.

Great, we've got a way to talk to our service now. Time to make it actually spread all the information.
We'll also be printing debug info regularly.

To begin with, let's initiate our context (that's always a good idea in the main function).
We'll also put a value into it, the name of our host, just for the debug logs.
It's a good thing to put into the context, as it's not something crucial for the functioning of our program,
and the context will get passed further anyways.

    ctx := context.Background()
    if name, err := os.Hostname(); err == nil {
        ctx = context.WithValue(ctx, "name", name)
    }

Having done this, we can set up our main loop, including the intervals at which we'll be sending state updates to peers and printing debug info.

    debugDataPrinterTicker := time.Tick(time.Second * 5)
    numberBroadcastTicker := time.Tick(time.Second * 2)
    for {
        select {
        case <-numberBroadcastTicker:
        // Notification code goes here...
        case <-debugDataPrinterTicker:
            log.Printf("Members: %v\n", cluster.Members())

            curVal, curGen := theOneAndOnlyNumber.getValue()
            log.Printf("State: Val: %v Gen: %v\n", curVal, curGen)
        }
    }

Ok, that seems to be it.

Just kidding. Time to finish up our service with the notification code.
We'll now get a list of other members in the cluster, set a timeout, and asynchronously notify a part of those others.

        case <-numberBroadcastTicker:
            members := getOtherMembers(cluster)

            ctx, _ := context.WithTimeout(ctx, time.Second*2)
            go notifyOthers(ctx, members, theOneAndOnlyNumber)

Now, let's look at the getOtherMembers function. It's actually just a function scanning through the memberlist, deleting ourselves and other nodes that aren't alive at the moment.

func getOtherMembers(cluster *serf.Serf) []serf.Member {
    members := cluster.Members()
    for i := 0; i < len(members); {
        if members[i].Name == cluster.LocalMember().Name || members[i].Status != serf.StatusAlive {
            if i < len(members)-1 {
                members = append(members[:i], members[i + 1:]...)
            } else {
                members = members[:i]
            }
        } else {
            i++
        }
    }
    return members
}

There's not much to it I suppose. It's using slicing to cut out or cut off members not conforming to our predicates.

Finally the function we use to notify others:

func notifyOthers(ctx context.Context, otherMembers []serf.Member, db *oneAndOnlyNumber) {
    g, ctx := errgroup.WithContext(ctx)

    if len(otherMembers) <= MembersToNotify {
        for _, member := range otherMembers {
            curMember := member
            g.Go(func() error {
                return notifyMember(ctx, curMember.Addr.String(), db)
            })
        }
    } else {
        randIndex := rand.Int() % len(otherMembers)
        for i := 0; i < MembersToNotify; i++ {
            g.Go(func() error {
                return notifyMember(
                    ctx,
                    otherMembers[(randIndex + i) % len(otherMembers)].Addr.String(),
                    db)
            })
        }
    }

    err := g.Wait()
    if err != nil {
        log.Printf("Error when notifying other members: %v", err)
    }
}

If there are only two members then it sends the notifications to them, otherwise it chooses a random index in the members array and chooses subsequent members from there on.
How does the errgroup work? It's a nifty library Brian Ketelsen wrote a great article about. It's basically a wait group which also gathers errors and aborts when one happens.

Now to finish our code, the notifyMember function:

func notifyMember(ctx context.Context, addr string, db *oneAndOnlyNumber) error {
    val, gen := db.getValue()
    req, err := http.NewRequest("POST", fmt.Sprintf("http://%v:8080/notify/%v/%v?notifier=%v", addr, val, gen, ctx.Value("name")), nil)
    if err != nil {
        return errors.Wrap(err, "Couldn't create request")
    }
    req = req.WithContext(ctx)

    _, err = http.DefaultClient.Do(req)
    if err != nil {
        return errors.Wrap(err, "Couldn't make request")
    }
    return nil
}

We craft a path with the formula {nodeAddress}:8080/notify/{curVal}/{curGen}?notifier={selfHostName}
We add the context to the request, so we get the timeout functionality, and finally make the request.

And that's actually all there is to the code.

Testing our database

We'll test our database using docker. The necessary dockerfile to put into your project directory looks like this:

FROM alpine
WORKDIR /app
COPY distApp /app/
ENTRYPOINT ["./distApp"]

Now, first build your application (if you're not on linux, you have to set the env variables GOOS=linux and GOARCH=amd64)
Later build the docker image:

docker build -t distapp .

And finally we can launch it. To supply the necessary environment variables, we'll need to know what ip address the containers will get.
First run:

docker network inspect bridge

Bridge is the default network containers get assigned to. You should get something like this:

[
    {
        "Name": "bridge",
        "Id": "b56a19697ed9d30488f189d5517fd79f04a4df70c8bbc07d8f3c49a491f10433",
        "Created": "2017-01-29T10:48:05.1592086Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1" <-- this is what we need
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

What's important for us is the gateway. In this case, our containers would be spawned with IP addresses from 172.17.0.2

So now we can start a few containers:

docker run -e ADVERTISE_ADDR=172.17.0.2 -p 8080:8080 distapp
docker run -e ADVERTISE_ADDR=172.17.0.3 -e CLUSTER_ADDR=172.17.0.2 -p 8081:8080 distapp
docker run -e ADVERTISE_ADDR=172.17.0.4 -e CLUSTER_ADDR=172.17.0.3 -p 8082:8080 distapp
docker run -e ADVERTISE_ADDR=172.17.0.5 -e CLUSTER_ADDR=172.17.0.4 -p 8083:8080 distapp

Next on you can test your deployment by stopping and starting containers, and setting/getting the variables at:

localhost:8080/set/5
localhost:8082/get/5
etc...

Conclusion

What's important, this is a really basic distributed system, it may become inconsistent (if you update the value on two different machines simultaneously, the cluster will have two values depending on the machine).
If you want to learn more, read about CAP, consensus, Paxos, RAFT, gossip, and data replication, they are all very interesting topics (at least in my opinion).

Anyways, I hope you had fun creating a small distributed system and encourage you to build your own, more advanced one, it'll be a great learning experience for sure!

The whole code is available on my Github.