Forem: Gianluca Brindisi

A Collection of Cloud Security Tools

Gianluca Brindisi — Sun, 18 Oct 2020 00:00:00 +0000

I’ve built a directory of open source cloud security tools.

A good part of my day to day is spent trying to automate away problems. Over the years I learned how to invest my time wisely, and I made a habit to research and use already made tools before start coding my own.

As a consequence I have a fairly large collection of utilities I keep nurturing, alongide references, commands and debugging adventures.

I thought I could as well make it public, so here we are: check it out.

Every tool has a page were I (will) store my own notes: see my very own docker-security as an example. I am still cleaning up most of them and I will publish a bit per time.

I’ll also commit a more compact version to github.

In the meanwhile if you have some tools to share please do!

Did you find this post interesting? I’d love to hear your thoughts: hello AT cloudberry.engineering

I write about cloud security on my blog, you can subscribe to the RSS feed or to the newsletter.

How to find and delete idle GCP Projects

Gianluca Brindisi — Tue, 13 Oct 2020 15:28:01 +0000

A constant source of pain in Google Cloud Platform (GCP) and everywhere else is the amount of unmaintained resources: idle virtual machines, old buckets, IAM policies, DNS records and so on. They contribute to the attack surface and the chance of a vulnerability increase with time.

Shutting off resources is a such a low hanging fruit from a risk perspective that as a security engineer you should make it a daily habit.

After all the most secure computer is the one that’s been turned off!

How to find the cruft

The bigger and complex a cloud infrastructure becomes, the harder it gets to find unmaintained stuff.

Having an inventory system in place, as early as possible, would prevent so many headaches but even the most enlightened leadership will have a hard time justifying the investment.

Eventually the problem will outgrow security and spill into other areas such as cloud spending (gasp!): that’s when everyone will start talking about inventories, accountability and resources lifecycle.

Until then how to find things to kill?

Start with the Projects.
The GCP model encourages the segmentation of the infrastructure logical areas into Projects, and a lot of audit facilities are aggregated on that level.
(Obviously Projects will also silently introduces cost multipliers such as VPCs but we will leave this for another rant.)

There are three sources one can query:

Activity Logs
Billing Reports
IAM Policies

Use these three and you can build your own personal heuristic that will answer the question: can I kill this?

Activity Logs

Events that change state and configuration of cloud services are collected in the Admin Activity and System Event audit logs.

While they both track configuration changes only the Admin Activity is the one that tracks manual changes driven by direct user action: creation of resources, change of IAM policies, etc.

The retention is ~400 days and I would check the frequency of these log entries to understand if services were being configured recently.

Usually an active project implies an active administrator.

Billing Reports

We can query the billing account(s) to get a per project cost/usage report.

If the cost graph is flat that could be an indicator that the project is idling.
In contrast plotting an active project’s cost will results in a bumpy curve as buckets will fill up, logs will be generated and resources will be add and removed over time.

It’s worth keeping in mind that we can also get usage reports for Compute Engine services. It’s mostly a data point about the lifecycle of resources rather than their usage - but can still contribute to our killer algorithm.

IAM Policies

Nobody knows a project better than its owner, so we can’t go wrong if we ask politely. The problem is finding that person.

The solution is to scrape the IAM policy.
I’d start by searching role bindings for Owner, Editor or Viewer as they are the basic roles in GCP.

If we are lucky we will get a Group or a User’s email.

If we get a Service Account (SA) we can investigate it.
When a SA is bind with a basic role, 99% of the time it’s been created in another project. So we can recursively scrape that project’s IAM and keep going until we find a human.

Look ma, no influencing skills

There are two things I learned the hard way as a security artisan:

Do not kill stuff without asking first
Do not flood people with alerts

As such my cruft hunting algorithm is called can_I_MAYBE_kill_this() and works like this:

I combine billing reports and admin activity logs to figure out if the project is idle since a while. I want to rule out projects that are obviously active.
I scrape the IAM policy and find potential owners.
I send them an email asking who is the technical contact for the project because I need to talk to them about a security situation. The combination of asking for someone accountable and mentioning security usually trigger a game of hot potato that ends with the project killed.
If I get no answer, I nudge that I will delete the project in X days. This is the part where is self reflect on how I ended up threatening good people for a living and I manually check the project again to find more evidence.
I delete the project.

Note that deleting a project will trigger a soft deletion, this means you have 30 days to change your mind before resources are actually decommissioned (although Cloud Storage services get decommissioned faster, usually ~ a week).

Conclusion

The take away is that finding and shutting down idle cloud resources is not straightforward and can’t be solved with a cron job.

Shut down other people’s resources at your risk and peril: be nice, ask, nudge and implore them to take care of their things.

Keep track of every time you have to track down owners. Make accountability and resource lifecycle a chapter of your threat model and build a case to lobby for an inventory system.

If you have someone in charge of keeping track of spending go and talk to them: change is never introduced in isolation, and there isn’t anything better than mixing cost savings and security to get ~~a budget~~ attention.

Happy hunting.

Did you find this post interesting? I’d love to hear your thoughts: hello AT cloudberry.engineering

I write about cloud security on my blog, you can subscribe to the RSS feed or to the newsletter.

Dockerfile Security Best Practices

Gianluca Brindisi — Sun, 04 Oct 2020 21:32:41 +0000

Container security is a broad problem space and there are many low hanging fruits one can harvest to mitigate risks. A good starting point is to follow some rules when writing Dockerfiles.

I’ve compiled a list of common security issues and how to avoid them. For every issue I’ve also written an Open Policy Agent (OPA) rule ready to be used to statically analyze your Dockerfiles with conftest. You can’t shift more left than this!

You can find the .rego rule set in this repository. I appreciate feedback and contributions.

Do not store secrets in environment variables

Secrets distribution is a hairy problem and it’s easy to do it wrong. For containerized applications one can surface them either from the filesystem by mounting volumes or more handily through environment variables.

Using ENV to store secrets is bad practice because Dockerfiles are usually distributed with the application, so there is no difference from hard coding secrets in code.

How to detect it:

secrets_env = [
    "passwd",
    "password",
    "pass",
 # "pwd", can't use this one   
    "secret",
    "key",
    "access",
    "api_key",
    "apikey",
    "token",
    "tkn"
]

deny[msg] {    
    input[i].Cmd == "env"
    val := input[i].Value
    contains(lower(val[_]), secrets_env[_])
    msg = sprintf("Line %d: Potential secret in ENV key found: %s", [i, val])
}

Only use trusted base images

Supply chain attacks for containerized application will also come from the hierarchy of layers used to build the container itself.

The main culprit is obviously the base image used. Untrusted base images are a high risk and whenever possible should be avoided.

Docker provides a set of official base images for most used operating systems and apps. By using them, we minimize risk of compromise by leveraging some sort of shared responsibility with Docker itself.

How to detect it:

deny[msg] {
    input[i].Cmd == "from"
    val := split(input[i].Value[0], "/")
    count(val) > 1
    msg = sprintf("Line %d: use a trusted base image", [i])
}

This rule is tuned towards DockerHub’s official images. It’s very dumb since I’m only detecting the absence of a namespace.

The definition of trust depends on your context: change this rule accordingly.

Do not use ‘latest’ tag for base image

Pinning the version of your base images will give you some peace of mind with regards to the predictability of the containers you are building.

If you rely on latest you might silently inherit updated packages that in the best worst case might impact your application reliability, in the worst worst case might introduce a vulnerability.

How to detect it:

deny[msg] {
    input[i].Cmd == "from"
    val := split(input[i].Value[0], ":")
    contains(lower(val[1]), "latest"])
    msg = sprintf("Line %d: do not use 'latest' tag for base images", [i])
}

Avoid curl bashing

Pulling stuff from internet and piping it into a shell is as bad as it could be. Unfortunately it’s a widespread solution to streamline installations of software.

wget https://cloudberry.engineering/absolutely-trustworthy.sh | sh

The risk is the same framed for supply chain attacks and it boils down to trust. If you really have to curl bash, do it right:

use a trusted source
use a secure connection
verify the authenticity and integrity of what you download

How to detect it:

deny[msg] {
    input[i].Cmd == "run"
    val := concat(" ", input[i].Value)
    matches := regex.find_n("(curl|wget)[^|^>]*[|>]", lower(val), -1)
    count(matches) > 0
    msg = sprintf("Line %d: Avoid curl bashing", [i])
}

Do not upgrade your system packages

This might be a bit of a stretch but the reasoning is the following: you want to pin the version of your software dependencies, if you do apt-get upgrade you will effectively upgrade them all to the latest version.

If you do upgrade and you are using the latest tag for the base image, you amplify the unpredictability of your dependencies tree.

What you want to do is to pin the base image version and just apt/apk update.

How to detect it:

upgrade_commands = [
    "apk upgrade",
    "apt-get upgrade",
    "dist-upgrade",
]

deny[msg] {
    input[i].Cmd == "run"
    val := concat(" ", input[i].Value)
    contains(val, upgrade_commands[_])
    msg = sprintf(“Line: %d: Do not upgrade your system packages", [i])
}

Do not use ADD if possible

One little feature of the ADD command is that you can point it to a remote url and it will fetch the content at building time:

ADD https://cloudberry.engineering/absolutely-trust-me.tar.gz

Ironically the official docs suggest to use curl bashing instead.

From a security perspective the same advice applies: don’t. Get whatever content you need before, verify it and then COPY. But if you really have to, use trusted sources over secure connections.

Note: if you have a fancy build system that dynamically generate Dockerfiles, then ADD is effectively a sink asking to be exploited.

How to detect it:

deny[msg] {
    input[i].Cmd == "add"
    msg = sprintf("Line %d: Use COPY instead of ADD", [i])
}

Do not root

Root in a container is the same root as on the host machine, but restricted by the docker daemon configuration. No matter the limitations, if an actor breaks out of the container he will still be able to find a way to get full access to the host.

Of course this is not ideal and your threat model can’t ignore the risk posed by running as root.

As such is best to always specify a user:

USER hopefullynotroot

Note that explicitly setting a user in the Dockerfile is just one layer of defence and won’t solve the whole running as root problem.

Instead one can — and should — adopt a defence in depth approach and mitigate further across the whole stack: strictly configure the docker daemon or use a rootless container solution, restrict the runtime configuration (prohibit --privileged if possible, etc), and so on.

How to detect it:

any_user {
    input[i].Cmd == "user"
 }

deny[msg] {
    not any_user
    msg = "Do not run as root, use USER instead"
}

Do not sudo

As a corollary to do not root, you shall not sudo either.

Even if you run as a user make sure the user is not in the sudoers club.

deny[msg] {
    input[i].Cmd == "run"
    val := concat(" ", input[i].Value)
    contains(lower(val), "sudo")
    msg = sprintf("Line %d: Do not use 'sudo' command", [i])
}

Acknowledgements

This work has been inspired and is an iteration on prior art from Madhu Akula.

Did you find this post interesting? I’d love to hear your thoughts: hello AT cloudberry.engineering

I write about cloud security on my blog, you can subscribe to the RSS feed or to the newsletter.

Stricter Access Control to Google Cloud Registry

Gianluca Brindisi — Sat, 26 Sep 2020 14:31:29 +0000

Google Cloud Registry (GCR) is the Docker container registry offered by Google Cloud Platform (GCP). Under the hood it's an interface on top of Google Cloud Storage (GCS), and it’s so thin that access control is entirely delegated to the storage layer.

There are no dedicated roles

In fact, there are no dedicated Identity Access Management (IAM) Roles to govern publishing and retrieval of container images: to push and pull we must use role bindings that will grant write (and read) to the underlying bucket that GCR is using.

According to the docs this bucket is artifacts.<PROJECT-ID>.appspot.com and the roles to use are roles/storage.admin and roles/storage.objectViewer (but any powerful primitive role such as Owner, Editor and Viewer will do).

The role binding can be applied to the IAM policy of the Project, Folder, Organization or the bucket itself.

What's the risk

While binding the role on the bucket's IAM could be fine, binding on an IAM policy higher in the hierarchy will result in a wider authorization grant affecting all buckets in scope.

Such grant could have an impact on your compliance posture.
A common example is if one of the buckets contains Personal Identifiable Information (PII) and your organization is subject to GDPR.

IAM is tricky and things get messier when the number of Projects we need to administer increases and there is a business case to give programmatic access to all GCR instances. For example if we have a centralized build system that needs to push container images, or if we need to integrate a third party container scanner.

In such cases, especially when a third party is involved, binding a Service Account with read/write permissions to the GCS layer is unacceptable as it will increase a potential attack's blast radius.

How to mitigate

While we wait for Google to implement a set of dedicated Roles (see Artifact Registry), there are a couple of solutions we can adopt to minimize the authorization grant.

The first is organizational: minimize the number of GCR instances.
Ideally, if you can use a single instance you can bind the Role on the associated bucket’s IAM policy.
A small number of instances could be managed that way but I let you decide what small means in your context.

The second solution is technical: leverage IAM Conditions to reduce the scope of the role binding to only the buckets that are used by GCR.
IAM Conditions is a feature of Cloud IAM that allows operators to scope down role bindings.

Luckily these buckets have similar names and we can use this pattern to set up a role binding that will be applied only when the bucket’s name match, like this:

{
    "expression": "resource.name.startsWith(\"projects/_/buckets/artifacts\")",
    "title": "GCR buckets only",
    "description": "Reduce the binding scope to affect only buckets used by GCR"
}

This solution is pragmatic and scale well with the number of Projects / Folders affected, as long as there are no other buckets named artifact*.

Keep in mind that you need to use the full bucket identifier in the condition and If GCR is configured to use explicit storage regions, the bucket name will be (eu|us|asia).artifacts.<PROJECT-ID>.appspot.com.

Did you find this post interesting? I’d love to hear your thoughts: hello AT cloudberry.engineering

I write about cloud security on my blog, you can subscribe to the RSS feed or to the newsletter.

Lateral Movement in the Cloud

Gianluca Brindisi — Wed, 16 Sep 2020 22:53:55 +0000

In the context of incident response lateral movement is how attackers are able to penetrate deeper inside a system. Understanding this concept is critical to contain an ongoing breach.

The Attack Lifecycle

There are many models that help to understand an attack lifecycle in depth, but from a practical perspective assume all attackers will do the following endlessly: compromise a resource , gain persistence , move laterally.

As a defender you will need to walk forward and backward this cycle to stop the bleeding, and the tricky part is indetifying the entry point : the resource that have been compromised first.

The entry point is the front door you left open. It is usually a resource that is publicly exposed like a virtual machine running a public website.

If you identify a compromised resource that is not publicly exposed, most probably the attacker reached it by moving laterally.

Modelling the movements

Lateral movement happens when the attacker compromise other resources from one which has already been breached.

It is enabled by two factors:

What other resources we be reached from a compromised one
What credentials we be found on compromised resources

These two factors defines the blast radius : the area that an attacker can traverse inside your infrastructure and, as a consequence, the impact of the attack.

The text book example of lateral movement is breaching a website, gain access to the virtual machine running it, finding the credentials to the database in the backend and compromise it.

In the cloud you can model attacker movements on three layers, which are loosely related to the way the cloud stack is segmented (IaaS, PaaS, SaaS):

The network layer.
The Identity & Access Management (IAM) layer.
The application layer.

Network Layer

The network layer is mostly tied to infrastructure services (the “I” in IaaS).

It boils down to a classic network security problem: which cloud services are connected to the same Virtual Private Network (VPC), which firewall rules are in place, etc.

In the worst case a compromised virtual machine’s blast radius is everything attached to the same VPC.

Identity & Access Management Layer

IAM is how access controls is governed in the cloud. It’s about authorising identities to certain actions on specific cloud services.

The concept of identity is fluid between cloud providers but it can be described as the authenticated entity to which authorisation grants are assigned. A user, a group, a service account, a workload.

Because identities can be very different things, and authorisation grants can be very granular, understanding the blast radius becomes complicated quickly.

From the attacker perspective compromising a user or compromising a random cloud service can have the very same impact if they have the same authorisation grants.

Instead from a defender point of view you will hardly be able to apply the same security controls, the same auditing and the same monitoring practices.

In short IAM is the layer where the economics of an attack skews in favour of attackers.

Application Layer

All the secrets and credentials that are bundled in the application layer of a service will also contribute to the blast radius, since they can be used to access and compromise further resources.

Think of passwords to databases, api tokens, credentials to third party services.

Conclusion

Modelling the lateral movements and the blast radius of an attack is an obligatory step for a successful containment of a breach.

An incident resolution will be as effective as how fast we will be able to answer the question: “what services am I running in my infrastructure, and how can I reach them?”

So be prepared: invest in an inventory of running cloud services you can query quickly and map their interconnections on the three different layers.

Did you find this post interesting? I’d love to hear your thoughts: hello AT cloudberry.engineering

I write about cloud security on my blog, you can subscribe to the RSS feed or to the newsletter.