Forem: Greg Bulmash

Five Ways Your CI/CD Pipeline Can Be Exploited

Greg Bulmash — Fri, 26 Jan 2024 17:08:58 +0000

We've talked about how Continuous Integration and Continuous Delivery (CI/CD) tools can be a source of secrets sprawl. While it's not as insecure as leaving them laying around in a publicly accessible file, CI/CD pipelines can be exploited in a number of ways and I'm going to share a few with you.

This article is not exhaustive. GitHub's Security Hardening Guide for GitHub Actions alone is 16 pages long if you try to print it. OWASP's Top 10 CI/CD Security Risks is 38 pages long. Protecting your CI/CD systems is not a tiny task, but it's an important one. To get you started, here's a quick read on five ways attackers can leverage your CI/CD to gain access to additional systems.

#1: When someone can find your credentials

In an NCC Group report on 10 exploits they ran, their first exploit starts with an improperly permissioned S3 bucket. The bucket had a script with a hardcoded Git credential. From there, it was a matter of steps to gain access to the CI/CD environment and get more credentials the CI/CD processes had access to… Pwned!

GitGuardian has found millions of hardcoded secrets in Git repositories, but secrets can be anywhere: S3 buckets, screencaps, you name it. Secrets hygiene should be followed everywhere. And given how many exploits have started in misconfigured S3 buckets, if you're going to use S3, make sure you're using it correctly.

#2 When GitHub has your AWS credentials

While environment variables are considered somewhat of a minimum standard for keeping secrets out of your code, if the code that uses them is compromised or is poorly configured, it can start revealing them through your logfiles.

GitHub and AWS actually have an ingenious workaround for this, using OpenID Connect (OIDC). In this case, AWS uses GitHub as an identity provider (IDP) for an Identity and Access Management (IAM) role you create. That IAM role becomes accessible only by your repository, and you can even scope it to a specific branch. GitHub and AWS negotiate temporary tokens between themselves and no password is stored.

When no password is stored by the system, no password can be revealed by the system.

#3 When access is poorly managed

The number two item in OWASP's Top 10 CI/CD Security Risks deals with IAM as well. It goes into a number of ways you can improperly manage this. To quote them: "Ensuring each human and application identity has been granted only the permissions required and only against the actual repositories it needs to access is not trivial."

This isn't limited to the service AWS calls "IAM," either. Identity and Access Management relates to everything from a keycard to get in the door of the building to an access policy assigned to a specific branch of a Git repository. Ensuring your CI/CD processes can only touch what they're supposed to touch and see what they're supposed to see is crucial.

Additionally, the number of people with access to your CI/CD logging and how they're able to access it matters. If your CI/CD process is leaking credentials into the logs and a contractor's login is shoulder-surfed in a Starbucks, the clock to a complete takeover starts ticking.

Some of the worst hacks don't start with someone finding the keys to the castle laying around. They start with someone who found the keys to the stable laying around. In the stable, they found the keys to the scullery in a saddlebag. And so on and so on.

#4 When you obfuscate your credentials

Many CI/CD logging systems have some method of identifying and redacting credentials in the logs. But if you make it so the redaction filters can't spot the credential, but an experienced human can, you might just clever yourself into getting hacked.

In many demonstrations of poisoning pipelines (getting a bit of bad code into your CI/CD process), the hackers have the bad code output your passwords in base 64. For example, PASSWORD='LetMeIn' in base 64 is UEFTU1dPUkQ9J0xldE1lSW4n. While redaction filters may get better and better at spotting these kinds of things, some systems' redaction filters are just a handful of regular expression patterns. Converting secrets into another format can help sneak them past the sniffers that are there to protect you.

In GitHub's security hardening advice, one of the primary points of guidance for secrets is "never use structured data as a secret." Packing up your secrets in XML, JSON, or YAML may help put the helpful secret-sniffing robots off the scent and let those secrets escape into your logs.

#5 When every file has the same write permissions

GitHub offers a feature called CODEOWNERS. We don't mean to bang the GitHub drum, but this offers a practical application that can be translated to other systems.

With CODEOWNERS, you can set the permissions on your .github/workflows directory such that whenever someone tries to change anything in it (creating a "poisoned pipeline"), an admin specified by CODEOWNERS is not only alerted, but has to approve the change.

Depending on your systems, it's possible to lock down important files or directories within a development environment and ensure only the people who need to be able to change build scripts or configurations can. Whether on a local build or the primary server, this helps ensure that you're not accidentally or intentionally sabotaged by an insider.

What else can you do?

After telling you all the ways you can be compromised, even though knowing them helps avoid it, we wanted to leave you with a couple of things you can proactively do (and if you're already doing them, you can finish this feeling good that you are).

Shift Left: This can relate to multiple things, but here, we mean don't wait until the build to test your code. Run more tests before code gets to the CI/CD pipeline. Tests need access and generate logs, and if those are being run in development environments, they're not providing an additional attack surface at the CI/CD level. Additionally, scan, scan, and scan again. There are multiple points, from a pre-commit hook in a developer environment to scanning the log files and artifacts from a CI/CD run, where you can integrate a secrets scanner like GitGuardian.
Use Vaults: Remember how AWS and GitHub can communicate without you giving GitHub an AWS secret? That doesn't just need to be limited to pushing builds to an EC2 instance or S3. It can also let you pull secrets for other resources from AWS Secrets Manager, so they're never stored in GitHub. You may not use AWS or GitHub, but you can explore how your favorite CI/CD tools and your favorite vault can integrate so you're not storing secrets in your build scripts, build tools, or the artifacts they produce.

We hope you're leaving this with some better understanding of how CI/CD tools can get exploited, how you can protect yourself better, and some ideas for exploring the topic further. To quote the excellent 1980s police drama, "Hill Street Blues": Hey, let's be careful out there.

Secure Your Secrets with .env

Greg Bulmash — Tue, 09 Jan 2024 17:00:52 +0000

Using environment variables to store secrets instead of writing them directly into your code is one of the quickest and easiest ways to add a layer of protection to your projects. There are many ways to use them, but a properly utilized .env file is one of the best, and I’ll explain why.

They’re project scoped

Environment variables are a part of every major operating system: Windows, MacOS, and all the flavors of *nix (Unix, BSD, Linux, etc.). They can be set at an operating system level, user level, session level… It gets complicated, and where/how you define them matters to the scope in which they can be accessed.

This variety of scopes also creates the distinct possibility of variable collisions. If you’re looking for an environment variable named API_KEY, that could be getting re-defined in each scope, and if you’re not steeped in that OS, it’s extra work to be sure you’re not clobbering something someone set at a different scope that some other app or service needs.

.env files are only consumed at runtime and only in the context of the app that’s consuming them. That prevents them from clobbering any other environment variables on the system that might be consumed outside your app.

They can be "ignored"

If you’re working on a JavaScript application in Node, you can’t ignore your index.js file in the version control system. It contains essential code. But you can set your .gitignore file to have the Git system ignore your .env file. If you do that from the inception of your repository, you won’t commit secrets to the project’s Git history.

A better option is to include a .sample.env file that sets the variable names, but only includes dummy data or blanks. People cloning/forking and using the repository can get the secrets via another route, then cp .sample.env .env (in a terminal), and assign the real values to the proper variables in the ignored .env file.

They’re relocatable

While most systems will default to looking for the .env file in the root of the app’s primary directory, you can always have it a level or two higher. So, if for example, a server configuration error or code bug leaves it possible to view all the files at the root of your web app as a directory, the .env will not be there for easy pickings.

This is not an uncommon practice. SSH keys are stored by default at ~/.ssh (a "hidden" subdirectory of the home directory of the user) on Windows, Mac, and Linux. You do not need to move them into the root directory of a project that uses them.

A quick .env demo in Node

Let’s say your working directory for the app you’re building is ~/Documents/work/projects/games/tictactoe and tictactoe is the root directory for the app you’re building and your Git repository. You can put the .env in the next directory up, games. And while we generally call the file type .env, you can call it .toecreds if you want to make it a distinct file that other processes would never even think to touch. We'll use that in the demo.

Here’s how you’d do that in Node.js.:

1. In your games/tictactoe directory, npm init (go with the defaults) and then npm install dotenv.

2. Create your .toecreds file in the games directory.

3. Fill the .toecreds file with information in the following format VARIABLE_NAME=VALUE (no spaces). You can also start a line with # for a comment.

# Leaderboard SaaS
LEADERBOARD_ENDPOINT=https://example.com/leaderboard/v1
LEADERBOARD_KEY=jknfwgfprgmerg…

4. At the top of your index.js (or whatever file is your launchpoint) in games/tictactoe, include the following lines:

require('dotenv').config({ path: '../.toecreds' })
console.log(process.env.LEADERBOARD_ENDPOINT)

Run your index.js and the endpoint URL will be output to the terminal. Meanwhile, the environment variables you set in it will not be available from the terminal and the file lives a level above your repository and can't accidentally be swept up if you misconfigure .gitignore.

Try adding a long timeout to the script and then running node index.js & to return control back to the terminal after invoking the script. While the script is running in that shell session, the environment variables available to the shell still do not contain the secrets. They are scoped to your running application.

You can have dev, test, and prod credential sets, having your CI/CD tooling pull the correct keys for the deployment target from a secrets manager and write the .toecreds (or .env) file to the same relative directory.

And there you have it

The use of a .env file helps you keep your app's secrets from ever being committed to your version control and provides an additional layer of protection against your secrets being discovered by hackers or other prying eyes. It's a great addition to your developer / dev-ops toolbox.

Ethical Hacking Q&A with Sonya Moisset

Greg Bulmash — Tue, 19 Dec 2023 16:54:32 +0000

GitGuardian’s Mackenzie Jackson had the privilege of hosting a webinar on ethical hacking with Snyk’s Sonya Moisset. What follows is a lightly edited transcript of the Q&A after her presentation or watch the Ethical Hacking Q&A video clip on YouTube.

Mackenzie (after describing the process for submitting and upvoting questions):

I'm just going to ask my question because I'm the host and I can do that.

So the question that I have too is… when we're doing something ethically, when we're ethically hacking something, we obviously have parameters that we have to work within. Is there anything that we have to look out for that could be a legal issue if we… if we perhaps access the wrong things? Do we have to be careful of that when we're starting our ethical hacking journey?

Sonya:

Yeah. This is why I stress the fact that when you go on those bug bounty platforms, it's really important to read their rules of the game because they will let you know which endpoint you can test, which URLs, which API you can actually test. And if you do test outside of this scope, this is where it's actually illegal.

And usually, when you test on these platforms, as I mentioned, you do sign an NDA when you're doing so, so you ask permission to actually test those. If you haven't signed the NDA and you actually find one vulnerability, then you can come up to these platforms and try to find those programs.

Usually, HackerOne, Intigriti, and other platforms will help you to actually convey this report and be in a safe place. But as I mentioned during the demo I wasn't supposed to have the codebase on my machine, because this is… you can consider it an IP, intellectual property, that I actually copied and had on my machine.

You should actually do all of the things with the Patch reversal to have the information. So just make sure to not have copies of databases, user information, and all of those. Make sure to stay in the scope of the testing and make sure to provide a detailed report of all of the commands that you've done, because potentially the security team/developer team will actually look at the logs. So yeah, they could potentially look at what you've done in their systems.

Mackenzie:

That's great. Thanks so much for that. We have another question here. Now there's actually kind of a lot of similar questions about this and I think that maybe Snyk Learn might be a good place. But where, if people want to get started… where do the beginners go to kind of… what would be your first steps, maybe to start with, and then what would be the journey beyond that, do you think?

Sonya:

Sure. So there are a couple of websites, good websites, and also free platforms where you can start this learning journey, which are Try Hack Me and Hack The Box. They’re two great platforms where you can create an account and actually try to “Pwn” those machines.

At Try Hack Me, I find it a little bit more beginner-friendly because they do have those rooms with step by step walk throughs. When you feel a bit more confident you can go on Hack The Box and try to hack the beginner machines. So, yeah, this is potentially a good one.

There's other platforms you can also try—the OWASP Juice Shop, which is an online retail shop that has a lot of vulnerabilities. There's also picoctf. There's a lot of those platforms. There's also, as you've mentioned, Hack The Box has an academy where you can actually learn a bit more. They do have modules on specific vulnerabilities, on how to do those ethical hacking methodologies and workflows, so this is to help you on that path.

Mackenzie:

Excellent. And there are a couple of more questions along that… I don't think we need to answer them… You know there are courses, there are certifications, but if you want to start, I think that those are the places to start.

Sonya:

Yeah, I would say if you want to also demonstrate those to recruiters, and you might not have the budget to actually do the certifications, is start with Try Hack Me, Then move to Hack The Box. And when you manage to pwn the box, write a detailed walkthrough on which tools you've actually used, the commands, what were the results, and also recommendations.

Then you can start working on creating a portfolio of walkthroughs. And this is something that [you] could potentially show to recruiters because you will have hands-on, specific tools, and the methodology, which is important because you want to demonstrate the skills.

Mackenzie:

Yeah, that's really cool, because perhaps people don't realize but there are certifications that you can get and they go on the end of your, you know, a lot of people have them on the end of their LinkedIn profiles. And a lot of them are great and it can show the recruiters that you know what you're talking about, but by golly, some of them are so expensive to do. I feel like what you're saying is that if you don't have the budget to do that, then if you document it, that can be a place you can save in.

That's fantastic advice.

Now you touched on AI and we had a question here from Abdullah, who says “How do you think generative AI will affect your day as an ethical hacker and the cybersecurity field as a whole? Are there real-world applications?”

So you've kind of touched on this, but is there anything more that you could kind of add on that? Are you using AI when you're doing ethical hacking today?

Sonya:

So, here we're talking about generative AI. This is the code that is generated through Copilot, through ChatGPT, and the other LLMs, so this would depend on how… what… what you're doing about this code.

So it's great to have those tools. Let's say if we're producing code through ChatGPT, what I would recommend is the same as when you copy/paste code from Stack Overflow.

Do you have security tools? It could be GitGuardian, Snyk… just to make sure that you have a pipeline to make sure that you're scanning the code that you will introduce into your code base. You can use these tools. Just make sure to double-check and make sure to understand what you're actually putting into your codebase. The problem that you might have copy/pasting Stack Overflow would be the same with the code generated through different LLMs.

But also this was an open question for not generative AI, but any of the ethical hacking AI tools or those that are powered by AI. This is interesting. It’s always the same, as with any tools, have a good understanding of the tools that you're using and also make sure that you're using them as an assistant. Don't lean only on your tool. Make sure to double-check every time the results, the outputs, just to make sure that it doesn't produce hallucinations or false positives or false negatives in a sense.

So just make sure to use it as an assistant when you're stuck and you need better ideas or explanations. For example, line by line for code or for an exploit. So, yeah, it depends on the usage that you're actually doing on AI.

Mackenzie:

Yeah, amazing. [pauses to promote the next webinar]

Maybe a couple more questions. This one I find quite interesting: “What overlap does an ethical hacker have in the day-to-day with a typical software engineer?”

And what I kind of think–perhaps I'm interpreting this wrong–but if you're in a company, and perhaps you're a red teamer, or you're on the appsec team, how do you interact with the software engineers? Do you have any advice for how to interact, perhaps, with software engineers and help them, and not just be the “opposite team,” let's say?

Sonya:

Sure, so the most important thing is the culture that you have in the organization. The security team should be enablers. They shouldn't be blockers or gates for developers. They should have that collaboration piece and communicate with the team.

It's usually a good thing, for example, doing these kinds of webinars with live hacking demos during lunch breaks or organizing this within the company so the developers have a better understanding of what is happening in the cyber security space because it's quite exciting.

When we're talking about hacking, everyone is interested to know, to have a better understanding of it, so this a good way, actually, to increase the culture and just to break the silos between both teams.

You can also gamify it. You can dedicate a day to making sure that you go through the backlog of security vulnerabilities. And you have the security team doing some training, explaining the vulnerability, sitting down with the developers, just you have that collaboration and communication piece.

This could also go through the Security Champion program, for example. A Security Champion is basically a developer that is acting as a bridge between security and developer teams. They could also upskill through this program. So there are several ways to do so, but the company should make sure the security team are enablers. That's the most important thing.

Mackenzie:

[Winds up the webinar]

“Do Not Push To Production” And Other Insecure Coding, Demonstrated By An Ethical Hacker

Greg Bulmash — Thu, 07 Dec 2023 23:47:50 +0000

In November, our own Mackenzie Jackson hosted a webinar called “Crack the Code: A Beginner’s Guide to Ethical Hacking” with special guest, Snyk’s Sonya Moisset. Viewers got to see some interesting vulnerabilities and coding practices that made her demo app pretty open to exploits.

I will warn you right now, if you watch the ethical hacking webinar, be careful about what you’re drinking, where you’re facing, and the size of your sips. I learned this the hard way when I almost sprayed my monitor with Coke Zero as she got to a specific point I’ll note when we get there.

What is “Ethical Hacking?”

To quote Sonya, ethical hacking is “identifying and exploiting vulnerabilities in computer systems or networks in a responsible and lawful manner.” It’s been around for quite a while. A friend of mine published a book about it over 25 years ago, called The Happy Hacker.

Basically, there are four main tenets that ethical hackers are supposed to follow:

Only hack with permission. If you’re hacking without permission, no matter what your motives, you could get arrested.
Follow relevant laws and regulations. Again, you can get arrested, even if your motives are altruistic.
Don’t be destructive/disruptive. Don’t try to nuke your target, crash it, or delete production data to prove your hack. Don’t use tools you know could do damage. Remember, if you’re hacking ethically, you’re a guest in the systems you breach, so be respectful.
Document your findings and provide a comprehensive report to the system owner.

Ethical hackers are very useful. They’re often the source of reports that lead to security patches for your browser, your OS, etc. For example, the Sakura Samurai ethical hacking group exposed (and helped fix) a security hole that provided access to sensitive United Nations data.

The heist!

Sonya’s demo went through hacking an app called Patch, which is a deliberately vulnerable demo application, written in Node.js. Thank goodness, she was running it locally, but had she put a tunneling service like ngrok in front of it, the app could have been available to the broader internet.

One of the first steps in hacking is reconnaissance. As part of her recon, Sonia tried some common and simple tests. She found that assets were being served from the public folder. She was able to take the URL to an asset, remove the file name and get access to the full directory listing of the folder.

This is pretty old and has multiple ways to solve it. Apache lets you turn off showing the directory listing in the server configuration. I simply include an index.html file in those directories. It has one line of text: “These are not the droids you’re looking for. Move along.”

But then she got a little fancier and tried a directory traversal exploit to see if she could get out of the public directory and into directories she shouldn’t be able to access. And she could.

She found her way to the top level of the app directory, giving her the ability to view and exfiltrate the entire source code. As she’d show later, directory traversal could go even farther, giving an attacker access to the ~/.ssh directory where secure keys are stored or the files where hashed system passwords are stored.

First thing she did, though, was browse through the package.json file, which provided a list of all the third-party NPM packages it was using. Using the Snyk plugin for VSCode, she was able to scan the app for vulnerabilities and find that the directory traversal weakness came from the app being pegged to the 0.2.5 version of a package that’s currently at version 3.0.0.

It may seem simple to fix that just by updating the version, but realize that there are likely to be breaking changes between those two versions because they’re so far removed. Extensive testing would be necessary. But it’s not her job to fix it. It’s her job to report it, so she recorded it with as much detail as possible, and moved on.

Remember though, she got access to all the code, not just that file. She started digging into the main code file, index.js, and scrolling through it. She saw all the packages being imported, then she was able to see all the routes for all the endpoints, including an endpoint for a chat feature she wasn’t aware of.

She scrolled further down to find the code associated with the chat endpoint and that’s when I almost…

// DO NOT PUSH TO PRODUCTION

For those of you who aren’t programmers, that’s a comment in the code. It’s a reminder to anyone who reads the code, but has absolutely zero impact on whether the code will actually get pushed to production. It’s not functional, simply informational, and might as well be a post-it note with “TO DO: Don’t!” on it for all the security it offers. This is why I almost covered my monitor in Coke Zero Sugar, because it’s the coding equivalent of a slapstick gag.

She checked that the endpoint actually worked, then started to probe it with API calls using CURL. She was able to add messages to the chat with information she found in the code. Then she tried to delete one and was blocked. But she went back through the code, found something interesting, traced it to another outdated package with a Prototype Pollution vulnerability, and was able to use it to inject code that gave her the ability to delete chat messages.

Note, she only deleted her own messages. Remember the tenet of not being destructive.

Wrapping Up…

When I did a demonstration of how to code a simple WebSockets chat at a developer meetup over 10 years ago, my app was so simple that it didn’t sanitize the messages. People were injecting HTML and CSS into their messages and the stuff they got up to was crazy. One of the better known XKCD comics is about a boy with a SQL injection exploit in his name.

It’s easier than you think to write vulnerable code and ethical hacking can help you find it when you have. But remember the four tenets: get permission, avoid illegal stuff, avoid destructive stuff, and most importantly document and report your findings to the system owner and give them time to fix things before you tell anyone else. If you ignore any of those, your hacking is not ethical and you could get in trouble.

If you want to learn more about ethical hacking, our friends at Snyk have a great set of ethical hacking resources for you to explore. And remember, an important bit of hacker recon is to go through public repositories, looking for plaintext secrets in your public code. To help you remedy that, try GitGuardian.