Forem: Mackenzie

Millions of .git folders exposed publicly by mistake

Mackenzie — Fri, 16 Dec 2022 13:19:27 +0000

2022 has been the year of source code leaks; Microsoft, Nvidia, Samsung, Rockstar, and many more companies have had their source code* involuntarily open-sourced*. But some new research by CyberNews has revealed that there are millions of private git repositories that are, in fact, not all that private. In this article, we will take a look at the research on exposed git repositories, review why this can be such a problem, and suggest what you can do differently.

Nearly 2 million exposed git repositories

Git is a technology that nearly all software developers use to collaborate and version control their software. You will likely be familiar with git repository hosts like GitHub, BitBucket, or GitLab, which all offer turnkey solutions to sign up and start pushing code to your own repositories and collaborating with others. Git can be a tricky technology and prone to user errors that can result in sensitive information being exposed. For example, when you create a new git repository on your machine, a .git folder is created. This folder contains all the information and meta-data about your project since it was created. If you made an edit from 2012 to your application, 10 years later, that edit is still hidden in that .git folder. If you commit an API key on a development branch 3 years ago, it's still inside this* .git *folder. Basically, unless you are certain you and no one on your team has ever committed anything remotely sensitive, which will be godly if true, this is likely a very sensitive folder.

Research Results

New research from CyberNews has uncovered that huge amounts of these .git folders are not only hosted remotely but also publicly accessible.

In a study to determine how many self-hosted git repositories were, in fact, unintentionally public, they discovered a shocking 1,930,000 repositories that were remotely accessible.

Exposed .git folders by country source

"Having public access to the .git folder could lead to the exposure of the source code. Tools required to get parts or full source code from the .git folder are free and well-known, which could lead to many more internal leaks or easier access to the system for a malicious actor,"\
Martynas Vareikis, a researcher at Cybernews, said.

CyberNews isn't alone in research like this. A smaller experiment was conducted by SDCat, a popular technology and security blogger who scanned 2.3 million domains to find git repositories and discovered

1053 fully or partially exposed git repositories
12 usernames with passwords in the git config data

These research projects, plus the countless related breaches we have had in the last two years show what a huge issue this truly is.

"Even after I parallelized the scanning script it took some days to scan the 2.6 million domains. I did not expect many results, but was surprised how widespread the problem is."\
SDCat

How do .git folders become exposed?

There are many ways .git sprawls into locations they might not be intended. It could be a misconfiguration of a backup or an attempt to host your own git server, but usually, it is a deployment issue. One example that occurred multiple times was with a static website, if someone is using an Amazon S3 bucket to host their site, instead uploading the current version they have uploaded an entire directory including the .git folder. For anyone who understands how sensitive these are it will seem unlikely and shocking that this would happen, but it happens, nearly 2 million times that we know of.

Why exposing git repositories is so problematic?

Git repositories are not designed to contain any sensitive information. They are designed to enable collaboration and sharing of source code between developers and sometimes the community at large. While source code can be a valuable asset to companies, arguably the most valuable asset a company has, source code in itself is often not so valuable to other parties and well-designed applications shouldn't become vulnerable just because their source code is exposed. So why you might ask, is it such a problem that source code gets leaked out? The answer is that source code often contains sensitive information that should not be there. Secrets like API keys, security certificates, and other credentials are very often exposed in source code. You can read lots more about this specific problem in some other blogs (......) but here are the key stats

An average-sized company with 400 developers will have 13,000 secrets (1,000 unique) inside their private repositories
GitGuardian scanned all public GitHub repositories and found over 6,000,000 secrets in 2021
3 out of every 1000 commits GitGuardian scanned contained at least one secret

There is a perfect storm resulting from the fact that git allows such easy collaboration of developers, secrets are meant to be programmatic, and that a git history never dies. While the research project by CyberNews didn't scan each repository for secrets in depth. They did find that 6% of the git repositories had their deployment credentials in the git configuration file..... I'm going to say that again, slowly.

6% of the exposed git repositories had the credentials to deploy their applications, publicly accessible to the world, in the configuration file!

Screenshot of a configuration file with deployment credentials

Often companies ignore the huge problem of exposed credentials inside git repositories because they hide behind the argument that the code is private and therefore shouldn't be exposed. Recent history tells us that this isn't the case.

Last year Twitch's git repositories were found to have a configuration error making them publicly accessible (In a similar way to the CyberNews study discovered) which led to the entire source code for all their projects (even the secret ones) being exposed along with about 6,600 secrets.

Even governments aren't immune to this problem with the Indian government having a massive breach that revealed there were hundreds of exposed git servers that revealed huge amounts of sensitive files including security certificates and even police reports. All because private git repositories weren't actually private.

Source code, nearly always, contains more than just source code. In the history of a project, on often forgotten development branches sensitive information is hidden. This is why even though source code might not be considered a security-critical asset, it needs to be protected and this is why private code repositories that are public are such a big concern.

What can we do?

The answer is obviously to make sure our git repositories are private right?

Well not quite. This research adds to the compelling pile of evidence that git repositories are not appropriate places to contain sensitive information. If your git repository is protected it becomes harder, but not impossible, for a bad actor to gain access to them. In 2021 the supply chain attack of CodeCov meant bad actors got access to up to 20,000 CodeCov users' private git repositories including HashiCorp, Twilio, and Rapid7 even though these were never exposed publicly. We have also seen companies like Uber have their repositories breached due to a compromised developer account. The point is that repositories have been exposed to bad actors as a weak point in our infrastructure and we need to secure them in more than one way.

Make sure they are private, segments between critical and noncritical projects, and developers have two-factor authentication, 2FA, enabled.
Ensure our repositories don't contain sensitive information like secrets in their histories through automated secrets detection and scanning. Tools like GitGuardian help a lot when it comes to this which can be used both on hosted repositories or on self-hosted repositories.
Scan our domains and infrastructure to reveal if we have exposed.git repositories and other critical infrastructure. You can scan your domains and subdomains with many tools such as Amass or dirsearch to name a couple.

Summary

While there is huge evidence to show that git repositories are high-value targets for adversaries, we can add to this evidence that these repositories are easily accessible to attackers via domain and IP scanning searching for .gitfolders. Yes, we must better protect these repositories and scan our own infrastructure for exposed weaknesses but we also must ensure sensitive data like secrets are not our repositories as a minimum effort for security.

The State of Security in Australia: HackSydney and BSides give insight into security post-Medibank and Optus

Mackenzie — Fri, 16 Dec 2022 13:17:39 +0000

Cybersecurity in Australia has moved well and truly into the focus of the mainstream media and the everyday public. This year we saw two catastrophic security breaches with Optus, an Australian telecom provider, and Medibank, one of the largest health insurance providers in Australia. Both breaches saw huge amounts of customers' personal information accessed by cyber criminals and actively used in various different attacks on the public. The reaction from these breaches has been bigger than anyone might have anticipated just a year ago, not just in the cyber communities, but also in the extended population as you hear stories of people's life savings being stolen by criminals conducting phishing campaigns and stealing identities. I recently traveled to the land down under and checked out some of the leading security conferences to see just how this has changed the security landscape in Australia.

I was fortunate enough to be invited as a guest to speak at two leading technical security conferences in Australia, HackSyndey and BSides Sydney. These conferences gave a great insight into the state of cybersecurity in Australia and what we can learn from it.

The underlying issues of security in Australia

In 2021 one of the leading voices in cybersecurity in Australia, Alastair MacGibbon, told 60 Minutes that there would be catastrophic consequences if Australia didn't take security more seriously. It appears that a year later, his prophecy came true.

"As we rely more and more on computers, as we stitch more of our lives and our economy and our society into technology, when that technology fails because of threat actor, it will have catastrophic consequences for us"\
Alastair MacGibbon

Not having been in Australia for many years I was curious about what had changed in the landscape and how it has been affected by recent events. What was clearly apparent is that the industry as a whole is under critical scrutiny, from within the security community extending to government and policymakers. This critical evaluation was best covered by Edward Farrell in his talk "A critical analysis of the Australian cybersecurity industry". This talk called for a reevaluation of the security industry, from the shortage of skills to the incentives set up by security consultants. Many organizations in Australia rely heavily on security consultancy companies to fill the gaps organizations are missing internally. But Farrell, who owns a security consultancy, stated that the standard practices of MSPs have not been effective in solving the issues of security in Australia. He outlined that "The chaos in the industry means old models do not work.......... the industry as a whole doesn't have incentives to solve issues quickly when we are being paid by the hour". He went on further to also challenge the reality of the skills shortage in Australia explaining that despite the hysteria, there isn't a shortage of upper and middle management roles in security, the shortage lies almost entirely at the level of technicians who are not being incentivized or targeted by the same jobs campaigns. Whilst certainly provocative, it is extremely refreshing to hear people, within the security community, challenge how it is set up and outline some failings.

The future of Australian cybersecurity

It doesn't take long talking to the people in attendance at security conferences in Australia to see that the community has really expanded with the mainstream reaction to security today. Something you don't often see at community lead conferences is government personnel, partially from a federal level, but this was a welcome change this year. BSides Sydney which is run entirely by volunteers not only had government representatives in attendance but also leading the conversations with presentations. This included Venessa Ninovic who gave detailed presentations on the latest state of phishing campaigns and data analyst Harriet Farlow who gave compelling insights into how machine learning and AI are being used by attackers. But, government and security are also converging on another level in the form of tough legislation including a new proposed bill that will change the fines of privacy breaches from $2.2million to $50 million.

Another shift that is noticeable is the increased interest in technical conversations around securing infrastructure, in particular API security, and with recent history, it is no surprise. In the case of the Optus breach it was an unsecured API (completely open to the public) that was used to steal millions of customers' data, and in the case of Medibank, attackers used internal APIs to automate the theft of customer data. It will be no surprise then, that there were few seats to be found when API security was the topic being presented. One such talk by Jason Kent "IOCsin your APIs" really took us through a terrifying journey of API security, not just abusing API endpoints to gain data but also how more traditional criminals are using these to locate valuable items in stores to conduct smash and grab type crimes. In a particularly entertaining story, he detailed how they lured criminals into stealing $500 Dyson hair dryers from a store with police waiting for them, by altering data being abused via an API. A talk that was perfectly complimented by Jayesh Bapu Ahire from the HackSydney conference which looked at how to implement API security testing into the development cycle. This one was a clear favorite.

So what is the current state of security in Australia?

So after endless conversations and 50 presentations, what can we learn from HackSydney and BSides about the state of security in Australia? One thing that was clear is the model where organizations rely heavily on security consultancy companies has failed to provide adequate security coverage and adequate incentives to provide better services. But as a result, directly from the biggest security breaches in Australian history, organizations, the government, and the security community are bringing about change. There has been a shift in the attitude of security, the community has grown to include stakeholders previously in the wings and the appetite for technical content has grown. If I could put it in a sentence I would say the state of security in Australia is a state of action and change. Change from within the security community, changes in how organizations will be served by security consultancies, action from organizations to take charge of security, and action from the government to introduce legislation to hold to account those that refuse to take security seriously.

Source Code as a Vulnerability - A Deep Dive into the Real Security Threats From the Twitch Leak

Mackenzie — Mon, 14 Feb 2022 12:56:59 +0000

Please note no information exposed here can be used to harm Twitch and we have provided them with a detailed report outlining the vulnerabilities discovered.

Watch video presentation of this text

Earlier this year Twitch became the latest in a growing list of companies whose source code was unintentionally made public. In this particular case, all source code belonging to Twitch was exposed to the forum 4chan which totalled 6,000 internal Git repositories, 3,000,000 documents with a combined unzipped size of 200GB.

Included in this leak was source code from various Twitch products, secret projects, subsidiaries, internal tools, and employees repositories. As well, there was exposed information from databases including the streamers' earnings.

We spent some time scouring the 6000 Git repositories for secrets and sensitive data and while most of the attention has been on the leaked streamers' revenues, our results show a much more serious problem that extends far beyond just this single breach.

Exploring the Twitch codebase

We found nearly 6,600 secrets inside the Twitch Git repositories. This includes 194 AWS keys, 69 Twilio keys, 68 Google API keys, hundreds of database connection strings, 14 GitHub OAuth keys, and even 4 Stripe keys to name only a few.

Perhaps what is the most troubling aspect of this breach, is that this amount of sensitive data we found isn't actually surprising. Considering the size of the leaked data, the magnitude of secrets we found is reasonable when benchmarked next to what we see in our new customers. We also saw a lot of evidence of security tools usage, indicating some maturity with AppSec showing how baked in the problem of leaked secrets in source code is for all organizations, big and small.

There have been many examples of private code being accessed by an attacker, what makes the Twitch breach unique however is that the malicious actors published source code publicly. This gave us an opportunity to put on our black hats and explore this breach from the perspective of a malicious actor. Doing so we could explore the security vulnerabilities of a real production codebase, starting with sensitive information like secrets.

First some disclaimers. Usually, GitGuardian would validate keys found in a repository to remove false positives. Because of the ongoing investigations that will undoubtedly be going on, GitGuardian decided not to validate any keys as not to mislead the forensics teams. This means we cannot give a percentage of the keys which were valid at the time of the leak.

Gaining initial access to the code

Even today we have not learned a lot about exactly how attackers were able to gather the information and their motives behind it. The only information from Twitch on this matter was below:

"The incident was a result of a server configuration change that allowed improper access by an unauthorized third party."

This doesn't give us too much information other than it is likely they accidentally made their servers accessible to unauthorized users. This could have been a backup server, a Git server, what we know is that the attackers were able to access Git repositories and also some databases.

We know from other examples of breaches there are many ways a bad actor can gain access to private code. In the case of Uber in 2016, attackers were able to gain access via poor password hygiene, in the case of the Indian Government breach attackers found misconfigured Git repositories by fuzzing domains, in the CodeCov breach attackers exploited a Docker image to gain access to code repositories. Point is, Git repositories are becoming high-value targets and it only takes a small crack in the security chain for an attacker to be able to find their way in.

The security implications

When we look at Twitch the main narrative in the media is that this doesn't present a security risk and that no significant customer data was leaked. Our findings directly challenge this, and while no customer information may have been compromised directly in the breach (other than the streamers' income), the vast amount of secrets we uncovered shows that there most definitely are big security concerns associated with this leak.

Out of the nearly 6,000 repositories, more than 1,100 had at least one occurrence of a secret candidate inside them (including their history). What we will look at now, is what an attacker would do if they had access to this information. Remember that in this case the attacker decided to publish the code to 4chan, but imagine a scenario where the attacker did not publish this information and Twitch was unaware of the breach giving an attacker ample time to find and exploit the sensitive information they found.

Simulating an attack

We decided to try and look at this the same way a malicious actor might and try to simulate the steps they would take. In this example, there is so much data to sort through that even though there are thousands of keys, it can be difficult to quickly identify the ones that are most interesting. In our case, we took an approach of:

Identifying and classifying different secrets into categories
Link high-value keys with high-value repositories

Identifying high-value and secondary secrets

The first step as an attacker would be to scan the entire code base for secrets. This can be very difficult depending on the tooling you have access to. Some keys like AWS keys follow specific patterns but most are simply high entropy strings which are much more difficult to find and will produce more false positives. Luckily for us we have the advantage of being able to leverage GitGuardian's powerful detection engine to accurately find these secrets.

Once we have uncovered them, we need to break them into three different categories.

Infrastructure & keys	Data keys	Secondary attack keys
example	example	example
Cloud service keys	Database Credential	* reCaptcha
Payment system keys	S3 bucket keys	* Messaging systems (Slack, Twilio)
	Encryption keys

Depending on your goal as an attacker you may focus on a specific category. If you are wanting to disrupt services, direct traffic, or use computing power for your own evil plan you can focus on the infrastructure keys. If your goal is to steal, encrypt or abuse data then you can focus on accessing the databases. If however, you want to infiltrate the organization further than the keys allow or conduct account takeovers, you can choose to focus on what we are calling secondary attack keys. These are keys that will let you launch targeted spear-phishing campaigns using tools like Twilio to send messages to users, bypass security features like reCaptcha or use the internal messaging system to try and elevate privileges.

In the example of Twitch we have no shortage of keys to uncover to fit whatever evil plan you may have. Once we have found and categorized the keys, however, we want to try and filter out the false positives. One simple way to do this is to validate the keys using a script that calls the backend. In our case, we opted not to do this so we don't disrupt the forensics that may be ongoing.

Therefore we used different methods to sort our findings:

Search for keywords indicating a testing purpose, such as 'Test'
Search for commits that are closest to the leak date.

Using PayPal Braintree keys (a payment service) as an example, we can quickly sort out the keys used in production and the keys used in SANDBOX environments.

Braintree SANDBOX key found in Twitch Data

Braintree Production Key found in Twitch Data

In the example below we can see an AWS key in a config.py file that was committed the same day as the breach occurred, making them high-value candidates for being true secrets.

Because a breach like this has such a large amount of sensitive information, each type comes with a different attack path an attacker might want to take.

Here are some examples: .

Database assignment credentials that could be used to access potentially sensitive information.

Cloudflare keys to disable protections on attacks like DDOS attacks.

Here we can see some reCaptcha keys that would allow an attacker to bypass protections in a DDOS or bruteForce campaign.

And of course a whole heap of credentials that appear to give access to numerous internal systems and tools. Another approach to finding high-value secrets is simply to look for the names of the repositories:

In this example, we can see 14 AWS keys in the repo with the name Backend/CloudServices, which most probably indicates a high-value repository. BINGO! This gives us a great indication of where we should focus our attention and which services we can attack first. All that is left for us, as the attackers, to do now is launch our evil plans starting with the most likely secrets to yield an immediate result and then working down to longer, more complicated attack paths using secondary attack keys.

The larger problem

Now it may appear here as if the core of this problem was because the source code of Twitch was leaked, but actually, this is not true. Source code getting into the wrong hands is a frequent event, this is because source code is a leaky asset, it transits through many servers, workstations and networks making a compromise not an unlikely event. Perhaps much worse than your code being made public is an attacker gaining access and not keeping quiet, working silently to exploit the secrets inside. In fact, the person or people that leaked this code likely did Twitch a favor as there were probably actors with more malicious intent who also discovered the misconfigured server.

The true security problem is not our source code being leaked, it's that our source code in and of itself is a vulnerability yet we do not acknowledge this until a security event forces us to. After diving deep into the Twitch source code, I can state again that Twitch wasn't an organization that didn't take security seriously or didn't show signs of maturity from an application security perspective. The problem is not on an organizational level, it is on an industry level. We must all assume our source code will be leaked and implement security measures to make sure our source code isn't a vulnerability.

Preventing secret sprawl

Secret sprawl is the unwanted distribution of secrets which is evident in the Twitch example. The simple truth is that it is extremely difficult to prevent secret sprawl entirely because it falls down to human error and the fact the secrets are actually very difficult to detect with high precision and recall (see an entire blog article about this). This does not mean however we cannot identify and take action before a breach happens.

Preventing secret sprawl needs to focus on three areas.

Tools to empower developers to prevent them from committing secrets in the first place;
Tools to detect secrets in real-time when they do leak (because they will);
And then tools that monitor places outside of your organization's control, such as on employees' public Git repositories.

GitGuardian has been taking on this challenge since 2017 and enables everyone within the software development lifecycle to take part in the solution to secret sprawl. GitGuardian Shield is an open-source developer tool taking a shift left strategy to allow the developers themselves to prevent secrets from leaking. Our application security platform for internal repository monitoring allows security and project leaders to make sure repositories are clean from secrets so if source code does leak it doesn't create a new security issue and lastly our public monitoring solution allows organizations to get visibility into how secrets are leaking outside of their scope of control.

Contact GitGuardian to find out how we can help your organization today.

Supply Chain Attacks: 6 Steps to protect your software supply chain

Mackenzie — Thu, 02 Dec 2021 16:11:10 +0000

Over the past few years, we have seen a massive increase in software Supply Chain attacks. What is a supply chain attack? This is a type of cyber security attack where adversaries slip malicious code or components into a trusted piece of software or hardware. The goal of such an attack is the ability to infiltrate organizations that are down the chain of the affected component. The focus on supply chain attacks in recent history has been on compromised hardware, but with a number of high-profile incidents, we have seen the conversation shift towards software. This article will look at software-related supply chain attacks specifically and use recent examples to show what companies can do to protect themselves from such attacks.

The software supply chain
Recent examples of supply chain attacks
- SolarWinds Supply chain attack
- Codecov - Attack through software deployment and testing tools
- EventStream - Attack through software dependencies
How to prevent a supply chain attack
*1. Only use trusted dependencies within your application
*2. Scanning open-source software for known vulnerabilities
*3. Patching intelligently (regularly not instantly)
*4. Segment your network
*5. Implement advanced authentication and least privilege
*6. Keep repositoriees free from secrets
Wrap up

The software supply chain

To understand this type of attack we need to be familiar with the concept that applications today are not monolithic pieces of software. They are built up of hundreds of building blocks, open-source libraries and packages, SaaS tools, deployment systems, cloud infrastructure and so on. This gets even more complicated though because each of these building blocks are also built up from a number of different building blocks. If you think about this in layers, a piece of software can be hundreds of layers deep with each component having dependencies until you get all the way down to the CPU instruction set. There is a phrase you will often hear developers use, “turtles all the way down” . This is an expression describing the problem of infinite regress, referring to the turtles standing on each other's back infinitely. Translated into software, this illustrates the perception of infinitely scoped issues related to software supply chain. some of the issues we face in the software supply chain.

Eric Doerr “The world is getting increasingly more interconnected every day, but in security, we don’t act like its interconnected” Black Hat 2019

What makes a supply chain attack so concerning is that the organizations that are impacted by it, often are powerless to prevent it. You have control over your source code, some control over the components and tools you choose to use but almost no control over the components and tools they choose for you. When you acknowledge that being 100% secure or ‘unhackable’ is impossible, then even when you take incredible care with the elements within your software supply chain, you can never remove the risk of a supply chain attack. However, you can reduce risk and in this article, we will take a look at 6 steps you can take to limit both the risk and damage of such an attack.
Here is a an example of the multiple attack surfaces:

“We all rely on the supply chain being fully immunized but the reality is it's not there” - Jeff Moss Black Hat 2021

Recent examples of supply chain attacks

There are lots of different types of software Supply Chain attacks that target different components to achieve a foothold in an organization. We will look at three recent examples of different types of such attacks to get a better understanding.

SolarWinds - Attack through a trusted system with privileged access

Unfortunately, you cannot write an article on supply chain attacks without bringing up the Solarwinds attack. It has become the poster child for the dangers they represent. In the case of SolarWinds, the attack used IT infrastructure monitoring with elevated privilege. This attack is one of the worst-case scenarios where the compromised system not only was used with tens of thousands of high profile clients, such as within the military and other areas within the government but because the software had admin level access on the networks it was installed on. The target called Solarwinds Orion (we will call it simply Orion from here on out) was compromised as early as September 2019. The first traces of code used by the attackers for testing the exploit were injected into the code in October 2019. It wasn’t until February 2020 when the malicious code, known as Sunburst, was injected into the Orion software and began to be deployed to its 18,000 users in March.

The Sunburst attack within the Orion code granted the attackers a back door into SolarWinds customers, some of the affected customers included Homeland Security, State Commerce and Treasury, FireEye, Microsoft, Intel, Cisco and Deloitte to name a few.

By targeting SolarWinds software and exploiting a security flaw (which is still debated), the attackers were able to gain access to some of the most protected organizations in the world. Many of these organizations were considered ‘unhackable’ (although any security professional worth their salt will tell you this is impossible), or at least had state-of-the-art security protections in place. But this was bypassed by exploiting part of their supply chain that had privileged access they were able to penetrate into these unhackable organizations and remain undetected for a significant amount of time.

Codecov - Attack through software deployment and testing tools

Next, we are going to look at an attack in software deployment tools, specifically the CI/CD environment. If you are unfamiliar with the CI/CD (Continuous Integration / Continuous Deployment) pipeline, it is a process by which software can be automatically tested before it is automatically deployed. This has become a fundamental part of the software supply chain and organizations leverage a number of different tools to test their software, this generally happens each time a new code is pushed to the production branch (among others) of a code repository.

One such tool that was again used by tens of thousands of customers was Codecov, a code coverage tool that lets users know how much of your application was being tested within your CI environment. Because this tool sat in the CI environment, it had access to the secrets (or credentials) the application used when it was being tested. For instance, it needed access to databases, third-party services and code repositories so the application could be tested.

In the example of Codecov, attackers were able to first gain access to Codecov’s private code repository and inject malicious code into their source code. This malicious code was only one line and did something very simple: It took the secrets in the CI environment (such as the git tokens) and sent them to the attacker’s remote server. Or in simpler terms, it took the credentials being used to build and test an application and gave them to the attacker. This allowed the attacker to access the private code repositories of many Codecov customers including Twilio Rapid7, HashiCorp and Monday.com to name a few. Private code repositories have become high-value targets for attackers because they often contain sensitive information including secrets. In some cases, with the right access attackers can even inject malicious code using these tokens.

EventStream - Attack through software dependencies

In the last example of a recent supply chain attack, we will look into open-source dependencies. Nearly all modern software applications use open-source dependencies. A study by Synopsys shows that 91% of modern applications are actually open-source code. Being able to use this open-source code is incredibly helpful as it means engineers don’t need to redesign components for each application they build and instead can focus on their unique value.

If an attacker can compromise a downstream dependency then they can achieve the critical step of initial access. A good example of this is the case of EventStream. This package was a toolkit to make streams easier to implement in Node.js, it was incredibly popular and was used by millions of applications. However, this EventStream itself had dependencies, including a module called FloodMap. In this case, an attacker posed as a maintainer of FlatMap and eventually took ownership over the open-source module. The attacker then injected malicious code into the module, turning it into a weapon to target downstream users. It also gave them a back door into EventStream and ultimately the EventStream users. So the attacker was able to gain access to targets by compromising a dependency of a dependency.

How to prevent a supply chain attack

Now that we understand some different ways a supply chain attack can take place, let's look at how we can harden our supply chain and protect ourselves against attacks. The difficulty with these types of attacks is that while you can do things to secure yourself, a lot of the time you are simply a passenger.

1. Only use trusted dependencies within your application

When choosing what dependencies and modules to use within your application you must make sure you are using software that:
Are well maintained with a continued track record of updates, this makes sure that any vulnerabilities discovered will be researched and patches updated. It also reduces the risk of a rogue maintainer injecting malicious code.
Is not misspelled! There is a type of attack called type-squatting where malicious actors will miss-spell a popular open-source dependency, usually, the module will perform the same job as the original, so it does not create suspicion, but with the added bonus of malicious code as well.

2. Scanning open-source software for known vulnerabilities

Open Source Scanning or OSS software like Snyk or WhiteSource are fundamental in hardening the supply chain. This software will search through the dependencies that your package uses and compare them to their extensive databases of vulnerable packages and versions to see if your application has any known vulnerabilities. If a dependency has a known vulnerability, especially one categorized as critical, then these packages can even automatically update this to the latest safe version in many cases. If however there is no update available, then it is extremely important that you use a different module or package as it is a sign that this software is no longer maintained.

3. Patching intelligently (regularly not instantly)

If you have ever read security articles, usually within that you will see a note telling you to update or patch your systems regularly. This of course makes sense, you want the latest version so that you can be sure any discovered vulnerabilities are no longer threats. While this is true, this does not mean you should update immediately. This may sound counterintuitive but if we take a look at all the examples we have in this article, the one similarity they have is that had the users not updated immediately, they wouldn’t have been affected. Now before you yell at your screen, I am not advocating that we don’t patch, or patch irregularly, but we need to apply thought to this.

Liran Tal, a developer advocate from Snyk said that it takes roughly 60 days for vulnerabilities to be discovered, reported and patched.

This is consistent with the examples given in this article. So with that in mind, you need to create an intelligent patching policy where you update to the latest version immediately if security flaws have been identified in dependent systems and otherwise follow an initial waiting period. OSS software can really help in doing this, it lets you know immediately of vulnerabilities in dependencies and if a patch is available, this means you can maintain good security without automatically updating everything as soon as possible.

4. Segment your network

If an attacker can gain access to your organization through a supply chain attack, they will want to move quickly into different areas of your network. Network segmentation is an effective way to limit the blast radius in case of an attack. It refers to dividing a larger network into smaller sub-networks with limited inter-connectivity between them. By controlling traffic flows between various sub-networks and by restricting attacker lateral movement, network segmentation prevents unauthorized users from accessing the organization’s data. Some common ways to segment a network include:

VLAN network segmentation: This involves creating segments in networks with VLANs or subnets using IP addresses for partitioning.
Firewall segmentation: Firewalls inside a network segment limit the attack surface and prevent threats from spreading. Since this method requires thousands of firewall rules, it introduces considerable complexity and cost to the system.
Segmentation with Software-defined Networking: SDN-based network segmentation supports greater automation and programmability. However, it focuses less on security visibility, and more on network policy implementation.
Micro-segmentation: This technique uses the host workload to enforce a segmented network, and whitelist models to block all traffic except what is permitted.

5. Implement advanced authentication and least privilege

There is a concept in security called Zero Trust. It means that just because someone has authentication credentials, we still don’t trust them. Examples of this include using multifactor authentication and limiting the IP addresses that can access systems. In a supply chain attack, we want to limit the amount of trust that the attacker ultimately has, implement strong authentication using principles of zero trust that can stop an attacker in their tracks.

It is also important to make sure we use the principle of least access, this means that users and also services only have the minimum amount of access to additional data and services. Because an attacker can launch an attack from a trusted system, often even using zero trust principles we cannot detect and block them, however, we can limit the amount of information that they have access to.

6. Make sure your repositories are free from secrets

It has become a classic playbook by attackers to target code repositories and backup servers through these types of attacks. This is exactly what happened in the case of Codecov where attackers were targeting git repositories. This is because these are the perfect initial access point to find credentials for the attacker to authenticate themselves and move deeper into an organization. We have to ensure that these systems are free from sensitive information that can be used by attackers. Namely, these are secrets like API keys and database credentials. Secret Sprawl, the unwanted distribution of these secrets, is a massive problem in modern applications and attackers rely on this to move laterally, elevate privileges and gain access to sensitive data. The simple truth is that even if we take all precautions and implement best practices we cannot always prevent a supply chain attack, so we must ensure we do not leave any gifts for attackers. You can read more on why secrets in git are such a big problem here. Using secret scanning tools like GitGuardian is essential, they will conduct a complete historical scan on the repositories including their history and continue scanning in real-time to ensure you never have any secrets in these repositories.
Get a historical scan of your repositories today

It is also fundamental to implement a secrets management solution to help you securely store, distribute and rotate secrets. This is something that can be seen as only a concern for large organizations but no matter the scale it is fundamental a solution is implace.

Wrap up

Supply chain attacks have been increasing dramatically and the simple truth is that no one is immune from them. The distributed architecture we use to build applications gives attackers the perfect opportunity to be able to target hundreds even thousands of organizations at the same time. This shift has completely changed the economics of the attackers, they can now assign significantly more resources to targeting the supply chain compared to targeting one individual company.
While it is impossible to be without the risk of being a victim in a supply chain attack, we can implement some best practices to reduce the risk of both being a victim and potential damage caused during an attack.

Codecov supply chain attack - Step by step breakdown

Mackenzie — Mon, 21 Jun 2021 14:06:23 +0000

Codecov recently had a significant breach as attackers were able to put a backdoor into Codecov to get access to customers' sensitive data. This article reviews exactly what happened, how attackers gained access, how they used sensitive information and of course, what to do if you were affected.

TL;DR

This breach was done by very sophisticated attackers who exploited a mistake in how Codecov built docker images. They used this to modify a script which allowed them to send the environment variables from the CI of Codecov customers to a remote server. While the attackers could have conducted multiple attacks from there, we can see based on other disclosures that one path they did take was accessing private git repositories from the git credentials in the CI environment, then exploiting secrets and data within. This shows the importance of keeping your git repositories clean and ensuring we don’t use production credentials in our CI environment where possible.

What is Codecov?

Codecov is a code coverage tool, essentially that means they check to see how much of your application is being tested. When we're building modern applications and we're using continuous integration (CI) and continuous deployment (CD) we want to make sure that we have automated tests in place so when we release a new feature, we can be confident that it works as intended and that it hasn't unintentionally broken any features within the application.

Now obviously we want to be able to test every line of code during this process, every function and every feature, but this requires quite mature testing automation and Codecov can help develop that because it lets you know what lines of code aren't being tested in your CI environment.

What happened - quick timeline of events

On January 31st 2021 malicious actors were able to update the bash uploader script in Codecov, they did this by leveraging credentials they were able to export from a docker image (more on this later).

Between January 31st and April 1st the attackers were able to squat inside Codecov and extract all of the environment variables of Codecov's customers

On April 1st it was actually one of Codecov's customers that noticed that the bash uploader had a different hash value to what was published on their website indicating that something was wrong.

Codecov investigated and were able to fix the issue on April 15th after some thorough investigations, Codecov then announced that they had been breached to the public and notified their customers.

Source: https://about.codecov.io/security-update/

So what does this all mean and how does it affect Codecov users and why is this type of attack a concerning trend for other CI tools?

Why is this type of attack Significant?

This type of attack is called a supply chain attack, this is because Codecov sits in your software supply line. And just like a supply chain in the physical world, each part of the chain deals with lots of different goods from multiple different customers. When attackers penetrate a chain in the supply line, they can breach multiple organizations.

Using the example above of an oversimplified modern software supply chain we can follow the different stages of a typical supply chain.

We create or modify our code
Commit and push this code into our repositories
New code goes to CI environment
- The applications is compiled
- We run tests on the application
- We produce reports on how our app performs
Code moves to CD pipeline
- Final changes reviewed
- Staging application deployed
- Production application deployed

Let's focus on the CI environment. We can do alot of powerful automation in this stage to test our application. But how we build applications has changed and we now rely on multiple external services; Databases, Payment systems, Cloud infrastructure……. All these components need to be accessed by the tools within the CI environment so they can build and test the application. For this reason, the CI environment needs to have access to the secrets or credentials that grant access to these systems. Hopefully, if we build a secure CI environment we are using staging infrastructure which is less critical. But it is still very common for production credentials to be used and most importantly, it is highly likely that the CI environment will have access to the git repository, which is known to contain a trove of sensitive information.

So by attacking Codecov the attackers now have access to all the credentials within the CI environment for ALL Codecov customers.

How the attackers breached Codecov

Now we understand why a supply chain attack can be extremely impactful, let's discuss the steps how the attackers were able to breach Codecov.

The attackers exploited an error in how Codecov created their docker images. This process actually allowed the attackers to extract a credential from the Docker image, this credential allowed them to be able to modify their Bash uploader script. A bash script is just a set of instructions similar to what you would write within your bash or terminal, but written out in a programmatic way. They added a single line of code to this bash, which was an additional step to send all the environment variables from the CI to an attacker's remote server. Essentially taking the sensitive information that makes your application run, and giving it to the bad guy. This single line of code was, if I can say so, beautifully executed and hidden on line 525 of a 1800+ line document. Without knowing it’s there it would be extremely difficult to find.

View the entire compromised bash uploader script

Who was affected by this?

Codecov has 23 000 customers/users, anyone that was using the compromised version of Codecov between January 31st and April 1st would have been affected. Large organizations such as Twilio, Hashicorp, Rapid7, Confluent have released their own statements about how this has affected them.

What did the attackers do?#

Because there are so many potential victims, we cannot be sure on all the ways the attackers leveraged the sensitive information they stole. However from the public disclosures we can get an idea. A good example is Twilio.

On April 22, 7 days after public announcement of the breach, GitHub had noticed suspicious activity relating to the Codecov breach and private repositories had been cloned with some Twilio user tokens exposed within these repositories.

While this example is very small in the scale of the breach, it clearly shows one attack path the attackers took.
Compromise Codecov
Use stolen git credentials from bash uploader
Access private repositories using stolen git credentials
Scan repositories for sensitive information and secrets
Exploit secrets

This clearly shows that private git repositories were a clear target by the attackers.

What should you do if you have been affected?

If you were using code carved between January 31st and April 1st then it's very important that you take action now.

Revoke secrets

The first thing that you should do is rotate all your credentials, this means all the credentials your CI environment has access to, even if they are not used in production environments as these can still be used to move laterally. But it also means to revoke access to any credentials that were stored with git repositories or other remote data stores that the CI environment had access to.

Check logs

The next thing is we want to analyze our logs to make sure that we can see any suspicious activity, this will give an indication whether or not the attackers have penetrated into your systems.

Scan git repositories

You should now agree it is very important to make sure our git systems are clean and free of sensitive information. These can be hidden deep in the git history of a project making them very difficult to find. This is why it is important to use detection tools to do this.
GitGuardian has a free version of their detection system which will quickly uncover any secrets, you can sign up here. If you have private shared repositories within an organization then these can only be scanned with the enterprise version. There is however a hack to do this for free. You can sign up for the 30 day free trial and use the time to audit your git history without needing to pay for the tool (but don’t tell anyone).

Add two way authentication for machines

The final step you may choose to take is adding two way authentication for machines accessing secrets. This means that you can grant access to your systems within your CI environment while adding another encryption and authentication step so attackers cannot use these even if they get exposed. This is a great step and fantastic products like Hashicorp vault exist that can do this. Bear in mind, these are often very complicated tools that are costly and complicated to install (even if the underlying tool is open-source). But this will ensure that in the event of an attack like this you are covered.

Is Codecov safe?

This is an uncomfortable question often, but I will provide my thoughts on this.
Firstly it is impossible to reduce the risk of the breach to 0. New vulnerabilities and exploits are discovered every day so there is always a risk that tools within your supply chain will be compromised. The attack on Codecov was clearly conducted by sophisticated attackers and while they were able to exploit a mistake, it was not a trivial exploit.

The other consideration is communication, Codecov were very upfront about the breach and have continued to provide new information. This is a good indication.

While I believe we need to be critical of tools we introduce into our supply chain, we can be certain Codecov have fixed the underlying problem and would have conducted a serious security audit following the breach.

The final comment on this falls back to the customers of Codecov. Of course we expect our vendors to take security measures seriously, but we also need to take responsibility for our own security. This means making sure we don’t use production credentials in our CI environment, ensuring our git repositories are clean and having response plans in place. If we can do this then we can

That's it!

Hopefully you found this article useful in understanding how this attack was conducted and If you have any questions, comments or want to request a breach review, reach out on Twitter to me at @advocatemack or use the hashtag #askmack.

To Secure Today’s Code, It’s Time to Shift Left

Mackenzie — Tue, 30 Mar 2021 06:39:08 +0000

The threat landscape is constantly evolving, and today’s DevOps professionals are under more pressure than ever to safeguard the software development lifecycle (SDLC). Protecting just one part of application security isn’t sufficient, because threats can appear at any stage of the process. To be fully effective, security must be integrated and baked into each step of the development process.

To extend the security they require across all their processes, many organizations are adopting a “shift left” approach. Shifting left encourages developers to take more ownership over security and security principles throughout their projects. It states that security should move from the “right” side, or end of the SDLC, to the beginning or “left” of the process.

Although shifting left might seem to place an additional burden on developers, it ultimately saves time and enables a more strategic approach. Incorporating stringent security best practices into the process frees developers from manually troubleshooting and chasing down bugs, so they can focus on innovating and improving their applications. Fortunately, a variety of free and open-source tools are available today to help developers get out in front of their security challenges.

A Broad Approach to Application Security
Application security vulnerabilities can extend across several phases of the development process, so it is important to remember that they may not all be addressed by just one product.

Ideally, an organization should be able to spot potential security gaps as early as possible in the SDLC, because issues discovered at this stage are easier and less expensive to remediate. However, discovering every vulnerability early in the development process is not always possible.

A robust approach to security will be based on a multilayered strategy that spans several different stages of the SDLC. Several free and open-source tools are available for:

-Static Application Security Testing (SAST)
-Secrets Detection
-Dependency Scanning
-Dynamic Application Security Testing (DAST)
-Integrated Application Security Testing (IAST)
-Run-time Application Self Protection (RASP)

Static Application System Testing

SAST is a commonly-used approach to automatic application security. SAST is also known as “white box testing,” and happens early in the development cycle. This type of testing scans an application's source code to discover any known vulnerabilities. Unlike DAST, SAST does not require an application to be compiled or running to start spotting vulnerabilities, so it can be implemented very early in the SDLC.

SAST also enforces coding standards and guidelines, without executing the underlying code. Several different SAST tools are available, so choose the solution that best fits your technology stack, and ensure that it is properly supported.

For example, a static, self-hosted code scanner can be integrated with CI/CD pipelines and its Docker ready. It should offer flexibility for use as a web-based, CLI, or Python API tool. A robust offering should scan for remote code injection, open redirect, SQL injection, XSS, and other threats.

Several different automated code review tools are also available. Some feature thousands of Static Code Analysis rules. For additional flexibility, choose a solution with support for dozens of multiple modern and legacy languages, to cover your entire project and its continuous development.

Secrets Detection

Secrets such as database credentials, API keys, and security certificates are critical for organizations, because they can provide access to highly sensitive and confidential systems. To protect them, secrets detection services scan source code, files, and logs for hidden secrets. The majority of secrets are high entropy strings, which are designed to appear random. However, most high entropy strings are not secrets; which makes secrets extremely difficult to detect. To spot secrets with a high degree of precision and recall, you’ll need to employ a highly specialized service that utilizes advanced classification algorithms.

Like SAST, secrets detection scans through source code. However, where SAST focuses only on the latest version of an application, secrets detection focuses on the entire history of the project. For example, version control systems such as Git keep track and store all changes to a project. If a source code was to contain hard-code secrets that were removed in later development stages, code reviews and SAST tools could miss those secrets. The secrets in turn could end up in a git repository and become compromised.

Some effective secrets detection solutions scan developers’ repositories for evidence of secrets. An optimal solution should cover hundreds of different types of secrets—from keys to database connection strings, usernames, passwords, and SSL certificates. Advanced tools can detect secrets through a combination of algorithms, including innovative pattern matching techniques. Consider a flexible, interoperable solution that’s fast and easy to configure and integrate with your GitHub account. It should also feature APIs to detect secrets in directories, email clients, Slack channels, or other services.

Dependency Scanning

Dependency Scanning lets you discover security vulnerabilities in your dependencies while you are developing and testing your applications. For example, it can be utilized when your application is using an external resource that may be vulnerable, such as an open-source library.

A variety of dependency scanning solutions are available, and many are free or open-source. Consider a tool that will help make security part of the development process from day one. The right solution should enable you to detect vulnerabilities from within your IDE and native Git scanning so you can test projects within the repositories.

To keep new security issues from passing through the build process and a production environment, consider a tool with a security gate. A security gate lets you test your running environment to confirm there is no exposure to existing vulnerabilities. A good dependency scanning tool should support multiple programming languages, and offer constant tracking of open-source vulnerabilities databases, such as the National Vulnerability Database (NVD).

Dynamic Application Security Testing

DAST, also called “black box” testing finds vulnerabilities in running applications, rather than discovering source code issues like SAST. Utilizing fault injection techniques on an app, it can identify common security problems such as cross-site scripting and SQL injection. It can also zero in on runtime issues that static analysis can’t. For example, DAST can identify server configuration and authentication issues, along with flaws that are visible only when a known user logs in.

When evaluating DAST tools, consider a solution that includes automated scanning for vulnerabilities, as well as manual features for hands-on, expert testing.

Interactive Application Security Testing

Sometimes called “grey box” testing, IAST brings together elements from both SAST and DAST. Most often, it is deployed as an agent within a test runtime environment, such as a .NET Common Language Runtime (CLR) or Java Virtual Machine (JVM). Once deployed, it observes operations or attacks, and identifies vulnerabilities.

One of the advantages of IAST is that it can dig deeper into vulnerabilities than other SDLC tools. Unlike SAST and DAST tools, it can observe the runtime context of applications such as the controller, data layer, presentation view, application server and logic, as well as user libraries and open-source components.

Runtime Application Self Protection

RASP is configured on a server and starts running when an application runs, to detect attacks in real-time. Once an application is up and running, RASP analyzes its behavior, as well as the context of the behavior, to protect it from malicious input. By using the app to continuously monitor its own behavior, your organization can identify and mitigate attacks immediately and automatically, with no need for human intervention.

The right RASP tool should cover major vulnerabilities identified by organizations such as the Open Web Application Security Project (OWASP), such as SQL injection, Server-Side Request Forgery (SSRF), and Cross Site Scripting (XSS). A strong solution will be able to leverage the execution logic of requests to block attacks with minimal false positives. It should also be flexible and easy to set up, with the ability to adapt to your applications’ specific stack, minimizing configuration tasks within your app.

The best time to start securing your processes is now

Security has long been top of mind for developers, and as the pace of new threats accelerates, it’s more important than ever to extend your best practices across your SDLC processes. Although the sheer volume of tools and solutions may seem daunting, taking time to understand how a particular solution will fit into your current workflow will enable you to better minimize risk and innovate faster.

Every application is different, so the tools we’ve discussed should be considered a starting point for applying the level of protection you need. With a bit of research, some due diligence, and the right solution, you can start shifting left immediately, to strengthen security and compliance across your entire development life cycle.

Parler wasn’t hacked: It just lacked the most basic security & privacy measures

Mackenzie — Wed, 13 Jan 2021 16:32:27 +0000

It was not surprising to read reports of a recent data breach of Parler, the right-wing version of Twitter, knowing that it would be a likely target for hacktivists following the capitol breach of January 6th. The most surprising thing after investigating the exploit was the cardinal sins Parler committed when it came to storing data on their application.

Firstly, to squash the misleading information coming out. It was widely reported that Twilio disabled 2FA for Parler which opened a security hole allowing the exploit to take place. This is false. Twillio did inform Parler they were in breach of their terms of service and planned to cancel their account, Parler then deactivated Twillio’s service. Twillio’s later investigation found no evidence that this resulted in a security flaw.
// Detect dark theme var iframe = document.getElementById('tweet-1348719143172927491-457'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1348719143172927491&theme=dark" }

What data was accessed

To make clear what data was accessed, only public posts on Parler were scraped and archived. Now undoubtedly this is not as significant compared to a breach where private data like user passwords or private messages were exposed. But there is still plenty of incriminating evidence of users along with absolutely shocking privacy and security mismanagements by Parler.

// Detect dark theme var iframe = document.getElementById('tweet-1348666166978424832-252'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1348666166978424832&theme=dark" }

What happened

Hacker and digital activist @dork_enby had in December 2020 reversed engineered the Parler API using Ghidra. Ghidra is a piece of reverse-engineering software designed and publicly released by the NSA. Basically it is a tool that helps to dig up the source code of a proprietary program. @dork_enby did not document the Parler API to explicitly archive the site but to be able to query the site to find public information.
Immediately after the capitol attack on January 6th, @dork_enby went into action to preserve what could be incriminating evidence in relation to the capitol attack. When it was announced AWS would be removing the application from their servers, activists worked together with @dork_enby to archive all public information on Parler. This resulted in 56.7 terabytes of data, which included every public post on Parler, 412 million files in all—including 150 million photos and more than 1 million videos.

Parler lacked the most basic security measures that would have prevented the automated scraping of the site's data. It sequentially ordered its posts URLs so that anyone could have easily, programmatically downloaded the site's millions of posts. This is known as an insecure direct object reference. Other social media sites use universally unique identifiers (UUIDs) for URLs where the ID’s have no relation to each other. Simply meaning you can not sequentially access or guess URLs. In addition to this, Parler had no limits placed on the public API’s that would have flagged an exploit like this (something all other social media platforms have).

The other sin Parler has commited was to leave, intact, the meta-data, such as the EXIF data stored within the raw media files. Most interesting of this data was the GPS coordinates. Again this is stripped from all other major social media sites during upload, for obvious privacy considerations.

image of the meta-data
how it was used to determine users location inside the capitol

This data could be used to uncover and bring criminal charges to the individuals involved in the storming of the capitol but also be used to uncover members of far right organizations, including their names, emails and locations.

Now despite some comments and my own title (hah more misleading headlines), this exploit used against Parler is still considered hacking because it was using systems in an unintended way to extract data. But it is not an illegal hack where unauthorized actors illegally breached into systems and services belonging to Parler accessing and or exposing sensitive private data.

Wrap up

So the Parler ‘hack’ as is widely being reported isn’t so much a Hack as it was irresponsible architectural design and mismanagement of public data by a social media provider. They used sequential numbering for post URLs which combined with a public API with no access limits, allowed all public posts to be archived by third parties. They also failed to remove meta-data of media files giving third parties access to information such as some GPS locations. This poor design was used in an exploit to archive all public data from the site.

Biggest security takeaway of 2020 - Don't leak secrets on GitHub

Mackenzie — Mon, 21 Dec 2020 16:28:51 +0000

2020 has been crazy, especially in Security.

We could list all the insane things that have happened this year, but, you were there, you lived through it. In the world of cybersecurity, we have seen some unprecedented malicious activity: Widespread phishing attacks utilizing the fear of Covid, attacks on hospitals, a huge increase in nation-state attacks..... Heck, even Jeff Bezos and Kayne got their Twitter profiles hacked. Picking a takeaway from the year is difficult, to say the least. But towards the end of the year in particular, we saw a massive increase in high profile attacks that were exploited with a common vulnerability. Leaked credentials in git, namely GitHub.

The recent headlines

One of the biggest data breaches we saw this year came from Brazil. Data from 16 million Brazilian Covid-19 patients were exposed online, a list that included the president of Brazil, Jair Bolsonaro. This leak contained a trove of sensitive information such as addresses, names, medication regimes and even medical history.

And more recently, the still-unfolding hack of SolarWinds. The company has acknowledged that hackers injected malware into a software update for its Orion platform, a suite of products broadly used across the U.S. federal government and Fortune 500 organizations to monitor the health of their IT networks. The complete extent of this hack is still unknown as it affects now thousands of SolarWind customers and the repercussions won't fully be known until well into 2021. What we do know right now:

Attackers gained access to SolarWinds update server and injected a small amount of malicious code into an update.
The hackers were able to use the injected malware to breach SolarWinds customers with the update.
Attackers gained access to email communications in the U.S. Treasury and Commerce departments
intrusion also had been used to infiltrate computer networks at the U.S. Department of Homeland Security (DHS).
Up to 18,000 customers (again government and fortune 500 organizations) have been affected by the intrusion.

So what do these two massive security incidents have in common?

Both involved employees leaking secrets into personal public GitHub accounts.

Analyzing the incidents

Analyzing all the breaches this year will be a very long series of articles and include some pretty big names like Starbucks and even Mercedes Benz. But if we take the two high profile examples above, which are possibly the biggest security events of the year. We can paint a picture of what set the incidents in motion.

Brazilian Covid-19 data leak

In this case, the leak came to light after a GitHub user spotted the spreadsheet containing the passwords to government medical systems within a personal GitHub. The GitHub account in question was owned by an employee of the Albert Einstein Hospital in the city of Sao Paolo. The user that discovered the spreadsheet later notified Brazilian newspaper Estadao, which analyzed the data and notified the hospital and the Brazilian Ministry of Health.

Among the systems that had credentials exposed were E-SUS-VE and Sivep-Gripe, two government databases used to store data on COVID-19 patients. E-SUS-VE was used for recording COVID-19 patients with mild symptoms, while Sivep-Gripe was used to keep track of hospitalized cases.

The two databases contained sensitive details such as patient names, addresses, ID information, but also healthcare records such as medical history and medication regimes.

Estadao reporters said that data for Brazilians across all 27 states was included in the two databases, including high profile figures like the country's president Jair Bolsonaro, the president's family, seven government ministers, and the governors of 17 Brazilian states.

SolarWind Breach

Disclaimer: SolarWind has yet to confirm the root cause of the hack that saw malicious code injected into a software update. And there is no way with the current information to know with certainty that a leaked credential was the entry point for the sophisticated and complex attack. But a SolarWind credential giving access to the update server was disclosed, not yet connected.

What we do know is that Security researcher Vinoth Kumar, gained access to a SolarWind FTP update server on 19th of November 2019 as a result discovering credentials to the server within the public GitHub repository of a SolarWind employee. Mr Vinoth notifiedSolarWind public disclosure team via an email (below), one line in particular stand out, "Via this [FTP credential] any hacker could upload a malicious exe and update it with release SolarWinds product", which is essentially what has been reported to have happened. The FTP credentials were exposed within a config file named PurgeApp.exe.config which was committed back in June 2018. This was confirmed by the SolarWinds team in a reply email to Mr Kumar.

The way the attack was carried out was no doubt carefully planned and executed, but attackers first needed to gain entry into the companies systems to be able to embed their malicious code into the software update. The leaked FTP credentials, though not confirmed, is certainly a vulnerability that could have been used by the attackers to achieve this objective.

"If a group of well-funded hackers can succeed in modifying just a bit of code somewhere and getting folks to install it as part of a legitimate software suite, they are gaining insider access to organisations which may be otherwise impenetrable, such as governments." Jackie Singh

Why this is such a predominant issue

Credential theft has long been a known and reported technique for attackers as outlined in the Mitre Att&ck framework. But there are some unique challenges that: git, a shift in software development and even the year that was 2020, have exasperated.

Leaks occurring outside of organizations control

The first thing to point out is that in both the mentioned cases, the secret leak was on employees personal GitHub accounts. Organizations have no authority to force employees to have any security measures in place on their personal profiles. it is after all outside of the companies scope and control. So even if organizations have detection capabilities within their own version control systems (which they absolutely should have), it's no guarantee that an employee won't make a mistake on their own repository. Leaked credentials is a human mistake we are trying to detect, and arguably these are harder to detect than malicious activity. Human mistakes will always happen and there is a huge number of ways secrets could leak:

Pushed to wrong repositories
Temporary code merged with secrets
Private repositories made public with secrets buried in history
Application logs, debug logs and config files committed with secrets

We can't prevent all human errors, but we can detect them and take action on it.

Rapid distribution of teams due to Covid-19

Another aspect that has influenced this issue is the dramatic nature in which many employees were suddenly working from home as we tried to deal with lockdowns throughout the world. Many no longer had the safety bubble of the secure closed network yet office systems still needed to be accessed and developers still needed access to secrets to be able to continue their work. While this is not an issue that cannot be overcome, the rapid rate at which organizations and people have had to adapt meant that the processes, documentation and tools were not able to be properly set up. As a result, more sensitive information was being shared between newly distributed teams and more sensitive information was being stored in multiple locations.

Challenges associated with detecting credentials

Finally, detecting secrets is hard, really hard! API keys are often high entropy strings that are random.

SECRET_KEY = 'wzftctmm@%c#ffp$v04i=mh8su*o!am^6op9+22xt2f2f#yrc*'

But high entropy strings are also used everywhere in code such as UUIDs (Universally unique identifiers), with URLs and even used to store assets. This is all to make the chance of a conflict almost impossible. But this does mean detecting secrets and not other high entropy strings is challenging and needs to factor in many different weak signals and to do this, algorithms need to be trained on massive amounts of data which is expensive in resources and time-consuming.

All this has meant that within the year of 2020, leaked credentials have proven to be one of the biggest security trends we have faced yet.

Mitigation

So how do we tackle the issue of secrets sprawling into git repositories so this isn't the takeaway of 2021? This is a complex issue, particularly at a large scale, but there are some general practices that we can put in place.

Manage secrets using vaults or key managers at an organization level

This first one is the most obvious. Secrets are the crown jewels of every organization. So they need to be tightly wrapped with fine grained auditing. There are many platforms that do this well and all companies should be implementing secure secret management as part of their security infrastructure. What system and how to implement it greatly depends on the organization, but HashiCorp vault has long been the gold standard in doing this. These systems help prevent secret sprawl and are the first step in trying to make sure your secrets do not end up in a git repository for a bad guy to find.

Tool recommendations:
Hashicorp Vault
Aws Secrets Manager
Azure key vault

Implement automated secrets detection within organizations assets

To prevent secrets being used in an attack, we need visibility into all the companies services and systems. Secrets sprawl because they are often not visible. Buried in git history, hardcoded into temporary source code and hidden within internal communication. Identifying the secrets that are sprawled within internal systems will allow you to minimize the risk of a potential secret leaking into a space outside of an organizations control.

Tool recommendation
GitGuardian private monitoring

Implement automated secrets detection on organizations perimeter

As discussed, the incidents originated within employees personal public GitHub accounts. Organizations have no authority to enforce security policies here, but they can monitor them (because the bad guys are). If you have 2 developers working for a company, this is a smaller problem, but if you have 500, even finding these developers on platforms like GitHub is a huge challenge. GitGuardian has some magic where they are able to find and link employees to companies easily and then monitor their repositories for company assets and secrets.

Tool recommendation
GitGuardian enterprise public monitoring

Rotate secrets regularly

Some secrets are easily rotated and some not. The secrets that can be rotated easily should be rotated frequently, a secrets management service like Vault will also help manage this. This way if there is a secret buried in a git repository that is made public the chances of that secret still being an active vulnerability has been reduced.

Set minimum permissions to secrets

The last thing we want to do is set our permissions to always be the minimum feasible. Once attackers have access to a system they are often able to elevate privileges and move laterally between services. So when secrets are made they should also have the minimum scope to ensure that in the worse case scenario if an attacker has a valid credential we can limit the damage and prevent movement.

Wrap up

2020 has been a crazy year for security, and just about everything else. But one of the biggest trends we have seen this year is the number of data breaches and attacks that have happened because of leaked secrets in public GitHub repositories. This is exasperated by the lack of visibility companies have within git repositories outside of their control, the rapid displacement of employees in 2020 due to Covid-19 lockdowns and the probabilistic nature of secrets detection.

While it is not an easy task, we can do things to prevent breaches from leaked secrets. This includes: implementing secret management systems, scanning for secrets within company assets and employees repositories, rotating keys frequently and setting minimal permissions for secrets.

FaaS Security- What you should know before deciding to go serverless

Mackenzie — Thu, 17 Dec 2020 13:03:48 +0000

Serverless architecture is becoming a compelling choice for developers and companies to host their applications. It is easy to see why with its ability to dynamically scale to meet load requirements as well as removing a lot of the complexity with deploying and maintaining applications, sometimes even removing the need for an Ops team. But what are the security considerations we should consider before choosing to go serverless?

What is serverless architecture?
Why choose to go serverless?
Serverless security overview
Security considerations in serverless

What is serverless architecture?

Serverless architecture (also known as serverless computing or function as a service, FaaS) is a software architecture where applications are hosted by a third-party service. This essentially means that your application is broken into individual services, which negates the need for server software and hardware management by the developers.

Why choose to go serverless?

When hosting an application on the internet, as most modern software is, requires some kind of server infrastructure. With options from cloud providers such as AWS, GCP and Azure it is more common today to have a virtual server which does remove a lot, if not all of the physical hardware concerns. But these platforms still require a lot of setup and management when it comes to the operating environment. For complex applications managing and maintaining these environments as well as deployments to them requires considerable resources and often done by a dedicated ops team.

Serverless architecture removes this need and it allows you to focus purely on the individual services and your application. It also means applications can be auto-scaled dynamically depending on the workloads. Developers only ever need to worry about their applications and code.

Who should use serverless?

You should especially consider using a serverless provider if you have a small number of functions that you need hosted. If your application is more complex, a serverless architecture can still be beneficial, but you will need to architect your application very differently.

Serverless security overview

There are pros and cons between all architecture designs and serverless architecture, despite all the comfort and elegance it has, serverless is not without some security considerations. This is not to say it is less secure in nature, but that there are new unique security considerations we must consider when choosing a serverless architecture.

It is true that with serverless architecture we can offload some of the responsibility onto the serverless providers. This includes the responsibility for: securing the data center, network, servers, operating systems and their configurations. However: application logic, code, data and application-layer configurations still need to be robust and resilient to attacks, which is the responsibility of application owners.

(Fig 1.1 Serverless Shared Responsibility Model)

Security considerations in Serverless

Increased attack surface

Serverless functions consume data from a wide range of event sources such as HTTP APIs, message queues, cloud storage, IoT device communications and so forth. This increases the attack surface, especially when such messages use protocols and complex message structures - many of which cannot be inspected by standard application layer protections such as Web application firewalls.

In addition the attack surface can become increasingly more complex in serverless architectures and can be difficult for some to understand. Many software developers and architects have yet to gain enough experience with the security risks and appropriate security protections required to secure such applications

Reduced system visibility

Visualizing and monitoring serverless architectures is still more complex than standard software environments. A lack of visibility can result in breaches going undetected for much longer periods of time.

Mitigation

There have become some recent tools to help visibility over serverless deployment. One such tool Dashbird attempts to help operate serverless applications. It offers failure detection, analytics, and visibility for AWS Lambda-based solutions. If not using AWS infrastructure another company offering a bespoke observability stack for serverless architectures is IOpipe. The platform provides fine-grained and near-real-time visibility into applications built using serverless computing.

Inadequate security testing

Performing security testing for serverless architectures is more complex than testing standard applications, especially when such applications interact with remote 3rd party services or with back-end cloud services such as NoSQL databases, cloud storage, or stream processing services. In addition, automated scanning tools are currently not adapted to scanning serverless applications:

DAST (dynamic application security testing) tools will only provide testing coverage for HTTP interfaces. This poses a problem when testing serverless applications that consume input from non-HTTP sources, or interact with back-end cloud services. In addition, many DAST tools have issues to effectively test web services (e.g. RESTful services) which don't follow the classic HTML/HTTP request/response model and request format.
SAST (static application security testing) tools rely on data flow analysis, control flow and semantic analysis to detect vulnerabilities in software. Since serverless applications contain multiple distinct functions that are stitched together using event triggers and cloud services (e.g. message queues, cloud storage or NoSQL databases), statically analyzing data flow in such scenarios is highly prone to false positives. In addition, SAST tools will also suffer from false negatives, since source/sink rules in many tools do not take into account FaaS constructs. These rule sets will need to evolve in order to provide proper support for serverless applications.

Added Complexity with Secret Management

As applications grow in size and complexity, there is a need to store and maintain "application secrets" -- for example:

API keys
Database credentials
Encryption keys
Sensitive configuration settings One of the most frequently recurring mistakes related to application secrets storage, is to simply store these secrets in a plain text configuration file, which is a part of the software project. In such cases, any user with "read" permissions on the project can get access to these secrets. The situation gets much worse, if the project is stored on a public repository.

In serverless applications, each function is packaged separately. A single centralized configuration file cannot be used. This leads developers to use "creative" approaches like using environment variables, which if used insecurely, may leak information. A common mistake is to store these secrets in plain text, as environment variables. While environment variables are a useful way to persist data across serverless function executions, in some cases, such environment variables can leak and reach the wrong hands.

Serverless deployment also limits ability to control access to secrets, particularly in deployment, leading to increased secret sprawl. If secrets are stored using environment variables - it's most likely that the people who deploy the application will have permissions to access the sensitive data

MITIGATION

It is critical that all application secrets will be stored in secure encrypted storage and that encryption keys be maintained via a centralized encryption key management infrastructure or service. Depending on the service you are using there are different tools to manage this: AWS Secrets Manager (link), Serverless secrets storage project on GitHub (link), Azure Key Vault (link).

It is also crucial to make sure that you have visibility into where secrets may be sprawled. For this, consider GitGuardian which can scan repositories and other locations secrets may be sprawled.

Wrap up

Serverless architecture can be an elegant solution that can reduce the need for dedicated Ops teams to manage infrastructure. Like everything within development it comes with pros and cons. Serverless applications are not by nature less secure, but come with a different set of challenges that need to be addressed before making the decision to go serverless.

What actually happens when you leak credentials on GitHub: The experiment

Mackenzie — Wed, 25 Nov 2020 13:21:46 +0000

The architecture of modern software development has changed. We now rely on hundreds of microservices, SaaS platforms and cloud infrastructure to build our applications. Secrets like API tokens, credentials and security certificates are the glue that connects these services. Think of these as the modern-day keys to the kingdom. Developers that understand how sensitive these keys are, also know that if these keys leaked onto public git repositories it can lead to serious data breaches. But how worried should you actually be if you leak secrets into public git? This is a question that was given a great answer recently by Andrzej Dyjak in a popular experiment posting to Twitter.

What was the experiment

To understand how attackers find and use secrets that they find, Andrezj set up a simple experiment:

Generate a secret
Commit to a public repository
Monitor the results

In order to be able to monitor the traffic from the leaked secret, a canary token was used. This is a great service that allows you to generate tokens for popular services such as Slack or AWS that track and alert when someone is trying to use them either through a webhook or email.

Try Canarytokens for yourself

For this experiment, an AWS token was generated from canary tokens which were then uploaded to public repositories in both GitHub and GitLab.

As it happened

Because this experiment happened across two platforms, we will break down the results separately.

See original timeline and post from Andrzej Dyjk

GitHub

15:27 The token was pushed into a GitHub repository.
15:34 (7 minutes) An email from GitGuardian was delivered to the commit email address alerting of a potential data breach.
15:38 (11 minutes) The AWS token was compromised for the first time.
17:40 (2 hours +) The token was compromised an additional 5 times with traffic coming from Germany, Netherlands, United Kingdom and Ukraine.

According to User-Agents the bots used by the malicious users were Python and Node.js SDKs

GitLab

16:24 The tokens were pushed to GitLab
17:26 (62 minutes) The token was compromised for the first and last time with traffic coming from France.

The attacker used a Python SDK bot according to User-Agent

Results

Firstly comparing the results from both GitHub and GitLab what we can see is that there is significantly more black hat activity on GitHub when we compare it to GitLab. There are two contributing factors to this. The first is most certainly the considerable size difference between the two repository hosts in both users and activity. GitHub boasts 50 million users with 2.5 million commits made per day while GitLab sits at only 1% of this with 500,000 users. The second contributing factor is GitHub's open API which has a read-only feature for all public events. While this API creates powerful community features that can be used for legitimate and helpful applications, it does make it easier for malicious actors to scan code. Although GitLab does have an API, they do not have an API for public events in the same way GitHub does making it more difficult to scan through public code at scale.

This experiment highlights what many security professionals know which is that GitHub and GitLab are both well known to malicious actors as places that contain a trove of sensitive information like secrets. This, of course, is backed up by many breaches including the now infamous data breach of Uber which had an amazon S3 bucket exploited when credentials were uploaded into a public git repository. But, this experiment also highlights something much more important, and that is malicious actors are indiscriminate in who they target. It is a problem that not only affects large corporations but one that also affects everyone, right down to an individual developer. This is why it is not only important to have security tools and infrastructure in place for organizations, but also community tools that alert developers too.

Prevention

Preventing such a data breach can be difficult because it is often the result of human error. While this is difficult to eliminate, there are fortunately tools out there that can reduce the risk and impact.

Check out this great resource for API best practices to help prevent a data breach in the future.

Secrets detection

Leaking secrets is a unique challenge because it is one caused largely from human error, this is very hard to defend against. In addition to implementing best practices, you should also have secrets detection in place. Not just in public repositories but also in private repositories. When it comes to secrets detection you have the options of going for commercial detection methods like GitGuardian, or, open-source solutions like Truffle-Hog.

While open-source projects are appealing, in the case of secrets detection you can get a large number of false positives which can disrupt workflow. They also come without alerting capabilities which means any vulnerability may be missed completely and as the experiment outlined, time is key. Check out this comparison for comparing Truffle-hog and Gitguardian and decide for yourself what way to go. The only thing to say for sure is to make sure a protection layer is in place.

Zero Trust

Zero trust infrastructure is a great way to add another layer of protection, but it in itself is not enough protection. Zero trust outlines that even someone with valid authentication credentials needs to still prove they are an authorized party. This can be done with IP range validation and 2-factor authentication for example. This, unfortunately, is not possible with all services and not a failsafe, but used within a strategy can prove to be effective. Read more about zero trusts with regards to leaked credentials.

Wrap up

What we can see without question is that malevolent actors are scanning GitHub and GitLab with automated tools to find and exploit leaked credentials. It doesn't matter if you are a fortune 500 company or developer working on personal projects, a leaked credential can affect everyone and it only takes a matter of minutes for attackers to find and exploit vulnerabilities.

To prevent this implement secret scanning within git repositories with automated alerts and where possible implement zero-trust environments.

Detecting credentials in source code: open-source or commercial solutions?

Mackenzie — Tue, 24 Nov 2020 17:50:40 +0000

In modern software development we rely on hundreds, sometimes thousands of different building blocks. The glue that connects all the different building blocks are collectively known as secrets. These are typically API keys, credentials, security certificates and URIs. These are the modern day master keys.They can provide access to cloud infrastructure, payment systems, internal messaging and user information to name a few. Once an attacker has a secret, they can move laterally between systems to uncover additional information and secrets*,* and because they are authenticated, they look and appear like valid users, making it extremely difficult to detect. (Read More)

But even having established how sensitive these secrets are and why they should be tightly wrapped, this next statement may surprise you:

These secrets are sprawled all over the internet, sitting in code repositories in public view.

For the proprietor of the code, these secrets are difficult to identify, but malevolent actors out to find them have developed simple and effective tools to uncover secrets deeply buried and long forgotten in git history.

There are plenty of articles, whitepapers and blog posts on the importance of protecting secrets, for example Hashicorp and GitGuardian have great resources on this topic. Instead, I want to focus on the different tools available for detecting secrets as well as their pros and cons. But of course it is up to you, the reader, to decide which tools will be best to protect your secrets.

Three options for secrets detection

When it comes to secrets detection, you can choose between 3 different approaches:

Building a custom solution in house
Using open-source projects
Using commercial products

Let's run through a few examples.

Building in house detection

For some of us, the problem of secret sprawl poses a perfect problem to unpack. I would be lying if I haven't played myself with building some fun regular expression (regex) scripts to detect sensitive strings inside code. But building a comprehensive reliable secrets detection script is a huge task.

First, you need to decide how to detect secrets. There are two main options for this: using regex to detect fixed string credentials (like Stripe API keys which begin with the same characters), or implement high entropy detection, which casts a large net but brings back a huge volume of results.

Method	Pros	Cons
Entropy: look for strings that appear random	Good for penetration testing, open sourcing a project or bug bounties because it brings a lot of results. These results must be reviewed manually.	Lots of false alerts (it is very frequent to see URLs, file paths, database IDs or other hashes with high entropy), which makes it impossible to use this method alone in an automated pipeline. Some keys are inevitably missed because the entropy threshold to be applied depends on the charset used to generate the key and its length
Regular expressions: match known, distinct patterns	Low number of false alerts. Known patterns make it easier to later check if the secret is valid or not or if this is an example or test key (see Step 2).	Unknown key types will be missing Credentials without a distinct pattern will be missed, which means lots of missed credentials! Think about passwords that can be virtually any string in many possible contexts, APIs that don’t have a distinct format, ...

When using regular expression, you have a very limited scope of secrets you can detect leaving you open to vulnerabilities. Using high entropy method, you will cast a wider net but also need to sort through more false positives. Of course, in an ideal world you want to use both, but then you'd need to build in post-validators that can sift through the results to exclude likely false positives.

If you are building this as an experiment for your personal projects, this can be a fun and exciting challenge. But when you bring in the challenges of detection at scale, you have to consider resources, alerting and mitigation. The challenge can quickly spiral into a huge project.

It is always best to first learn from a real-life example, I would encourage anyone going down the path of building a secrets detection solution to first read about how SAP built its internal secrets detection solution.

If you are fixed on building a personal solution, I would have to advocate for beginning with one of the many open-source projects available to build upon. I know this can be less exciting than a personal challenge, but when you begin to unpack the scope of the problem, it will save you a ton of work.

Using open-source tools

Open-source tools are not just a good starting point for building your own custom decision patterns, but there are actually also great projects available that provide immediate value with minimal setup.

Popular open-source tools

There is a huge list of open source detection tools available on GitHub. Below are a few that are both popular and well-maintained.

Tool	Description
Truffle Hog	One of the popular utilities to find secrets everywhere, including branches, commit history. Truffle Hog search using regex and entropy, and the result is printed on the screen.
Git Secrets	Released by AWS Labs, as you can guess by the name – it scans for the secrets. Git Secrets would help prevent committing AWS keys by adding a pattern.
Gitrob	Gitrob makes it easy for you to analyze the finding on a web interface. It’s based on Go, so that’s a prerequisite.
Git Hound	A git plugin based on GO, Git Hound, helps prevent sensitive data from being committed in a repository against PCRE (Perl Compatible Regular Expressions). It’s available in a binary version for Windows, Linux, Darwin, etc. Useful if you don’t have GO installed.

Pros and cons of open-source tools

Pros	Cons
Ability to define custom detectors	Slow performance at scale
Can be installed locally canceling the need for a third party investigation	Large number of false positives can result in disruptions to the workflow
Typically supported and maintained by interested parties (Yelp for example)	Hard to enforce throughout an organization
	No alerting features and cannot be integrated into a SEIM
	Does not allow for team collaboration and incident investigation

While the detection reliability and efficiency vary between solutions, the detection systems all lack enterprise features such as alerting, audit trails and in-depth investigation.

Open-source solutions, in my opinion, are best used for bug bounty and one-off pen testing exercises where high volumes of positive results can be sorted through and evaluated. When these systems are put in place in regular production, particularly within organization, the results can be overwhelming and extremely restrictive to the workflow. That being said, there are still some clear advantages over commercial systems in some situations.

Using commercial tools

Along with many high profile cases of secrets being discovered inside git repositories including Uber, many vendors have come to the party with solutions to combat this.

From the many conversations around secrets detection, the biggest concern is vendor trust. You are essentially allowing a third party to find and detect the most sensitive information that you or your organization own.

Many vendors, including the big players like GitGuardian, do offer an on premise version of their products. But this comes usually with an enterprise license which is costly for developers and smaller companies.

The idea of allowing a third party to scan for secrets inside source code can be concerning, and there are definitely some considerations to take into account. The first is that secrets inside git repositories, private and public, should already be considered compromised. Git provides the perfect platform to facilitate secret sprawl, because code is a leaky asset and git provides no audit log of who has access to it or where it has been cloned. So if secrets exist in code repositories, using a third-party application to scan for them, does not really increase the risk vector.

Commercial vendors also have larger teams and time dedicated to detecting secrets, making them more reliable in large scales but also offers additional enterprise features such as alerting, dashboards to allow investigation and remediation, as well as much easier set-up. All this means that the tool will fit into your workflow much better.

The best example of a comparison between the two most predominant open-source and commercial vendors can be found here.

Commercial secrets detection solutions

While there are additional vendors in the market, below are the four core competitors in the space that are the current market leaders.

Tool	Description
GitGuardian	The most predominant detection solution with both private and public monitoring. With over 200+ secrets supported it has the largest detection capabilities on the market. GitGuardian is a developer first company with excellent support as well as a product completely free for developers and open-source organizations.
GitHub Token Scanning	GitHub offers a commercial token scanning solution that covers 25 different secrets. This is currently only available for commercial clients with an advanced security license at a cost of $80 per developer but does come with other security features.
GitLab secrets detection	GitLab also offers limited secrets detection capabilities which come free on all plans since version 13.3. Currently, they offer detection for 12 secret types by default. You can however add custom rules and regex.
Nightfall	AI-powered scanner to detect API keys, secrets, sensitive information. Watchtower Radar API lets you integrate with GitHub public or private repository, AWS, GitLab, Twilio, etc. The scan results are available on a web interface or CLI output.

Pros and cons of commercial secrets detection

Pros	Cons
Sophisticated detection algorithms with greater number of support secrets	Third party access to source code
Real time detection	Closed source (with the expectation of some GitGuardian products)
Alerting mechanisms built in
Credential validity checks (limited to GitGuardian)
Contextual analysis of the code to reduce false positives
Audit trail of secrets and remediation steps
Role based authentication

Wrap up

Implementing secrets detection should always be part of the threat mitigation strategy of all developers and organisations. There are many available solutions on the market for both open-source and commercial vendors, all with their own considerations. While commercial vendors offer more sophisticated detection without buying commercial licenses, they come with the consideration of needing to provide third-party access to source code. Although open-source solutions are a cost-effective solution, they can provide such a large number of false positives they become prohibitive to workflow. Or you can build your own, but beware of the big task ahead of you. But in the end, it comes down to what works best for you and your organisation.

Additional Resources

To continue on this topic, check out the links below.

Secrets Detection Learning Center (GitGuardian)

How to securely use secrets (HashiCorp)

Academic research paper on secrets inside git repositories

Understanding Secret Sprawl (GitGuardian)

Why secrets in git are such a problem

Mackenzie — Tue, 08 Sep 2020 11:07:14 +0000

Secrets in version control systems (VCS) like git is the current state of the world despite widely being recognized as a bad practice. Once source code enters a git repository, it can organically spread into multiple locations. This includes any secrets that may be included within. But why then are secrets in git repositories so common?

*Why secrets end up in git
*Why secrets in git are dangerous
*Real world examples: recent data breaches
*Detecting secrets in reviews (or not)
*Using automated secrets detection in git
*Wrap up

Why secrets end up in git

A seasoned developer may be scratching their heads wondering why anyone may put secrets inside a git repository. But the fact is, secrets inside git repositories is the current state of the world.

Previously I have talked about how it is common to choose the path of least resistance when it comes to accessing and distributing secrets. Git acts as the central point of truth for a project, so it makes sense, at least from a convenience point of view, that secrets are stored inside a private git repository to make distribution and access easy.

But storing secrets like this is playing with fire, it only takes a very small incident to get burnt.

In addition to intentionally storing secrets in git, when secrets are not managed properly, it is very easy to lose track of them. Secrets may be hardcoded into source code, stored as text file, shared over Slack or buried inside a debug application log. In addition, developers can be in large distributed teams with access to a plethora of secrets while being faced with reduced release cycles and an ever growing number of technologies to master.

Why secrets in git are dangerous

Source code, we have to remember, is very leaky. Code is copied and transferred everywhere. Git is designed in a way that allows, even promotes, code to be freely distributed.

Projects can be cloned onto multiple machines, forked into new projects, distributed to customers, made public so on and so forth. Each time it’s duplicated on git, the entire history of that project is also duplicated.

Why storing secrets in public repositories is bad will be obvious. They are freely available to everyone on the internet and it is very easy to monitor public repositories, GitHub has a public API to fetch all public commits for example.

But what about private git repositories?

Private repositories don’t publish your source code to the internet openly, but it doesn’t have adequate protection to store such sensitive information either. Imagine if there was a plain text file with all your credit card numbers within it, you hopefully wouldn’t put this into the companies git repository, secrets are just as sensitive.

A few things to consider when storing secrets in private repositories:

Everyone in the organization with access to the repo has access to the secrets within (one compromised account can provide an attacker access to a trove of secrets).
Repositories can be cloned onto multiple machines or forked into new projects.
Private repositories can be made public which can have secrets buried in the git history.

Another important consideration is that code removed from a git repository is never actually gone.

Git keeps track of all changes that are made. Code that is removed - or more technically correct: code that is committed over - still exists within the git history.

Interestingly enough, code is removed from a project at near equal volume that is added. This means that the code within repositories are much deeper than the first layer and secrets could be buried deep within the git history under a mass of commits that have been long forgotten.

https://github.com/hashicorp/vault/graphs/code-frequency

Comment: The contributions graph that you see above from HashiCorp Vault repository is a typical view of a project's history. The regularity you find in project contribution graphs is both surprising and interesting (check out some projects graphs, it seems to be a rule of nature).

Real world examples: recent data breaches

Secrets being leaked into public places happens with surprising regularity.

If you perform a search on GitHub for the commit message ‘removed aws key’, you will find thousands of results. And that's just within public repositories.

https://github.com/search?q=removed+aws+key&type=Commits

GitGuardian detects over 3,000 leaked secrets each day within public GitHub alone, there are thousands of examples for this but below are a couple recent or noteworthy examples.

Publicly disclosed examples of recent data breaches through leaked credentials.

Starbucks Data Breach - January 2020
JumpCloud API key found in GitHub repository

Equifax Data Breach - April 2020
leaked secrets in personal GitHub account granted access to sensitive data for equifax customers

Uber Data Breach - October 2016
Poor password hygiene allowed intruders to access Uber’s Amazon S3 Datastore using an AWS access key posted in a private GitHub repository.

If this seems like an issue for only large companies to worry about, it’s not. Attackers are constantly exploiting personal services through secret keys too. In one example, bad actors scanned GitHub for AWS keys and used them to mine cryptocurrency, leaving developers with thousands of dollars in debt.

Detecting secrets in reviews (or not)

One great advantage of git is to be able to quickly and clearly see changes made and compare previous and proposed code states. It’s therefore common to believe that if secrets are leaked in source code, they of course will be detected within a code review or in a pull request.

Code reviews are great for detecting logic flaws, maintaining good coding practices and keeping code quality high. But they are not adequate protection for detecting secrets.

This is because reviews generally only consider the net difference between the current and proposed state. Not the entire history of a branch. Branches are commonly cleaned before being merged into the master branch, temporary code is added then deleted, unnecessary files added then removed.. But now these files, which are high risk candidates for containing secrets, are not visible to the reviewer (unless they want to go through the entire history of a branch).

Let's walk through the example above. While this is oversimplified it tells a familiar story.

Commit B a file named main.py is added. A new branch is created to add a new function to main.py in commit C, this feature uses an API key so to save time for testing this is hardcoded. Once the feature is working the hardcoded API key is replaced with an env variable and the file is cleaned. Finally, a pull request is made and accepted because the reviewer looks at the net difference between commit B and D, ignoring commit C. Now undetected secrets are buried in the git history of the project.

While this scenario is very basic, add in hundreds of commits and files between master and a development branch and you can see how easy it is to miss secrets in code reviews.

Using automated detection to find secrets in git

Taking in consideration all we have just discussed about secrets inside git, it is clear that this is a problem that will persist and one we cannot solve with human code reviews. While automation is not always the answer, detecting secrets, in particular secrets inside git, automated secrets detection is a clear solution to this widespread problem.

Unfortunately detecting secrets in git is not quite as easy as it first seems because of the probabilistic nature of secrets. This makes it hard to distinguish between a true secret and other random-looking strings like database IDs or other hashes.

The good news however, is that GitGuardian has built powerful tools for developers to detect secrets in git. A great dashboard with native GitHub and GitLab integrations, a CLI tool called GG-Shield or you can even build custom your own git secrets scanner using the GitGuardian API.

Implement Automated Secrets Detection in your Git Workflow

Wrap up

Let’s have a quick review of what we have gone through. Git repositories are very common places to find secrets and they remain the perfect incubator for secrets to sprawl into multiple locations. Git keeps a track of a project's history which can be deep making finding secrets difficult. Because of the workflow git creates, it is common for any secrets to be missed during manual checking procedures and automated secrets detection should be introduced into the SDLC.

Forem: Mackenzie

Millions of .git folders exposed publicly by mistake

Nearly 2 million exposed git repositories

Research Results

How do .git folders become exposed?

Why exposing git repositories is so problematic?

What can we do?

Summary

The State of Security in Australia: HackSydney and BSides give insight into security post-Medibank and Optus

The underlying issues of security in Australia

The future of Australian cybersecurity

So what is the current state of security in Australia?

Source Code as a Vulnerability - A Deep Dive into the Real Security Threats From the Twitch Leak

Exploring the Twitch codebase

Gaining initial access to the code

The security implications

Simulating an attack

Identifying high-value and secondary secrets

The larger problem

Preventing secret sprawl

Supply Chain Attacks: 6 Steps to protect your software supply chain

Table Of Contents

The software supply chain

Recent examples of supply chain attacks

SolarWinds - Attack through a trusted system with privileged access

Codecov - Attack through software deployment and testing tools

EventStream - Attack through software dependencies

How to prevent a supply chain attack

1. Only use trusted dependencies within your application

2. Scanning open-source software for known vulnerabilities

3. Patching intelligently (regularly not instantly)

4. Segment your network

5. Implement advanced authentication and least privilege

6. Make sure your repositories are free from secrets

Wrap up

Codecov supply chain attack - Step by step breakdown

TL;DR

What is Codecov?

What happened - quick timeline of events

Why is this type of attack Significant?

How the attackers breached Codecov

Who was affected by this?

What did the attackers do?#

What should you do if you have been affected?

Revoke secrets

Check logs

Scan git repositories

Add two way authentication for machines

Is Codecov safe?

That's it!

To Secure Today’s Code, It’s Time to Shift Left

Static Application System Testing

Secrets Detection

Dependency Scanning

Dynamic Application Security Testing

Interactive Application Security Testing

Runtime Application Self Protection

The best time to start securing your processes is now

Parler wasn’t hacked: It just lacked the most basic security & privacy measures

What data was accessed

What happened

Wrap up

Biggest security takeaway of 2020 - Don't leak secrets on GitHub

The recent headlines

Analyzing the incidents

Brazilian Covid-19 data leak

SolarWind Breach

Why this is such a predominant issue

Leaks occurring outside of organizations control

Rapid distribution of teams due to Covid-19

Challenges associated with detecting credentials

Mitigation

Manage secrets using vaults or key managers at an organization level

Implement automated secrets detection within organizations assets

Implement automated secrets detection on organizations perimeter

Rotate secrets regularly

Set minimum permissions to secrets

Wrap up

FaaS Security- What you should know before deciding to go serverless

What is serverless architecture?