Forem: Uleska

The Challenge of running too many security tools in CI/CD

Gary Robinson — Wed, 11 May 2022 09:26:13 +0000

DevSecOps involves setting up many different automated security tools to cover all bases. It’s not uncommon for organisations to run tons of security tools due to the many types of AppSec scans that are needed to find all the types of vulnerabilities. This means a mix of commercial tools from different vendors, various open-source tools and many homegrown scripts and checks.
This ‘Tool Sprawl’ fragments the DevSecOps automation and holds it back. As the number of security tools grows and the DevSecOps processes become more complex, companies are realising the greater challenge is to make sense of all of the test results within a short period of time, like during the pipeline run.

Automating manual tasks in the process around automated security tools is very important for DevSecOps in CI/CD. The quality and effectiveness of the data from your DevSecOps processes and tech are going to be the difference between people using it and your continuous improvement, or teams simply bypassing it and it all being a waste of time.

Let’s go through the challenges in manual software testing and examine the issues when automating this DevSecOps process.

Choosing which tools to run in CI/CD
Running the various tooling in or out of CI/CD
Collect the test results from all the security tools into one place
Working out what’s changed in this software release
Triaging all your security issues
Prioritisation of your security issues
Communication with various teams
Security Metrics

CHOOSING WHICH TOOLS TO RUN IN CI/CD

We already know you’ll likely need at least 10+ security integrated tools to get enough security coverage for a single project. Chances are, your company will have different teams running different tech stacks. You may have different CI/CD pipelines designed for them. Trying to shoehorn a set of security tools as ‘one size fits all’ will not work for anyone. For example, running python security tools against a C# codeline will be a waste of time and offer a false sense of security.

Different types of tools are better at finding different types of vulnerabilities. Companies are combining DAST, SAST, SCA, IAST, Container, Cloud, and many more types of tools to find issues.

That’s why you’ll need to introduce lots more tooling to cater for the range of security checks varying teams will need. Your DevSecOps tech will not only need to handle those tools but also easily configure which are being used for a team or project. However, trying to configure these within the CI/CD pipeline logic is complicated and rigid.

The Uleska Platform allows you to pre-configure security ‘Toolkits’ that combine relevant security tools according to your projects’ tech stacks and dependencies. This is easily configurable without modifying the CI/CD logic and gives you more flexibility and control over the testing to be done, by simply adding two lines into your CI/CD workflows.

RUNNING THE VARIOUS TOOLING IN OR OUT OF CI/CD

Security tools come in all shapes and sizes and you’ll likely need logic for each and every one to get them to run. So where do you put this logic? Many teams initially try to place it in CI/CD, but as it grows in complexity, it tends to fit better outside the CI/CD platform so it can be updated and extended easily.

Kubernetes and dockers tend to suit the needed architecture of running these security tools well, but creating that setup will become a project in itself. This is why we have built the Uleska Platform architecture on top of Kubernetes and containers. Users can benefit from the sizing, queuing and monitoring of running security tests during the CI/CD builds in real-time - without having to get their hands dirty.

COLLECT TEST RESULTS FROM ALL THE SECURITY TOOLS INTO ONE PLACE

If you were to use three SAST tools in your pipeline, it’s likely that each of them is going to report issues in different ways. As there’s no industry standard on how to report security issues, they may do the following:

Report different fields
Report severity in different ways
Barely report much at all

From those three tools, one consolidated list needs to be retrieved. A call must be made on how to fix the issues at hand and what tools are required. This is where flexibility is vital, as you’ll need to add in more tools.

Essentially, this means implementing DevSecOps vulnerability management. Now your tooling extends to dynamic infrastructure, containers and other types. There’s likely no code fields with them, but now we’re handling port numbers, vulnerable HTTP requests and more.

For some companies, the task of bringing together the results from all of these tools is a full-time job, but one ideal for automation. That’s why the Uleska Platform does exactly that, with a single taxonomy for describing security issues that all results are mapped into.

For consumers of this information, including development and security teams, having a single and consistent pane of glass to flag security issues in near real-time during the CI/CD pipeline is invaluable.

WORKING OUT WHAT’S CHANGED IN THIS SOFTWARE RELEASE

Engineering and operations teams don’t have time to manage hundreds of issues across lots of security tools in every CI/CD run. DevSecOps for software security focuses on the ‘what changed?’ question. Consolidated sets of issues from the last change don’t need to be a War and Peace of every problem in the project. Remember, you’re building an efficient process to flag security issues.

Security tools are built to find every issue they can and report back. Yes, we want a baseline of all issues at the start of the project, but once we do that first triage, remove the false positives, duplicates and non-issues, we then want to move those known issues to our backlog and focus on anything new.

That means there are two things we want to know when security tools are applied to this run:

Have any NEW security issues come up?

New issues can come from code or config changes introducing new bugs or a security tool (or database) has been updated and found a new flaw in our existing containers and libraries.

Have any issues been fixed?

It sounds simple, but if a security tool has found an issue before, someone has applied a fix. Now the tool doesn’t find that issue so we can consider the fix assured and remove it from our lists.

With the right software, such as the Uleska Platform, historical security toolkit runs are stored and can be compared against the current run, so you can know instantly if new issues have been introduced.

TRIAGING ALL THE SECURITY ISSUES

How and where do you record false positives, duplicates and nonsense issues? Some commercial tools allow you to do this, while many don’t. The vast majority of open source and custom security tools can’t track this either.

Yet non-issues are the biggest problem in DevSecOps. Development teams don’t appreciate being given 100s of non-issues from security runs.

Market-leading platforms allow you to set false-positives and non-issues centrally, for all security tools and results it runs. These are sticky, meaning the next time you run the tools, all the non-issues are remembered and don’t show up in any reports, shielding development from these in communications.

Here at Uleska, we’re working on a number of new ways to make false-positive handling much more efficient, so security and development teams don’t need to waste time on this.

PRIORITISATION OF YOUR SECURITY ISSUES

If you’ve ever tried running many tools against the same target, you’ll find a few interesting things:

Not all tools will find all the same issues (you can use this to determine the effectiveness of the tools).
Multiple tools that do find the same issue will likely prioritise it differently.

This difference in prioritisation is not only frustrating, but it’s typically only based on the technical bug and completely ignores risk aspects of the project. It neglects the number of users, the sensitivity of data processed, the criticalness of the project to the business and so on.

Maybe low-risk issues can be passed through without intervention, but the highest top 10% of risks should go back to development for fixes before release. Software like the Uleska Platform provides this automatically within the CI/CD pipeline and further metrics - speeding up the decision making process.

COMMUNICATION WITH VARIOUS TEAMS

We’ve already examined how you can reduce the number of issues being flagged in each CI/CD run, however, there will still be an increase in the number of real issues that are flagged - DevSecOps runs at increasing speed and scale.

Requests to help describe the issue and communicating best practices or suggested fixes already in place can consume a lot of time, taking security teams away from other important tasks and slowing down development.

The Uleska Platform has built-in advisory functions that allow categories of custom remediations to be automatically incorporated into communications to development teams. This gives teams quick access to the advice, fixes, and education they need and allows security teams to scale their communication without consuming their time.

SECURITY METRICS

Even from a metrics point of view, the stakeholders in your company don’t want to have to go to lots of different security tools and make up the statistics. It’s inefficient and again becomes someone’s job, which prevents them from doing something more productive. It can take days to bring all the info together, meaning it’s already out of date.

Automating this during the CI/CD pipeline not only gives near real-time metrics on security, but since it’s being updated every release, you have a lot more granularity on the measurements. Seeing the difference between month one and month three, daily, is a lot better than measuring the difference once a quarter.

To discover more about the challenges of automating DevSecOps and how to overcome them, check out our playbook.

LEARN MORE ABOUT THE CHALLENGES OF DEVSECOPS

The problem with DevSecOps is incorporating many layers of security tasks into the fast-paced software development cycle. Thankfully, there are a variety of things you can do to overcome the challenges faced. In our playbook, we cover the top 10 challenges of automating DevSecOps, while also delivering actionable advice on how to overcome them.

Download it here!

Can DevSecOps Tools Open Security Testing To Everyone?

Gary Robinson — Fri, 19 Nov 2021 11:00:37 +0000

THE EXPERIMENT: JAVASCRIPT SOURCE CODE SECURITY TESTING TOOLS

To begin our experiment, we chose 1,000 Javascript projects on GitHub that were recently updated and had between 50 and 200 stars in rating. Next, we went to UpWork.com and paid non-professionals £5/hour to use the Uleska Platform to automatically run three Javascript source code security testing tools. We consolidated the results generated by the platform to the project teams and made sure they didn’t have security or programming skills.

We didn’t set up any of our cyber value-at-risk enumerations, as they would apply to how an open-source library is used within the end-user application (i.e. types of data processed, quantity of data, environment etc.).

We also made the decision not to triage the results, as the Uleska Platform has functionality to automatically remove false positives and duplicates, a feature heavily used by our customers.

Since this experiment was for open source projects, instead of enterprise software teams, we wanted to see how difficult (or easy) reports containing a mix of real and false-positive issues were for open source teams.

DEVSECOPS TOOLS OUTCOMES

The experiment was a great success, providing a variety of valuable insights, including some improvements we can make to our DevSecOps orchestration usage. Some of our main learnings were:

With simple instructions, non-skilled people were able to onboard hundreds of projects. They were also able to run a number of security tools easily, mainly due to the abstraction the Uleska Platform provided. Instead of starting command lines or setting up profiles, this was done easily by the click of a button. Plus, the onboarding/execution only took a few days.
As operations run so frequently, we discovered ways to speed up our own UI/UX and our API and make it even simpler, aiming for 2-3 clicks to set up a project/application test. The feedback we received from the group showed the team using the Uleska Platform UI to kick off the testing instead of any triggers from GitHub or DevOps tools or continuous integration. However, the effect would be the same.
There were over 35,000 issues registered by the tooling, some of which were false positives, with others acknowledged by the project teams as issues to be fixed. Around 10% of the projects tested didn’t return any issues at all.

In the next stage of this experiment, we will look to automatically remove the false positives.

DEVSECOPS CHALLENGES WITH THIS EXPERIMENT

During this experiment, we ran into some logistical challenges. Firstly, we created new GitHub accounts to find/extract the GitHub URL for the project to pass that into the Uleska Platform so the codeline can be tested.

These new GitHub accounts were also used to update the projects with the report of security issues. This meant these GitHub accounts were now creating GitHub projects or code which were later flagged by GitHub. This meant they were no longer able to submit issue reports to projects. For this reason, we stopped short of the full 1,000 projects, ending at around the 730 project mark.

We also had a few projects react negatively to being passed security reports out of the blue. Sometimes this was because the false positives weren't removed, other times our reports were perceived as spam. Now we understand where we need to improve, in our next iteration of this experiment, we’re looking to remove the likely false positives using the Uleska technology and look forward to helping more open source projects stay secure.

To discover more about the challenges of automating DevSecOps and Uleska can help overcome them, check out our playbook.

HERE: >> https://www.uleska.com/download-our-guide-on-overcoming-devsecops-automation-challenges

DISCOVER HOW TO OVERCOME THE CHALLENGES OF DEVSECOPS

Tell me how would you speed up Pipelines Using Automated Risk-Based Decisions

Gary Robinson — Fri, 15 Oct 2021 15:52:57 +0000

The speed of software development today allows us to get products to market at a faster pace than ever before. This is great because you can get releases out there, see the changes and get the updates. But security is typically being left behind, and there are a number of challenges facing agile security.

Increasing regulations like the New York CRR 500, and BaFin in Germany are enforcing timely verification and risk-based approaches. With a need for more frequent and automatable checks, how do we make sure we understand the impact as well as the vulnerability?

With more and more high profile breaches, clients today demand to know the risks from their software supply chain. This challenges security, development and DevOps teams to shift left, to have software security checks in their SDLC so security can keep pace and have verification measures before the product goes live to customers.

It’s important that we’re not doing security automation for the sake of it, we’re answering the question: Between releases, what has changed in terms of our security profile and the release we’re about to put out?

Why automate security testing?

Security is driven by regulations and standards. A large organisation is going to have numerous security documents that will list lots of technical or procedural controls that a security review should encompass. Even if your organisation doesn’t have those standards, you’ve still got systems like NIST or OWASP like the OWASP Top 10 or the OWASP ASVS (Application Security Verification Standard) which lists about 300 technical controls that check different things. When we say we want to secure something, we’re running verification against controls like these.

It’s a blind approach to simply take our favourite security tool and stick that API into our CI/CD. We probably don’t know what the tool is actually checking out of the 300 controls mentioned and which of those controls apply to us. It’s crucial to be able to pinpoint exactly what these tools are doing for us in terms of automation.

According to our research, even some of the most popular security tools that can be plugged into a CI/CD system only cover a small percentage of source code checks you’d need to do. That coverage grows when you combine automated security tools, such as combining the five example tools (Bandit, SonarQube, OWASP ZAP, nodejsscan and OWASP Dependency Check). This would cover 36% of checks over 87 controls - building up an overlap of controls. This differs depending on the organisation’s tech stack, but one thing to keep in mind, regardless of size, is that some off-the-shelf tools have very good coverage, but because of the nature of the controls, they tend not be able to cover many controls. These controls might be quite specific to your product, like a business logic check, but something specific to authorisation or authentication, that wouldn’t be discoverable by any off-the-shelf tool.

When building a security automation program, a typical organisation will likely need several security tools to get adequate coverage of their security standards or regulations. For teams building out custom checks, one of the challenges they face is making security tools an effective part of the security automation profile.

Challenges of Automated Security Tooling

It may sound ironic, but running automated security tools in an automated security program can have a lot of manual work involved, and getting the tools to interact isn’t so simple. And interpreting outputs and making decisions based on what’s critical isn’t always obvious.

When you bring in multiple tools and custom checks for greater coverage, you suddenly have many different checks happening in the CI/CD cycle. In the background, there will be different types of code and scripts to allow these API interactions. For organisations with hundreds of projects, it becomes unmanageable.

For each tool, teams need to understand the vulnerabilities and how each tool outputs them. Since there’s no standardisation in the industry, how do we get to a single source of truth for all of your security issues?

The tooling can get quite segregated - different teams could be working on different languages in different frameworks. Security programs need to be consistent for your tools to work around the security program, not the other way around.

Again, the crucial thing to know is what's changed from build to build. The challenge here is how, across these different builds across all projects across all the different tools, can we easily and quickly in real-time understand what our backlog was before, and what it is now, and more importantly, what's the related risk so that we can make automated risk-based decisions before releasing our software into the wild when those vulnerabilities become real.

How should we think about risk?

It’s interesting to look at risk versus vulnerabilities. A vulnerability is pretty much static in terms of the technical problem, but the risk that a vulnerability represents to the different types of products can really differ. For example, many tools will flag an SQL injection—a common vulnerability—as a high-priority issue. But not all of these vulnerabilities will be high risk; not all of them lead to important data breaches or even run the risk of being exploited. This vulnerability in an internal project, with no sensitive data, with a few users doesn’t pose a high-priority risk. The same vulnerability in an external project, with lots of sensitive data, will be of high priority (e.g. a project that includes passports or financial data). It’s the same technical vulnerability but has vastly different impacts.

There are a number of risk frameworks to help us think about the overall impact of vulnerabilities to our business. Frameworks like the FAIR Institute is one of them, and it lends itself to measuring risk values against technical flaws in software development.

We’ve implemented this framework inside our product here at Uleska. Whenever any of the tools we run bring back vulnerabilities, we automatically run through that algorithm to produce an impact and risk score. This helps understand across projects and vulnerabilities what the risks are and helps to issues, measure what’s happening and move forward.

How we do it at Uleska

We implement a security pipeline alongside an organisation’s normal CI/CD pipeline. This consistent and repeatable pipeline automates the usually manual tasks in security teams or, more often--as organisations are trying to shift left--in the development teams. It takes the task of running the security tests, kicks off the security tools and brings the results back into one place, giving that single pane of glass view.

Doing these tests in real-time means you get results back into the CI/CD pipeline and can make automated decisions based on risks or vulnerabilities found. The toolkits may differ, but the security process remains consistent.

Running the process across all projects means every metric is captured. We can tie them to OWASP ASVS categories to see the results changing week to week, helping you inform customers and stakeholders about the current state of vulnerabilities and risks, and letting you see trends to inform your processes.

Get up and running for free

What can you do right now with Uleska for free? Everything. The product allows you to use all the risk testing tools you need. The free account only limits your testing tools to one hundred tests in a month, but that's plenty for people to get started with and understand how they can get going from zero with this sort of security automation.

Fitting DevSecOps into CI/CD Pipelines

Gary Robinson — Wed, 21 Jul 2021 21:51:34 +0000

Put simply, the goal of CI/CD pipelines is automation and a key goal of DevSecOps is to alert someone to a problem as early in the automated-delivery process as possible. It’s no wonder it’s a business priority, with many quickly realising DevSecOps is not just about plugging a few APIs into a CI/CD pipeline and calling it a success.

For those entangled in laboursome, traditional security measures, DevSecOps is a breath of fresh air. Security can no longer be secondary, here’s how to incorporate it into your pipeline.

How not to incorporate DevSecOps
Implementation of effective security guardrails
Embracing the agile approach

HOW NOT TO INCORPORATE DEVSECOPS

Failure is part of the tech transformation process. Only through trial and error can you keep innovating. It’s a key part of the journey to DevSecOps.

Don’t let your vision be limited by a silo. We’ve seen so many companies that have automated a portion of the security process in a pipeline realise that other valuable parts of the security process were missed and have caused a failure to make the automation effective.

There’s a number of tasks that would need to be performed in any security testing. Between the time a developer pushes their code to the time it goes live you need to make sure you’ve successfully:

Ran all the relevant security tools against the relevant assets
Collected all the issues together from each individual tool
Triaged the issues to understand any new risks introduced
Prioritised the new highlighted issues
Communicated the issues, stats and data back to the CI/CD process and to any other messaging systems

We’ve seen examples of DevSecOps pipelines that don’t make it as far as step two, by hooking in a number of security tool APIs. What this results in is a mass of issues sitting in a bunch of security tools that no-one is looking at.

If a security vulnerability comes to light and no-one is around to view it - does it even exist?

In effect, it doesn’t.

This is one of the reasons why the recent US Executive Order directly mentions the checking and remediation of security issues before release. See section 4 (e) (iv):

employing automated tools, or comparable processes, that check for known and potential vulnerabilities and remediate them, which shall operate regularly, or at a minimum prior to product, version, or update release;

Similarly, what can cause friction when shoehorning DevSecOps into your pipeline is completely automating vulnerability management, which comes only after security tools have been run manually by a separate security team.

With CI/CD pipelines running in minutes and the response time of overworked, understaffed, security teams being measured in weeks - you’ve guessed it, this tends to not work either.

To keep everything sweet between development and security teams, you need to implement effective security guardrails to provide low friction and low effort automation security checks.

IMPLEMENTATION OF EFFECTIVE SECURITY GUARDRAILS

During the CI/CD pipeline there are a few questions being asked of DevSecOps, which the process and tooling need to be able to answer.

Nearly every security tool out there is built with a different mindset from DevSecOps. They run, find a bunch of issues and return something convoluted back.

Okay, you’ve got some new issues being flagged up, what do you do with that?

Do you go to the security team for input?

No, because that’s a manual step and your aim here is to automate DevSecOps.

You need to develop solutions that don’t overwhelm CI/CD pipelines, yet allow flexibility for differing tech stack, security tools and environments. There’s a bunch of risk prioritization steps you can take to quickly and automatically determine if a new reported issue is a ‘must fix’ or a ‘who cares’.

“Yes, we already know there are already a few hundred issues in this project, we’ve accepted that and it’s in the backlog!“

What we’re asking in DevSecOps is, have the latest changes caused any new and important security issues that we should report and worry about before going live?

To answer this, not only do you need to collect lots of info from typically lots of tools (in whatever format they return their results in), but you also need to work out what the difference is from the last run.

To facilitate the automation of this prioritisation, the Uleska Platform includes cyber risk modules that assign effective risk against every issue returned. This risk prioritisation logic is then configured outside of the CI/CD pipeline logic, meaning it can be changed seamlessly.

The CI/CD pipeline logic can have easy and consistent logic applied to make decisions based on the information returned. It’s just one way we can secure your software at speed and scale.

EMBRACING THE AGILE APPROACH

Traditional security professionals operate in a silo, with capacity limited by the number of security personnel inside it. Scaling manual processes is already an uphill battle so embracing the agile nature of DevSecOps has never sounded so good.

The framework focuses on the leverage of automation throughout the process. Leveraging DevSecOps practices and CI/CD pipelines enables organisations to respond to security and reliability events quickly and efficiently. Producing resilient and secure software on a predictable schedule and budget is every dev's dream.

Always remember, no standard implementation of a DevSecOps environment or a CI/CD pipeline exists that works for everyone.

Automation, armed with security, ensures that the best days of software delivery are ahead of us. Overall, your security arsenal enhances your credibility in the market and builds trust with consumers. With that in mind, there’s no better time to start preempting any challenges you might come up against.

PIPELINES AND PROBLEMS: EMPLOYING DEVSECOPS SUCCESSFULLY

Devising a framework that supports the ultimate goal of adopting DevSecOps practices mature enough to support fully automated CI/CD pipelines is no easy feat on your own.

That’s why we’ve provided practical guidance for software security teams looking to save time when scanning and testing software - all without slowing down the delivery. Download your guide today to prepare for the security challenges your team might come up against.

Read the whole playbook here