Forem: Daniel

GitOps and Kubernetes – Secure Handling of Secrets

Daniel — Wed, 18 Jan 2023 16:11:28 +0000

Especially in well established companies, it may be the case that a certain way of handling secrets has become established, which is convenient, but not necessarily the safest solution. One example of this is the storage of secrets directly in the CI server. The use of a key management system (KMS) for storing secrets is a more secure solution, but it initially requires a lot of effort for the changeover. That is why KMS are not always used in the Kubernetes environment at present. GitOps with its declarative approach forces a rethink in the handling of secrets.

Kubernetes and Secrets

With Kubernetes, Secrets are basically stored unencrypted in the API server's underlying data store (etcd). This means that anyone who has access to the etcd can also obtain or modify secrets. Therefore, at least these measures should be taken:

Enable encryption of Secrets using Encryption at Rest,
implement a least-privilege approach for Secrets by configuring RBAC rules,
restrict access to Secrets to specific containers by mounting Secrets only in containers where they are needed,
finding a way to get Secrets into the cluster (for example, using the CI server), and
evaluating the use of an external provider for storing Secrets (KMS).

This shows that even without GitOps, secure and appropriate handling of Secrets in Kubernetes is not trivial. This is reinforced by the declarative approach of GitOps, since Secrets must also be stored or at least referenced outside the cluster, namely in the source code management. This enforces a secure handling of Secrets. Most of the solutions described in the following are not only exclusively applicable with GitOps, but are basically applicable when using Kubernetes.

GitOps enforces different handling of secrets

With GitOps, the state of a project is described declaratively in the source code management and the deployment environment continuously synchronizes with Git. The CI server is only used for the build, the deployment is done by the GitOps operator. This approach allows to take automation to a new level, but also leads to the need to find a new way of handling secrets. By storing the entire state in Git, a way must inevitably be found to store secrets in a secure manner in Git, because stored directly and unencrypted in Git offers a large attack surface. For the secure handling of secrets there are basically two approaches:

Encrypted storage of the Secrets in Git
Storing the secrets in a Key Management System (KMS) and referencing them in Git.

Encrypted storage in Git

Sealed Secrets (Operator)

An option that easily works with GitOps is the Operator Sealed Secrets from Bitnami. Secrets encrypted with it can only be decrypted by operators running inside the cluster, not even by the original author. For encryption, there is a CLI (and a third-party web UI) that requires a connection to the cluster. The disadvantage of this is that the key material is stored in the cluster, the secrets are bound to the cluster and one has to take care of backups and operation.

Use of Key Management System (KMS)

When using a KMS, the first step is to choose a suitable tool. Since the major operators of public clouds such as Google, Microsoft, Amazon, etc. usually offer a KMS that is easy to use and Hashicorp Vault has established itself as a solution for on-premises operated clusters, the first step is quickly completed.
The second step is then to integrate the KMS into the cluster. There are several ways to do this:

Running a special operator
Mounting secrets in the file system using Container Storage Interface (CSI)
Injecting a side car into pods
GitOps operator with either native support for KMS or via plugin

External Secrets (Operator)

External Secrets is an operator that integrates external KMS such as Hashicorp Vault or those of the major cloud providers. It reads secrets from the external APIs and injects them into Kubernetes secrets. The operator is a new implementation after the merge of similar projects from GoDaddy and ContainerSolutions.

Secrets Store CSI Driver

Mounting secrets into the file system of pods is another exciting way to go, as the CSI Driver is an official part of Kubernetes. It is developed by the Kubernetes Special Interest Group, making it potentially very durable. The CSI Driver provides providers for popular vendors, which in turn are developed by the vendors themselves.

Hashicorp Vault k8s (Sidecar Injector)

Hashicorp Vault k8s is an operator that modifies pods via a mutating webhook to connect between vault and pod via sidecars (additional containers) to provide secrets. This has the major advantage that no secret objects are created in Kubernetes here. The disadvantage is that this way only works with Vault.

SOPS

Mozilla Secret Ops (SOPS), already known from the time before Kubernetes, offers even more options – at the expense of a more complex configuration. Here, the key material can come from the key management systems (KMS) of the major cloud providers, an own HashiCorp Vault or from self-managed PGP keys. Since it is not Kubernetes-native, SOPS does not include an operator, but there are several different ways to use it with GitOps.

Flux v2 provides native support.
ArgoCD supports SOPS with the vault Plugin.
There is also the helm secrets plugin, which can also be used in ArgoCD with manual configuration.
There is also a third-party sops-secrets operator available.

Example: Hashicorp Vault with External Secrets Operator in the GitOps Playground

You can see first hand and try out the management of secrets with Hashicorp Vault and synchronization into the cluster with the External Secrets Operator in the GitOps Playground.
The GitOps Playground is an open source project for trying out GitOps including a sample application. To the GitOps Playground

Conclusion

The declarative approach of GitOps leads to the need to rethink the handling of secrets. The curse and the blessing is that there are many options that enable secure handling of secrets. Thanks to the different approaches there are suitable operators for different requirements. We would be happy if you post questions, feedback or suggestions for other operators, the GitOps Playground or GitOps in general in our GitOps community.

DevOps Toolchain Cloudogu EcoSystem DevStarter

Daniel — Thu, 18 Aug 2022 15:30:00 +0000

DevOps is a methodology or collection of approaches, practices and methods to deliver new software features faster. Depending on how the previous ways of working were in an organization, adopting DevOps can mean a fundamental change in ways of working and, consequently, can take a long time. It is important to note that DevOps cannot be introduced by simply using certain tools, because tools only support the implementation of the ways of working, the mindset however needs to be developed independently. Nevertheleess, choosing the right tools can help significantly with the introduction of DevOps.

Since working methods and processes are often changed when DevOps is introduced, new tools are usually also required to support these changes and support the automation of processes. That’s why it’s important to have both a flexible infrastructure that allows tools to be added easily and to use flexible tools per se that are intuitive and support cross-functional sharing.

That’s why we’re introducing the Cloudogu EcoSystem DevStarter, a platform that, even in its basic form, runs a variety of tools that users need for DevOps. The following graphic shows you which tools can be used in which DevOps phases.

Easy Redmine (Plan, Code and Operate)

The Easy Redmine project management tool offers, in addition to classic project management functions for Agile and Waterfall, other helpful features:

Issue tracker
Time tracking
Resource planning
Dashboards
Project planning via Agile board, Gantt chart and more.

Also, Easy Redmine offers a help desk extension, making it particularly helpful in the phases plan and code (task planning and processing) as well as operate (help desk).

BlueSpice MediaWiki (Plan and Code)

BlueSpice is an easy-to-use wiki that can be used in any department or task area in a company to document information or collect and discuss ideas. In software development, the wiki can be used to collect detailed requirements in the plan phase so that they can then be referred to during development (code phase). Of course, the wiki can also be used in other phases of the software lifecycle to record information.

SCM-Manager (Code)

The source code management tool SCM-Manager can be used to manage Git, Mercurial as well as Subversion repositories. This makes the tool much more flexible than other solutions that only support one kind of repositories, e,g, Git. This makes the tool particularly valuable for companies that still have Mercurial repositories, for example, but would like to switch to Git. But even if only one repository type is used, SCM-Manager has many advantages:

Variety of integrations with issue trackers like Easy Redmine available.
Easy integration into continuous development processes
Many functions of the tool can also be used via a GUI
Extensive code review process available
Simple operation

The tool is used by teams in the code phase to version the source code of software applications and then use it for builds.

Jenkins (Build and Deploy)

The continuous integration server Jenkins is the central building block of the development pipeline. The tool can be connected to a variety of other tools and used to automate steps. Automated builds as well as deployments of applications are only the most obvious functions. A development pipeline can consist of these steps:

Pulling the current state of the source code from the version management (SCM-Manager).
Performing the build and running unit tests using artifacts (Nexus Repository)
Start of a static code analysis (SonarQube)
Upon successful code analysis, storing the built version (Nexus Repository)
Deployment of the built version

SonarQube (Test)

SonarQube is an open source tool that can be used to perform static code analysis. The tool provides a variety of tests for many programming languages and allows quality gates to be set to enforce minimum code quality requirements. Examples of quality gates are

% coverage of code with unit tests
Rate of comments in the code
Number of (potential) bugs in the code

Nexus Lifecycle (Test)

Recently we have been shown several times the extent that security vulnerabilities in open source components can have. In the case of security vulnerabilities in very popular components, the news even makes it to the big media. For the majority of open source components, however, it can be very time-consuming to keep up to date on possible vulnerabilities. This is where Nexus Lifecycle comes in. The tool analyzes dependencies in your software projects and automatically informs you about known vulnerabilities. It also gives you information about possible licensing restrictions of open source software.

Nexus Repository (Build and Release)

The Nexus Repository tool is an artifact repository management tool that can be used, among other things, to manage and store build artifacts before they are deployed. It can also be used to manage binaries, providing a centralized source of information within the organization and speeding up builds.

Elasticsearch (Monitor)

Elasticsearch can be used to collect and process a wide variety of monitoring data and to use that data for alerting.

Further DevOps tools

As an addition to the tools included in the Cloudogu EcoSystem DevStarter, there are of course other tools widely used in the DevOps environment to enable even better collaboration between development and operations:

Kubernetes, a system for deploying, scaling and managing container applications.
Docker, a tool for isolating applications through container virtualization.
Terraform, an infrastructure-as-code tool that can be used to create, modify and improve infrastructure.
Selenium, a framework for creating automated tests for web applications.
Nagios, a tool for monitoring services in complex IT infrastructures.

Kubernetes least privilege implementation using the Google Cloud as an example

Daniel — Fri, 06 May 2022 14:43:58 +0000

Everyone knows it: granting privileges is always a balance between security, usability and maintenance effort. If permissions are granted very generously, the effort is very low and there are rarely any hurdles to use; however, security is compromised. If permissions are granted sparingly, security is higher, but there are costly processes and a lot of administrative overhead.

Kubernetes offers many possibilities with its Role-based access control (RBAC), which are also extensively documented (https://kubernetes.io/docs/reference/access-authn-authz/rbac/). Unfortunately, however, there are not many practical tips for actual implementation. To break out of this predicament, we have written plugins that allow you to use the Kubernetes-sudo context to get a simple but effective entry point for managing permissions. In doing so, this blog article illustrates how to install the plugins and configure the cluster using a managed Kubernetes from Google Cloud Platform as an example.

Permissions of developers in the cluster

The solution presented here refers to permissions of humans, since applications should basically only have read-only access to their own secrets and the configmap in the cluster. So this is straightforward from the point of view of assigning permissions. However, when it comes to permissions for developers, it becomes much more complex, because people’s tasks and roles can change over time and because permissions can be used to protect people from making careless mistakes. This results in the high maintenance effort mentioned at the beginning.

A solution with minimal maintenance effort is to give all developers the same, extensive authorizations. However, this creates the risk that harmful changes can be made accidentally at any time. Especially in productive environments, this can lead to critical downtimes and even data loss.

That’s why we decided to take an approach that briefly uses additional privileges to execute commands, similar to the sudo command in Linux.

Implementing the least-privilege approach using sudo-context

To implement sudo-style permissions, these things must be implemented:

Using the impersonate feature of Kubernetes
Setting up the sudo context
Granting permissions in the cluster

We will now describe these steps in detail.

Setting up the impersonate feature

The sudo function is Kubernetes’ internal “impersonate” feature of the Kubernetes API. It allows commands to be executed as a different user, group, or service account. The first step is to enable the sudo function in the cluster. There are different ways to do this depending on the use case. How these are finally installed and configured on client and cluster side follows in the following.

kubectl-sudo plugin

With the kubectl-sudo plugin, kubectl commands that require more extensive rights can be executed explicitly as a member of the admin group. This reduces the chance of accidentally modifying or deleting resources on the cluster, for example when running scripts or being in the wrong namespace.

The plugin only works for kubectl, but other tools that use kubeconfig (helmet, fluxctl, k9s, etc.) cannot use it. Here is a simple example of how to use the plugin kubectl sudo get pod.

helm-sudo plugin

In the Kubernetes environment, Helm charts are very important. So, to be able to use the functionality for Helm as well, we have developed a corresponding plugin that can be used analogously to kubectl-sudo. Analogous to the usage of the kubectl-sudo plugin, here is an example for the helm plugin helm sudo list.

sudo context for other tools

Alternatively and for all other tools like fluxctl or k9s there is the possibility to create a sudo context in kubeconfig. This can then be used as follows:

kubectl --context SUDO-mycontext # alternative to kubectl-sudo    
kgpo --context SUDO-mycontext # also works with aliases!     
helm --kube-context SUDO-mycontext   
fluxctl --context SUDO-mycontext     
k9s --context SUDO-mycontext # Changes also in k9s possible ":ctx"

If auto-completion features have been installed for certain tools, they will automatically detect the available contexts and can be selected with Tab.

Attention: When using the sudo context, always make sure to specify the namespace in which a command is to be executed. By default, only the current namespace is stored in kubeconfig when setting up the context. If a command is to be executed in a different namespace, this must be explicitly specified with a parameter: kubectl --context SUDO-mycontext --namespace mynamespace get secret

The sudo context should only ever be passed as a parameter. It should never be set as the active context, as this would grant permanent admin rights and thus undermine the protection against accidental changes. This is analogous to the sudo su command under Linux, with which a user has all permissions and is not stopped from performing risky actions.

However, both kubectl sudo and helm sudo do not require the namespace to be specified each time. Here the commands are always executed in the current namespace of the current context. Therefore, for helm and kubectl, the use of the sudo plugins is preferable.

Setting up the local tools

To create a sudo context this script is available: wget -P /tmp/ "https://raw.githubusercontent.com/cloudogu/sudo-kubeconfig/0.1.0/create-sudo-kubeconfig.sh"
With it, only these steps are necessary to interactively create a kubeconfig for the currently selected context:

chmod +x /tmp/create-sudo-kubeconfig.sh
/tmp/create-sudo-kubeconfig.sh

kubectl --context SUDO-mycontext get pod

If needed, the two plugins already mentioned, kubectl-sudo and helm-sudo, can be installed via bash:

Optional: install kubectl-sudo

sudo bash -c 'curl -fSL https://raw.githubusercontent.com/postfinance/kubectl-sudo/master/bash/kubectl-sudo -o /usr/bin/kubectl-sudo && chmod a+x /usr/bin/kubectl-sudo'    

kubectl sudo get pod

Optional: install helm-sudo

helm plugin install  https://github.com/cloudogu/helm-sudo --version=0.0.2

helm sudo list

Technical realization of the authorization

Now that the prerequisites for using the sudo function have been created on the local computer, the authorizations must be set up. We will show the steps for the technical realization as an example using a managed Kubernetes of the Google Cloud Platform.

RBAC

With the sudo function, we can now slip into the role of users, groups and service accounts to execute commands (impersonate). In order for us to get broader rights through the “impersonate”, they must be authorized using Kubernetes’ Role-based access control (RBAC).

The “impersonate” is implemented by:

kubectl-sudo: kubectl --as=$USER --as-group=system:masters "$@"
helm-sudo: helm --kube-as-user ${USER} --kube-as-group system:masters "$@"
and create-sudo-kubeconfig.sh: as-groups: [ system:masters ]

In an existing k8s cluster, two resources must be created to give users access to the impersonate feature:

A ClusterRole, which allows to use the impersonate feature.
A ClusterRoleBinding to allow individual users or groups to use the previously created ClusterRole.

# sudoer.yaml
# Creates a ClusterRole which allows to impersonate users,
# groups and serviceaccounts
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sudoer
rules:
  - apiGroups: [""]
    verbs: ["impersonate"]
    resources: ["users", "groups", "serviceaccounts"]

# cluster-sudoers.yaml
# Allows users to use kubectl sudo on all resources in the cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-sudoers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sudoer
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: admins@email.com
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: user1@email.com

In the current ClusterRoleBinding, everyone listed there has sudo permissions in all namespaces. It is also possible to use multiple ClusterRoleBindings and thus have permissions only for certain namespaces. This is a good approach if, for example, different teams have separate namespaces. To do this, another attribute namespace: namespace: namespace-name must be listed in the ClusterRoleBinding under metadata.

At first glance, it may look like anonymous changes to the cluster are now possible because a different role is assumed. However, in the audit logs in the Google Cloud Platform, the actual user principal who made a change can be seen. This means that it is possible to trace which user made a change. This works similarly in managed clusters of other cloud providers.

Google Cloud Platform (GCP)

In order for RBAC and the adjustments just described to be effective in the GCP at all, the original authorization per Google Cloud must first be bypassed. People who have the role Owner, Editor or Kubernetes Engine Admin in the GCP are normally allowed to execute everything in the cluster, even if they are not explicitly permitted by RBAC.

Under IAM, therefore, a custom role must be created once in the GCP that only allows authentication to the Kubernetes cluster. This role, which we call “Kubernetes Engine Authentication”, is assigned the following permissions:

container.apiServices.get
container.apiServices.list
container.clusters.get
container.clusters.getCredentials

This role now gets assigned to all users that need access to the cluster. The other permissions are then assigned by RBAC in the cluster. The role can also be assigned to entire groups. The groups are again managed via the GSuite groups. However, to do this, propagation of groups must be enabled when the cluster is created (Google Groups for RBAC). Unfortunately, this setting cannot be activated retrospectively for an already existing cluster. For this purpose, it must be rebuilt.

Conclusion

By using RBAC and sudo context, the effort for maintaining permissions and security are in a good balance. On the one hand, there is no need to maintain permissions for each person and on the other hand, the risk of unwanted changes, e.g. because one is in the wrong namespace, is significantly reduced.

Let’s take this scenario: A developer is testing changes to a deployment in his local dev cluster. During the course of the workday, minor work is done on the productive cluster in the GCP. Now the developer forgets to switch back to his local context and wants to delete the test deployment at the end of the day. Something like this has certainly happened to some people before. This can lead to downtimes or in the worst case to data loss.

However, if changes can only be applied to the production cluster using the sudo context or SUDO plugin, the accidental deletion will fail and the developer will notice his mistake. So, the vulnerability to accidental errors becomes less while maintaining ease of use, ease of implementation and high security. Since we have been using RBAC and the associated sudo context ourselves at Cloudogu, we have been working much more securely on our Kubernetes clusters.

GitLab DevSecOps Report 2021 - Proactively prevent vulnerabilities

Daniel — Mon, 20 Sep 2021 10:40:20 +0000

Web security or security-aware software development should no longer be a luxury. That's why terms like DevOps or DevSecOps have become an integral part of our industry. In other words, agile software development that is focused on security is one of the most important approaches to modern development. Or is it?

What does GitLab's DevSecOps Report 2021 have to say about this?

In it are some very interesting findings for the importance of security in software development:

99% of applications contain at least 4 vulnerabilities, 80% even have more than 20.
More than 90% of participants say that security scans run for more than 3 hours, with about a third running for more than 8 hours.
For more than two-thirds of participants, it takes more than 4 hours to fix a vulnerability.

These numbers show two things:

All applications contain vulnerabilities, although automated tests are already used to prevent them.
It is quite costly to fix vulnerabilities.

In addition, the report shows that a large percentage of companies have already been victims of successful attacks:

More than 70% have lost critical data.
Two-thirds have experienced operational disruptions and
More than 60% have seen negative impacts to their brand.

Based on these serious impacts, we might assume that security is becoming a higher priority. However, nearly 80% of DevOps teams reported just the opposite, saying they were under pressure to shorten release cycles. As a result, more than 50% of organizations reported sometimes skipping security scans to meet deadlines.

Preventing cyber-attacks and IT vulnerabilities

These results show that companies are in a dilemma: meeting deadlines or repercussions from successful cyber-attacks against themselves or their products. The simple solution to this would be to simply value security over new features. But nothing is simple when you constantly must innovate to succeed in today's fast-paced world.
Another solution to the dilemma is to equip development teams with the knowledge and tools to prevent security vulnerabilities from the start, when the code is first written. There are several ways to do this.

Continuing education in any form to proactively improve IT security.

There are a variety of different offerings in the area of in-person training: Training courses, eLearning, micro-learning, self-study, competitions, etc. Each of these forms of learning has a right to exist, as everyone has different preferences and strengths when it comes to learning. In addition, the different forms of learning have advantages with different levels of prior knowledge. Often a combination is very helpful. For example, in a classic training course, the basics can first be learned, which are then internalized through micro-learning or a competition. The important thing is to bring the training to the developers and not the other way round.

Classical training courses have, among other things, the advantages that they impart knowledge in a short period of time without distractions and that individual questions and requirements can be addressed. A disadvantage is that they often do not take place directly.
eLearning offers the freedom to work on the learning content at one's own pace, even in between. However, this often leads to the problem that continuing with lessons can easily be lost in the daily work routine alongside other tasks.
The situation is similar with micro-learning, in which learning content is broken down into small modules and ideally integrated into the daily work routine in a context-related manner. An example of this is the Secure Code Warrior plugin for SCM Manager (see below). The contextual integration of learning content has the advantage that the learning units are not in competition with other tasks because they are integrated into the tasks.
In self-study there is no fixed curriculum. This has the advantage that developers only acquire exactly the knowledge they really need. The disadvantage is, that all content must be researched independently.
At first glance, competitions are only suitable for deepening existing knowledge. However, they also offer the opportunity to gain new knowledge by working on problems that are new and have to be solved in a creative way.

Learn more about the free tournament here.

Micro-learning: improving safety through continuous and contextual learning

Contextual learning offers the opportunity to closely integrate practice and theory to improve learning outcomes. For this purpose, suitable learning content, e.g. in the form of micro-learning, is displayed during the processing of tasks. An example of this is the integration of videos and tasks on security vulnerabilities in the code review process.

Through such integrations, learning content is provided exactly when team members are working on tasks with potential security vulnerabilities. An example of this is the Secure Code Warrior plugin for SCM Manager mentioned earlier.

Conclusion

GitLab's DevSecOps Report 2021 shows that software security, while perceived as an important issue, is prioritized lower than the development of new features in many organizations. This prioritization is unlikely to change much in the future. Therefore, it is necessary to change from a reactive to a proactive approach in order to meet the security requirements while keeping release cycles short. This can be achieved through different types of training.

Shorter release cycles through improved security

Daniel — Tue, 07 Sep 2021 15:55:43 +0000

The world’s reliance on software is already great and will continue to grow. That's why the security of applications will also become increasingly important. This development is further reinforced by the global pandemic, as more businesses and services have increased their online availability. The U.S. Federal Bureau of Investigation (FBI), for example, has reported a 300% increase in cybercrime since the beginning of the pandemic: This shows that with the growing reliance on software and applications, the risk of attacks is also increasing.

Software developing companies are therefore in a dilemma: On the one hand, they have to release new products and upgrades quickly in order to be successful on the market. On the other hand, these should also be secure in order to avoid successful attacks. However, security is often seen as a brake on rapid development - and therefore unfortunately neglected.
Note: Some tests, "penetration tests" for example, can take a very long time (2 weeks) - a contradiction to short development cycles.

This opinion comes from the fact that for a very long time development and security teams worked separately. Development teams write new code and get it ready for release. Before release, the security team looks at the code again and makes security requests before GoLive. This loop sometimes makes the development process extremely prolonged. It can also create tension between teams, as it is seen more as criticism than a security improvement.

DevSecOps for better integration of security in software development

The DevSecOps approach offers an answer to these challenges. It is an evolution of the DevOps approach that allows development and operations teams to work more closely together. In DevSecOps, security teams are integrated into DevOps teams.

From being reactive to being proactive in security.

No matter how teams are organized, in most cases (security) mistakes are corrected reactively. This is usually done in these ways:

Automated tests are written to check for known vulnerabilities.
A bug is reported that was discovered either in manual tests or in production.

The goal should be to detect bugs as early as possible so that the effort to fix it is as low as possible. The shift-left approach does exactly that and has been around for a very long time. However, no matter how early a bug is found in the development process, it always requires that someone from the team has to go back into the old code to correct the bug. Reducing the effort further only works by proactively preventing bugs. For more information about how expensive it is to fix security issues, see here.

Learn to write secure code and not discover vulnerabilities

In order to proactively prevent (security) bugs, developers must be provided with appropriate security guidelines and knowledge about security vulnerabilities. The Open Web Application Security Project (OWASP) has defined 10 measures for this purpose:

Define security requirements
Use security frameworks and libraries
Secure access to databases
Encrypt and escape data
Validate all input
Implement digital identities
Enforce authorization systems
Protect data everywhere
Log and monitor security-relevant events
Handle all errors and exceptions

Learning approaches to increase security in software development

In order for developers to be able to implement these measures, they must of course have the necessary knowledge. There are a wide variety of approaches for this, which can be used for learning depending on the topic and personal preference. The important thing is, that the learning material should not only be easily accessible, but also timely.

Security training in DevSecOps

According to traditional understanding, security is not one of the core tasks of developers, namely the implementation of new functions. That's why training on security topics needs to be fun, challenging, and engaging in order to impart critical knowledge. It is important that trainings are conducted using real code, in the respective language and according to prior knowledge in order to have a direct knowledge transfer.
Since the security awareness of the developers should be strengthened, it makes sense to conduct trainings regularly in small units and not, for example, to attend a training once a year.

Tournaments for more web security

Learning, as mentioned before, should be fun, challenging, and engaging for developers. All of this is achieved through contests where participants compete to win by fixing security vulnerabilities. Such contests can last as little as a few hours or as long as several days. And because the topic of web security is so important to us, we have planned a competition on the topic of security to coincide with the launch of the Secure Code Warrior plugin in our SCM-Manager.

Learn more about the free tournament here.

Automated real-time coaching

Meanwhile, there are extensions for development environments that check the code in real time against specified security policies. This way, developers get feedback on whether or not their code meets security specifications right as they are writing the code. Ideally, they are given direct suggestions for corrections, as with a spell checker. In addition, suitable exercises can also be suggested. This is very much in the spirit of gamification or e-learning.

Assessing the security leak

In order to be able to define measures to improve security, it is first necessary to determine what the current status is. This is the only way to identify weak points. Ideally, the status quo should not be determined only once, but the development should be monitored continuously.
One way to see the current state of knowledge of the team is to introduce badges or awards for developers who have completed certain trainings, for example.

Conclusion

To make software applications more secure without extending the development time, it is necessary to proactively write secure code. To achieve this, traditional structures must be broken down and more responsibility for the security of applications must be transferred directly to the developers. The basic prerequisite for this is that the developers have the necessary knowledge. Fortunately, there are now many interactive and engaging ways of imparting knowledge, such as training, tournaments, real-time coaching and assessments, which provide knowledge with a high level of practical relevance in small learning units.

Scrum vs. Kanban – How to select the best agile methodology for you

Daniel — Wed, 28 Jul 2021 11:56:23 +0000

During the last years it became common to use agile methods in software development. The most widespread ones are Scrum and Kanban. The “State of Agile” survey for 2020 for example found that a vast majority of companies (~75%) uses Scrum or Scrum hybrids. The second place is held by Kanban and “Scrumban” with about 15%. That is why we want to compare those two methodologies.

This post will be about Scrum and Kanban in general. For detailed information about the tools visit for example www.scrum.org or http://limitedwipsociety.ning.com/.

Why Agile?

What are the reasons to use agile approaches in projects? Let’s answer this question with some figures: The State of agile survey found that the greater majority of people (70%) using the agile approach improved their ability to manage changing priorities. Before people start working with agile tools they have expectations about the benefits, and after the projects completion they can say whether their expectations were met, or not. The following image shows the importance of several aspects and whether they improved by using agile methodologies or tools. (The data is from the 8th State of Agile survey, but the results stayed almost the same over the years.)

You can see that it is important for 75% of the participants of the survey to accelerate the “time to market” (blue bar). For 83% of projects this aspect improved by using agile methodologies (yellow bar). This behavior can be seen for all of the shown aspects: important aspects improved for the vast majority of projects. What the survey didn’t talk about, are the reasons why for some participants those aspects didn’t improve.

The Agile Manifesto

There are a lot of agile methodologies and tools that have similarities and differences. All are based on the same principles, which are written down in the “Manifesto for Agile Software Development”. The manifesto states that certain aspects are more important than others and that it is necessary to internalize those facts.

For example the manifesto says that responding to change is more important than following a plan. An important factor for the success of agile projects is that the participants have the right mindset to work in an agile project environment. Another important advice is that you shouldn’t limit yourself to just one tool. Combine aspects from different tools that fit your needs, but be aware of the fact that you are combining several tools.

There is a large number of agile methodologies. Many of them have quite similar approaches and use the same principles. Some of them use a lot of restrictions, others leave a lot of free space. The Agile Manifesto states the basic principles and each methodology uses it’s own set of tools and rules. The challenge is to find the methodology and tools that best fit your needs.

Comparison of Scrum and Kanban

After showing the most common reasons for using agile approaches and their basic principles, we can now compare the two methodologies that are being used the most: Scrum and Kanban. We want to provide a first insight to the mindset of Scrum and Kanban teams and show similarities and differences of the two tools.

It is said that working on projects with agile tools helps organizations to complete projects faster. This is because tools like Scrum or Kanban are process tools – they use transparency to show optimization potential and thereby help to work more effectively. An expectation to the users is to experiment by continuously adjusting to chaning circumstances and by customizing the environment. Another similarity of the two tools is, that they are both not very prescriptive – they provide a framework and leave space to adjust the methodology to your conditions. Nevertheless, Scrum is more prescriptive than Kanban.

Workload

One difference between the tools is, that Scrum limits the workflow by limiting the work in progress (WIP) indirectly whereas Kanban limits the WIP directly. In Scrum the limit of WIP is managed by the workload during one iteration, or sprint. In case of the following board the maximal workload is 4 (there can be a maximum of 4 cards in the WIP column), because there are only four cards. The Kanban board is different in one little detail, the 2 in the WIP column. This limits the WIP to two simultaneous tasks. Therefore it would be allowed to start with task C immediately, but task D can only be started after either B or C are finished. The Scrum team on the other hand is allowed to start the tasks C and D immediately.

Response Time for new Requirements

Another difference of the two tools is the way they respond to new requirements. Let’s say you have the following situation in your Scrum or Kanban project and someone turns up and wants to add task E to the board. In which way do the teams react to the new card?

The Scrum team typically says something like: “Sorry, but we have committed us to A, B, C and D for this sprint. Of course you can add E to the next sprint.”

The Kanban team would say something like: “Of course you can add E to the board. But to do that you have remove either C or D, because the limit of 2 tasks is reached right now. Or you wait until we finished A or B.”

Depending on the timing for the new requirement, in Scrum the response time to new requirements can be something between the full length of the sprint and one day (in case the requirement comes up one day before the new sprint starts). Therefore the average response time is half the sprint length. In Kanban the response time is as long as it takes to get capacity available. This can be instantly (by removing another task) or the time it takes to complete another task.

Comparison

The following tables will show you several similarities, differences and the basic work rules of the two methodologies to point out their strengths and weaknesses. The comparison should also help you to find the approach (or certain aspects) you can use in your projects.

Similarities

Both…
… are pull scheduling systems. This means that the team chooses when and how much work to commit to.

… are based on continuous and empirical process optimization.

… emphasize responding to change over following a plan.

… emphasize responding to change over following a plan.

… are Lean and Agile.

… limit WIP.

… use transparency to dive process improvement.

… focus on delivering releasable software early and often.

… are based on self-organizing teams.

… require breaking the work into pieces.

… help to continuously optimize the release plan based on empirical data.

Work Rules

The different approaches of the two tools can be shown best by showing their basic work rules.

Scrum	Kanban
Split your organization into small, cross-functional, self organizing teams. Split your work into a list of small, concrete deliverables. Sort the list by priority and estimate the relative effort of each item. Split time into short fixed-length iterations with potentially shippable code demonstrated after each iteration.	Visualize the workflow: Split the work into pieces, write each item on a card and put it on the wall. Use named columns to illustrate where each item is in the workflow.
Optimize the release plan and update priorities in collaboration with the customer, based on insights gained by inspecting the release after each iteration.	Limit Work in Progress (WIP) – assign explicit limits to how many items may be in progess at each workflow state.
Optimize the process by having a retrospective after each iteration.	Measure the lead time (average time to complete one item), optimize the process to make lead time as small and predictable as possible.

Differences

To emphasize the differences between the two tools a bit more the following table shows some aspects that are prescribed or optional in the tools.

Scrum	Kanban
Timeboxed iterations prescribed.	Timeboxed iterations optional. Can have separate cadences for planning, release, and process improvement. Can be eventdriven instead of timeboxed.
Team commits to a specific amount of work for each iteration.	Commitment optional.
Uses velocity as default metric for planning and process improvement.	Uses Lead Time as default metric for planning and process improvement.
Cross-functional teams prescribed.	Cross-functional teams optional. Specialist teams allowed.
Items must be broken down so they can be completed within 1 sprint.	No particular item size is prescribed.
Burndown chart prescribed.	No particular type of diagram is prescribed
WIP limited indirectly (per sprint)	WIP limited directly (per workflow state)
Estimation prescribed	Estimation optional
Cannot add items to ongoing iteration.	Can add new items whenever capacity is available.
A sprint backlog is owned by one specific team	A Kanban board may be shared by multiple teams or individuals
Prescribes 3 roles	Doesn't prescribe any roles
A Scrum board is reset between each sprint	A Kanban board is persistent
Prescribes a prioritized product backlog	Priority setting is optional

How to choose between Scrum and Kanban?

As you can see, the two tools have basic similarities, but they are very different in the details. If you want to use one of them, or certain aspects, you should be aware that both are more than just boards with a lot of cards and talking about those cards. Of course, using a board is a start, but there are so much more opportunities to improve a project. If you stick to the rules and tools of the methodologies they can help you a lot. An important thing you should keep in mind is that it is not important to stick to one methodology. You are free to combine aspects, tools and rules of different methodologies to set up your project management system.

So how do Scrum and Kanban look in real life? We will show an example of how projects proceed on the different boards. This again shows some advantages and disadvantages of the two methodologies and can help you to find the solution for your own project.

Examples of Scrum and Kanban in use

The following example represents a project that is less trivial than the one in the previous paragraph. There is a product backlog containing several tasks and a production process that consists of several steps.

In case of Scrum the team is in the first sprint, which consists of the tasks A, B, C, D, E and F. A is already done, B to E are at the moment in progress and F is still to do. During the upcoming sprints the team will commit to the tasks G to N.
The Kanban board consists of the backlog from which maximal 2 tasks can be selected for priorization. The development section allows max. 3 tasks at once (tasks from both columns – ongoing and done – count). After the development the features need to be tested and after that they will be in the production environment and are thereby done. You can see that the board represents the different states of the production process.

The Scrum board will be reset after each sprint, whereas the Kanban board is persistent.

After a few Scrum sprints or simply after some time the two boards could look like this:

Note: For Kanban it is not necessary that there is a maximal amount of tasks for each column.

The Kanban board, which contains more columns than the Scrum board, can help to optimize the process steps, because it is necessary to think about them in order to draw the board. It can also be very helpful to optimize the amount of allowed cards per column, because this can reveal bottle necks. The Scrum board and the partition of tasks into sprints helps to organize tasks and to think about reasonable packages. But remember, it is always possible to combine aspects of different methodologies and tools to find the board, rules and tools that fit your project best.

Other Agile Tools and Methodologies

Besides the two tools Scrum and Kanban there is a huge number of other agile tools and methodologies that serve as an inspiration for you. The more you read about them, the more you will see that they are based on the same principles (which are stated in the Agile Manifesto) and that some use the same approaches and methods. Here is a short list of some wide spread approaches that you could take a look into.

Extreme Programming (XP)
Lean Software Development
Feature Driven Development (FDD)
Crystal Family
Test Driven Development (TDD)

Bottom line

A Kanban board represents the workflow of a project whereas a Scrum board always consists of the same columns. Another big difference is that it is possible to limit the number of cards in certain columns when using Kanban while Scrum limits the WIP by restricting the number of cards in the Sprint Backlog.

Besides the two methodologies that were presented here there are numerous other agile approaches. Some of them are based on Scrum or Kanban. A very widespread approach is a combination of Extreme Programming and Scrum, because both supplement each other very well. It’s up to you to find the methodologies, tools and rules that you want to use to accomplish the aims of your projects. Since there is no “one fits all” for all projects you have to experiment and try variations.

How to define software requirements

Daniel — Mon, 26 Jul 2021 12:03:07 +0000

In order to prevent misunderstandings like this there are 2 important things you need to know:

how to define requirements clearly and
how to keep track of changes during the development.

How to Define Requirements

Depending on the way you work you need to describe requirements in different ways. In Scrum for example you create User Stories, in waterfall projects use cases or other forms of requirement documentation. Regardless of formal aspects you need to ensure that you have all information that are required for the implementation. To achieve this you only need to follow these 5 steps:

Identification of stakeholders: To assure that you get all relevant requirements you need to make sure that you know all stakeholders.
Collect requirements: There are different ways you can find out about requirements, e.g. you can use interviews, case scenarios or prototypes. No matter which method you’re using, you should always find answers to these questions:
- What is the product supposed to do?
- How well should it do it? (e.g. +/- limits, quality, measurable terms)
- Under what conditions? (e.g. environment, states)
Categorization of requirements: To get a good overview of the requirements you should categorize them. The two major categories are functional and non-functional requirements.
Interpretation and recording of requirements: Only well defined requirements can be considered adequately in the product. For each requirement you should…
- … define the requirement in detail.
- … prioritize the requirement.
- … analyze the impact of change.
- … resolve conflicting issues by talking to the stakeholders.
- … analyze the feasibility.
- … specify test cases.
Sign off: Before implementing a requirement you should get the ‘Go’ from the stakeholders.

After you talked to the stakeholders, identified requirements, defined and prioritized them, you have an evaluated list of aspects that need to be considered in the product.

Keep Track of Changes

It is one thing to capture all requirements for a project. The other thing is to ensure that all the requirements are met by the product. The Requirements Traceability Matrix can help you to keep track of all the requirements and associated test cases. The matrix links requirements with derived product specifications and test cases. This is what the matrix looks like:

ID	Priority	Description	Type	Risks	Specifications	Test Cases
Unique identifier	Priorization	A short description	Functional or non-functional	Associated risks	Link to the detailed description	List of associated test cases (unit, integration, system, user tests, etc.)

To make sure that a requirement is truly considered in the product, it is necessary that it is described in the product specifications and that it is associated with at least one test case.

If requirements change you have two different options:

Adjust the requirement specification accordingly and document the changes.
Add the changed requirement as a new item to the list and treat it like a new requirement.

No matter how you’re dealing with changes, the Requirements Traceability Matrix provides an overview of requirements and associated specifications. So if a specification changes, you can easily see which requirements are involved, and vice versa.

Requirements need to be tracked

Finding out about all requirements of a product and ensuring their implementation is the key to a happy customer. Therefore it is advisable to invest enough time into investigating and defining requirements. In addition to that it is necessary to ensure that the final product meets all requirements. This can only be achieved by considering them in the product specifications and by testing the specifications. This can be ensured by using a Requirements Traceability Matrix.

IT compliance in practice – correctly containing and deleting data and projects in B2B software development

Daniel — Fri, 23 Jul 2021 14:22:58 +0000

For many, compliance is an abstract concept that only managers For many, compliance is an abstract concept that only managers have to deal with. But it’s not. Compliance is everyone’s business.

What is compliance and why is it so important?

Briefly, compliance is the observance of laws, rules, standards, contracts and moral values, which can come from both outside and inside a company. The tricky thing about compliance is that non-compliance is not always immediately apparent. However, once it is discovered, things can quickly become unpleasant. Compliance affects not only management, but every employee, since anyone can knowingly or unknowingly violate it in the decisions they make. Examples of this include not using software under the terms of its license or failing to comply with data protection guidelines by not deleting data as required.

A special case: deletion or handing over of data

Since compliance is about adhering to regulations of any kind, this is very complex and individual issue. That’s why this post will be dealing with a very specific topic: the deletion of data when the contract ends.

In B2B software development, it is customary for contracts to contain a clause on how to hand over or destroy all records and documents related to the project.

At first glance, this does not seem to pose a problem. Yet, as is so often the case, the devil is in the details. Depending on how meticulous tidying up at the end of the project is, it can be very time-consuming to actually find and delete all of the information and data.

Inadequate isolation of projects can be a compliance issue

The clause referred to above means that at the end of a project all systems would have to be searched to find all of the data relating to the project. This ranges from obvious data such as repositories, artifacts, Wiki entries, documents such as conceptual designs or documentation to less obvious data such as tickets in the issue tracker or source code stored on secondary systems. How extensive this search is essentially depends on two factors:

The scope and duration of the project
Isolation of data from different projects

The scope and duration of projects

The longer and larger a project is, the more data is accumulated, increasing the likelihood that some of the stored data will be forgotten and that more and more systems will be used. This means that the “search radius” must be considerably expanded. It also means that there’s a lot more to do.

A wide dispersion of data can be reduced, for instance, by organizational rules on the storage of data. But rules like these often provide a lot of leeway and are hence only helpful to a limited extent.

Isolation of projects

In addition to the size and duration of a project, its isolation from other projects is also a factor influencing what has to be done to identify all of the relevant data. Here is a simple example of different levels of project isolation:

Low isolation: The data from different projects are all stored on one network drive. There is no prescribed folder structure.
Medium isolation: All of the data is on the same drive, but each project has its own folder.
High isolation: Data from different projects are stored on their own network drives with separate hard disks.

It is immediately apparent how much easier it is to identify the data in a highly contained project. If all of the data for one project is stored on a separate hard disk, it can easily be handed over to the customer or erased. In medium isolation, only a few folders need to be copied or deleted. The work involved in a low isolation situation is something everyone is free to imagine on their own.

It is also important to keep in mind that data may be present in any system used for the project. This quickly makes it relatively apparent why even small differences can have a big effect.

Backups – a story on its own

But that’s not all. Backups of systems are created to prevent data loss during a project. Depending on the level of project isolation, what needs to be done at the end of the project can be multiplied again. Because here too, the lower the level of isolation, the more complex the search will be. If the backups are still compressed or encrypted, the task becomes even more time-consuming, and because that is not enough, it must also be remembered that every change made to the backup puts its integrity at risk.

Separate infrastructure for each project

The good news is that it can also be very easy to abide by the contract: Using the Cloudogu EcoSystem, a completely independent instance can be used for each project. It contains all of the data such as repositories, issues, documentation, artifacts, etc., making it easy to delete all of the data at the end of the project or hand it over to the client. Backups are also easy to delete, since they are also created separately by project.

More security thanks to micro-learning and gamification – Secure Code Warrior plugin for SCM-Manager

Daniel — Fri, 09 Jul 2021 20:31:48 +0000

The regularity of media reports on cyberattacks shows that security is, or should be, a key issue for software development teams these days. Experience also shows that security vulnerabilities are usually not created by highly specialized functions. Rather, many successful attacks exploit well-known security vulnerabilities. For this reason, we are very pleased that the learning platform Secure Code Warrior is now integrated into our version management tool SCM-Manager.

An example of well-known security vulnerabilities is SQL injections, where arbitrary code is injected into database queries, allowing unauthorized information to be read (for more on this, see the Wikipedia article). Such attacks are very popular because they can be carried out very easily. That’s why SQL injections have consistently ranked first in the Open Web Application Security Project’s (OWASP) top 10 security risks since 2010.

These vulnerabilities are actually easy to close. Often, there just seems to be a lack of awareness, or the necessary time to perform appropriate security checks and design processes in such a way that security aspects are taken into account on an ongoing basis. Awareness can be created either classically through targeted training or through continuous learning, e.g. by means of microlearning or gamification. Secure Code Warrior is a very good example of the latter. By combining Secure Code Warrior with the version control management tool SCM-Manager, security aspects can be integrated into processes easily and in a time-saving manner.

Learning with Secure Code Warrior

The Secure Code Warrior platform makes it possible to use microlearning and gamification to gain knowledge about widespread security vulnerabilities and thus close them. The platform offers learning content on almost 150 security topics such as SQL Injection, Cross-Site Scripting (XSS), Memory Corruption or Client Side Injection for all common programming languages such as PHP, JSP, JavaScript, C++, Java Spring, .NET and many more. The content is taught in the form of videos (see example below) and programming exercises (challenges).

In combination with the plugin for the version control management tool SCM-Manager, the information is integrated directly into the software development process.

Open Source Version Management Tool SCM-Manager

SCM-Manager is an open source version management tool that Cloudogu took over in 2020 (to the official announcement of the acquisition). In the same year, we released the completely revised version 2 of the tool. SCM-Manager can be operated on-premises and offers, in addition to repository management, a complete review process for changes, the ability to edit files directly in the browser, and many other features.

The Integration of learning content about security vulnerabilities with the plugin for Secure Code Warrior is the latest enhancement of the tool.

Secure Code Warrior Plugin for SCM-Manager

With the free plugin, videos and links to security vulnerability challenges are displayed directly in pull requests. This way developers directly get all important information about the security vulnerability. For this purpose, the description of pull requests as well as comments and tasks from reviewers are searched for keywords.

For example, the pull request shown in figure 2 contains the keyword “SQL Injection” in the description. Therefore, the corresponding learning content is displayed.

This integration offers the possibility to use the information from Secure Code Warrior in different ways.

SCM-Manager makes the pull request a “security issue” with Secure Code Warrior

When a security vulnerability is found and fixed in the application, the pull request can be used to educate other team members on the topic – by performing the review. By mentioning the security vulnerability in the pull request’s description, information about the topic is displayed. This can be used to learn the theory. At the same time, the learned basics can be comprehended in the context of the expert’s changes in the own application. This approach spreads the knowledge of the topic over several people.

To have information on security topics displayed in pull requests, it is sufficient to mention the topic in the description or title of the pull request.

Security-related comments directly in the SCM-Manager review process

In SCM-Manager, reviewers can provide feedback on pull requests to point out potential security vulnerabilities. All that is required, is to mention a security topic in comments or in tasks. Thus, the corresponding Secure Code Warrior content is displayed in an automatically generated comment.

Note: Only “root” comments are searched for keywords, not replies to comments.

Take a look at the instructions if you want to try out the plugin.

Conclusion

The Secure Code Warrior plugin for SCM-Manager integrates information about vulnerabilities directly into the creation and approval process for changes. All that is required is that a person involved in the process mentions the security vulnerability. All the necessary information for implementation is then provided automatically. The advantage of this approach is that knowledge about security vulnerabilities is spread throughout the team without additional effort, and team members can educate themselves through self-study using micro-learning and gamification.