Forem: Omer Levi Hevroni

How to store a secret on Kubernetes?

Omer Levi Hevroni — Wed, 09 Jan 2019 20:21:09 +0000

At Soluto, we're using Kubernetes platform in production for more than a year. One of the challenges we faced is secrets management, which is not a simple task at all (did you experienced similar issues?). Kubernetes Secret object has its own limitations (especially, how to store the manifest files). We also tried various solutions (like Sealed Secrets and Helm Secrets to name a few), but none of them was a good fit for us. We wanted a solution that is:

GitOps (so it can support our decentralized ops culture)
Zero trust (once a secret is encrypted, there is no need and no way to decrypt it)

This is why we built Kamus - our secret encryption solution. Kamus let you seamlessly encrypt secrets, that can be decrypted only by the app running in production.
I'll be more than happy to answer any question you might have regarding Kamus, and of course - look forward to hearing your feedback!

Can Kubernetes Deployments Be Fun And Simple?

Omer Levi Hevroni — Thu, 12 Apr 2018 16:48:50 +0000

Short answer: YES (scroll to end to find out, or check out the sample repo).

Long answer: Read along to find out!

Kubernetes deployment seems pretty simple: all you need is just a bunch of YAML files, and by using kubectl (the Kubernetes command line utility) you’ll have your service up and running in your Kubernetes cluster.

Although deploying one service is an easy task, how do you deploy hundreds of microservices? At Soluto, we have more than 100 live microservices and the number keeps growing – so as we started thinking about shifting workload to our Kubernetes cluster, we faced a few challenges:

Kubernetes deployment is complex, and there are many moving parts that you need to set up correctly: pod autoscaler, pod resources, ingress, etc. Those parts require some experience with how Kubernetes works, and not setting it up correctly might cause issues in production. Ideally we’d have a way to simplify this, so developers can focus on writing their code and worry less about deployment.

Security is another challenge. All services in production should have certain things, like Transport Layer Security (TLS), which are not necessarily complex but need to be taken care of nonetheless. We would like to pre-configure them so that any new deployment will be secured by default.

Finding a solution

To solve these challenges and speed up and ease the adoption process, we looked for a way to create a template for Kubernetes. Something that any developer would be able to use, and would require just a few parameters (e.g. the Docker image of the service) to get the service up and running in production.

On the other hand, we needed to be careful not to hide too much – developers must be able to understand what’s going on so they can handle production issues. We had to find the right level of abstraction that makes it easier to deploy to Kubernetes, without hiding too much detail.

With that in mind, we started to look for a solution. After trying a few things, we found Helm. Helm is a package manager for Kubernetes.You can use it to install any app on your cluster, and Helm will take care of getting all the required configuration files and installing them on your cluster. Helm also support updating deployments, rollbacks, and many other cool features. Each Helm package is called a “Chart”, and the charts are stored in a repository. With helm, installing MongoDB, for instance, is as easy as helm install stable/mongodb.

Sound like an excellent solution! We can define a chart for each type of service – like one for all our web APIs, which will handle things like load balancer and TLS – and the developer simply needs to specify the required parameters using Helm’s configuration files.

Helm: Let’s see how it’s done

In order to use Helm, we first need to install it (Helm has two components – Helm client running on your computer and Tiller, a server-side component running on your cluster). Then, we need to create the chart, by simply using this Helm CLI command: helm create web-api

After running this command, you’ll notice the creation of a new folder named “web-api”. Within this folder you’ll find all the familiar Kubernetes configuration files: deployment, service, ingress etc.. Now it’s time to customize a bit: we can add a horizontal pod autoscaler, define the default resources the pod requires, and of course, enable TLS by default. Everything is highly customizable based on the Go templating mechanism, so anything we add can be overridden later by the developer, in case the default configuration doesn’t work as needed.

So now we have a chart – but how can we consume it? The chart has to exist in a Helm repository, which is basically a server with a few Zip archives (which are all the charts in the repository) and one index file that is consumed by the CLI. You can manually set up your repo using any storage service like Azure Blob or AWS S3, but the simplest option is Chart Museum.

Chart museum is a Helm repository with a CRUD API to manage your charts. It supports basic authentication so you can restrict who can push new charts to your Helm repository. Helm doesn’t supply any museum-as-a-service solution, so you’re gonna have to roll your own – but it’s dead simple – simply by using its docker image.

Now we can build a CI/CD pipeline for our web-api chart, to ease the process of modifying it:
– Run some kind of tests, to make sure that the new version is not broken. I’ll discuss how in the next paragraph.
– Pack the new chart, using Helm CLI.
– Push the new package to our Chart Museum instance, using Chart Museum’s API.

Testing it out

Now our chart is ready to be used by developers! But wait… how can we know that the chart actually works? And how can we make sure it will continue to work? This is why we need to test our chart, and there are basically two things we want to test.

1) We want to test our template – for example, if an ingress is supposed to exist with a TLS and specific rules (defined by the developer), we should test the generated template and make sure the ingress was created correctly. 2) We want to test that the files are valid Kubernetes configurations and that they work as expected.

Testing the first thing is relatively simple – check out this sample repo to see just how simple it is. This allows us to test the generated Kubernetes files, by using kubetest. This is great but can be complex and messy, especially when having a lot of branching in your template files. A better solution is required – that will allow us to do unit testing for the templates, without generating Kubernetes files. This wasn’t a problem until recently when we started to have a lot of branching in our templates and we’re now looking for options.

The second thing, testing that Kubernetes files are valid, is a bit more tricky. For now, at Soluto we’re using Helm’s version mechanism: each chart has a version, and all of our services will use the latest stable version. When a new chart version is pushed, we can test this version on a specific service. If it works correctly, update the rest of the services. Another option is to test that using minikube, but it was too complex for our needs.

Finally: Deploying!

So now we have a CI/CD pipeline for our Helm charts, and we have prepared a Helm chart that the developers can use. Now, when a new developer wants to deploy a new service to production, all they have to do is:

Add our repository to their local Helm using helm repo add chartmuseum http://<chart-museum-url>
Create a new Helm config file and specify the required parameters (e.g. docker image of the service)
Run helm upgrade —install <service-name> chartmusuem/web-api -f <path_to_config_file> and that’s it – the service is alive.

And to make it even easier to understand, I’ve created a sample repository. The repository contains all the things that I’ve discussed in this blog post: A generic chart for a web app, simple app that can be deployed with this chart and chart museum. Check it out to better understand what’s going on – just follow the walkthrough in the repo.

Thank you for reading along. If you have any questions, or you need help getting started with Helm, feel free to reach out either via the comments here or via Twitter.
Happy Helming!

Originally posted on Soluto Blog

Testing A WordPress Plugin Can Be Fun!

Omer Levi Hevroni — Tue, 27 Mar 2018 18:47:46 +0000

Do you have any experience with WordPress? Until recently, I never tried to develop something in the WordPress ecosystem. A few weeks ago, I found a cool plugin that I wanted to improve. While working on it, I realized how complex it is to write tests for WordPress — and I wanted to share my experience with you. Part of the complexity was running those tests in Docker containers — to make it easy to run them with a real WordPress environment. Because it was a bit complicated, I’ve created a template that you can use — so you could focus more on writing your tests, and less on how to run them. The template (which is actually also a sample project) is available on GitHub.

Using the Template

To use this template to test your plugin, you first need to modify a few things so it will test your actual plugin instead of the demo plugin:

Clone the repo.
Copy all the source code of your plugin to the src folder.
Modify test/bootstrap.php: change to require your plugin instead of my plugin (Look for the TODO at line 21).
Start writing tests for your plugin! You can either modify the basic tests under tests/tests-plugin-security-scanner.php or delete this file and start from scratch. The whole WordPress test SDK is available for you.

To run the tests, you first need to build the containers using:

docker-compose up --build

And then we can run the tests using:

docker-compose run wordpress vendor/bin/phpunit

Then, just sit back and watch the tests as they run.

Cool, right?

Now go ahead and start testing your plugin, it is really easy and improves the quality of your code!

So what exactly is going on behind the scenes?

Continue reading to get into the bits and bytes of how the template works.

Plugin Security Scanner

Before we dive into the details, let’s start with a little story — how I ended up developing a WordPress plugin. As you probably know, WordPress is a great platform for building almost everything- blogs, shopping sites etc. One of the things that makes WordPress so powerful is its ecosystem — WordPress has more than 50,000 plugins and thousands of themes that you can use to customize your site. Most of your specific WordPress needs have already been packed into a plugin or a theme, which is awesome! But not everything is perfect. Have you ever wondered how secure that plugin is that you’re using? After all, the plugin code is running on your site, so a vulnerability in a plugin (or, even worse — an intentionally malicious plugin) could harm your site. Luckily for us, we have WP vulnerability DB that catalogs vulnerabilities in WordPress, along with its plugins and themes. This is great — but doing it manually each time you install a new plugin or theme is not feasible. Also, it will not alert you when a new vulnerability is found in a plugin you are using.

Luckily for us, again, there is also a plugin for that — it will query WP vulnerability DB once a day with all your plugins, themes and WordPress version for vulnerabilities. If a vulnerability is found, it’ll report it to you via mail. This is great, but ideally, I’d want it to integrate with a monitoring system (we are using icinga, but any monitoring system should work). It is possible to achieve that integration with email, but webhook support will make it a lot easier.

One of the reasons I am a fan of open source is that if there is a missing feature, you can contribute to the project and add it. As this is an open source plugin, I’ve done just that. And as I was typing code, I asked myself — what about testing it? This is a pretty important plugin — and I want to be confident that it does what it’s supposed to do, and that it will keep doing it in the future.

Current WordPress Testing Tools

If you have an experience with WordPress development, you’ve probably asked yourself the same question — how to write tests for WordPress plugins (or themes, this is usually the same)?.

The good part is that WordPress can take care of that too — WordPress CLI can generate everything you need to start working on your tests. Although it’s called a unit test, integration tests seem like a more suitable name for them — as they run against real WordPress. For unit tests scenarios, it is better to use WordPress mock.

Because those are integration tests, they require an environment with PHP and MySQL installed — which makes sense — but is not ideal: (1) I don’t want to install PHP and MySQL on my machine (and on any other developer’s laptop) only to run those tests. (2) I want to have the local environment as close as possible to the environment in the CI server. Even more importantly, it needs to be close to the environment on the production server as well.(3) I wanted to avoid version hell — I don’t want to worry about installing the correct version and such.

I solved this by running the tests in Docker, although other solutions (like Vagrant) could work too. It was pretty complicated to do that — and that’s why I created a template that anyone can use for their tests (like I said in the beginning).

As I mentioned before, you can use WordPress CLI to generate all that is required to run the tests. The template is based mostly on what’s generated by WordPress, with changes made so I could run it in Docker.

Let’s take a deep dive into it so you could understand what’s going on.

Template File Structure

Before looking at the code, let’s discuss the Docker solution. The template is based on 2 containers: One will run MySQL DB (using the official docker image), and the other will run our tests (we will look into the Dockerfile soon). This could be easily done by using Docker Compose, another tool from Docker that allows you to easily define and manage multiple containers. The containers are defined using the docker-compose.yml file, which I will not discuss here – but you are more than invited to take a look!

The most interesting folder is the tests folder. Under this folder you will find 2 files:

bootstrap.php: As the name implies, this file is taking care of bootstrapping – loading WordPress SDK so it will be available for your code and your tests code, and loading your plugin code so you could test it.
tests-plugin-sample.php: This file contains the actual tests for your plugin – feel free to rename it.

In order to test WordPress functions, your test classes have to inherit from WP_UnitTestCase class. Other than that, this is a regular PHPUnit test class – so all the regular rules for PHPUnit apply here. You can call any function of your plugin, and of course – the full WordPress SDK is available for you. Also, in WP_UnitTestCasethere are a lot of utility methods that will help you in testing your plugin – for example, to generate users or posts. WordPress has a really awesome testing SDK – but the documentation is not so good. There is an unofficial documentation, but you can always read the code.

Now let’s take a look at the bin folder. It contains 2 scripts that install all that is required for our tests. The WordPress CLI generates one file – install-wp-tests.sh. I chose to split it into 2 files so it will play more nicely with Docker – and you’ll understand why when we dive into the Dockerfile.

The last folder is the src folder, which contains the code of your plugin. I chose to move the plugin code to a folder to make things a bit cleaner. The root folder contains many files now, and it is a bit hard to understand what’s going on.

The Dockerfile

Let’s take a look at the Dockerfile, to get a better understanding of what is going on:

ARG PHP_IMAGE_TAG
FROM php:$PHP_IMAGE_TAG
ARG WORDPRESS_DB_PASSWORD
ARG WORDPRESS_VERSION

The first lines of the Dockerfile define a few build arguments that can be used to customize the test environment — the PHP and WordPress version. This allows you to make sure your code is compatible with all the relevant versions. The last argument is the DB password — which is controlled by the compose file: it has to be the same as the one passed to the DB container.

The PHP image has to be based on Alpine Linux — because starting at PHP 5, you need to install an extension in order to interact with MySQL. for some reason, this did not work well on Ubuntu Linux and moving to Alpine Linux solved all the weird MySQL issues. By looking here, it looks like the process of installing extensions works better on Alpine Linux. This is also the reason I did not use the PHPUnit image — it does not come with the docker-php-ext-enable utility that allows you to easily enable PHP extensions.

RUN echo "http://dl-3.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories &&\
apk add --update --no-cache subversion mysql mysql-client git bash g++ make autoconf && \
set -ex;&& \
docker-php-ext-install mysqli pdo pdo_mysql pcntl \
&& php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" \
&& php composer-setup.php --install-dir=/usr/bin --filename=composer
&& docker-php-source extract \
&& pecl install xdebug \
&& docker-php-ext-enable xdebug \
&& docker-php-source delete \
&& echo "xdebug.remote_enable=on" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.remote_autostart=off" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.remote_port=9000" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.remote_handler=dbgp" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.remote_connect_back=0" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer \
&& rm -rf /tmp/*

This chunk installs all that is required to run the setup scripts and our tests. The script uses SVN to download WordPress, and mysqladmin to manage the MySQL DB. You will notice I’m installing a few extensions, mainly the MySQL extension. XDebug is installed to enable code coverage and for remote debugging (see here to learn how to debug over docker. I personally did not try it). And finally, Composer installed to manage dependencies for our test code, including PHPUnit

WORKDIR /tmp
COPY ./bin/install-wp-tests.sh /tmp/install-wp-tests.sh .
RUN /tmp/install-wp-tests.sh wordpress_test root $WORDPRESS_DB_PASSWORD mysql $WORDPRESS_VERSION
COPY ./db-error.php /tmp/wordpress/wp-content/db-error.php .

Those lines take care of installing WordPress and the testing SDK. The installation process takes some time, especially because it downloads a lot of things. That’s why I am running it as soon as possible, so it will be cached and not have to run each time I trigger a build on my machine. The db-error is used so we will have meaningful DB errors, but it is not required.

WORKDIR /wordpress
COPY composer.json /wordpress
RUN composer install
COPY . /wordpress

After all that, all we need to do is copy composer.json and install the dependencies using composer – again, to keep it cached when changing only test files. You’ll notice that I am using a slightly older version of PHPUnit (5.5.0) – there is a breaking change since PHPUnit 6 that broke WordPress test SDK (see here for more details), so I had to use an older version. And finally – copy all other files to the container.

CMD bin/install-db.sh wordpress_test root $WORDPRESS_DB_PASSWORD mysql

You will notice that the CMD is the DB init script: This is why I’ve split the original script into 2 different files. All the WordPress initialization could happen early in the building stage so it will be cached and will not have to be run every time our test code changed. The DB initialization scripts require an actual DB to work with — so it has to be run only after the DB is ready.

Conclusion

I hope that now you have a better understanding of how the template works. I’m already looking forward to hearing about your experience with it: Were you able to use it? What worked and what didn’t work for you? Share your thoughts here in the comments, by opening an issue on the repo — or by tweeting at me (@omerlh).

Originally posted under soluto-engineering blog

Find security issues before the code goes into production

Omer Levi Hevroni — Sun, 11 Mar 2018 20:15:30 +0000

Are you running security tests in your CI?

You might be wondering — what does running security tests even mean? What does it do? Security tests just test your code for known vulnerabilities, to make sure hackers will not be able to hack into your system. This might sound complicated — but actually, it is pretty simple. There are many existing tools that you can use for running security tests — and in this blog post, I will introduce one of them.

At Soluto, we have a few open source projects, and the biggest one is Tweek. Tweek is a feature management solution, it allows you to do things like A/B test or use feature flags (e.g. open a feature to only part of your users to see how it behaves). It is a critical component of our infrastructure, and therefore, make sure it is secure is really important. Also, as it is an open source project, so anyone can contribute code to it — which could introduce new vulnerabilities. Read along to find out how I easily added security tests to Tweek — or if want to skip the explanation, you can read the PR.

What is Zap and how it can help us

To build our security tests, I used a tool named Zaproxy or just Zap. Zap is an open source and a free hacking tool developed and maintained by OWASP. The coolest part about Zap is that it has an API — which means that I can run all of its hacking functionality in the CI.

The basic feature of Zap is that it functions as a web proxy — e.g. it inspects the traffic from the client to the server. Zap can inspect the request and response, and look for various security issues, like missing security headers. So, I can use existing tests (either UI automation or integration tests), proxy them through Zap and then query Zap for the alerts — let’s see how!

Proxy Tweek’s API smoke tests

Tweek is a complex system, composed of many micro-services and UI. In this blog post, I’ll focus on Tweek’s API, but the same methodology can also be applied to Tweek’s UI. Tweek’s API (and the integration tests) are developed in .NET core (C#). All the integration tests are using HttpClient to send requests, and we can easily configure a proxy:

var baseUrl = Environment.GetEnvironmentVariable("TWEEK_API_URL") ?? "http://localhost:4003";
var proxyUrl = Environment.GetEnvironmentVariable("PROXY_URL");
var handler = new HttpClientHandler();
if (proxyUrl != null)
{
    handler.Proxy = new WebProxy(proxyUrl, false);
}
output.WriteLine($"TWEEK_API_URL {baseUrl}");
var client = new HttpClient(handler);
client.BaseAddress = new Uri(baseUrl);
return new TweekApi(client);

Now, all we need to do is run the tests and look for interesting findings in Zap. Keep in mind that .NET will not proxy requests to localhost — so this will not work when running Tweek locally.

Bringing it all together

Tweek is a containerized app, developed and deployed using Docker. As Tweek contains multiple micro-services, I am using Docker-compose to run Tweek locally. Docker-compose allows you to define all the containers you need in one single yaml file, and then it takes care of the rest. This is the relevant part of the docker-compose yaml:

api:
 image: soluto/tweek-api
 logging:
   driver: "none"
zap:
  image: owasp/zap2docker-bare
  command: zap.sh -daemon -host 0.0.0.0 -port 8090
smoke-tests:
  build:
    context: ../../
    dockerfile: TweekApiSmokeTestDockerfile
  depends_on: 
    - api
    - zap
  working_dir: /repo/tweek
  environment: 
    - TWEEK_API_URL=http://api/
    - PROXY_URL=http://zap:8090
  command: /bin/bash -c "wget --tries 20 --timeout=15 --read-timeout=20 --waitretry=30 --retry-connrefused http://api/status && /src/services/api/Tweek.ApiService.SmokeTests/test.sh"

I omitted all the irrelevant services and tests. You can take a look at the full file here (Tweek has multiple compose files, I referenced the relevant one). In this snippet, you can see we have 3 services:

The API (using Tweek’s official image ) running Tweek’s API code
Smoke tests (using test container, this is the dockerfile) which run our test code
Zap (using Zap’s official bare image — small image, ideal for CI).

You might also notice that the smoke tests are configured to run against our Tweek API, and proxy the requests using Zap. The test.sh script just runs the tests, and queries Zap for interesting alerts — we will deep dive into it soon.
So now all we need to do is run:

docker-compose up --build --exit-code-from smoke-tests

And watch the magic happen. Docker-compose will set the exit code according to the tests and Zap results — so if we have a bug or a security issue, the exit code will be set to 1 — which means we can fail the build! Hurray!

Disabling rules

So now let’s take a look at the results. When I first run the tests through Zap, I noticed that Zap has many alerts. Most of them are because of missing X-Content-Type and Cache-Control headers. I thought that those missing headers were not real vulnerabilities for Tweek API, so I wanted to ignore them. Luckily, Zap's API allows to disabling rules (you can find all the rules here). Let’s take a look at the test.sh script file to see how:

#!/bin/bash
# Abort script on error
set -e
./wait-for-it.sh api:80 -t 4000
./wait-for-it.sh zap:8090 -t 4000
# Disable X-Content-Type scanner - not relevant for API
curl --fail http://zap:8090/JSON/pscan/action/disableScanners/?zapapiformat=JSON\&formMethod=GET\&ids=10021
#Disable Cache control scanner - not relevant for this API
curl --fail http://zap:8090/JSON/pscan/action/disableScanners/?zapapiformat=JSON\&formMethod=GET\&ids=10049
dotnet test

The script uses wait-for-it.sh to wait until Zap and Tweek’s API complete loading. Then, I disabled those two rules that I discussed before using Zap’s API. Now, I can simply call dotnet test, which will run our smoke tests and proxy them through Zap. Now let's continue to the next part - querying Zap for the alerts and failing the build.

Filtering results with Glue

So disabling those rules was easy, but that is not enough. There are cases when you want to ignore or postpone specific issues, but not turn off the entire rule. This was the case with Zap’s last two findings. Zap reported requests that indicate usage of Base64 and SHA-1 — those are the cases that need manual review. When I reviewed Tweek’s case, I found it was clearly a false positive — so I wanted to ignore those findings, but not turn off the rule.

Luckily for us, we have another open source project from OWASP that can help with that — Glue. Glue aims to ease “gluing” security tools into a CI/CD pipeline. It already has support for more than 15 security tools, including Zap. Glue also supports various filtering — like filtering false positives. It also supports various types of reporting — like reporting to Jira. It is open source, so you can easily add more filters or reporters. For example, you can report each new finding to Slack, open a GitHub issue, or report in TeamCity format.

One of the built-in filters in Glue is the file filter — it will write all of the findings to a JSON file, and you can set, for each finding, whether it should be ignored or postponed. You can checkout Tweek’s glue.json file. You’ll notice that I marked those two issues as ignore.

So now let’s take a look at the end of the test.sh file:


ruby /usr/bin/glue/bin/glue
  -t zap
  --zap-host http://zap --zap-port 8090 --zap-passive-mode
  -f text
  --exit-on-warn 0
  http://api
  --finding-file-path /usr/src/wrk/glue.json

Here, I tell Glue that I want to run only Zap, without any other tool. I provide it the host and port that Zap is listening on. Then, I can specify that I want a text report (other reporters currently supported are JSON and Jira) and that I want to set the exit code to 1 on any finding (0 is the lowest level, 3 is the highest). The next parameter is our target — Tweek’s API URL (“http://api"). And the last parameter is our glue.json file, with the ignored findings.
Now if we run it using docker-compose, the exit code will be 0 — no findings! That means we can run it on each new commit — knowing that Zap is watching our back.

How can you use it?

As you saw, this was pretty simple — you can also check out the PR I created in Tweek’s repo to add those security tests. In addition, there is a sample repo demoing this solution — using OWASP’s vulnerable Web App called “Juice Shop”. This repo demonstrates how you can add security tests to Juice Shop — and since it is a vulnerable app, there are some findings. You can clone this repo and play with it to get a better understanding of how this solution works.
I hope you now understand how you can easily add security tests to your CI — but in case something was unclear, feel free to reach out to me. Twitter is the best option, but you can also leave a comment here — I would love to help you integrate Zap into your CI.

Orignaly posted on Soluto Engineering Blog.