Forem: Ryan Blunden

Doppler Encrypted Secrets Snapshots for High Availability

Ryan Blunden — Wed, 23 Nov 2022 05:36:42 +0000

Ideally, fetching secrets at runtime by your application is avoided altogether using Doppler integrations (e.g. Doppler Kubernetes Secrets Operator), which continuously sync secrets to your hosting environment ahead of time, but this won't always be possible.

For legacy applications, on-prem, and virtualized environments where secrets must be fetched at runtime, bundling Doppler CLI secrets snapshots into your application provides a robust failsafe in the event Doppler's API is unreachable.

The result is similar to a GitOps secrets approach, but without the messy business of committing encrypted secrets to Git repositories can be avoided.

Build. Encrypt. Deploy. Fallback.

Doppler secrets snapshots are bundled into the application build during CI/CD with a single command:

doppler secrets download secrets.json.enc

Then to fallback to a secrets snapshot when running your application:

doppler run --fallback secrets.json.enc -- ./app-start.sh

Secrets snapshots aren't just for high availability. They're perfect for network-restricted environments and offer protection if Doppler's API rate limit is exceeded.

The addition of the --fallback-only instructs the CLI only to use the snapshot:

doppler run \
 --fallback secrets.json.enc \
 --fallback-only -- ./app-start.sh

Now let's look into passphrase usage for encrypting and decrypting secrets snapshots.

Passphrase

By default, the CLI configuration is used to construct the passphrase, which is typically the Service Token value exposed via the DOPPLER_TOKEN environment variable. While this is the simplest solution, setting the passphrase explicitly has the benefits of:

Only using the DOPPLER_TOKEN for authentication
Allowing the passphrase and DOPPLER_TOKEN values to be rotated independently

Most importantly, your implementation must guarantee the passphrase remains constant from build to deployment during a release. As these are infrastructure secrets, we recommend a dedicated infra project for each application containing only the DOPPLER_TOKEN and (should you use a passphrase) a DOPPLER_PASSPHRASE.

‎Doppler integrations then keep infrastructure secrets in sync between your build and deployment environments, eliminating the risks associated with manual copying and pasting.

The following video demonstrates how to set up an infra project in three simple steps:

%[https://vimeo.com/769739281]

With the required pieces in place, let's move on to creating and using secrets snapshots.

Build

With the DOPPLER_TOKEN and DOPPLER_PASSPHRASE injected into your build environment (e.g. GitHub Action Secrets), we can now create the secrets snapshot file:

doppler secrets download \
  --passphrase "$DOPPLER_PASSPHRASE" \
  secrets.json.enc

Secrets snapshots also support name transformers and download formats:

# ASP.NET Core secrets snapshot
doppler secrets download \
 --passphrase "$DOPPLER_PASSPHRASE" \
 --format dotnet-json \
 appsettings.json.enc

Ensure the secrets snapshot file is included in your build, then you're done!

Deploy

With the DOPPLER_TOKEN and DOPPLER_PASSPHRASE injected into the deployment environment via automation (e.g Ansible), the Doppler CLI will attempt to fetch the latest version of secrets using the secrets snapshot as a fallback:

doppler run \
  --passphrase "$DOPPLER_PASSPHRASE" \
  --fallback secrets.json.enc -- ./start-app.sh

Summary

Doppler secret snapshots are your high-availability solution to ensure applications can always access their secrets.

Using Docker? Check out our dedicated Docker High Availability documentation to learn more.

How To Prevent Secrets From Ending Up On Developers' Machines

Ryan Blunden — Tue, 21 Jun 2022 00:28:05 +0000

Even with environment variable storage offered by modern hosting platforms and secrets managers provided by every cloud, developers' machines are still littered with secrets in unencrypted text files because local development was left out of the picture.

But we can remedy this situation using dynamic secrets injection and ephemeral secrets files, and in this post, we'll be using the Doppler CLI to demonstrate how this is possible.

But before diving in, we need to solve the problem of secure and encrypted storage for secrets used in local development.

SecretOps and Local Development

Development scoped secrets simply don't exist in traditional solutions because secrets are siloed within the confines of their respective cloud or platform.

While multi-cloud capable secret managers such as HashiCorp Vault showed great promise, the prohibitively steep learning curve and unavoidable complexity involved with fetching secrets create a significant barrier to adoption as teams are left with no incentive to switch.

A SecretOps Platform builds upon the idea of centralized secrets storage but differs from existing solutions by providing a CLI and integrations for syncing secrets to any environment, machine, platform, or cloud secrets manager. It's the best of both worlds, providing a single source of truth for management while development teams choose the best secrets access method on a per-application basis.

How does this help local development? In a SecretOps world, each application has a Development environment specifically for use on developers' machines, solving the storage problem.

Using the Doppler CLI to demonstrate, let's dive in to explore the mechanics of dynamic secrets injection and ephemeral secrets files.

Dynamic Secrets Injection

The Doppler CLI uses the same secrets injection model as platforms such as Heroku and Cloudflare Workers by injecting the secrets as environment variables into the application process.

You can use a command:

doppler run -- npm start

Use a script:

doppler run -- ./launch-app.sh

Create a long-running background process in a virtual machine:

nohup doppler run -- npm start >/dev/null 2>&1 &

Use the Doppler CLI inside a Docker container:

…
FROM ubuntu

# Install Doppler CLI
RUN curl -Ls --tlsv1.2 --proto "=https" --retry 3 https://cli.doppler.com/install.sh | sh

CMD ["doppler", "run", "--", "npm", "start"]

And inject environment variables consumed by Docker Compose:

doppler run -- docker-compose up

It's also possible to pipe secrets in .env file format to the Docker CLI, where it reads the output as a file using process substitution:

docker run \
  --env-file <(doppler secrets download --no-file --format docker) \
  my-awesome-app

The same technique applies to creating a Kubernetes secret:

kubectl create secret generic my-app-secret \
  --from-env-file <(doppler secrets download --no-file --format docker)

But if a secrets file is what your application needs, we've got you covered.

Ephemeral Secrets Files

The Doppler CLI enables the mounting of ephemeral secrets files in .env, JSON, or a custom file format using a secrets template that is automatically cleaned up when the application process exits. Imagine how happy your Security team will be when they learn that secrets will never live on any developer's machines again!

To mount an .env file:

# Node.js
doppler run --mount .env -- npm start

# PHP
doppler run --mount .env -- php artisan serve

To mount a JSON file:

doppler run --mount env.json -- npm start

You can specify the format using --mount-format if the file extension doesn't map to a known format:

doppler run --mount app.config --mount-format json -- npm start

Or you can use a custom template, e.g. configure Firebase functions emulator using a .runtimeconfig.json file:

# 1. Create the template
echo '{ "doppler": {{tojson .}} }' > .runtimeconfig.json.tmpl

# 2. Mount the .runtimeconfig.json and run the emulator
doppler run \
  --mount .runtimeconfig.json \
  --mount-template .runtimeconfig.json.tmpl -- firebase emulators:start --only functions

You can even make things more secure by restricting the number of read requests using the --mount-max-reads option, e.g. caching PHP configuration which only requires the .env file to be read once:

doppler run --mount .env --mount-max-reads 1 --command="php artisan config:cache && php artisan serve"

If you're wondering what happens to the mounted file if the Doppler process is force killed, its file contents will appear to vanish instantly. The mounted file isn't a regular file at all, but a Unix named-pipe. If you've ever heard the phrase "everything is a file in Unix", you now have a better understanding of what that means.

Named pipes are designed for inter-process communication while still using the file system as the access point. In this case, it's a client-server model, where your application is effectively sending a read request to the Doppler CLI. If the Doppler CLI is force killed, the .env file (named pipe) will still exist, but because no process is attached to it, requests to read the file will simply hang. Just delete the file from your terminal, and you're good to go.

Summary

Thanks to SecretOps, we now have the workflows required to prevent secrets from ever living on a developer's machine again. All you need is dynamic secrets injection, ephemeral secrets files, and a single source of truth to pull it all together.

Manually Updating .env Files Isn't DevOps

Ryan Blunden — Tue, 07 Jun 2022 05:28:43 +0000

Managing .env files and keeping them in sync across environments is painful when done manually.

Doesn't it seem odd that Slack is still a common way for .env files to be synced between team members? Shouldn't we be concerned that syntax errors from .env file edits are so prevalent that dotenv linting tools are needed?

It's time to bring the DevOps principles of automation and ephemeral resources to managing environment variables and .env files. This post is a compilation of Doppler's best tips and tricks to do just that as is broken into three sections:

Centralized Environment Variable Management
Dynamically Injected Environment Variables
Dynamically Created Ephemeral .env Files

Although the examples are Doppler centric, the goal is to give you fresh ideas for improving app config and secrets automation in your workplace. We've got lots of stuff to cover, so let's dive in!

1. Centralized Environment Variable Management

It's simply not possible to automate the syncing of environment variables across teams, hosting platforms, and development environments without a centralized source of truth at the organization level.

Modern platforms such as Heroku and Vercel provide built-in environment variable storage, but unless all of your applications are hosted on a single platform, they can only function as a source of truth for individual applications. And if you're using cloud-based virtual machines such as those from DigitalOcean or AWS EC2, you're on your own to figure out environment variable storage and access.

With the exception of Vercel's development scoped environment variables, first-class local development support is missing from every modern platform and cloud provider, explaining why so many teams still rely on .env files, even if not used in production. We know how crucial it is for local environments to closely mirror production, but it seems we're willing to make tradeoffs when it comes to environment variables.

In the past, secrets managers such as HashiCorp Vault were seen as the solution. But replacing the simplicity of environment variables with complex SDKs often resulted in siloed secrets and teams going rogue by managing environment variables their own way. Cloud secrets managers also didn't improve the local development story.

Essentially, we need a new way of managing environment variables that reflects the needs of modern application development.

Why we need SecretOps

SecretOps is designed for multi-cloud deployments and combines the strengths of traditional solutions while addressing their weaknesses.As a starting point, a SecretOps platform must:

Centralize the storage and management of secrets
Provide flexible and secure environment variable injection for every platform and cloud
Provide a first-class local development experience
Increase Developer productivity through secrets automation workflows
Decrease the complexity of secrets management

Using Doppler as an example, we're tackling these requirements by providing:

A fully-hosted solution with collaborative secrets management workflows and fine-grained access controls, all from a single centralized dashboard.
A CLI for injecting environment variables and ephemeral .env files in any environment, from local development to CI/CD, Docker, Kubernetes, Virtual Machines etc.
Integrations that sync secrets to platforms such as Vercel, Netlify, Heroku, and GitHub Secrets, so applications don't have to integrate with Doppler directly.

Doppler's operating model is that managing secrets should be centralized, but fetching and syncing secrets should be tailored to every customer's needs. For example, many of our customers enjoy the Doppler dashboard's superior features and developer experience but sync secrets to Azure Key Vault so production secrets access remains as is.

Our vision for SecretOps is constantly evolving. Our goal is to share, inspire, and help move our industry forward with new ideas that take secrets automation to the next level.

2. Dynamically Injected Environment Variables

Using Doppler to illustrate, let's look at several methods for environment variable injection.

Platform Injected Environment Variables

Having a platform or infrastructure tool to inject environment variables into your application is the best solution, as you can then say goodbye to .env files altogether.

So while that removes the need for .env files in select platforms, additional tooling is needed for virtual machines, local development, and Kubernetes, just to name a few.

CLI Application Runner

This method uses a CLI to run your application, injecting environment variables directly into the application process.

Here is how the Doppler CLI can be used to inject environment variables into a Node.js application:

doppler run -- npm start

You can also use a script:

doppler run -- ./launch-app.sh

Create a long-running background process in a virtual machine:

nohup doppler run -- npm start >/dev/null 2>&1 &

Or use the Doppler CLI inside a Docker container:

…

# Install Doppler CLI
RUN curl -Ls --tlsv1.2 --proto "=https" --retry 3 https://cli.doppler.com/install.sh | sh

CMD ["doppler", "run", "--", "npm", "start"]

A CLI application runner with environment variable injection should have the following properties:

Be open source
Support every major operating system
Installable via package managers
Correctly passes signals to the application (e.g SIGINT) so your application can terminate gracefully

The doppler run command is just one way of accessing secrets, and you can find more examples in our CLI Secrets Access Guide.

3. Docker Container Environment Variables

This method injects environment variables into a Docker container at runtime, removing the temptation of embedding an .env file in the Docker image (yes, it happens) and avoiding the host creating and mounting the .env file inside the container.

The Doppler CLI pipes secrets in .env file format to the Docker CLI where it reads the output as a file using process substitution:

docker run \
  --env-file <(doppler secrets download --no-file --format docker) \
  my-awesome-app

Here we get the benefits of .env file configuration but without an .env file ever touching the disk.

You can see other use cases in our docker-examples GitHub repository.

Docker Compose Environment Variables

Docker Compose differs from Docker as it accesses environment variables from the host when docker compose up is run.

Docker Compose sensibly requires you to define which environment variables to pass through to each service as variables such as $PATH are host-specific:

version: '3.9'
services:
    web:
        image: my-app
        # Host environment variables passed through to container
        environment:
            - AUTH_TOKEN
            - DB_CONNECTION_URL

Because of how Docker Compose access environment variables, we can use the Doppler CLI as an application runner:

doppler run -- docker compose up

Other Docker Compose use cases can be found in our docker-examples GitHub repository.

Kubernetes Environment Variables

Kubernetes provides excellent support for injecting environment variables into containers using Key-Value pairs stored in a Kubernetes secret.

Doppler provides two options for syncing secrets to Kubernetes:

Doppler CLI Created Secrets
Doppler Kubernetes Operator Managed Secrets

Doppler CLI Created Secrets

The first step is to create a generic Kubernetes secret (the first argument being the secret's name) where just like Docker, secrets in .env file format are piped to kubectl where it reads the output as a file:

kubectl create secret generic my-app-secret \
  --from-env-file <(doppler secrets download --no-file --format docker)

Doppler Kubernetes Operator Managed Secrets

Our Kubernetes Operator is designed to scale and fully automate secrets syncing from Doppler to Kubernetes. Once installed and configured, it instantly creates and updates Kubernetes secrets as they change in Doppler with support for automatic deployment reloads if the secrets they're consuming have changed.

As it's a more advanced solution that requires Kubernetes cluster administration experience, we won't be covering it here but check out our Kubernetes Operator documentation to learn more.

Kubernetes Deployments and Environment Variables

Injecting environment variables into a Deployment from the Key-Value pairs in a Kubernetes secret is done using the envFrom property of a container spec:

...
    spec:
      containers:
        - name: awesome-app
          envFrom:
            - secretRef:
                name: my-app-secret # Kubernetes secret name
…

Hopefully, you've learned some new tricks for environment variable injection! Now let's move on to.env files.

Dynamically Created Ephemeral .env Files

Environment variable injection is always preferred, but sometimes, an .env file is the only workable solution.

Protective measures such as locking down file ownership, file permissions, and heavily restricting shell access should go without saying. But the risk of .env files existing on the file system indefinitely has always been a concern and why we've been hesitant to recommend .env file usage in the past.

But thanks to the Doppler CLI, we can now mount ephemeral .env files that are automatically cleaned up when the application exits. Imagine not having to worry about anyone in your company accidentally committing an .env file again!

One of the most popular use cases is for PHP developers building Laravel applications:

doppler run --mount .env -- php artisan serve

The file extension is used to automatically set the format (JSON format is also supported):

doppler run --mount secrets.json -- npm start

You can set the format if the file extension doesn't map to a known type:

doppler run --mount app.config --mount-format json -- npm start

To increase security, you can also restrict the number of reads:

doppler run --mount .env --mount-max-reads 1 --command="php artisan config:cache && php artisan serve"

If you're wondering what happens to the mounted file if the Doppler process is force killed, its file contents will appear to vanish instantly. This is because the mounted file isn't a regular file at all, but a Unix named-pipe. If you've ever heard the phrase “everything is a file in Unix”, you now have a better understanding of what that means.

Named pipes are designed for inter-process communication while still using the file system as the access point. In this case, it's a client-server model, where your application is effectively sending a read request to the Doppler CLI. In the event the Doppler CLI is force killed, the .env file (named pipe) will still exist, but because no process is attached to it, requests to read the file will simply hang.

It's also named pipes that enable us to restrict the number of reads using the --mount-max-reads option as once the limit is exceeded, the CLI simply removes the named pipe from the file system.

Summary

I hope you take away some new automation ideas to bring back to your team so you can spend less time updating .env files and more time shipping software.

Why syncing .env files doesn’t scale for secrets management

Ryan Blunden — Wed, 19 Jan 2022 07:59:09 +0000

Learn why using a Universal Secrets Platform is the key to managing environment variables at scale and eliminates the need for syncing .env files.

The benefits of using environment variables to keep secrets out of source code are well established. But are .env files the best method for managing them?

Secrets management has evolved beyond the limited Key-Value storage that .env files provide. However, most developers are either unaware of .env file’s shortcomings or have simply become numb to the pain from the many years of usage and lack of innovation.

This post aims to highlight the risks of continuing to use .env files, why the major cloud vendors and hosting platforms offer a built-in secrets or environment variables store which can be used instead, and how secrets managers such as Doppler and HashiCorp Vault are providing the much-needed management and secrets automation layer on top of encrypted and access controlled secret storage.

A brief history of .env files

The usage of environment variables and .env files for application config and secrets largely began around 2013. It was a long overdue and important step towards more secure secrets management practices.

Libraries such as Python’s dotenv and Node’s dotenv made it easy for developers to use .env files and environment variables for application configuration, giving developers a simple path to removing secrets from their source code for good.

To think that .env file usage has remained practically unchanged for over eight years is quite remarkable. It should come as no surprise then, that it’s time to say goodbye to .env files in exchange for alternatives that better meet the needs of modern application development.

The problems with .env files

Using .env files allowed us to move secrets out of source code. Unfortunately, they introduced a new set of challenges:

Scaling issues related to syncing .env file changes across environments and different cloud providers, increasing the risk of infrastructure misconfiguration and potential downtime.
Easy for .env files to contain syntax errors, requiring additional tools such as dotenv-linter to be added to pro-commit hooks or GitHub checks.
Sharing of unencrypted secrets in .env files over Slack when secrets change or new developers join a team risks breaking the principle of least privilege by exposing secrets to potentially unauthorized users.
Inconsistent format of environment variables can cause issues, e.g. Docker and GitHub require unquoted values while other packages do not.
Patchy and inconsistent support for multi-line secrets such as TLS certificates, SSH keys, JSON, and YAML.
Secrets used in multiple applications must be duplicated in every .env file (instead of dynamic secrets referencing), making updating and rolling credentials tedious and repetitive.
If persisted to disk in plain text, they may be readable by unauthorized users with access to the system and threat actors if file restrictive file permissions aren't used.
Easy to accidentally expose to malicious bots if placed in the webroot of a web server or S3 buckets.
Local development environments break whenever team members forget to share updates that need to be applied to their .env files, e.g. when a feature branch is merged that requires a new secret.

It's clear .env files have serious implications for application security Next, we’ll take a closer look at why the productivity impacts of using .env files may be worse than you think.

The hidden productivity costs from using .env files

Small repetitive problems, such as manually updating .env files across several servers as part of the deployment process, while perhaps initially frustrating and annoying, can easily become just an expected part of the application deployment lifecycle.

While some developers would argue that the papercuts associated with using .env files are minor, one thing we can all agree on is that interruptions can have serious productivity implications for writing code.

According to a recent study, the average lost time per serious interruption is 23 minutes:

"You need to get into the mindset for development and then slowly trace back to where you left off. This can easily take more than 30 minutes." - Gloria Mark, Professor of Informatics California, Irvine.

The cost of misconfiguration errors is not just the time spent fixing an .env file related issue. It's the impact of unexpected context switching and the challenge of getting back into a state of deep work, also known as “flow”.

Why developers have ignored traditional secrets managers

Traditional secrets managers such as Azure Key Vault or AWS Secrets Manager provide encrypted storage and fine-grained access controls, specially designed for storing secrets such as API keys, database credentials, SSH keys, and TLS certificates.

They are incredibly secure, robust, and enterprise-ready. But unfortunately, secret managers such as HashiCorp Vault are built for security teams, not developers.

As a result, they can be complex to implement correctly and often require secret-fetching implementation details to leak into application code—the exact opposite of the benefits afforded by using language-agnostic environment variables.

Even security-minded developers motivated to use a traditional secrets managers have typically given up for one primary reason: Using .env files was much easier.

Instead of environment variables, a vendor-specific SDK, platform integration, or custom application code for fetching secrets from a vendor’s API is often required.

For example, take this AWS Secrets Manager SDK for Node.js sample code for fetching secrets:

// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "<<{{MyRegionName}}>>",
    secretName = "<<{{MySecretName}}>>",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});

// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
// We rethrow the exception by default.

client.getSecretValue({SecretId: secretName}, function(err, data) {
    if (err) {
        if (err.code === 'DecryptionFailureException')
            // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InternalServiceErrorException')
            // An error occurred on the server side.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidParameterException')
            // You provided an invalid value for a parameter.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidRequestException')
            // You provided a parameter value that is not valid for the current state of the resource.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'ResourceNotFoundException')
            // We can't find the resource that you asked for.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
    }
    else {
        // Decrypts secret using the associated KMS CMK.
        // Depending on whether the secret is a string or binary, one of these fields will be populated.
        if ('SecretString' in data) {
            secret = data.SecretString;
        } else {
            let buff = new Buffer(data.SecretBinary, 'base64');
            decodedBinarySecret = buff.toString('ascii');
        }
    }

    // Your code goes here. 
});

It's this level of complexity compared with using environment variables that turns most developers off from the start.

As product teams are incentivized to ship software and new features as fast as possible, migrating to a traditional secrets manager usually only occurs due to regulatory requirements or security mandates.

But is it possible to still use environment variables for modern applications without .env files?

Modern platforms with native environment variable storage

Modern hosting platforms such as Netlify, Vercel, DigitalOcean, Cloudflare Workers, Fly.io, and Railway all come with secure environment variable storage built-in.

This not only shows how easy it is to migrate away from .env files but confirms that environment variables are still the best language and platform-agnostic method for injecting secrets into an application.

Do you need .env files for local development?

It may seem we’re still reliant on .env files for local development if hosting platforms only manage environment variables for applications running on their infrastructure. But this is beginning to change.

Every developer understands that inconsistencies between local and production environments are a recipe for unexpected issues. That’s why a USP provides first-class support for managing secrets in every environment. A trend modern hosting providers are also beginning to follow.

Vercel, for example, offers environment variable storage specifically for local development that is fetched and injected into the Node.js application via the Vercel CLI:

vercel dev

But what about developers using hosting providers without such functionality? This is where a USP such as Doppler fills in the gaps, eliminating the need for manually managed .env files.

Once developers have created a project and installed the Doppler CLI, secrets can be injected as environment variables into any application process:

doppler run -- npm run firebase-local-functions

Developer tooling is rapidly improving to provide a better integrated local development experience that will eliminate the differences between local and production environments and the need for manually managed .env files on developer machines.

A Universal approach to taming secret sprawl

Taming secret sprawl is a growing challenge for every development team and one that is only made worse as the number of .env files increase. We need an entirely new approach to secrets management that goes beyond incremental improvements—A Universal Secrets Platform.

Taking a “Universal” approach means being able to manage and sync secrets to every application on any platform by avoiding the problems associated with siloed secrets and antiquated solutions that don’t scale such as trying to sync dotenv files across platforms.

This can be achieved through a hub-and-spoke model where the USP acts as a single source of truth for secret storage and management with integrations automatically syncing secrets when they change to any external platform, including other secrets managers.

We hope our vision of a Universal Secrets Platform serves as inspiration for other secrets managers to create a more developer-friendly experience in order to make migrating away from .env files a more attractive option to developers.

Summary

We don’t need to sync .env files. We need the developer-specific workflows that a Universal Secrets Platform such as Doppler can provide.

The simplicity of .env files, while attractive at first, is also its greatest weakness. The demands of modern application development and the explosion of microservices across multiple clouds and platforms provide scalability challenges .env files simply can’t address.

The use of .env files was certainly an improvement over hard-coded secrets. But better options now exist for secrets management, and not only will your infrastructure be more secure without .env files, you’ll be more productive without them too.

Doppler: How to Set Environment Variables for a Python Django Application using Apache and mod_wsgi in Docker

Ryan Blunden — Tue, 17 Aug 2021 07:00:00 +0000

Using environment variables to configure Django and other Python applications is awesome, but using them with Apache and mod_wsgi in Docker is a tricky thing to get right.

That's why I created this step-by-step tutorial and sample application to put all the info you need in one place.

Although this tutorial is for Docker and Django, the same steps apply, whether you're using a Virtual Machine or a different Python framework.

Prefer just to read the code? Head to the accompanying repository at https://github.com/DopplerUniversity/django-apache-mod-wsgi

Environment Variables, Apache and mod_wsgi

When hosting a Python WSGI compatible framework like Django in Apache with mod_wsgi, the only environment variables populated in the os.environ dictionary are those that exist in the environment of the script that starts Apache. But instead of having to mess with Apache's service manager settings (e.g. systemd or systemctl), there's a better way.

Most Apache distributions provide a shell script specifically for the purpose of setting environment variables that will be made available to modules such as mod_wsgi.

It's then a matter of knowing the location of this shell script as it can be different depending on the Linux distribution. For example:

Debian/Ubuntu: /etc/apache2/envvars
CentOS: /etc/sysconfig/httpd

We'll be using the python:3.9-slim-buster Docker image is Debian based.

Appending App Config and Secrets to the Environment Variables File

Essentially, it boils down to fetching the secrets as key/value pairs and writing them to the envvars file in the typical shell environiment variables format:

export FIRST_NAME="The"
export LAST_NAME="Mandalorion"

But where from and how do we fetch the app config and secrets to populate that file?

As I'm the Developer Advocate for Doppler, I'll start with a Doppler CLI example, but the mechanics of "fetch secrets, then append to file" can easily be adapted.

First, you would need to set up your project in Doppler and you can use the following button to get you started if you want to follow along.

Then use the Doppler CLI inside the Docker container to fetch the secrets (requires a DOPPLER_TOKEN environment variable with a Service Token value):

# Transform JSON key:value pairs into export statements using jq
if [ -n "$DOPPLER_TOKEN" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars using Doppler CLI'
    doppler secrets download --no-file | jq -r $'. | to_entries[] | "export \(.key)=\'\(.value)\'"' >> /etc/apache2/envvars
fi

Notice I've used single quotes, not double quotes around the values?

That's because it gives you the flexibility of storing secrets with double quotes such as JSON in Doppler which you could use for example, to dynamically set Django's ALLOWED_HOSTS setting dynamically for any environment.

ALLOWED_HOSTS = json.loads(os.environ['ALLOWED_HOSTS'])

You could also use a .env file but I wouldn't recommend it and instead, I'd look into using a secrets manager.

But that aside, here is how you could do it using an .env file:

if [ -f "$PWD/.env" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars from .env file'
    cat "$PWD/.env" >> /etc/apache2/envvars
fi

Now that we know how to pass environment variables from Apache to mod_wsgi, let's move onto getting this working in Docker.

Docker Configuration for Apache and mod_wsgi

Let's breakdown the task of configuring a Python Django Application using Apache and mod_wsgi in Docker into three steps:

Custom Start Script
Apache Site Config
Dockerfile

If you only want to see the working code examples, head to the accompanying repository at https://github.com/DopplerUniversity/django-apache-mod-wsgi

As this isn't a Docker or Apache tutorial, I won't be diving too deeply into the Dockerfile or Apache site config file, but if you've got questions, head over to the Doppler community forum and I'll be able to help you there.

1. Custom Start Script

Running your application in Docker is usually a case of setting the CMD, for example:

CMD ["python", "src/app.py"]

But it's trickier here as we first need to append the environment variables to /etc/apache2/envvars before running Apache.

As this requires multiple commands, we'll create a custom script:

#!/bin/bash

# apache-doppler-start

set -e

echo 'ServerName localhost' >> /etc/apache2/apache2.conf # Silence FQDN warning

# Doppler CLI
if [ -n "$DOPPLER_TOKEN" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars from Doppler CLI'
    doppler secrets download --no-file | jq -r $'. | to_entries[] | "export \(.key)=\'\(.value)\'"' >> /etc/apache2/envvars
fi

# Mounted .env file
if [ -f "$PWD/.env" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars from .env file'
    cat "$PWD/.env" >> /etc/apache2/envvars
fi

# Run Apache
apache2ctl -D FOREGROUND

2. Apache Site Config

Here is an example Apache site config file for a Django application:

# wsgi.conf
<VirtualHost *:80>
    ServerName django-apache-mod-wsgi
    ServerAlias django-apache-mod-wsgi
    ServerAdmin webmaster@doppler

    # Defining `WSGIDaemonProcess` and `WSGIProcessGroup` triggers daemon mode
    WSGIDaemonProcess django-apache-mod-wsgi processes=2 threads=15 display-name=%{GROUP} python-path=/usr/local/lib/python3.9/site-packages:/usr/src/app    
    WSGIProcessGroup django-apache-mod-wsgi
    WSGIScriptAlias / /usr/src/app/doppler/wsgi.py

    <Directory /usr/src/app/doppler/>
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>

    # Redirect all logging to stdout for Docker
    LogLevel INFO
    ErrorLog /dev/stdout
    TransferLog /dev/stdout
</VirtualHost>

3. Dockerfile

The Dockerfile is reasonably straightforward, installing the Doppler CLI and Apache dependencies before copying the Django source code, custom script, and Apache site config:

FROM python:3.9-slim-buster

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

# Install Doppler CLI and related dependencies
RUN apt-get -qq update && apt-get install -y apt-transport-https ca-certificates curl gnupg jq && \
curl -sLf --retry 3 --tlsv1.2 --proto "=https" 'https://packages.doppler.com/public/cli/gpg.DE2A7741A397C129.key' |  apt-key add - && \
echo "deb https://packages.doppler.com/public/cli/deb/debian any-version main" | tee /etc/apt/sources.list.d/doppler-cli.list && \
apt-get -qq update && apt-get install doppler

# Install Apache and related dependencies
RUN apt-get install --yes apache2 apache2-dev libapache2-mod-wsgi-py3 && \
    apt-get clean && \
    apt-get remove --purge --auto-remove -y && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /usr/src/app

COPY requirements*.txt .
RUN pip install --quiet --no-cache-dir --upgrade pip && \
    pip install --quiet --no-cache-dir -r requirements.txt

# Application source
COPY src/ ./

# Custom CMD script
COPY apache-doppler-start /usr/local/bin/

# Apache site config
COPY wsgi.conf /etc/apache2/sites-enabled/000-default.conf

EXPOSE 80 443

# https://httpd.apache.org/docs/2.4/stopping.html#gracefulstop
STOPSIGNAL SIGWINCH

CMD ["apache-doppler-start"]

With all the pieces in place, we can nowbuild the Docker image (clone the sample repository to follow along):

docker image build -t django-apache-mod-wsgi:latest .

Now we're ready to run the container!

Running the Django Application with Apache and mod_wsgi in Docker

We'll start with a Doppler example, then with an .env file.

With Doppler, you'll first need to set a DOPPLER_TOKEN environment variable to the value of a Service Token. This is what provides read-only access to a specific Doppler config in production environments.

Usually, this would be securely set by your deployment environment (e.g. GitHub Action Secret) but for completeness and simplicity, we'll set it manually below:

export DOPPLER_TOKEN="dp.st.xxxx" # Service token value created from Doppler dashboard

Now run the container:

docker container run \
    -it \
    --init \
    --name doppler-apache-mod-wsgi \
    --rm \
    -p 8080:80 \
    -e DOPPLER_TOKEN="$DOPPLER_TOKEN" \
    django-apache-mod-wsgi

.env File

To run the .env file version, we'll use the sample.env file from the sample repository:

# sample.env
export DJANGO_SETTINGS_MODULE='doppler.settings'
export DEBUG='yes'
export ALLOWED_HOSTS='["*"]'
export SECRET_KEY='bf5e1b31-6ba7-48e2-9175-f2293671e6df'

Then to run the container:

docker container run \
    -it \
    --init \
    --name dotenv-apache-mod-wsgi \
    --rm \
    -v $(pwd)/sample.env:/usr/src/app/.env \
    -p 8080:80 \
    django-apache-mod-wsgi

Summary

Nice work in making it to the end!

Now you know how to configure Python applications hosted with Apache and mod_wsgi running in Docker using environment variables for app configuration and secrets.

Feedback is welcome and you can reach us on Twitter, our Community forum, or send me an email at ryan.blunden@doppler.com.

Why Secrets Management Is NOT Just a Key-Value Store

Ryan Blunden — Tue, 10 Aug 2021 00:28:34 +0000

For those who’ve managed secrets via .env files or have briefly looked at secrets managers such as HashiCorp Vault , it’s easy to get the impression that “secrets management” is simply the storage and retrieval of secrets from a Key-Value data store.

But recurring tasks such as instantly comparing secret values between environments when troubleshooting, or being alerted when a new secret is introduced are crucial features every secrets manager should provide. Without them, development teams are forced to cobble together their own solutions, often in silos on a per-application basis.

The reality is that .env files and traditional secrets managers were never designed to meet the needs of modern application development teams, where microservices and multi-cloud deployments are the new normal.

Essentially, we want the flexibility of a Key-Value secret storage solution but with an operating model and feature set that decreases complexity, increases developer productivity, and standardizes how secrets are managed for applications in every part of the business.

In this post, we’ll explore the most important aspects of secrets management and why Key-Value storage is only the beginning.

Access Control and Permissions

Who needs access to secrets? What should their level of permissions be and which applications and environments should they have access to? Is access provided according to org structure or application ownership?

If access controls aren’t fine-grained enough, you risk violating the principle of least privilege but if too rigid, can impede the ability of teams to embrace a culture of DevOps and “shifting left” to bring security earlier into the application development process.

The following is a starting point for secret and access permission requirements:

Ability to segment parts of the business into separate workplaces so for example, an acquired start-up can onboard to using the secrets manager while remaining entirely separate from the organization’s existing secrets.
Consider how access will be managed across cloud and platform boundaries, e.g. AWS, GCP, and Vercel?
Ability to grant access to secrets for specific applications and environments within each application.
In most cases, developers should only have access to secrets belonging to their applications and only for the appropriate environment (e.g., dev and test). DevSecOps will have greater permission levels so that they can work cross-functionally in any environment.
Temporary secrets access to external entities (e.g. contractors) should be scoped to a subset of application secrets via an auth/service token.
One-off sharing of secrets to external entities (e.g. TLS Certificates to an external firm) should be captured in an activity log and sent via encrypted means using a service such as Doppler Share.
Mandated MFA for secrets manager dashboard access.
Privilege elevation (e.g. developer viewing production secrets) be time-limited and captured in an activity log.
Seamless integration with existing Single Sign-on solutions such as SAML and SCIM to allow default access permission levels to be assigned based on role and/or group membership.

A further consideration is restricting machine access to a CIDR IP address range, preventing leaked service tokens from use outside of trusted corporate and cloud networks.

There’s a lot to consider for access control and permissions and your secrets manager should make it straightforward to meet these requirements.

Versioning and Rollback

Versioning and the ability to roll back changes is a must and shouldn’t come at the cost of reduced performance or additional storage fees.

While most secret managers support versioning and rollback, it’s critical to evaluate the actual process.

Being able to roll back a misconfigured change in a single click with support for triggering an automatic application reload or deploy is something teams should come to expect from a secrets manager, enabling a misconfiguration fix to be applied in seconds.

Secret Documentation

There are wildly different perspectives on documenting code, but the one thing all developers can agree upon is that useful code comments provide context that explains the “why”, not just what the code does.

Such context and comments can be especially valuable for secrets, but where should this be documented?

A reasonable option is a README or inline code comments where the secret is in use, however, the context may be lost if a secret’s value is changed in your secrets manager by someone who hasn’t viewed the code.

Being able to notate secrets directly in the secrets manager reduces the chance of misconfiguration errors that could’ve been easily avoided, had the appropriate comments (and context) be provided.

Local Development

The closer your development environment is to production, the less likely unexpected problems are to arise during deployment, which is why secrets should be accessed the same way in development, as they are in production.

Your secrets manager should treat Development as a proper environment in its own right and be responsible for supplying secrets to developers locally. This saves each developer from manually updating and patching their own environment variables list in their IDE or .env file.

Development environments also present a unique challenge where secret values will often be specific to each developer, e.g. when working on a feature branch that uses a new secret.

Developers need the flexibility to override secrets during development without affecting other developers on the team. This is an essential feature a secrets manager should provide to ensure secrets access remains consistent for every environment.

A Single Unified View for Application Configuration

One of the reasons developers love .env files is that it gives them a single unified view of how an application is configured as they tend to hold both secrets and config data such as API keys, port bindings, and feature flags.

When assessing the transition from .env files to a secrets manager, teams are often confused as to whether secrets and config should be separated. For example, should secrets be migrated to the secrets manager while config continues to exist in .env files?

While secret managers are designed to store sensitive data, it makes sense to store both config and secrets together, as there is no logistical or financial advantage gained by separating them.

If secrets and config are simply displayed as Key-Value pairs in the secrets manager’s dashboard, it’s up to the viewer to define a way to visually organize secrets by application, then further segment into separate environments.

Your secrets manager should provide a simple built-in mechanism to view secrets for a specific application and environment, and storing config as well as secrets in your secret manager ensures you’ll still get a single unified view of how your application is configured.

Reduce Friction to Increase Adoption

While implementing a secrets manager is often a security endeavor, it's better viewed as a productivity one, as every project team can appreciate time and effort saved. And if security posture is improved, well even better!

But if the focus is on security at the cost of productivity (even if short term), gaining widespread adoption will be significantly more difficult, as product teams are incentivized to ship new features, not improve security (unless mandated to do so).

Therefore, the secrets manager that is likely to have the biggest impact in improving security, is the one that provides a nearly frictionless experience for teams to implement and use on a daily basis.

Preventing Misconfiguration Issues

Misconfiguration issues are much easier to deal with if caught before changes are applied in production. To achieve this, developers and DevOps Engineers need access to a secrets activity log that pushes secret events to where they naturally communicate such as a Slack or Microsoft Teams channel.

The secrets management dashboard should also provide indicators of potential problems such as if a secret was added to the staging environment but not production, as well as troubleshooting features such as the ability to instantly compare a secret value across every environment.

A secrets manager must therefore help to prevent known issues associated with application configuration and deployment scenarios to increase production stability and reduce time spent firefighting issues caused by misconfiguration.

Automate Application Reload and Redeployment

The Infrastructure as Code movement put automation at the heart of deployment workflows and an essential feature of any secrets manager should be the option to automatically trigger an application to reload or redeploy if its config or secrets change.

This again highlights the need for secret managers to go beyond simply Key-Value storage, as, as they must know which application to trigger a redeployment for, based on the application and environment the changed secret belongs to.

Secrets Management Goes Beyond Secret Storage

By now, it should be clear that secrets management extends beyond the secure storage of secrets as Key-Value pairs. We’ve touched on processes, best practices, as well as various risks, challenges, and issues that can arise and what a secrets manager should reasonably do to mitigate them.

These relate to key elements of storage, access, visibility, integrations, workflows, and troubleshooting processes for not just secrets management, but application configuration generally.

Doppler is simply the result of building the universal secrets manager we wish had but simply didn’t exist.

Doppler enables you to stop using old ways to store, share, and access secrets via .env files.

It's your centralized source of truth for managing secrets across multiple clouds, platforms, and packaging formats, from cloud-native build packs to containers, serverless, and more.

What is a Secrets Manager?

Ryan Blunden — Tue, 10 Aug 2021 00:18:02 +0000

Secrets and credentials management is widely considered to be the most overlooked aspect of software development. Many teams struggle daily to organize and sync secrets between environments, with manually maintained .env files being one of the most common sources of frustration for developers and DevSecOps.

Not only does this impact developer productivity, but without a centralized data store with fine-grained access control and audit logs, secrets like APIs and login credentials are at risk of exploitation.

Security teams want developers to “shift left” and incorporate security best practices into the application development lifecycle and tooling. Still, too often, .env files fall into the “if it ain’t broke, don’t fix it” category” simply because it appears to be the best solution available.

The good news is that a relatively new breed of security infrastructure tooling known as “Secrets Managers” are here to make .env files and other insecure secrets storage practices a thing of the past.

In this article, we’ll unpack Secrets Managers, what they do, why they matter, and what to look for when choosing a Secrets Manager for your team and organization.

Your Secrets Are At Risk – And Here’s Why

For example, a developer is working on an infrastructure-as-code solution that requires access to database credentials and API keys as part of the application deployment process.

A straightforward solution could be to store the required secrets in a plain text file that the deployment code can consume. The developer knows it’s not ideal, but it works, and they’re under pressure to finish the job so they can get back to shipping new features. At that moment, it’s the simplest solution. And sometimes, simplicity wins out, but for the wrong reasons.

Some organizations consider the simplicity of .env files to be its strength, but the underlying maintenance issues such as the manual syncing of changes between environments, as well as the security risk from unencrypted secrets storage on application servers, often go unnoticed or ignored.

In the same way, we learned that storing secrets unencrypted in application code was insecure; we need to apply that same thinking to the storage of unencrypted secrets on the file system.

This is why forward-thinking organizations are embracing Secret Managers, so let’s now focus on what they are and how they work.

What is a Secrets Manager?

Today, organizations depend on commercial, open-source, and internally developed applications, automated IT infrastructure, and DevOps methodologies to support innovation and business growth. Regardless of your development environment, every automation tool, script, and application you use consumes data of a sensitive nature in order to perform its job.

A Secrets Manager is a storage and management solution for storing any type of sensitive data your application requires, such as:

Database credentials
API keys
SSH Keys
TLS Certifications

A Secrets Manager provides a centralized source of truth that empowers organizations to securely and efficiently store and control access based on role, machine identity, environment, and other factors to apply the principle of least privilege.

Why Secrets Managers Matter

Cybercriminals often seek to exploit vulnerabilities in plain text methods such as .env files as they are easy to scan for. The initial breach of stealing credentials is often only the beginning of a broader, more devastating attack.

Failing to use the protective powers of a Secrets Manager puts you further at risk from the following security challenges.

Mitigating the Risk of Cybercrime

Cybercrime is big business, with successful breaches costing companies upwards of $200,000, on average. Without a Secrets Manager, your organization’s external threat mitigation strategy is severely lacking.

For example, if you don’t have a centralized secrets repository, how do you know who can access specific credentials? Can you guarantee that all your secrets are encrypted at rest? Can you say with confidence that you have done everything in your power to prevent your secrets from falling into the wrong hands?

It’s much better to acknowledge immediate improvements are possible and act on them now instead of later in the aftermath of a breach.

Managing Secrets Sprawl Across your Applications and Environments

Secrets are widespread across your application deployment environments, platforms, and cloud infrastructure. Without a single source of truth for secrets storage, tracking secret usage becomes ad hoc, making operations such as mandatory rolling of credentials at set time intervals difficult and time-consuming.

A secrets manager can provide answers to questions every engineering leader should be asking from a risk management perspective, such as:

How do we roll back a secret change in the event of a production outage caused by misconfiguration?
Which applications are using a shared secret such as an API key?
How fast can we roll a compromised credential and reload applications to pick up the change?
Where is the audit log for tracking secret access and value changes?
How are we sharing secrets with external contractors?

Maintaining Security Without Access Controls

Secrets contain highly sensitive information, and without a Secrets Manager, anyone that can access .env files, Slack messages containing secrets, or other internal documentation where secrets may be stored, risk unintended access by unauthorized users or, worse, threat actors with malicious intent.

To mitigate this risk, you need tight access controls that enable you to select who can manage secrets for each environment (e.g. only DevSecOps for live environments) and assign appropriate permissions and policies to groups, individual users, and machines.

A Secrets Manager also provides a detailed audit log, critical when reviewing secret change history and administrative operations that document what users and machines accessed secrets and when.

Creating Secure Environments for Machine Identities

In Gartner’s round-up of security risks and trends for 2021, the management of machine identities as an integral security capability was listed in the top eight.

Through cloud automation and infrastructure as code, organizations are grappling with greater numbers of non-human entities – containers, servers, applications, and devices – meaning the management of machine identities is becoming increasingly important to overall security.

Gartner notes that “establishing an enterprise-wide strategy for managing machine identities, certificates and secrets will enable the organization to better secure digital transformation.”

A Secrets Manager should be a foundational component of that management strategy.

Managing and Accessing Secrets

You shouldn’t be sharing secrets via IM, zip files, git, or email, embedding credentials in code or databases, or using simple, albeit insecure, .env files. That much is clear. Secrets Managers are fundamental to streamlined workflows and security best practices, but how can organizations be sure that teams will use the chosen solution?

In short, the selected Secrets Manager must be easy and intuitive to use, as any anticipated friction can hurt widespread adoption.

Teams can manage secrets via an access-controlled dashboard, CLI, or language-specific SDKs. Accessing secrets is then performed at runtime, ensuring an application always has the latest version of the secrets when deployed.

And assuming application developers are following the industry best practice of using environment variables to access secrets in memory (not the file system), the security benefits are instant and significant.

Organizations seeking to replace .env files with a secrets manager should ensure it has built-in support for injecting secrets as environment variables, eliminating the need for application code changes for a fast and painless migration.

Environment variables have the benefit of being language and framework agnostic, which is why platforms such as AWS Lambda, Heroku, Vercel, Netlify, and others have standardized on environment variables for compatibility and because they require no third-party packages to consume.

How to Select the Ideal Secrets Manager for Your Organization

Your security is only as good as your Secrets Manager. When weighing your options, consider the following factors:

Speed of implementation

Leading Secret Managers allow you to hit the ground running, minimizing migration time to secure your secrets sooner rather than later.
Versatility across environments

Your team works in all environments – from development to production and your secrets manager should as well.
User experience

What do your team’s favorite developer tools have in common? They solve a specific problem exceptionally well while providing a delightful developer experience. Choose a secrets manager that meets your security standards that your developers will love to use.
Onboarding process

A quick and streamlined onboarding process makes it easy to add teams and applications. A secrets manager should make life easier for development teams, not harder.
Application access

Choose a secrets manager that meets your access policy requirements without additional overhead and complexity. The most secure secrets management implementation is the one that’s most widely adopted.
Secrets Injected via environment variables

If possible, choose a Secrets Manager that supports injection via environment variables to ensure secrets access is language and framework agnostic without the need for third-party libraries or SDKs.

Level Up Your Organization’s Security with a Secrets Manager

The security risks posed by ad hoc and insecure secret storage and access are too pressing to ignore. Secrets stored in encrypted formats such as .env files with no centralized source of truth to control access are putting themselves at risk unnecessarily.

A Secrets Manager empowers your team to apply the principle of least privilege with fine-grained access controls for restricting what secrets and environments are accessible based on user groups and machine identities. All of which can be automated.

Using environment variables to inject secrets into applications at runtime maximizes language and framework compatibility, and the easier your chosen secrets manager solution is to implement, the better chance it will be adopted organization-wide.

Don’t leave your organization’s security to chance.

Start your Secrets Manager journey today by signing up for Doppler—The multi-cloud Secrets Manager your team will love that takes minutes, not hours to use.

Doppler: Truffle Security Recommends Doppler For Remediating Leaked Secrets

Ryan Blunden — Mon, 09 Aug 2021 07:00:00 +0000

If an engineering leader asked you "How we are protecting ourselves against the accidental leakage of secrets?", what would you say?

It's not an easy question to answer, especially for those more on the Developer side of DevOps who often don't have a background or focus on security.

The answer we see emerging is a two-pronged offensive strategy:

Secure Secrets Management
‍Sensitive data such as API keys and database credentials must be stored in an encrypted, audited, and accessed controlled secrets manager. ‍
**Secrets Leakage Detection **Continuous scanning of environments for leakage of credentials in places such as Slack channels, Git repositories, and other internal systems (e.g. task trackers and Wiki's)

While Doppler addresses secrets management, Truffle Security, the company behind the popular free and open source secrets scanner TruffleHog, is on a mission to detect and prevent the accidental leakage of secrets and credentials.

Dylan Ayrey, Co-Founder of Truffle Security recently published a blog post explaining why Doppler is their recommended for solution for enterprise companies looking for an instant remediation strategy once secret leaks are detected and no secrets manager is present:

There’s a lot of different secrets management solutions out there, but many of them aren’t very user friendly, and can be difficult to set up. One we’ve had our eye on recently is Doppler.

It's not just about mitigating the risks of secrets being leaked, as that will inadvertently happen, but what is critical, is how fast you can respond, revoke, and roll any leaked credentials. Again, from Dylan:

The more approachable and usable your secrets management solution is, the quicker leaked keys can be rotated out, and the less exposure time they have to bad actors.

Truffle Security and Doppler are a formidable team in keeping your secrets secure, giving attackers the least possible chance of getting their hands on valid credentials before their leakage has been detected and credentials revoked.

Kubernetes Security Hardening Guidance

Ryan Blunden — Wed, 04 Aug 2021 19:00:00 +0000

How to increase the security posture of a Kubernetes cluster is always top of mind for cluster administrators.

While we recommend starting with the documented Kubernetes security best practices, a new resource to add to your must-read list is the Kubernetes Hardening Guidance Report.

Produced by the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA), it details the most critical threats to the security of Kubernetes environments, providing guidance for cluster configuration in order to minimize risk.

More specifically, the report focusses on hardening techniques in three main areas:

Container and Pod scanning for vulnerabilities, weaknesses, and misconfiguration
Running containers and Pods with the most restrictive set of privileges possible
Network security recommendations for firewall configuration, network separation, authentication, and log auditing

You can download the report from https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF

Secrets Sprawl is Hurting Your Productivity. Here’s How to Fix It.

Ryan Blunden — Sun, 01 Aug 2021 02:29:00 +0000

Passwords, tokens, encryption keys, and API keys start to pile up as we spin up new cloud resources and release new apps. Because different systems and people need to access this authentication information, these secrets suddenly become duplicated across various services — even in some places they shouldn’t be!

A typical IT environment may use container-related secrets managers, such as Kubernetes Secrets or Docker secrets. Or, they might use platform-specific secrets managers such as AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager. They may even use key-value stores, such as Hashicorp Vault and CyberArk Conjur.

Organizations frequently use environment variables in individual virtual machines (VMs) and serverless functions to configure applications and provide secrets such as API keys. Unfortunately, some developers are still hard-code sensitive data into their application’s source code, or using plain text unencrypted files such as a .env file. Of course, this is never a good idea, and most teams realize this, but a viable alternative that won’t impact productivity is often not apparent.

It becomes increasingly challenging for companies to distribute, organize, and secure these secrets as IT infrastructure grows. This haphazard storage of secrets is called secrets sprawl.

Secret sprawl is such a common problem because secrets hold together these different application components. Most modern applications consist of smaller building blocks and use a growing number of technologies.

As developers ensure each application has everything it needs, it’s easy for our secrets to get out of control. Everything seems fine until we need to make a change or — even worse — there’s a security breach. Suddenly, tracking down those credentials becomes a time-sucking, trying task that we never expected.

Let’s look at some of the pain points secrets sprawl causes and how we can get our secrets back under control.

Secrets Sprawl’s Pain Points

When our secrets are sprawled all over our infrastructure, the development and operations teams (and, really, the whole organization) face some pain points. First, it’s challenging to keep track of where all those secrets are. When we need to find specific secrets, we might end up digging through various secrets managers, code, or even someone’s desk for that elusive USB stick.

Did our credentials just change? Now we have to track down every duplicate copy of that secret across our entire infrastructure.

Second, security is a notable secrets sprawl pain point. When our secrets spread across a wide area, it’s challenging to track who has access — let alone manage that access and enforce access rules. We should protect and encrypt all these secrets locations, but there are just so many. And they’re growing every day!

We know we should frequently change our secrets to reduce the chance of compromise (especially when an employee leaves), but secrets sprawl makes this a ton of work. Maybe we really wanted to work on that new feature instead, so we decided to just change our credentials next month.

Third, secrets sprawl makes it more challenging to share secrets. It’s great that our credentials are nice and secure in Kubernetes secrets. But that doesn’t help when we need to use that secret for an Elastic Compute Cloud (EC2) virtual machine or in our continuous integration and continuous deployment (CI/CD) pipelines.

Also, if secrets are in multiple locations, maybe only a few employees know where they all are. As a result, the organization becomes dependent on those specific employees, making it even harder to get new team members up to speed on the secrets architecture. There’s also the potential that the keepers of the secret might feel a sense of ownership and be reluctant to share. When they do share the secrets, doing it in plain text isn’t ideal.

Most organizations plan to scale their development team and architecture. Without transparency in secrets management, secret sprawling becomes a major scaling bottleneck. Growth also means more secrets, and more secrets means more locations in which secrets are stored.

Now that we know some of the pain points secrets sprawl causes, let’s look at some solutions.

Managing Secrets Sprawl with a Secrets Manager

Centralization helps keep secrets sprawl in check. When you keep all your secrets in one central location, you know where to find them and can easily make changes and monitor who and what machines and services have access. Secrets managers help centralize your secrets by storing them in one place where all your services connect.

If your company is small enough, you might decide to deploy all applications as Kubernetes containers and store all secrets in Kubernetes Secrets. This centralization tightens access control and makes it easier to audit secrets. Maintaining a centralized and encrypted secrets store ensures the organization applies consistent threat protection. It reduces unauthorized access and boosts security posture.

In a large company, one administrator might manage multiple Kubernetes clusters as you scale. In this situation, individual users can’t enforce application-wide policies. This means secrets are better managed.

Sticking to a single cloud provider and using its built-in secrets manager helps. But, this really limits what you can do with your infrastructure. A single provider might not have all the services you need for dynamically growing infrastructures. Many organizations are pivoting to the multiple cloud computing services approach for better disaster recovery and better servicing users distributed over a larger area (or even internationally) with edge computing.

Choosing a Multi-Cloud Secrets Manager

Multi-cloud capable secret managers such as Doppler and HashiCorp Vault reduce secrets sprawl by centralizing secrets storage with integrations for a wide range of platforms.

Vault’s Key-Value store, while flexible, wasn't designed for organizing secrets by application or microservice. Developers must determine how secrets will be stored and extracted from Vault and injected into their application. Some organizations will need and appreciate Vault’s flexibility but it could be prohibitively complex for others, making them avoid using it and defeating its purpose. Its storage and maintenance costs can also add up. Above all, secrets management is not just about Key-Value storage.

But Doppler offers a better solution. It was built for today's microservice world, organizing secrets by application with a customizable list of environments. It enables you to control all your secrets centrally, and you can set it up across every environment, from development to production.

Doppler delivers a seamless integration experience for every cloud provider and platform. While Doppler provides centralized secrets storage management, you can still use the built-in secrets managers your favorite cloud services provide, such as AWS Lambda, Azure Key Vault, Heroku, Vercel, and a growing list of others. Secrets automatically sync to external secrets stores, saving developers time so they can focus on building products and features.

With Doppler, it’s a completely managed service, saving teams time and money otherwise spent on deploying, updating, and supporting a self-hosted secrets manager.

Conclusion

Because of the growing trend to deploy microservices to multiple clouds and platforms, every organization faces secrets sprawl, even if they don’t realize it yet. It’s a natural consequence of growth. As developers, we can be so immersed in our secrets sprawl, or only focussed on our particular application that we aren’t even aware of the bigger picture and impact secret sprawl is having across teams. It becomes the status quo despite hindering productivity.

Secrets managers reduce secrets sprawl by securing all these credentials in a central location. Change your credentials everywhere with just one click and see exactly who and what can access any given secret.
‍

Whether you’re part of a fresh startup or a growing organization, it’s never too early or too late to wrangle your secrets into one manageable dashboard.

Doppler: Visual Studio Dev Containers For Node.js Apps on AWS

Ryan Blunden — Thu, 08 Jul 2021 07:00:00 +0000

If your Node.js applications are deployed as a Docker container in production, doesn't it make sense to develop inside a container as well?

Follow along with this live-coding session to learn how to build a Node.js remote container-based development environment on AWS using Visual Studio Code Dev Containers, including support for step-debugging and development-specific package installation.

Also check out our Visual Studio Code Remote Dev Containers on AWS Set Up Guide to learn how to configure your local and remote machine and you can find the code used throughout the video on GitHub.

If you have questions, you can find me on Twitter or send me an email at ryan.blunden@doppler.com.

Video Transcript

I'm Ryan Blunden, developer advocate at Doppler and I'm really excited to be presenting to you today, a pretty ambitious topic, really. Not just dev containers on the VS Code, which is a very new technology, but how to actually run dev containers remotely on AWS. So, I'm going to share my screen and we're going to get straight into it.

Now, because this isn't your typical webinar, we've got a bit of a to-do list to get through. Now, you don't need to necessarily read all of these things, but this just gives you an idea of what we're going to try and tackle in the 45 minutes or maybe 42 minutes that are left for the session today.

So, just to start off with a really quick intro in terms of the application. The application is called a Mandalorian Gifs Generator. And it's a really simple app that, as you might guess, randomly generates Mandalorian Gifs. I'm a big believer that if you want to learn something, you should build a real application and test it out that way. So, if we go to localhost:8080, and take a look. This is the sort of thing that we're going to be deploying in our remote container. Cool. So, let's stop that and then heading back to our list. The other thing I want to show you quickly is what does it actually look like to launch a dev container? Now, I'm going to be using this with Docker locally because that's a bit of way to save time.

And so, if we bring up the command palette in Visual Studio Code, we can see here command to rebuild and reopen in the container. So, this is the exact flow that you'll do if you've got your code cloned locally, and then you want to launch that code in a container, that is the workflow that you will do. And let's see. Ah, we have some sort of error. Now, what happens is when you've got an error, you will then bounce back to Visual Studio Code, so you can make that change and then go back into the container flow. And I have a suspicion that the reason why this is happening is to do with volume mounting. Now, if there is one area of Visual Studio Code in getting this all to work that is the most challenging, it is by far mounting your code in the container.

I'm going to get into this code a lot more later, but as you can see here, I'm using the workspace mount that will be in the virtual machine. Here, I'm going to be using the local version of that. And if you are going to start off with dev containers, I definitely recommend doing it locally first with a local installation of Docker, just because there are a lot of moving pieces. What we can see here is once we have opened the container on Visual Studio Code, this is the really cool part. This is why Visual Studio Code and Microsoft and GitHub are going to succeed in that dream of having a completely remote development environment that still has all of the features that we're used to. So, we can see here that I've got all of my project files. I've got an integrated terminal except this terminal is actually running inside the Docker container.

And what's really cool is because it's Visual Studio Code, I still have all of the features in terms of step debugging and all of those sorts of things that I would normally, if I was, using Visual Studio Code and everything was running locally. So, I'll reopen this code locally. Now, I'm from Doppler and Doppler is a secrets manager. The purpose of this all isn't really going to be to feature Doppler but just we'll be touching on some things related to dotenv files, which can make things a little tricky with Visual Studio Code. So, if you are going to use Doppler with Visual Studio Code, which I'd highly recommend you check out, what you'll need to do is launch Visual Studio Code with some environment variables that Doppler needs in order to configure itself so it knows which secrets to pull in for which project.

To give you just a really quick peek at what Doppler looks like, essentially you have a list of projects, this one's Mandalorian Gifs, and just like an env file, here are all of the keys and the secret values over here as well. So, we'll see if we can get to doing a brief demonstration of Doppler later. Okay. So, I'm going to [inaudible 00:06:18] this out and now let's get back to our presentation. And I keep checking on time. All right. So, what I want to do is just really run you through the steps that you'll need to go through in terms of setup. Ideally, if you want to follow along, or if you want to do this afterwards, you'll clone the repository. And if you want to check that out, it's at Doppler HQ and then just search for the Mandalorian Gifs node repository. So, once you've done that and you've cloned it locally, then what you would do is just do what I did then.

You will use Docker and the Visual Studio Remote dev containers to try this out locally. And the reason you want to try it out locally initially, is because as you saw, every time it goes from standard Visual Studio Code to the dev container it's launching containers in the background, it's doing a bunch of setup. And as soon as you move this to a remote host, it becomes a lot slower. So, it's just like when you're developing applications, you want a really fast feedback loop. You want that for the dev container development process as well.

The next thing that you'll need is you'll need SSH auth set up. Now, if you don't know a great deal about open SSH, that's a topic in and of itself. Microsoft does have some good resources on learning how to use open SSH and the reason why we need this is because the easiest way for our local Docker CLI to control the Docker host that's going to be on AWS is through an SSH tunnel. We can do it, otherwise, using TLS and TCP and sockets, but that's kind of really complicated. So, we're going to be using SSH. So, ideally, you understand, you've used SSH before and you already have an SSH identity set up because that's what we're going to use to be able to connect to the machine.

Next, you're going to need some AWS credentials. Now, the good news for running this in the cloud is you don't necessarily need to do things like EC2 instance with security groups and assumed IAM roles and stuff like that. That's perfectly good for production workloads. All you need and what I would recommend is a Lightsail instance. So, Lightsail is kind of like DigitalOcean on AWS, where it makes it easy just to create an instance. You can directly edit the network ports right in this UI here, and it just makes it really, really simple. So, that's what I would recommend, and it's not too expensive as well.

And then the optional step is installing the Doppler CLI locally. If you want to do that, it's really, really easy and what's good is it's free to get started. So, you don't have to enter a credit card or anything like that. And you still get all the features you need if you wanted to use Doppler in the production. And if you had to install page, you'll see how to install the Doppler CLI for, basically, every operating system imaginable and we've got first-class support for windows as well.

Okay. Now, let's get into some concepts. Now, I'm going to presume that because you are attending this webinar or you're watching it after the fact that you know what Docker is, you know what remote development is and you're on board. You want the future sooner. Now, if you're not sure, the reason why remote development is so compelling is for a couple of reasons. One is that if your application is deployed in production in a container, then it makes sense that your development environment is as close to that as possible. Because if you're using a different version of node locally, because that's what comes with Homebrew and that's different from what you're using in your container, that's just like an example of an opportunity where unexpected things can happen when you deploy your application in a production environment. Now, in terms of why development containers as opposed to virtual machines, GitHub and GitHub Codespaces is going to work in a container.

And one of the cool things that they're going to be able to do as a built-in service I imagine is you will be able to check out a teammates branch as part of reviewing a pull request. That's going to open in a remote dev container. And as long as there's a way to get the secrets that that application requires, so it can start off and it configure itself, you can actually preview the application for that pull request is for, without having to bring that down locally, do a whole dance and all that sort of stuff.

So, that's what Visual Studio Code and GitHub is moving towards, and it is incredible for that. You don't necessarily need to have a project locally and rebuild the container. You can optionally just say I don't want you to build a container. I just want you to use an existing image and then just launch me into that. So, it's pretty amazing technology and that's where the future is as well. Being able to spin up a container development environment in minutes without having to do much work at all, because the hard work is just getting Visual Studio Code to be able to control Docker. All right. So, we've talked about why remote containers.

Now, if you go to Visual Studio Code's site, there is an entire section dedicated to just remote development in general because there's a couple of options to get started. One is developing just remotely on the VM itself versus in a remote container. Now, I would say that remote containers are where you want to end up or perhaps just local containers, but doing remote development over SSH is a good way to start because at least it'll expose you to tools such as the Remote Explorer, you'll get to configure open SSH. So, that is a great way to get going. In terms of, you know, which one is better, containers are great because they're more lightweight. You can run different sorts of applications and you can run applications that have multiple containers as well.

But remote VM, that's still remote development. So, that's super cool. Now, really briefly, to sort of demystify the magic that you saw before, I want to really briefly touch on how Visual Studio Code itself is architected because it's not necessarily working the way that you think. Now, focusing on this diagram here, the way that it works, Visual Studio Code is running on... Obviously, it's running on my machine. But only the themes that it needs to run locally when we're actually running a container. So, things like any theme I have, that is going to be running locally, but the things that are going to be running remotely is the VS Code server. And here, when we see open exposed port, it is open SSH that is providing a tunnel into the VM and then into the container itself.

This is really powerful. This is why for instance VS Codes, JavaScript and TypeScript language server is able to run remotely and give us all of that code intelligence. So, all of the heavy lifting is being done here. This is essentially almost just a thin client text editor at this point. Visual Studio Code is also taking care of doing things like port-forwarding from our local machine to inside the container. So, we can just hit our application localhost:8080 but it's tunneling that through to the actual container. So, the amount of complexity that Visual Studio Code is taking care for us is huge and the fact that we can run extensions from Visual Studio Code just about anything inside our containers as well is just phenomenal. So, this isn't a stripped paired back version of remote development in VS Code, they are trying to give you the full, entire experience.

Okay. The next step and I won't have a chance to go into this in too much detail, but it's a similar thing with Docker. So, if you've used Docker locally, it's easy to think that Docker is just like an encapsulated application, but it also has a client and a host. And so, what happens is, at the moment on my machine, let's say, when I do Docker run, everything is running in a virtual machine on my own machine.

But what I can do is I can configure the client and say, "Client, I actually want you to control this different Docker host over SSH." And there's a way to configure Doppler... Sorry, Docker, with things called context. That's what enables you to say, "Hey, Docker, I want to use this remote host," or, "I want to switch back to doing something local." So, that's just a really cool feature to learn about Docker that is going to come in handy because obviously when VS Code is running at dev containers, we want to run the dev containers on our remote VM.

We're all good. All right. Let's keep cracking on. How am I going for time? Yeah, not too bad. Okay. Now, some of these steps for you, they might be very pedestrian, very boring, but I want to try and set everything up from scratch, A, because if something goes wrong, that's an opportunity to learn. And, B, it's important to see how all these pieces fit together because that's not something I've seen in any other webinar or tutorial so far. And that's why I was so excited to share this with you.

Okay. So, the first thing that you need to do is when you go in and create the instance, go into your account and add your SSH key. Now, the reason that you want to go and add your own SSH key, it's so that you don't have to use the one that they provide, you have to download it, set the right file permissions, use the right flag on open SSH. You don't want to have to do any of those things, or you can learn how to add it to your identity, but then that's tricky because it doesn't do it over machine restarts. So, chances are, if you're a developer, you've already got an SSH key. So, just upload that, it's going to make things so much easier.

Now, if we head back, we're going to create our instance. Now, in terms of how big your instance is going to be, well, I guess it depends on what you're going to be doing. If you're running all the services in a monorepo, then you're going to need something really beefy. And that is probably one of the biggest benefits of developing remotely as well. Even if you've got a massive specter MacBook Pro that still may not be able to launch every container that you need. So, in terms of which machine, it really depends on the demands, it depends on how much your boss will let you spend, it depends on a lot of things. So, I'm going to choose just this one. It's pretty cool for CPU's. That should definitely do it. Check that the SSH key that you have uploaded is the one that's selected here. It's going to make it a lot easier. Let's do DopplerDevContainer and then the other thing that we'll want to do is add a launch script because what we need to do is we need to install Docker and get that all going.

And so, the easiest way to do this... This script, by the way, all of this code is available here, if you go into the AWS... Oh, that's interesting, and then you can just grab it here. All right. So, what we're going to do is I'm going to grab all of this code, go back to Lightsail, paste it in here, and then it's going to execute that as part of setting the machine up. So, we'll create the instance and this will take about three to five minutes, something like that. Okay. So, still booting up, still installing Docker and things like that, what I'm going to do is I'm going to copy this IP address and I'm going to create a host alias. Now, the reason why I do this is I'm not good at remembering things in general, so, I don't have much hope of remembering an IP address but what I can do is go into a regular terminal and it's not code.

Sudo nano, and I'm just going to make an entry in the host's file. Now, maybe you haven't used a host file before but it's a really cool way of just taking an IP address and adding an alias. So, instead of having to refer to this when we set up our SSH connection, we just use this instead and let's save that. Cool. All right. So, now this isn't going to work because I don't think the port for ping has been set, but you can say that it did resolve to that IP address. All right. So, let me keep going back to my list. We've done this. We're creating the virtual machine, that's a work in progress. We've created our host alias, and then the next step it's configuring the VM. So, what we can do is, by now, we should probably be able to connect to it over SSH. User is Ubuntu because that's the Ubuntu distribution and our machine alias is AWS dev.

Alright. And we are in. Let's see if Docker is installed. Hey, it's installed. This is a good sign. Let's now verify, if I go back to my instructions that it is working. Let's see. Let's go back here. That's not what I wanted to do. Let's try that again. And do we get, "Hello world?" Hey, we get, "Hello world." Okay. Now, for those of you that are systems admin inclined, you'll notice that I was able to run Docker, which is a privileged thing to be able to control on a system, but it's using the Ubuntu user. Now, the reason why we need to do that is Visual Studio Code will be using the open SSH connection, which uses the Ubuntu user. So, therefore, we had to give Ubuntu user permissions to be able to control Docker, and we certainly didn't want to make it possible for the root user to connect over open SSH.

So, that's just a bit of a necessary thing that we had to do. All right. I think we've gone pretty well on time. Let's see. So, we've tested that. We've configured our VM. And the last thing I should check is that we have the code for this on the virtual machine. Okay, which we did not. So, let's go and clone that down. Oops. Okay. Now, the reason why we're cloning the code for this repository, even though we're going to be working in a dev container is because one of the most challenging aspects of dev containers is when you've got your repo locally, and then you want to be able to develop on it remotely, that code has to get to the remote Docker server somehow.

Now, when Visual Studio Code builds your dev container, it's not doing anything magic, it's basically looking for the Docker file, and from scratch, it's going to go through all of these steps. Now, because we can see here that it's copying the code that's inside source, it is going to do that initially locally, but then once you're in the cloud, once you're running your container remotely, well, it doesn't have access to your local files anymore. And there's no way to mount your local files to the remote source. That's only something you can do if you're using Docker locally. This is probably the most challenging aspect of this, so if you've got any questions, send me an email at ryan.blunden@doppler, and I can point you in the right direction.

Okay. So, the reason why I've done this is a couple of reasons. One is I can authenticate this machine with GitHub or my code host, so I can go through all of my Git workflows here. And what we're also going to do is we're going to just Mount this Mandalorian Gifs node path into the container, because that's so much easier than using like a volume where it's difficult to then get those code changes in. It's already got a Git repository so Visual Studio Code is going to be able to pick up those changes. It's just a much, much easier way to go. I'm going to send you some links after so you can delve into the different ways of doing this, but to get started with remote development, which is definitely in the advanced part of things, [inaudible 00:23:18] in the VM and then mount that code from the VM into the dev container.

All right. There's a lot to remember, which, hopefully, this to do list concept is working for you. All right. So, now that we've got our VM in the cloud, now we need to configure Docker so it is able to control that remote host because that's what Visual Studio Code is going to do for us when it creates the dev container. All right. So, remote SSH, we've already tested that, that works. Now, let's create the Docker context. So, because I have my AWS dev alias, that's what I can use here. If you're using an IP address, then you would put your IP address in here. So, I'm going to copy this code and then run in the shell. Now, the reason why I'm not running it, I can just as easily run it in the shell, so, let me start doing that.

Okay. So, what this is going to do is it's going to create a new context, which is a new configuration for what the CLI is pointing out to control in terms of a Docker host. And here we are saying that we're going to connect to it via SSH. Now, I will say that sometimes there can be performance challenges using open SSH and chances are, if you're like DevSecOps team or your DevOps platforms team is setting this up for you, they may also set it up using TLS. So, you're communicating directly with the Docker daemon, but for our purposes, and this is also great for security, this is how we're going to go. Now, in terms of being able to inspect the different contexts that we have, we can see that we have AWS dev and then the default is from Docker desktop, which is where... And we can say that that's the one I'm using because it has the little asterisks next to it.

In terms of Docker configuration, you can either do it this way or you can do something called using the Docker host setting in VS Code. Now, this is where things get a little bit tricky too. The reason why you might want to use docker.host in VS Code instead of creating the Docker context is because if you do it this way, it means that Visual Studio Code will respect the setting that you put in here, but it's only specific for Visual Studio Code. So, for instance, if you want it to use dev containers remotely in here, but then when you run Docker containers in your local shell, that can still point to your local Docker instance. Personally, I try to stay away from Visual Studio Code settings if they're not for the workspace as much as possible. And it's really not difficult to be able to set the Docker context. You can just say docker context use AWS dev. And now we're using AWS dev context.

Then if you wanted to switch back to use your local Docker instance, then you just do that. And the great thing is that this setting is consistent between Visual Studio Code and your system as well. So, I think it makes much more sense just to use it directly here.

Okay. So, then the next check, let's actually try running a container, running it locally, as in triggering it locally, but we should be able to observe that it is actually running remotely on our VM. So, we'll see if this works. You'll notice now that there's a bit of a delay and that's because it has to go through open SSH and we can see, yeah, it seems to be working. Actually, I just want to do this again, just to prove this is a real demo into Docker PS, and once that is up and running, see, hey, we're in the machine and we can see that container running.

Cool. All right. So, that is basically everything covered in terms of Docker configuration. As I said, the remote SSH part and configuring that authentication between your machine and the VM, on Windows, this is a lot tricky than it is on Linux or Mac, depending upon if you're using WSL, depending upon the [inaudible 00:28:05] SSH client, if you're using a terminal such as Git Bash or the integrated terminal in Visual Studio Code, there's a lot of different options, and so be prepared to go on a bit of a wild ride to get that set up, but then once it's set up, it is super easy. Okay. So, now we're kind of getting to the good stuff, the dev container related stuff. Let's go through and show you all of the extensions that you need to install. To save you having to manually search the extension store, I can just share with you this. This is what I want to do. It's not Q&A messages. If that was message, where would I be?

[inaudible 00:29:01] There we go. All right. [inaudible 00:29:10] Okay. All right. So, I'm not going to install a whole bunch of extensions. It's really just these three. And let me close that. And the extensions that you absolutely have to have, this is going to be done for you, so you don't really need to worry about that, but the Docker extension and the remote containers extension, obviously, so we can have remote containers. Now, the Docker extension is really cool because if you want to do things like clean up the dev containers, clean up any images that are created, then you get this random little view here and you can do things like remove this image or remove a container. We're not really going to be using that too much. So, we've done that. We did the dev container locally before, so we don't have to worry about that.

Let's see here. Let's now take a look at the remote UI as well. This remote UI is a relatively new feature and it is also brilliant. Now, it's different because the Docker UI is really just showing us things about Docker, surprise, surprise. The Remote Explorer, what's really cool about this is that it understands dev containers opens as stage VMs as well. Things like WSL instances. So, this is your sort of like map for anything that you might be doing in Visual Studio Code that's happening remotely. Okay. Let's do that. Now, let's quickly talk about the dev container workflow. What I found really challenging when I was learning about dev containers is that there is so much documentation and it's really overwhelming. But the thing is, there's a lot of things that Microsoft are doing to make things as easy as possible for you.

So, while it is a bit of a steep learning curve, it's not too bad once you get your head around the overall concepts, which is really the point of this webinar. So, essentially what you will do, I'd recommend that you're using Docker locally. So, you'll be modifying your dev container file. And just to quickly show you what that looks like. So, you'll be doing things like... You'll be, perhaps, modifying your Docker file. You'll be creating a command that runs when the container first starts like installing dev packages, configuring ports, all that sort of stuff.

So, there is a lot of like tweak a dev container, rebuild it, it didn't work, go back to the code, tweak it, rebuild. And essentially that's what is going on in this workflow here. So, you open it, did everything work? Did you get an error? Okay. Do some success, then reopen it again. That's why doing it locally you'll get a much faster feedback loop and be prepared for a lot of these when you first get going.

Okay. Dev container syntax. Let's go cover that. So, the spec for dev containers is huge, and that's because there are so many different things that you can do. You've got build args, build targets, a lot of the same things that you've got when you actually build your container. You've got the container env, you've got remote env and you've got things where it's like, what is the VS Code server should start as? Should that be different to the container that it's actually going to be running in? One thing that they advocate for, which is good, is running as a non root user, reason being is that if you run as a root user in your container, because we are mapping the code from the host into our container, we have... We'll just go in here, you can see that all of this code is owned by Ubuntu in the Ubuntu group.

If we run our container as root, any files that we create in the container are going to be owned by root here as well. Now, in the resources I'm going to email you, there is an entire section on container user scenarios, such as making the user IDs match. Yeah, it's a whole thing but I've got a link in there to help you out. All right. So, getting back to the dev container syntax. It's reasonably simply to get going once you understand how everything works. So, let me just go through the most essential things for you to understand when it comes to the syntax here. First of all, you need to choose whether you're going to be building your image from scratch each time, or whether you just want to use an off the shelf image. Most of the time you'll want to do this.

This might be a good option if you're just wanting to quickly review a pull request and you don't need to necessarily rebuild the image just to do that. You can change the container user to be root, but what I would do is I would stay away from this until you know you need to. The reason why you might need to, in my example, all my command is doing, it's just installing the dev packages, such as Es Lint and Prettier and things like that. So, I don't need necessarily root access, but if you are installing things like a compiler like GCC, then you'll need root access to be able to install those system level packages. This is where we specify the command that we are going to run. Forward Ports, pretty straightforward, no pun intended. These are the ports that your application is going to be listing on.

Now, you can define the workspace folder. Now, by default, I believe this is just root workspace, which is this. Now, this is fine. Don't feel like you need to change this. The reason why I'm doing this is because I set up a path that is looking for the nodemon binary in node modules. That's why I need these to be the same. That's just me, you don't have to do this. Then, this is probably the most important part. You have essentially three ways. I touched on this really briefly earlier, you have three ways in which you can mount code in here. One is use a volume. This is great to get started. It's also awesome because you don't even need access to the file system here because Docker is going to be creating an inner volume.

And then what happens is Visual Studio Code takes that volume and then mounts it inside the container. Now, this is cool, but it's quite advanced because then you need Git credentials inside the container in order to be able to create [inaudible 00:35:44] Git operations as well. So, that's definitely on the more advanced side of things. This is the version that we're going to be using because it's going to be mounting the code that's in here into this here. And if you've ever used a volume mount into a running container for like local development purposes, this is exactly how you would do that. If you're doing things locally, then you just want this because you just say Visual Studio Code, wherever my workspace is on my machine, then just map that into here. All right.

So, yeah, as I said, because we're going to be running this remotely, we'll go with option number two. Now, you can also set environment variables for your container. Now, in this instance I've got the Doppler CLI installed locally. And the reason why I use Doppler and I'm not just being the developer advocate, but it makes it so much easier to manage your secrets. If I can just give you a really brief overview of why to demonstrate what I think is like the killer features to use in development, A, you get a really nice UI where you can go and change things, you can do things like add notes to secrets. And what's great is because Doppler is like a central source of truth for secrets. This is what your teammates are also going to be pulling from as well.

So, for instance, if they have just merged code and their code relies on a new config value called new secret. Well, if you were using the dotenv files, you have to have a way of getting this new secret variable out to everyone on your team. This might be over Slack. You might even be uploading just your whole env file for one to look at. That's not great for security, and it's just a manual process. Everyone has to then go and grab that and put it in their dot.env file. If you use Doppler and you make this change and save it, then the next time that you use Doppler to run your application, it's always going to fetch to the latest version of your secrets. So, that just makes it so much nicer. The other thing that you can do as well, which is particularly handy in development, because what happens is I'm working on a project stuff and Brian, they're working on different, different features, and they might need to override a particular setting.

Now, if we go in here, yes, they can change it here, but this is going to change it for all of the developers. We have a concept of branch config. And what that enables me to do is I can then come into this branch config and let's say, I can change it to G, I hit save. And then if I'm pulling my secrets from my particular config, it means that I will see this rating, but my teammates will still see what is in the dev config. And you can see that this little icon here saying that I've overridden that value. So, just a couple of reasons why you might want to consider using a secrets manager instead of dotenv files. And we're going to be talking about dotenv files in future articles on Doppler site, so, definitely go and check those out.

If things start to get weird and they just don't work in terms of step debugging, this is something you can try. As I said, this really is bleeding edge technology, so sometimes things don't always go as planned. And then here's just an example of an extension that's actually going to be run in the remote container. So, I'm going to be able to use Prettier to look out my Prettier [inaudible 00:39:05] I config and reformat my code, but this is all happening remotely inside the VS Code server. Okay. So, that is the basic syntax. And look at that, we're almost out of time, but that's okay. I only got a few more things I really want to talk about. So, we've talked about the container syntax. I've given you a crash course on the code mounting, but now let's actually do something cool, and look at step debugging.

So, to do step debugging, let's actually rebuild and reload this container, this time in the remote environment. So, what's going to do, it's going to show you all of the logs. For the most part, this is not going to be very interesting. Here we can see that it is rebuilding the container and it does this from scratch in order to make sure that if you've changed anything in the Docker file, that's going to be reflected in the container that it ends up running for you. While that is doing that, I want to talk about a couple of other things as well. So, the Visual Studio Code is cool in that it... Let me move this. It's cool because it sets up the local tunneling. So, I just lost my train of thought. It sets up the local tunneling, which is cool, but that can be a little bit slower because it's going from your machine tunneling over open SHH to the remote host and then tunneling into the container.

So, there's a lot of sort of like jumps that happen in there. If you haven't used it before, there's a great little product called ngrok. It's free to use initially, but they do have paid plans as well. And what this does is it's tunneling as a service. So, what's particularly cool is if you wanted to do something like test out Doppler webhooks for order reloading when secrets change, you'll need an HTTPS address for that. And so, you can run just your app on port 8080 just over HDP, and then ngrok will actually take care of creating a tunnel directly into that. And because it's a tunnel from ngrok servers directly into the container, it's much faster to preview your app and I'll be able to give you a demo on how that works.

Okay. So, this is definitely going to be a bit slower, the first time you start out, because there's a lot of machinery the Visual Studio Code needs to install. So, for us, not just the dev container, but all of the VS server infrastructure as well. Okay. So, far so good, now it's running our dev container set up command and, hey, now we can start to see our code in Visual Studio Code. This is, I think, the holy grail that developers have been looking for. We have a very consistent editor experience, including being able to have our launch configurations, but this is all running on the server.

Now, when I'm doing like training on these sort of things, I always like to prove that there's no magic. So, if we head back over here and we go into the Mandalorian Gifs code, if I do that, and then if I touch new file and we head back in here, what we should see is new file. And if I delete that, we should see it deleted from here as well. It's super cool, right? Almost like magic, but because you guys know what's going on, it's not magic. Okay. So, let's go back. What was my next step? Okay. Now... We'll, wait. So, the way to think about this is it doesn't necessarily look at your Docker file, its goal is not to automatically start off the container. What it's done is it's essentially created a development environment for you. And so, now we create a terminal. This terminal is also in the context of the container.

And so, now what we want to do is we want to run our application. So, let me just see if... Oh yeah. So, there's weird things like this that happen, bleeding edge technology. Let's see. Okay, great. So, Doppler is working. So, I'm going to run this application using Doppler because it takes care of an issue with dotenv files related to the host name, environment variable and I'll show you that problem in a second. Okay. So, we are launching our application and fantastic. Everything is working brilliantly. Now, we can open this in the browser, and this is tunneling from our local machine through the open SSH connection, then through the container on the host. So, the performance isn't terrible, but it's not the best. And so, this is why using something like ngrok can speed things up immensely.

So, let's go and set that up. And also, one thing I should say too is... Yeah, we're definitely over time. I'm going to keep going for five minutes. If you need to drop off, no worries. This is all going to be covered in the recording, and we'll send you a link to that. So, one thing that I'm a big fan of is making sure that everything is self-contained in your package of [inaudible 00:44:38] and so here we've got all of the instructions for setting up ngrok and the way I've built this repository, obviously this isn't a real app that we're going to deploy and create a business around, but all of the code that you'll need to learn, I make sure that everything is here, including things like how to set up your Lightsail instance with Ubuntu. So, if you are someone that just wants to go and just check out the code and learn that way, go for it.

So, this is working. This is all well and good. Now, I want to show you how do you use ngrok in order to make this even better. So, I'm going to say npm run ngrok setup. That's going to install everything. It's going to ask for my authtoken. And the authtoken is how I get features such as a stable remote URL, so I don't have to remember the random one that ngrok is going to create for me. Obviously, I'm using a password manager. Please use a password manager. Please don't ever use just normal passwords.

There's just too many breaches and things like that. You need a way to create super robust passwords that you'd never be able to remember. Okay. So, ngrok has created a forwarding interface from this to local host in the container. So, if I click this, we'll see it's much faster. So, this is really the speed and workflow that you would expect. This isn't bad and you don't necessarily need ngrok. But for me, I don't think this performance is ideal. And it's simply a limitation of just using remote SSH and all of that sort of tunneling as well. So, this is for sure what I would go with.

Okay. If things get weird, there's a couple of things you can do. You can tell Visual Studio Code that you simply want to rebuild the container. You can also ask Visual Studio Code to reload the window. And then if you get into some sort of like weird state where Visual Studio Code can't connect to like the container, even though you can verify the container is running, you can just disconnect, go back to your local view, and then you can go into using... Where's it going? Oh, once you reload Visual Studio Code locally, and you get access to Docker, then you can just manually delete the container that Visual Studio Code has created for you. You can delete the image that it's created for you. So, you can just get back to a clean slate. So, there's a few different things that can go wrong, but, hopefully after this guide it'll be reasonably smooth sailing for you.

Okay. So, that's everything that I really wanted to cover in this webinar. We went a little bit over time, but not too bad. So, hopefully, by the end of this obviously, there's a lot to get your head around and a lot to set up, but I wanted to demystify the magic of dev containers. There's a lot of technology the Visual Studio Code is taken care of for you, but essentially the VS Code is a client. The Docker CLI is a local client as well. We're using open SSH to communicate with a remote host, in this case, Ubuntu VM on AWS. And that is what is allowing Visual Studio Code to control containers on that remote host. We've gone through some of the challenges and benefits of remote development in containers.

The benefits are huge, but they're not without a bit of a learning curve and some configuration issues. So, whether this is the right thing for your team, it's up for debate. But I think it's worth experimenting with this, for sure. Taking you through how to set up a VM in AWS and I'd recommend Lightsail just because it's built for this kind of thing. You just want a simple way to create a VM. We've learnt how to configure Docker locally using the open SSH configuration. And we've learnt with Docker contexts, how we can switch between the AWS version and our local Docker version. You've got a basic understanding now of the dev container syntax at least in terms of the most important things that you're going to definitely need in terms of getting things working. And then we've also covered knowledge of source code mounting options.

Now, as a quick side note, I just wanted to touch on why using dotenv files can just be really problematic in this sort of scenario. So, I'm going to stop our debug server here. What I'm going to do is I've got a sample dotenv file. I'm simply going to copy this and paste it. Okay. So, we've got a dotenv file. And now when I run our application, instead of using the Doppler CLI, I'm just going to run it as normal. So, we have all of our app configuration in dotenv file. Things seem pretty good, pretty straightforward. Let's bring up the debug console. Okay. Now, for those of you that have used Docker and have had issues with being able to reach into the application inside of Docker from the outside world and have had troubles, you'll know exactly what the problem is.

If I now go back to here, it's not working. But we can see that the code is up, what's going on? Well, it turns out that the container itself has a host name, environment variable. And so, if we're using the dotenv [inaudible 00:50:28] which we are, let's see if there's a way that we can... All right, environment variables, so it makes the dotenv file the source of truth. Okay. So, it seems like this is something that's already been asked.

Okay. So, essentially this is not something they're planning on fixing at all. So, that's really challenging. I think on the homepage, they actually do have a section for how to override. There we go. This is really odd and we don't want to have to go through all of this issue. So, you can work around this. You can go into your launch configuration and you could always specify that for here. We also want to find env. We set this to HOSTNAME. We set this to 0.0.0.0, and now if we run it with node... Oops, What I probably wanted to do is just reload that. Cool. Let's run that again.

And, hopefully, we should see... Okay, now we're seeing HOSTNAME. And if we get back here, I know I've got things to work. Now, this is just... You might think, "Well, that's not that big a deal," but what we and our customers have noticed, it's that it's little paper cuts like this, that just add up to a lot of pain over time. You really shouldn't have to hack your launch configuration just to work around an issue with dotenv. So, that's why using a secret manager that has the source of truth to your secrets remotely is just going to be the way to go, regardless of whether it's Doppler or a different secrets manager, you really should look at moving away from dotenv files, not just for the productivity benefits, but for the security benefits as well.

All right. Well, sorry, I did go 10 minutes over time. I am going to send out a link after this session with some recommended reading, some troubleshooting tips, things to consider such as Git credentials that we didn't have time to go into today. And you should now have all of the knowledge to at least confidently start in bringing dev containers to your own remote environments and your applications. So, I hope you enjoyed that, and all of that made a lot of sense. It is a lot to get your head around, but I think Remote dev containers are the future. That's what we've always wanted. And once GitHub Codespaces come out of beta, this will be exciting because then everyone to a certain extent will get the benefits of Remote dev containers for things such as easily previewing a pull request in a container, in a live environment.

Doppler: Visual Studio Code Remote Dev Containers on AWS Set Up Guide

Ryan Blunden — Wed, 07 Jul 2021 11:28:00 +0000

This guide will provide a list of system requirements and links to resources to get you set up and ready for remote development on AWS using Visual Studio Code Dev Containers.

While the remote Ubuntu configuration steps are AWS-specific, the process will be almost identical for other cloud providers.

This guide assumes you’re familiar with the Terminal, SSH, Docker, Linux (Ubuntu more specifically), and AWS, so if you’re not experienced using these technologies, be prepared to spend more time getting things set up.

If you’re entirely new to Dev Containers, I’d recommend watching my webinar on Visual Studio Dev Containers for Node.js apps on AWS to give you a solid overview of what’s involved.

The Visual Studio Code Remote Development in Containers guide and tutorial are also great resources to help you get started.

Let’s get to it!

Example Repository

If you’re looking for an actual Node.js application using Dev Containers, check out the Mandalorion Gifs Node.js repository, including an example devcontainer.json file and launch configuration for step-debugging.

Alternatively, you can use one of Microsoft’s sample repositories with examples in Go, Python, Rust, NET core, and other popular languages.

Step 1. AWS Credentials

Your AWS credentials must have permissions required for managing a Lightsail instance and SSH keys, or if using EC2, you’ll need permissions for SSH Key Pairs, Security Groups, and EC2 instances.

If using EC2, I’d recommend checking if your organization has a base AMI you should be using.

Step 2. Install Visual Studio Code Remote Container Extensions

The following extensions are required for remote container-based development with Visual Studio Code:

Step 3. Install Docker Locally

A local installation of Docker is required for Visual Studio Code to manage containers on the AWS instance from your local machine.

Mac and Windows users should install Docker Desktop 2.0+, and Linux users will need a distribution of Docker CE/EE 18.06 or newer.

Step 4. Install OpenSSH Compatible SSH Client

You will need a Visual Studio Code supported OpenSSH compatible SSH client installed locally to create an SSH tunnel that Visual Studio Code will use to control Docker on the remote host.

For Windows 10 users, if you’re currently using Putty, I strongly encourage you to use Open SSH instead, as it will save you from frustrating troubleshooting experiences now and in the future.

Step 5. Create SSH Key

If you haven’t already, you’ll need to generate an SSH key and add it to the list of known identities for your SSH client.

Step 6. Create Ubuntu 20.04 AWS instance

Configuring your AWS Ubuntu instance consists of two steps: Uploading or importing your SSH key, then creating the instance.

Step 6.1: Upload Local SSH Public Key to AWS

AWS Lightsail is recommended as it’s the easiest option for creating an Ubuntu 20.04 instance running Docker, but an EC2 instance works just as well if that’s your preference.

Before creating your virtual machine, upload or import your local SSH public key (usually at ~/.ssh/id_rsa.pub) to either Lightsail or EC2.

You can optionally use an SSH key created by AWS, but I recommend using your own if possible.

Step 6.2: Create Ubuntu Instance

The most crucial step is to select your uploaded public key when choosing the SSH key for your instance. If you have an issue connecting via SSH later, it’s almost certainly because you didn’t upload or select your SSH key when creating the instance.

Next, you can use the following code as the launch script that will be run as part of initializing the instance, but you can always run these commands later once logged in via SSH.

#!/usr/bin/env bash

export DEBIAN_FRONTEND=noninteractive
export HOME=/root

# System dependencies
apt-get update && apt-get install -y make nano

# Install Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io

# Install Docker Compose
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

# Enable standard user (ubuntu) to manage containers (required for Remote Containers)
groupadd docker
usermod -aG docker ubuntu
newgrp docker

Step 7. SSH to the Ubuntu Instance

Once the instance has been initialized, get the IP address from the AWS console, then log in using the following SSH command:

ssh ubuntu@$INSTANCE_IP_ADDRESS

If you’re having trouble connecting, check that the public key you imported into AWS is attached to your instance, and if not, the easiest path forward is to delete, then recreate the instance with your SSH key selected.

If you’re still having connection issues, re-run the ssh command with the -v\ flag to enable the SSH client’s debug mode.

If the debug output still isn’t helping, check out DigitalOcean’s SSH troubleshooting guide, which will hopefully offer additional solutions to try.

Once logged in, you’ll need to install and configure Docker if you haven’t already.

Start by switching to the root user (sudo su\), then copy and paste the commands from our Ubuntu installation script.

Step 8. Check Docker Works on AWS

Once the installation commands have been run, log out, then SSH back in and try running a container:

docker run —rm hello-world

If this doesn’t work, try switching to the root user (sudo su\), and try running the container.

If you can run the container as the root user, it’s likely a permissions issue with the ubuntu user. In this case, I recommend reading Docker’s documentation on managing Docker as a non-root user, which should fix it.

If all else fails, try rebooting the instance, SSH in again and try running the container as ubuntu again, and if still having trouble, check out Visual Studio Code’s Troubleshooting Guide.

Step 9. Create a Docker Context Locally

It’s time to start putting the pieces together!

Now create a new Docker context locally that will communicate with Docker running remotely via an SSH tunnel.

Open a terminal locally, then run the following command to create a devcontainers context:

docker context create devcontainers --docker host=ssh://ubuntu@$INSTANCE_IP_ADDRESS

Next, you’ll switch from the default context (Docker locally) to devcontainers by running:

docker context use devcontainers

Then in the same terminal window, check that the Docker CLI can communicate with Docker on the remote machine by running:

docker run —rm hello-world

If you could not run the container, check out Visual Studio Code’s SSH Tunneling for Docker guide to help you troubleshoot connection issues.

If you want to switch back to using the default context, run:

docker context use default

Step 10. Try Running a Dev Container

Phew! If you’ve made it this far, you’re now ready to try running a Dev Container remotely!

Open Visual Studio Code, then run the command:

Remote-Containers: Try a Development Container Sample...

Then select one of the available options.

This time you run a Dev Container, it can take anywhere from 5-10 minutes. Visual Studio Code needs first to set up the extension host and Visual Studio Code server before building the Dev Container environment.

Presuming all goes well, you’ll eventually see the files in the container in the Explorer panel, and clicking on Terminal will open a command-line prompt inside the container.

Awesome work!

You’re running Dev Container remotely in the cloud with the experience of coding locally in Visual Studio Code!

Now Bring the Dev Container Workflow to Your Application

This guide has covered the required steps to set up and run Dev Containers remotely on AWS, although the same process applies regardless of which cloud provider you’re using.

Below is a list of additional resources you'll need if implementing at a team level, such as configuring Git credentials so you can push commits from your Dev Container environment.

If you’re new to Dev Containers, I recommend starting with the following videos from Microsoft, then progress to the documentation, as understanding the big picture helps before diving into the documentation.

Feedback is welcome, and you can reach us on Twitter, our Community forum, or send me an email at ryan.blunden@doppler.com.