Forem: Doppler

How we built our Secrets Rotation Engine

Nic Manoogian — Mon, 12 Dec 2022 15:24:41 +0000

Teams understand the value of rotating secrets:

Secrets are used in applications which often send logs and other metadata to third-party services, making accidental leaks a significant risk
Some employees need access to certain application secrets but should only have access while it’s needed
Secrets are often shared between several applications (perhaps even across multiple teams), making the exposure surface area massive

Great, rotation is a best practice. But when is the last time your team rotated a third-party service token or even your database credentials? If it was recently, how long did that process take?

Performing manual rotation is risky and cumbersome. Individuals with access to production running manual workflows for every secret, every N days? Not going to happen. Like any good “thing that has to be done over and over” — this process is ripe for automation, provided it can be done safely.

Doppler is in a unique position to build rotation automation because teams already:

Use Doppler as their source-of-truth for secrets
Configure their applications to load secrets from Doppler
Rely on Doppler’s access control, observability, and auditing features

However, a secret rotation feature needs to check some very important boxes before it can trusted for production use.

Requirements for Doppler’s Rotation Engine

Encryption and Storage

All rotation state data, including credentials and sensitive parameters, must be encrypted to the same standards as any other secret in our system.

Atomic Operations

Rotation must occur in “one shot”. Keeping all rotation operations atomic ensures that applications that fetch secrets from Doppler will always receive a valid (i.e. working) credential — even if Doppler’s infrastructure is interrupted (e.g. power or network failure).

Graceful Error Handling

Doppler will facilitate rotation via third-party integrations and integration points which are infamous for being the most brittle parts of any software system. Some errors that the engine encounters can be retried, while others will require user intervention.

Above all, data loss must not be possible. If an error or interruption occurs during any point in the rotation process, Doppler’s rotation engine must be able to recover without an issue.

No Public Access Required

If a secret source (e.g. a database) isn’t publicly accessibly, it won’t need to be made public for Doppler to rotate its secrets. Automatic rotation is meant to improve your security posture, not trade one set of risks for another.

We used an interesting strategy to meet this requirement and the solution deserves its own post. Keep an eye out for our next rotation post on Proxied Rotation to learn more.

The Two-Secret Strategy

To make automatic rotation safe and atomic, we can’t just go revoking old API keys and generating new ones. We need to generate a new key, transition applications to use it, and revoke the old key when it is no longer in use.

Doppler accomplishes this by always maintaining two valid credentials: the ***active*** and the **inactive*. Every N days (we call this the *rotation interval), the active credential becomes the inactive credential and the inactive credential is updated or replaced with a new active credential.

Let’s take database rotation on a 15-day rotation interval as an example:

During setup (day 0), the user provides Doppler with two valid starting credentials (username/password pairs) which will be used for rotation. Let’s say they’re called appuser1 and appuser2.
- At this point, when an application fetches secrets from Doppler, it’ll get back appuser1 with the original password
On day 15, Doppler will update the password (more on how in our Proxied Rotation post) for appuser2 and make it the active credential
- When an application fetches secrets, it’ll receive appuser2 and its new password
- The credentials for appuser1 are still valid but the credential is inactive and cannot be requested from Doppler
On day 30, Doppler will update the password for appuser1 and make it the active credential
- When an application fetches secrets, it’ll receive appuser1 and its new password
- The credentials for appuser2 are still valid but the credential is inactive and cannot be requested from Doppler

With this strategy, each credential is valid for two rotation intervals (2*N days). As long as applications fetch secrets from Doppler with at least this frequency, they’re guaranteed to always be holding a valid credential.

Types of Rotation

Doppler supports two types of rotation: Updater and Issuer

The database rotation example above is updater rotation. The user provides Doppler with two starting credentials and Doppler updates them “in-place”.

Not all credentials can be updated in place; some need to be freshly created and revoked. With issuer rotation, the user provides parameters for creating new credentials and Doppler will revoke and reissue credentials to perform the rotation.

Let’s take SendGrid API key rotation with a 15-day rotation interval as an example:

During setup (day 0), the user provides Doppler with the SendGrid API scopes that the API key needs
- Doppler immediately creates a new API key with the requested scopes and this key is available to applications. Let’s call it key1.
On day 15, Doppler creates a new SendGrid API key, again with the requested SendGrid scopes. Let’s call it key2.
- Applications fetching secrets from Doppler now get key2
- key1 is still valid but becomes inactive and cannot be requested from Doppler
On day 30, Doppler revokes the inactive credential (key1) and issues a new one (key3) to be the active credential
- Applications now get key3
- key2 is still valid but becomes inactive and cannot be requested from Doppler

Like the update strategy, applications are guaranteed to always fetch a valid credential as long as they re-fetch at least once every two rotation intervals. The difference with issuer rotation is that we’re continuously revoking old credentials and issuing new ones.

Rotation State

To meet the atomic rotation and encryption requirements, Doppler stores rotation state as a tokenized JSON string.

For those unfamiliar with tokenization: All sensitive data in Doppler is stored and managed by an isolated tokenization service in our infrastructure. Our web applications exchange sensitive data for tokens and vice versa. There’s more information on tokenization in our security facts sheet.

Here’s an example JSON state object for a SendGrid rotated secret:

{
  "parameters": {
    "scopes": [
      "mail.batch.create",
      "mail.send"
    ]
  },
  "activeIndex": 0,
  "credentials": [
    {
      "NAME": "Doppler-sendgrid-dev-SENDGRID-1668029618931",
      "ID": "NuINAi1TThiDDuXQUD1pfw",
      "SECRET": "SG.NuINAi1TThiDDuXQUD1pfw.tptg7Dt2CtGBAG2OFgI-17vBvhJOWn22mchOGtGChD8"
    },
    {
      "NAME": "Doppler-sendgrid-dev-SENDGRID-1668029625563",
      "ID": "4rY4TZbPSX-EBoAkZrOkJA",
      "SECRET": "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"
    }
  ],
  "pendingCredential": null
}

The parameters object contains static information about how to connect with the secret source and/or how to create credentials (in the case of issuer rotation).
The activeIndex field contains the index of the active credential in the credentials list.
The credentials field contains the list of valid credentials. Depending on the type of secret, the credential may contain several fields, some sensitive and some informational.
The pendingCredential field contains transient state that is used during the rotation. We’ll discuss this more in the next section.

We found this structure to be sufficiently generic to accommodate all types of rotated secrets, while still allowing our core logic to perform rotation operations abstractly.

When a rotation is performed, the rotation engine applies the necessary changes, builds new state, and commits the tokenized JSON object to the database.

Two-Phase Commits

The rotation state must always be accurate to meet our atomicity requirements, but interruptions (e.g. power or network failure) make this tricky for updater rotations.

To update a database password, the rotation engine needs to:

Generate a new password in memory
Update the DB user’s password in the source database
Commit the new rotation state to the Doppler database

This leaves a critical period between steps 2 and 3 where our engine could experience an interruption (e.g. power or network failure). When rotation is attempted after the interruption, our state might be out-of-sync with the true state of the database. Big yikes.

To avoid this, we generate a new password and immediately commit the credential to the Doppler database in the pendingCredential field. This occurs before we attempt to update the DB user’s password in the source database. If the engine encounters an interruption, it can check the pendingCredential field during the next attempt. If a pending credential is present, the engine can test the pending username and password to verify whether or not the credential was truly updated.

Issuer rotation isn’t affected by this problem, but the engine is careful to ensure that the old credential is revoked (or missing, to be resilient to interruptions) before issuing the next credential. This ensures that used credentials are not “abandoned” in error scenarios.

Delivering Secret Fields to the Application

Whenever rotated secret state is created or updated, the rotation engine saves the resulting secrets as “injected secrets” into the config. This allows advanced functionality like config logs, sync integrations, webhooks, secret references, and access logs to work for rotated secrets the same way they do for static secrets.

For example, if our rotated database secret was named DB_USER, then the secrets DB_USER_NAME, DB_USER_PASSWORD, and DB_USER_SECRET would all be available in the config.

Automatic Redeployment

Integrating Doppler into your deployment process is the most reliable way to ensure that your applications are always consuming valid secrets. When a secret is rotated, the new injected secret values are immediately synced to any integrations you’ve configured, just like any other secret change in Doppler. Automatic restarts can be implemented with the Doppler Secrets Operator, External Secrets Operator, or a webhook automation.

Handling Errors

In order for automatic rotation to be useful, error states need to be friendly and the dashboard needs to provide clear directions to users on how to resolve problems.

Doppler distinguishes between:

Authentication Errors: These disable the whole integration between Doppler and the service, which effectively disables all rotated secrets that use the integration to authenticate. Users are prompted to correct their authentication to continue.
Access Errors: These disable just the rotated secret. Users are prompted to grant appropriate access back to the credential that Doppler is using to facilitate rotation.
Transient Errors: These should be retried at some point in the future.
- Doppler will automatically retry these errors with an exponential backoff, eventually disabling the rotated secret and notifying the user.
- When a rotated secret is in an error backoff state, this information is available in the dashboard. This includes the time of the last attempt with the error message and the time of the next attempt.

Whenever an error requires user intervention, we notify users with the appropriate permissions to correct the issue via email.

Closing Thoughts

Manual rotation is complex, time consuming, and error prone. We believe that automation is the answer — but only if the tooling is secure, reliable, and fails gracefully when necessary. After all, a tool is only as good as its failure modes.

Why not kick the tires and see for yourself?

Doppler Encrypted Secrets Snapshots for High Availability

Ryan Blunden — Wed, 23 Nov 2022 05:36:42 +0000

Ideally, fetching secrets at runtime by your application is avoided altogether using Doppler integrations (e.g. Doppler Kubernetes Secrets Operator), which continuously sync secrets to your hosting environment ahead of time, but this won't always be possible.

For legacy applications, on-prem, and virtualized environments where secrets must be fetched at runtime, bundling Doppler CLI secrets snapshots into your application provides a robust failsafe in the event Doppler's API is unreachable.

The result is similar to a GitOps secrets approach, but without the messy business of committing encrypted secrets to Git repositories can be avoided.

Build. Encrypt. Deploy. Fallback.

Doppler secrets snapshots are bundled into the application build during CI/CD with a single command:

doppler secrets download secrets.json.enc

Then to fallback to a secrets snapshot when running your application:

doppler run --fallback secrets.json.enc -- ./app-start.sh

Secrets snapshots aren't just for high availability. They're perfect for network-restricted environments and offer protection if Doppler's API rate limit is exceeded.

The addition of the --fallback-only instructs the CLI only to use the snapshot:

doppler run \
 --fallback secrets.json.enc \
 --fallback-only -- ./app-start.sh

Now let's look into passphrase usage for encrypting and decrypting secrets snapshots.

Passphrase

By default, the CLI configuration is used to construct the passphrase, which is typically the Service Token value exposed via the DOPPLER_TOKEN environment variable. While this is the simplest solution, setting the passphrase explicitly has the benefits of:

Only using the DOPPLER_TOKEN for authentication
Allowing the passphrase and DOPPLER_TOKEN values to be rotated independently

Most importantly, your implementation must guarantee the passphrase remains constant from build to deployment during a release. As these are infrastructure secrets, we recommend a dedicated infra project for each application containing only the DOPPLER_TOKEN and (should you use a passphrase) a DOPPLER_PASSPHRASE.

‎Doppler integrations then keep infrastructure secrets in sync between your build and deployment environments, eliminating the risks associated with manual copying and pasting.

The following video demonstrates how to set up an infra project in three simple steps:

%[https://vimeo.com/769739281]

With the required pieces in place, let's move on to creating and using secrets snapshots.

Build

With the DOPPLER_TOKEN and DOPPLER_PASSPHRASE injected into your build environment (e.g. GitHub Action Secrets), we can now create the secrets snapshot file:

doppler secrets download \
  --passphrase "$DOPPLER_PASSPHRASE" \
  secrets.json.enc

Secrets snapshots also support name transformers and download formats:

# ASP.NET Core secrets snapshot
doppler secrets download \
 --passphrase "$DOPPLER_PASSPHRASE" \
 --format dotnet-json \
 appsettings.json.enc

Ensure the secrets snapshot file is included in your build, then you're done!

Deploy

With the DOPPLER_TOKEN and DOPPLER_PASSPHRASE injected into the deployment environment via automation (e.g Ansible), the Doppler CLI will attempt to fetch the latest version of secrets using the secrets snapshot as a fallback:

doppler run \
  --passphrase "$DOPPLER_PASSPHRASE" \
  --fallback secrets.json.enc -- ./start-app.sh

Summary

Doppler secret snapshots are your high-availability solution to ensure applications can always access their secrets.

Using Docker? Check out our dedicated Docker High Availability documentation to learn more.

State of Kubernetes Secrets Management in 2022

Tiexin Guo — Wed, 23 Nov 2022 03:40:24 +0000

Secrets are a fundamental building block of the modern Software Development Lifecycle (SDLC). Applications, CI/CD systems, API access, Databases, etc., all require some form of secrets/tokens/credentials. Keeping secrets out of source code, e.g. using the Twelve-Factor App methodology, is just one of many ways to ensure secrets are managed securely.

While Kubernetes secrets are relatively simple to understand, there are many factors to consider when deciding how they will be managed and injected into containers.

This blog post is a deep dive into the most popular approaches available for Kubernetes Secrets management in 2022, so without further adieu, let's get started!

Side Note: Enable Encryption of Kubernetes Secret Data

Because Kubernetes Secrets are stored as unencrypted base64-encoded strings, it's strongly recommended to enable encryption at rest for your cluster to provide additional protection in the case a malicious actor gains access to the underlying etcd data store.

Kubernetes Secrets Usage and Challenges

Kubernetes essentially provides two methods for injecting secrets into a container:

Secrets as files
Secrets as environment variables

The application can then either read the file contents of the secrets (in the case of mounting secrets as data volumes) or consume them as environment variables.

Secrets as Files

Accessing secrets from the file system is common in frameworks such as ASP.NET Core and Java Spring Boot; however, some consider it an outdated approach in preference to environment variables.

Because an application's config will likely vary between environments (e.g, development, staging, production, development), managing secrets in files pose several security questions:

Should you check the secret files in your version control system?
- If so, how to encrypt them (since they are secrets)?
- If not, where to securely store the secret files?
Should secrets be stored in CI/CD and shipped with the deployment artifact?
What if secret files are scattered across different repositories?
How is access to secret files managed and audited?
Is there a single source of truth for all secrets?
What if the secrets files are language/framework-specific, but an app in another language/framework needs access to the same secrets?

Kubernetes secrets can address most of these concerns by storing the contents of the secrets file in a single key and value and mounted as a single file inside a container.

To mount a Kubernetes secret into a Pod's container within a deployment:

...
    spec:
      containers:
        - name: web-app
          volumeMounts:
            - name: app-secrets
              # Mounted directory path in container
              mountPath: /usr/src/app/secrets
              readOnly: true
      volumes:
        - name: app-secrets
          secret:
            # Kubernetes secret name
            secretName: dotnet-webapp-appsettings
            items:
              # Key in secret containing appsettings.json contents
              - key: APP_SETTINGS
                # Results in file mounted at /usr/src/app/secrets/appsettings.json
                path: appsettings.json

The downside is that most secret files are framework specific, so environment variables may be preferable for teams wanting a consistent approach to secrets injection.

Secrets as Environment Variables

Environment variables from a Kubernetes secret are injected into a Pod's container process with precise control that allows for all or only a subset of secrets to be exposed.

The most straightforward approach is to expose every secret key-value pair using the envFrom property:

...
    spec:
      containers:
        - name: web-app
          envFrom:
            # Kubernetes secret name
            - secretRef:
                name: app-secrets

Or to only expose a specific list of key-value pairs, use the env property with one or more valueFrom items:

...
    spec:
      containers:
        - name: web-app
          env:
              # Environment variable name
            - name: MY_APP_SECRET
              valueFrom:
                secretKeyRef:
                  # Kubernetes secret name
                  name: doppler-test-secret
                  # Key name in Kubernetes secret
                  key: MY_APP_SECRET

Using environment variables for secrets comes with the benefit of not having to worry about teammates accidentally committing secrets to repositories. Unlike custom secret file formats like Java System Properties, environment variables are language and framework agnostic.

Kubernetes Secrets Source of Truth

So you've decided Kubernetes Secrets are the way to go! Awesome! But because Kubernetes doesn't provide a great experience managing and updating secret values, we'll need to devise our own strategy.

If we create Kubernetes Secrets using YAML files, where do we store those files? Encrypting them within a Git repository is one option. You'd then have the difficult task of managing encryption keys across different repositories and multiple environments and sharing secrets between teams with different needs and permissions. While tools such as Mozilla SOPS and Bitnami Sealed Secrets provide solutions for encrypted secrets, the operational overhead and complexity of managing secrets in version control is not the easiest solution to adopt and scale.

Suppose we choose not to use encrypted YAML files and instead use Kubernetes Secrets as the single truth source. This simplifies things as secrets are only ever stored within Kubernetes, but it also presents new management and access control challenges.

Imagine when you need to migrate to another Kubernetes cluster. You'd have to export and then import every secret, not to mention that multiple manual kubectl commands are required when updating a secret, and we know that manual operations are error-prone.

So, Kubernetes Secrets as the single source of truth is also a no-go.

We need an external and centralized single source of truth with encrypted storage and fine-grained access controls for managing application secrets that can then be synced to Kubernetes via automation.

Secrets Managers and Kubernetes

Secret managers provide a secure and encrypted source of truth for secrets storage at enterprise scale.

There are multiple choices on the market, such as Doppler, AWS Secrets Manager, HashiCorp Vault, etc. Although every secret manager has its quirks and features, they all meet the required security standards for storing and accessing sensitive data.

Secrets Managers solve the security, permissions, sharings, single source of truth, and operational overhead issues once and for all, leaving only one challenge: how to synchronize secrets from the Secrets Manager to Kubernetes.

Let's now look at the four most common secret sync approaches, working towards the ideal solution as we go.

1. Secrets Manager API or SDK

With this method, you don't need to use Kubernetes Secrets, as they are fetched directly from the Secrets Manager via its API or SDK.

The obvious downside is that this doesn't utilize Kubernetes Secrets at all. Plus, you are vendor locked-in (think of the tremendous overhead if you wanted to change to another Secrets Manager: you'd have to touch the code of all the apps).

Kubernetes Secrets provide a native and vendor-agnostic method for applications to access secrets, so this is the least preferred option.

2. Secrets Agent/Sidecar Injection

Using an agent or sidecar also accesses secrets at runtime via an API but provides a more native Kubernetes secrets access experience by dynamically mounting secrets into a Pod's container as part of the deployment process. It's emulating the same behavior as a Kubernetes mounted secret, but in a vendor-specific way and without actually using a Kubernetes Secret. Let's use HashiCorp Vault as an example to illustrate.

The HashiCorp Vault has an Agent Injector that dynamically alters pod specifications during deployment to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates (phew!). The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the Pod if annotations exist within the request. This functionality is provided by the vault-Kubernetes project. For more info, refer to the Vault Kubernetes Injector documentation.

By rendering secrets to a shared volume, containers within the Pod can consume Vault secrets without being Vault-aware.

There are, however, a few downsides. The first and most significant is that installing and using this method requires in-depth knowledge of Kubernetes and HashiCorp Vault. The beauty of Kubernetes Secrets is that they're simple to understand and consume. In the quest to integrate a Secrets Manager more tightly into the application deployment process, we now have something much more complex to wrap our heads around.

The second is understanding how the agent injects the secrets as a volume. Every container in the Pod can be configured to mount a shared memory volume. This volume is mounted to /vault/secrets, meaning the containerized app must read its secrets from files, losing the benefits of environment variable injection.

Third, for the injection to work, the user must add various Vault-specific secret injection annotations, for example:

vault.hashicorp.com/agent-inject-secret-foo: database/roles/app
vault.hashicorp.com/agent-inject-secret-bar: consul/creds/app
vault.hashicorp.com/role: 'app'

The first annotation will be rendered to /vault/secrets/foo with the second annotation rendered to /vault/secrets/bar.

And this brings operational overhead as you'll need to update all your Deployments' annotation. Also, you are vendor locked in now, and if you want to move to another secret manager, you'd have to rework all of your Deployment manifests.

3. Kubernetes Secrets Operators

A Kubernetes Operator is a specific type of application designed to extend the functionality of Kubernetes, such as the Doppler Secrets Operator and External Secrets Operator. Here, we'll use the External Secrets Operator to show how to add new secrets sync functionality to your Kubernetes Cluster.

The External Secrets Operator integrates external secret managers such as Doppler, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, and many more. The operator is configured to sync specific secrets from one or more Secrets Managers to various Kubernetes Secrets while continuously syncing any secret changes on a set interval.

The Kubernetes External Secrets Operator has many advantages over the previous methods:

It supports multiple secrets managers with no vendor-lockin.
It syncs secrets to plain old Kubernetes Secrets, so there is no need to update your Deployment manifests.

It does introduce additional operational overhead in that even though you don't need to maintain Kubernetes Secrets anymore, you do have to manage the new Kubernetes objects specific to the External Secrets Operator.

The External Secrets Operator works by using a SecretStore or ClusterSecretStore to provide authenticated access to a secrets manager and an ExternalSecret that references one of these stores and controls how and which secrets are synced to Kubernetes:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: doppler-auth-api
spec:
  provider:
    doppler:
      auth:
        secretRef:
          dopplerToken:
            name: doppler-token-auth-api
            key: dopplerToken
-
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: auth-api-all
spec:
  secretStoreRef:
    kind: SecretStore
    name: doppler-auth-api
  target:
    name: auth-api-all
  dataFrom:
    - find:
        name:
          regexp: .*

This is much better than maintaining native Kubernetes Secrets YAML and is significantly more straightforward than the Secrets Agent/Sidecar Injection method.

We'll also be covering the Doppler Secrets Operator later in this article.

4. Secrets Store Container Storage Interface (CSI)

A relatively new and emerging method uses a Kubernetes Secrets Store CSI Driver, which allows secrets stored in secret managers to be mounted into pods as a volume. Once the Volume is attached to the Pod, the data is mounted into the container's file system.

While this seems similar to the Agent/Sidecar method where you'd still have to read files, Secrets Store CSI also supports synchronizing secrets as native Kubernetes Secrets with an extra SecretProviderClass custom resource, allowing for secrets to be injected as environment variables:

apiVersion: secrets-store.csi.x-Kubernetes.io/v1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  # accepted provider options: Azure, Vault, or GCP
  provider: vault
  # [OPTIONAL] SecretObject defines the desired state of synced Kubernetes secret objects
  secretObjects:
  - data:
      # data field to populate
    - key: username
      # name of the mounted content to sync. this could be the object name or the object alias
      objectName: foo1
    # name of the Kubernetes Secret object
    secretName: foosecret
    # type of the Kubernetes Secret object e.g., Opaque, kubernetes.io/tls
    type: Opaque

This is the same level of operational overhead as the External Secrets Operator, as you need to manage one YAML per secret.

Auto-Updating Deployments When Secrets Change

Now that we've covered the most common methods for syncing secrets to Kubernetes, let's get to a significant pain point. When a secret value changes, how do you reload the Kubernetes Deployments consuming those secrets?

The Secrets Store CSI Driver can periodically update the Pod mount and Kubernetes Secret with the latest content from external secrets managers (e.g., auto rotation of database credentials). However, the CSI driver does not trigger a reload for the affected deployments—it only handles updating the Pod mount and Kubernetes Secret, similar to how Kubernetes updates secrets mounted as volumes.

Unfortunately, there isn't one out-of-the-box, elegant solution for triggering deployments to be reloaded with the solutions we've shown. You'd need to rely on open source tools such as Reloader, which watches for changes in ConfigMap and Secrets and does rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet, and DeploymentConfig.

Kubernetes Secrets Management Essentials for 2023 and beyond

It's now easier than ever to effectively manage Kubernetes secrets, and whether you're just about to move your workloads to Kubernetes or you're revisiting how to manage secrets, here's the list of features that I'd prioritize:

Easy to use, intuitive secrets management experience.
Easy to sync secrets to Kubernetes Secrets without a complex setup and installation process.
Environment variables for application config and secrets.
Low operational overhead that makes best practices easy to implement.
Multi-cloud capabilities to avoid vendor lock-in and siloed secrets.
Last but not least, the killer feature: automatically reload a Deployment/Pod when secrets change.

How the Doppler SecretOps Platform Could Change the Game

Because traditional secrets managers are just key-value stores (e.g. HashiCorp Vault and AWS Secrets Manager), most teams still have a frustrating experience when it comes to managing secrets.

Managing and syncing secrets must be as easy as possible because as friction and difficulty increase, so does the number of developers and teams that implement their own (and likely insecure) solution instead.

A Secrets Manager like this should've existed years ago, but now, a new kid is on the block: Doppler.

Simply put, Doppler is trying to make it as easy as it should be to manage secrets on Kubernetes thanks to the Doppler Secrets Operator. Easy to use? Check. Easy to sync secrets as Kubernetes Secrets? Check. Twelve-Factor App, cloud-native? Check. No overhead? Check.

And the big one: Automatic reload upon secrets update? Check! And it only takes a few minutes to install and configure:

If you're interested in trying Doppler, you can check out my quick tutorial.

Summary

The evolving Kubernetes Secrets landscape now provides development teams with many choices for storing, managing, syncing, and injecting secrets into containers.

I hope this post has given you some new ideas for improving your secrets management workflows in Kubernetes.

Managing Kubernetes Secrets with the External Secrets Operator and Doppler

Tyler Langlois — Tue, 25 Oct 2022 15:48:28 +0000

API keys, credentials, and other types of sensitive information are the primary way your application calls external services and interacts with the world outside its own runtime. Ensuring that your secrets are secure is one of the most important operational tasks that you need to address, but it's not just rigorous security that's important providing an ergonomic, audited, and maintainable flow for the management of those secrets is crucial as well. If managing credentials is confusing or difficult, mistakes can undo all of the careful work that goes into keeping secrets secure.

Container orchestrators like Kubernetes offer helpful abstractions to address the need for sensitive values with native Secrets. However, a Kubernetes Secret is a somewhat rudimentary object it lacks encryption by default and normally exists as a plain key/value within etcd. By contrast, services like Vault or Doppler are intentionally designed to provide strong guarantees and well-defined access controls. Can we combine the operational power of Kubernetes with the assurance of a fully managed secret provider?

Yes! The External Secrets Operator is a Kubernetes operator that bridges the gap between Kubernetes' native secret support and external systems that provide a canonical source of truth for secret storage. It does this by leveraging custom resources that define how to retrieve external secrets in order to manage the lifecycle of Secret resources in your Kubernetes cluster. In this tutorial, we'll use Doppler as the external secret provider to illustrate using external secrets in a real-world application.

Outcomes

The end goal of this guide will be to leverage well-managed secrets in an application deployed in Kubernetes. To achieve this, we'll use:

The Kubernetes External Secrets Operator
Doppler to store and manage external secrets
minikube or kind for a local Kubernetes API

At the conclusion of this tutorial, you'll understand how to store and manage secrets in Doppler, use those same secrets as native Kubernetes secrets, and ultimately use them in a running application. Let's begin!

Prerequisites

This article assumes that you'll follow along with your own Kubernetes cluster. You may choose to operate in an existing cluster (potentially within a sandboxed namespace), but if you'd like to use a sandbox instead, we suggest using either minikube or kind to create a local Kubernetes installation which will provide a safe environment for testing. In order to provide a clean and pristine environment, this tutorial will use minikube as the Kubernetes target.

In addition to a functional Kubernetes cluster, ensure that the following command-line tools are installed:

helm, which we'll use to install the External Secrets Operator Kubernetes resources. Use the helm quickstart guide to install helm.
The Doppler command line utility doppler using the Doppler CLI Guide documentation.
- You will need to register for a Doppler account to create projects and store external secrets.
kubectl to interact with Kubernetes. kubectl is available for a wide range of operating systems, and the Kubernetes documentation provides comprehensive installation guides for kubectl and other tools as well.
If you choose to use a local Kubernetes sandbox, install minikube and follow the instructions to create a local instance of Kubernetes. This can be as simple as minikube start, but consult the documentation if you need additional assistance.
- minikube bundles a version-compatible installation of kubectl if you choose to use minikube for this exercise. It's a handy time saver.

Once installed, confirm that your environment is ready to go:

kubectl version should confirm that kubectl is present and communicating with the Kubernetes API successfully. If you aren't using minikube, simply use the plain kubectl command for the remainder of the tutorial.

$ minikube kubectl -- version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:38:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

The helm and doppler commands should be available:

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.13"}

$ doppler --version
v3.44.0

With the prerequisites installed, let's proceed with building and running our program.

Sample Application

Before we start creating secrets, let's begin by illustrating the external secrets workflow with a simple example service. The following guide will assume the use of minikube to build and load an application container image, so if your Kubernetes environment is different, you may need to adapt the instructions to work with your specific container registry strategy.

First, perform some initial steps to bootstrap a python Flask application:

$ mkdir -p ~/tmp/secret-sauce
$ cd ~/tmp/secret-sauce
$ git init
$ echo flask > requirements.txt
$ python3 -m venv venv
$ ./venv/bin/pip install -r requirements.txt

Create a file named app.py that contains the following simple web application:

from flask import Flask
from os import environ

app = Flask( __name__ )

sauce = environ.get('SECRET_SAUCE')

@app.route("/")
def index():
    if sauce:
        return f"The secret sauce is: {sauce}!"
    else:
        return "You'll never find my secret sauce."

Define a small Dockerfile that defines how to build a container image for our application:

FROM python:3

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "-m" , "flask", "run", "--host=0.0.0.0"]

You can run this application locally with:

$ ./venv/bin/flask run

Try accessing the running application at http://localhost:5000 to see a response. The root route (/) will render a static string when no secret is present in the environment but will print the secret when the indicated environment variable is defined. Don't print secrets in production! This application is just a demonstration of how to read the value in a real program.

We're ready to load this application into Kubernetes. Assuming that you're following along with minikube, proceed to build the application container in the Kubernetes node under the name "app":

$ minikube image build -t app .
...lots of output...
Successfully tagged app:latest

Create a new Kubernetes Deployment file named deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: webapp
          image: app
          imagePullPolicy: Never
          ports:
            - containerPort: 5000

Take note of a few assumptions in this Deployment:

imagePullPolicy has been set to Never because we're running a locally-built image. By default, Kubernetes would attempt to pull the app image, which doesn't exist. As previously mentioned, if you're following this tutorial in an environment other than minikube, you may need to push the container image to a registry and adjust these settings slightly.
We've exposed port :5000 which we can access later.

Load this Deployment into your running cluster:

$ kubectl apply -f deployment.yaml
deployment.apps/app created

Finally, forward the port in another terminal window in order to access the running application in a simple tunnel.

$ kubectl port-forward deployment/app 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000

Send a request to your Kubernetes application to see it in action:

$ curl http://localhost:5000
You'll never find my secret sauce.

Fantastic! Note that we've received the response that indicates no secret value has been injected into the environment. How can we add a secret to the application and see the secret sauce?

Doppler

By using Doppler, we can achieve a great deal of control over application secrets and manage them at each step of our application's lifecycle, from in our local development environment to within a Kubernetes workload.

If you haven't signed up with Doppler, you can do so now. Once you have an account, proceed to use the login command to set up your account locally:

$ doppler login

Setup Doppler for the demo application by defining a doppler-template.yaml file in the root of the application directory. This YAML file defines a template that we can import to create a new Doppler project easily from the command line:

projects:
  - name: secret-sauce
    description: Kubernetes demo app
    environments:
      - slug: dev
        name: Development
        configs:
          - slug: dev
      - slug: stg
        name: Staging
        configs:
          - slug: stg
      - slug: prd
        name: Production
        configs:
          - slug: prd
    secrets:
      dev:
        SECRET_SAUCE: tartar
      stg:
        SECRET_SAUCE: horseradish
      prd:
        SECRET_SAUCE: tzatziki

Enter the directory in your shell and run the following doppler command in order to bootstrap your Doppler project. This will create a new project called secret-sauce, set up a few different environments, and load an initial secret value for SECRET_SAUCE:

$ doppler import

You'll see output similar to the following:

┌──────────────┬──────────────┬─────────────────────┬──────────────────────────┐
│ ID           │ NAME         │ DESCRIPTION         │ CREATED AT               │
├──────────────┼──────────────┼─────────────────────┼──────────────────────────┤
│ secret-sauce │ secret-sauce │ Kubernetes demo app │ 2022-10-11T22:10:36.567Z │
└──────────────┴──────────────┴─────────────────────┴──────────────────────────┘

View the secrets for this project with doppler secrets:

$ doppler secrets

In addition to some default variables that begin with DOPPLER_, you'll also find the secret value we'd like to inject, SECRET_SAUCE:

┌─────────────────────┬──────────────┐
│ NAME                │ VALUE        │
├─────────────────────┼──────────────┤
│ DOPPLER_CONFIG      │ dev          │
│ DOPPLER_ENVIRONMENT │ dev          │
│ DOPPLER_PROJECT     │ secret-sauce │
│ SECRET_SAUCE        │ tartar       │
└─────────────────────┴──────────────┘

Run a quick test to confirm that our application behaves as expected when a secret is present in its environment variables. To do this, we can use the doppler run command to easily inject the variable into our application.

$ doppler run -- ./venv/bin/flask run

Then issue a request to the listening port to see whether the response has changed:

$ curl http://localhost:5000
The secret sauce is: tartar!

Great! Let's continue on to install the External Secrets Operator.

External Secrets

The External Secrets Operator provides the translation layer between Kubernetes' native secrets and external secrets. The operator leverages custom resources in order to model external secrets, which it then retrieves as necessary and translates into native Kubernetes secrets that your workload can easily consume.

The operator is provided as a Helm chart, so first add the upstream external-secrets Helm repository to access the chart:

$ helm repo add external-secrets https://charts.external-secrets.io
external-secrets has been added to your repositories

Next, install the chart into your Kubernetes cluster. The following command:

Instructs helm to use the external-secrets chart,
passes -n to install the chart into the external-secrets namespace,
creates the namespace as necessary with --create-namespace, and
configures the requisite CRDs as well (--set installCRDs=true)

$ helm install external-secrets \
     external-secrets/external-secrets \
     -n external-secrets \
     --create-namespace \
     --set installCRDs=true

You should see output similar to the following:

NAME: external-secrets
LAST DEPLOYED: Tue Oct 11 12:27:18 2022
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully!

In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets

With this operator installed, we're now ready to set up Doppler to serve as the source for external secrets.

Integrating Doppler with Kubernetes

The first step is to create a service token which serves as the authentication mechanism for the external secrets operator against Doppler. It should be stored inside of a generic Kubernetes secret that the operator will consume.

You can generate and store the token in a single step with some fancy footwork in the shell to avoid copying and pasting the token around. The following command creates a new Doppler token and then immediately loads it into Kubernetes:

$ kubectl create secret generic \
    doppler-token-auth-api \
    --from-literal dopplerToken=$(doppler configs tokens create --config prd doppler-auth-token --plain)
secret/doppler-token-auth-api created

Note that we're passing the --config prd flag to doppler in order to create a token scoped to the prd (production) configuration of our Doppler project. In our application directory, we defaulted to dev, which has its own secrets. With this technique, we're only interacting with development secrets locally while loading production secrets into our container runtime, which keeps the risk of exposure for production secrets minimal.

Next, create a SecretStore CRD that points the operator at your Doppler service token secret. This step sets up Doppler as a source for external secrets that we can call upon later.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: doppler-auth-api
spec:
  provider:
    doppler:
      auth:
        secretRef:
          dopplerToken:
            name: doppler-token-auth-api
            key: dopplerToken

Create the SecretStore in Kubernetes:

$ kubectl apply -f secretstore.yaml
secretstore.external-secrets.io/doppler-auth-api created

We're ready to inject our secret sauce! A new ExternalSecret is the necessary resource that enables us to synchronize one of our Doppler secrets to a related generic Kubernetes secret. Create a new file called externalsecret.yaml with the following YAML:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: secret-sauce
spec:
  secretStoreRef:
    kind: SecretStore
    name: doppler-auth-api

  target:
    name: secret-sauce

  data:
    - secretKey: SECRET_SAUCE
      remoteRef:
        key: SECRET_SAUCE

Load it into Kubernetes:

$ kubectl apply -f externalsecret.yaml
externalsecret.external-secrets.io/secret-sauce created

You can confirm that the secret was loaded by the operator by listing it with kubectl:

$ kubectl get secret secret-sauce
NAME TYPE DATA AGE
secret-sauce Opaque 1 3s

Success! Note that if you do not see the new secret, you can also inspect the External Secret Operator's logs to debug any issues:

$ kubectl -n external-secrets logs -lapp.kubernetes.io/name=external-secrets

Generic secrets can be seen in the Kubernetes dashboard as well. If you're using minikube, you can quickly install and view the Kubernetes dashboard with the following command:

$ minikube dashboard

Your browser will open to the dashboard landing page. You may use the left sidebar to navigate to the "Secrets" link to view your new secret:

The only task left to do is to consume the secret from our application. Here's an updated Deployment that references the newly-created Secret. Note the new env key, which injects the variable SECRET_SAUCE drawn from a secret named secret-sauce under the key SECRET_SAUCE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: webapp
          image: app
          imagePullPolicy: Never
          env:
            - name: SECRET_SAUCE
              valueFrom:
                secretKeyRef:
                  name: secret-sauce
                  key: SECRET_SAUCE
          ports:
            - containerPort: 5000

Apply the updated Deployment:

$ kubectl apply -f deployment.yaml
deployment.apps/app configured

The changed Deployment manifest will recreate any necessary pods, so invoke a new forwarded port once the pods have restarted to forward requests to the new containers:

$ kubectl port-forward deployment/app 5000:5000

Finally, Issue an HTTP request to the running container to see the injected Doppler secret in action:

$ curl http://localhost:5000
The secret sauce is: tzatziki!

Note that the content of this secret differs from the one we saw in our local development environment. The --config prd flag to the doppler command loaded a token with rights to the prd configuration in Doppler, which injects the correct secret for the configuration that the Doppler token has been scoped to.

Congratulations! You've successfully:

Installed the External Secrets Operator into your Kubernetes cluster
Managed an application secret with Doppler
Connected Doppler with the External Secrets Operator
Used a Doppler project secret within a Kubernetes Deployment

What's Next?

There's even more you can do with the External Secrets Operator and Doppler, so if you'd like to learn more, dive into the documentation to learn how to use features like JSON processing, filtering, and more.

If you're actively using Kubernetes secrets stored in etcd today, you should also configure encryption at rest so that your secrets are secure no matter where they're stored. Whether you load them with kubectl or the External Secrets Operator, configuring encryption at rest is still an important step to address every step of your secret management lifecycle.

How To Prevent Secrets From Ending Up On Developers' Machines

Ryan Blunden — Tue, 21 Jun 2022 00:28:05 +0000

Even with environment variable storage offered by modern hosting platforms and secrets managers provided by every cloud, developers' machines are still littered with secrets in unencrypted text files because local development was left out of the picture.

But we can remedy this situation using dynamic secrets injection and ephemeral secrets files, and in this post, we'll be using the Doppler CLI to demonstrate how this is possible.

But before diving in, we need to solve the problem of secure and encrypted storage for secrets used in local development.

SecretOps and Local Development

Development scoped secrets simply don't exist in traditional solutions because secrets are siloed within the confines of their respective cloud or platform.

While multi-cloud capable secret managers such as HashiCorp Vault showed great promise, the prohibitively steep learning curve and unavoidable complexity involved with fetching secrets create a significant barrier to adoption as teams are left with no incentive to switch.

A SecretOps Platform builds upon the idea of centralized secrets storage but differs from existing solutions by providing a CLI and integrations for syncing secrets to any environment, machine, platform, or cloud secrets manager. It's the best of both worlds, providing a single source of truth for management while development teams choose the best secrets access method on a per-application basis.

How does this help local development? In a SecretOps world, each application has a Development environment specifically for use on developers' machines, solving the storage problem.

Using the Doppler CLI to demonstrate, let's dive in to explore the mechanics of dynamic secrets injection and ephemeral secrets files.

Dynamic Secrets Injection

The Doppler CLI uses the same secrets injection model as platforms such as Heroku and Cloudflare Workers by injecting the secrets as environment variables into the application process.

You can use a command:

doppler run -- npm start

Use a script:

doppler run -- ./launch-app.sh

Create a long-running background process in a virtual machine:

nohup doppler run -- npm start >/dev/null 2>&1 &

Use the Doppler CLI inside a Docker container:

…
FROM ubuntu

# Install Doppler CLI
RUN curl -Ls --tlsv1.2 --proto "=https" --retry 3 https://cli.doppler.com/install.sh | sh

CMD ["doppler", "run", "--", "npm", "start"]

And inject environment variables consumed by Docker Compose:

doppler run -- docker-compose up

It's also possible to pipe secrets in .env file format to the Docker CLI, where it reads the output as a file using process substitution:

docker run \
  --env-file <(doppler secrets download --no-file --format docker) \
  my-awesome-app

The same technique applies to creating a Kubernetes secret:

kubectl create secret generic my-app-secret \
  --from-env-file <(doppler secrets download --no-file --format docker)

But if a secrets file is what your application needs, we've got you covered.

Ephemeral Secrets Files

The Doppler CLI enables the mounting of ephemeral secrets files in .env, JSON, or a custom file format using a secrets template that is automatically cleaned up when the application process exits. Imagine how happy your Security team will be when they learn that secrets will never live on any developer's machines again!

To mount an .env file:

# Node.js
doppler run --mount .env -- npm start

# PHP
doppler run --mount .env -- php artisan serve

To mount a JSON file:

doppler run --mount env.json -- npm start

You can specify the format using --mount-format if the file extension doesn't map to a known format:

doppler run --mount app.config --mount-format json -- npm start

Or you can use a custom template, e.g. configure Firebase functions emulator using a .runtimeconfig.json file:

# 1. Create the template
echo '{ "doppler": {{tojson .}} }' > .runtimeconfig.json.tmpl

# 2. Mount the .runtimeconfig.json and run the emulator
doppler run \
  --mount .runtimeconfig.json \
  --mount-template .runtimeconfig.json.tmpl -- firebase emulators:start --only functions

You can even make things more secure by restricting the number of read requests using the --mount-max-reads option, e.g. caching PHP configuration which only requires the .env file to be read once:

doppler run --mount .env --mount-max-reads 1 --command="php artisan config:cache && php artisan serve"

If you're wondering what happens to the mounted file if the Doppler process is force killed, its file contents will appear to vanish instantly. The mounted file isn't a regular file at all, but a Unix named-pipe. If you've ever heard the phrase "everything is a file in Unix", you now have a better understanding of what that means.

Named pipes are designed for inter-process communication while still using the file system as the access point. In this case, it's a client-server model, where your application is effectively sending a read request to the Doppler CLI. If the Doppler CLI is force killed, the .env file (named pipe) will still exist, but because no process is attached to it, requests to read the file will simply hang. Just delete the file from your terminal, and you're good to go.

Summary

Thanks to SecretOps, we now have the workflows required to prevent secrets from ever living on a developer's machine again. All you need is dynamic secrets injection, ephemeral secrets files, and a single source of truth to pull it all together.

Manually Updating .env Files Isn't DevOps

Ryan Blunden — Tue, 07 Jun 2022 05:28:43 +0000

Managing .env files and keeping them in sync across environments is painful when done manually.

Doesn't it seem odd that Slack is still a common way for .env files to be synced between team members? Shouldn't we be concerned that syntax errors from .env file edits are so prevalent that dotenv linting tools are needed?

It's time to bring the DevOps principles of automation and ephemeral resources to managing environment variables and .env files. This post is a compilation of Doppler's best tips and tricks to do just that as is broken into three sections:

Centralized Environment Variable Management
Dynamically Injected Environment Variables
Dynamically Created Ephemeral .env Files

Although the examples are Doppler centric, the goal is to give you fresh ideas for improving app config and secrets automation in your workplace. We've got lots of stuff to cover, so let's dive in!

1. Centralized Environment Variable Management

It's simply not possible to automate the syncing of environment variables across teams, hosting platforms, and development environments without a centralized source of truth at the organization level.

Modern platforms such as Heroku and Vercel provide built-in environment variable storage, but unless all of your applications are hosted on a single platform, they can only function as a source of truth for individual applications. And if you're using cloud-based virtual machines such as those from DigitalOcean or AWS EC2, you're on your own to figure out environment variable storage and access.

With the exception of Vercel's development scoped environment variables, first-class local development support is missing from every modern platform and cloud provider, explaining why so many teams still rely on .env files, even if not used in production. We know how crucial it is for local environments to closely mirror production, but it seems we're willing to make tradeoffs when it comes to environment variables.

In the past, secrets managers such as HashiCorp Vault were seen as the solution. But replacing the simplicity of environment variables with complex SDKs often resulted in siloed secrets and teams going rogue by managing environment variables their own way. Cloud secrets managers also didn't improve the local development story.

Essentially, we need a new way of managing environment variables that reflects the needs of modern application development.

Why we need SecretOps

SecretOps is designed for multi-cloud deployments and combines the strengths of traditional solutions while addressing their weaknesses.As a starting point, a SecretOps platform must:

Centralize the storage and management of secrets
Provide flexible and secure environment variable injection for every platform and cloud
Provide a first-class local development experience
Increase Developer productivity through secrets automation workflows
Decrease the complexity of secrets management

Using Doppler as an example, we're tackling these requirements by providing:

A fully-hosted solution with collaborative secrets management workflows and fine-grained access controls, all from a single centralized dashboard.
A CLI for injecting environment variables and ephemeral .env files in any environment, from local development to CI/CD, Docker, Kubernetes, Virtual Machines etc.
Integrations that sync secrets to platforms such as Vercel, Netlify, Heroku, and GitHub Secrets, so applications don't have to integrate with Doppler directly.

Doppler's operating model is that managing secrets should be centralized, but fetching and syncing secrets should be tailored to every customer's needs. For example, many of our customers enjoy the Doppler dashboard's superior features and developer experience but sync secrets to Azure Key Vault so production secrets access remains as is.

Our vision for SecretOps is constantly evolving. Our goal is to share, inspire, and help move our industry forward with new ideas that take secrets automation to the next level.

2. Dynamically Injected Environment Variables

Using Doppler to illustrate, let's look at several methods for environment variable injection.

Platform Injected Environment Variables

Having a platform or infrastructure tool to inject environment variables into your application is the best solution, as you can then say goodbye to .env files altogether.

So while that removes the need for .env files in select platforms, additional tooling is needed for virtual machines, local development, and Kubernetes, just to name a few.

CLI Application Runner

This method uses a CLI to run your application, injecting environment variables directly into the application process.

Here is how the Doppler CLI can be used to inject environment variables into a Node.js application:

doppler run -- npm start

You can also use a script:

doppler run -- ./launch-app.sh

Create a long-running background process in a virtual machine:

nohup doppler run -- npm start >/dev/null 2>&1 &

Or use the Doppler CLI inside a Docker container:

…

# Install Doppler CLI
RUN curl -Ls --tlsv1.2 --proto "=https" --retry 3 https://cli.doppler.com/install.sh | sh

CMD ["doppler", "run", "--", "npm", "start"]

A CLI application runner with environment variable injection should have the following properties:

Be open source
Support every major operating system
Installable via package managers
Correctly passes signals to the application (e.g SIGINT) so your application can terminate gracefully

The doppler run command is just one way of accessing secrets, and you can find more examples in our CLI Secrets Access Guide.

3. Docker Container Environment Variables

This method injects environment variables into a Docker container at runtime, removing the temptation of embedding an .env file in the Docker image (yes, it happens) and avoiding the host creating and mounting the .env file inside the container.

The Doppler CLI pipes secrets in .env file format to the Docker CLI where it reads the output as a file using process substitution:

docker run \
  --env-file <(doppler secrets download --no-file --format docker) \
  my-awesome-app

Here we get the benefits of .env file configuration but without an .env file ever touching the disk.

You can see other use cases in our docker-examples GitHub repository.

Docker Compose Environment Variables

Docker Compose differs from Docker as it accesses environment variables from the host when docker compose up is run.

Docker Compose sensibly requires you to define which environment variables to pass through to each service as variables such as $PATH are host-specific:

version: '3.9'
services:
    web:
        image: my-app
        # Host environment variables passed through to container
        environment:
            - AUTH_TOKEN
            - DB_CONNECTION_URL

Because of how Docker Compose access environment variables, we can use the Doppler CLI as an application runner:

doppler run -- docker compose up

Other Docker Compose use cases can be found in our docker-examples GitHub repository.

Kubernetes Environment Variables

Kubernetes provides excellent support for injecting environment variables into containers using Key-Value pairs stored in a Kubernetes secret.

Doppler provides two options for syncing secrets to Kubernetes:

Doppler CLI Created Secrets
Doppler Kubernetes Operator Managed Secrets

Doppler CLI Created Secrets

The first step is to create a generic Kubernetes secret (the first argument being the secret's name) where just like Docker, secrets in .env file format are piped to kubectl where it reads the output as a file:

kubectl create secret generic my-app-secret \
  --from-env-file <(doppler secrets download --no-file --format docker)

Doppler Kubernetes Operator Managed Secrets

Our Kubernetes Operator is designed to scale and fully automate secrets syncing from Doppler to Kubernetes. Once installed and configured, it instantly creates and updates Kubernetes secrets as they change in Doppler with support for automatic deployment reloads if the secrets they're consuming have changed.

As it's a more advanced solution that requires Kubernetes cluster administration experience, we won't be covering it here but check out our Kubernetes Operator documentation to learn more.

Kubernetes Deployments and Environment Variables

Injecting environment variables into a Deployment from the Key-Value pairs in a Kubernetes secret is done using the envFrom property of a container spec:

...
    spec:
      containers:
        - name: awesome-app
          envFrom:
            - secretRef:
                name: my-app-secret # Kubernetes secret name
…

Hopefully, you've learned some new tricks for environment variable injection! Now let's move on to.env files.

Dynamically Created Ephemeral .env Files

Environment variable injection is always preferred, but sometimes, an .env file is the only workable solution.

Protective measures such as locking down file ownership, file permissions, and heavily restricting shell access should go without saying. But the risk of .env files existing on the file system indefinitely has always been a concern and why we've been hesitant to recommend .env file usage in the past.

But thanks to the Doppler CLI, we can now mount ephemeral .env files that are automatically cleaned up when the application exits. Imagine not having to worry about anyone in your company accidentally committing an .env file again!

One of the most popular use cases is for PHP developers building Laravel applications:

doppler run --mount .env -- php artisan serve

The file extension is used to automatically set the format (JSON format is also supported):

doppler run --mount secrets.json -- npm start

You can set the format if the file extension doesn't map to a known type:

doppler run --mount app.config --mount-format json -- npm start

To increase security, you can also restrict the number of reads:

doppler run --mount .env --mount-max-reads 1 --command="php artisan config:cache && php artisan serve"

If you're wondering what happens to the mounted file if the Doppler process is force killed, its file contents will appear to vanish instantly. This is because the mounted file isn't a regular file at all, but a Unix named-pipe. If you've ever heard the phrase “everything is a file in Unix”, you now have a better understanding of what that means.

Named pipes are designed for inter-process communication while still using the file system as the access point. In this case, it's a client-server model, where your application is effectively sending a read request to the Doppler CLI. In the event the Doppler CLI is force killed, the .env file (named pipe) will still exist, but because no process is attached to it, requests to read the file will simply hang.

It's also named pipes that enable us to restrict the number of reads using the --mount-max-reads option as once the limit is exceeded, the CLI simply removes the named pipe from the file system.

Summary

I hope you take away some new automation ideas to bring back to your team so you can spend less time updating .env files and more time shipping software.

Crash course in Web3 Application Development with Python

Tyler Langlois — Tue, 31 May 2022 13:00:00 +0000

In the world of cryptocurrency, private keys are king. Mathematical mechanics treat correctly signed transactions as valid regardless of intent, which makes account keys and passphrases a particularly lucrative target for malicious actors. While end-user best practices for securing keys tend to be plentiful, the development side of blockchain work takes place amid a sea of API tokens, key files, and service credentials that can sprawl out of control if not managed correctly.

Ensuring that these sensitive values remain secure is critical and outdated solutions such as .env files are no longer sufficient.

This tutorial will start from the basics of interacting with blockchain APIs from joining as a peer upwards. There are numerous third-party services and APIs that sit atop various blockchains, but establishing secure management practices for primitives such as private keys is fundamental.

No prior experience interacting with blockchain or web3 services is assumed, and we'll use a testnet in order to limit the potential exposure of sensitive information. Doppler will be used to store and retrieve credentials associated with private key material, which is a secure and convenient way to manage secrets without the risks of storing them on disk in unencrypted text files.

Before downloading any programs or generating account keys, there are a few concepts to establish heading into the unique paradigms that accompany working with web3 networks.

While plenty of unique blockchains and products abound in the web3 space, this guide will focus on the Ethereum blockchain, which is one of the more commonly used networks. Like most cryptocurrency blockchains, Ethereum operates in a decentralized way with a vast network of computers that can be interacted with as a peer. However, any sort of operation that requires some proof of ownership, e.g. signing a transaction, does require the authoritative stamp of a private key.

This is the core of how the network operates, which makes the method of communication and management of privileged information a chief concern.

This tutorial will follow a basic outline to work directly with the Ethereum blockchain and its API:

Connecting to the testnet
Generating and securing public keys
Receiving tokens
Sending tokens securely

Let's begin!

A Chain Primer

When choosing how to connect to the Ethereum blockchain, there are two general approaches, each with distinct advantages and disadvantages.

The first is by connecting to a third-party API that serves as a frontend to low-level blockchain network protocols. Services like Infura offer a web-based JSON RPC API, which is widely compatible with a range of libraries and languages and allows developers to interact with the blockchain without downloading the entire ledger, which can take up significant compute resources and disk space (around 500GB at the time of this writing).

Using a third-party API means that your code can run from a variety of environments without being tethered to a running node and the accompanying resource requirements that are necessary. The downside to this approach is reliance upon a third party when one of the core advantages of blockchain technology is decentralisation. Routing communication with the aid of an intermediary party is convenient, but means that clients must trust the provider and are subject to the inherent properties of using a hosted API instead of communicating directly via peer protocols (such as availability and uptime). These tradeoffs are often the right choice, but are tradeoffs nonetheless.

The second approach is to run a native node to communicate to the chosen blockchain directly. As previously mentioned, there are significant computational costs involved, but this means that the client speaks directly to the chosen peer network which brings with it benefits like smaller risk exposure, high availability of the decentralised network, and the ability to self-validate the ledger.

There are additional considerations to take into account when operating a dedicated node, such as choosing whether to download the entire ledger during the initial loading process or instead electing a more lightweight method that incurs a smaller disk usage cost.

This guide will opt for the self-hosted node technique, though in practice, either approach is a reasonable choice.

Finally, a node operator must select which network to connect to when initialising their connection.

Normally, users will choose to connect to the live, "production" chain when interacting with a public ledger, but during local development and testing, a testnet is a better choice. In addition to less traffic (resulting in smaller ledgers and transaction costs), acquiring tokens to send and receive is usually free and does not require any initial funds. This is a significant advantage when sending and receiving tokens without risk of making mistakes that could incur significant consequences.

First Steps

We'll use geth (Go Ethereum) to connect to the Ethereum network, which is a golang implementation that can function not only as a client but a fully featured node as well.

First, follow the instructions for installing geth for your operating system. Once installed, the geth command should be available from the terminal.

geth version
Geth
Version: 1.10.17-stable
Architecture: amd64
Go Version: go1.16.13
Operating System: linux
GOPATH=
GOROOT=/nix/store/j1x3cy8g2cqcr54rg98gw0mdc28jlyc8-go-1.16.13/share/go

The command-line flags passed to the geth CLI dictate which network it will connect to. There are a number of testnets within the Ethereum network to choose from, and this guide will use the Goerli proof-of-authority network as its newer than some of the others, and is not coupled with the computationally-expensive proof-of-work networks.

The --syncmode flag controls the method that geth will use when downloading the historical ledger of the blockchain. On one end of the extreme, the full method will download the entirety of the blockchain ledger which includes all the relevant transactional data. On the other, the light method will only fetch recent block headers and retrieve other information on-demand. When communicating with the production Ethereum network, the full method offers the most comprehensive assurance that the local copy of blockchain data is valid and legitimate. However, in this exploratory case on a testnet, the light method is sufficient.

Begin the geth process in a background terminal on your machine. The initial syncing process will take time, but using the light method on the goerli testnet will not use excessive space (at the time of this writing, the entirety of the chain requires about 500MB of disk space). The following command will store the Goerli chain data in the ~/.goerli directory:

geth --syncmode light --datadir ~/.goerli --goerli
INFO [04-05|15:54:49.916] Starting Geth on Goerli testnet...
...

After some time, the daemon will stop importing historical blocks and the latest blocks will be available.

The logging output from geth will include the age of a block when performing initial synchronisation. When you see blocks being imported with output such as age=4m04w1d, this indicates old blocks being retrieved to construct the local chain. Once older blocks stop appearing, this indicates that the initial synchronisation is complete.

You can continue with the rest of this guide while the blockchain synchronises; full synchronisation will only be necessary once we reach the portion that covers transactions.

Accounts and Keys

At the beginning of this guide, we spoke at length about the importance of keys when interacting with the blockchain.

Public-key cryptography forms the backbone of the cryptographic methods for forming and authenticating transactions on public ledgers. Private keys prove ownership of public addresses and sign transactions to send tokens to other public addresses.

While high-level, user-facing cryptocurrency applications may leverage account information for wallets in the form of traditional username/password combinations, the core values that comprise an "account" on a blockchain like Ethereum is a cryptographic key - tokens belong to an addressed coupled to a private key. The geth CLI includes utilities to generate keys, which it calls an account.

To begin, create a new account. Remember that this "account" is really a locally encrypted file and not an account that can be "logged into" from another machine - if you are using an account tied to a production blockchain, it's critical to ensure that this account is secured by a strong passphrase backed up regularly.

In our case, the local account and keypair will be associated with a testnet, but we'll still use best practices for managing the passphrase by storing it in encrypted secrets store (Doppler) and treat the account as sensitive. Because the running geth process is active, we'll connect to the running daemon via attach instead of using the geth account subcommand.

Enter the geth console:

geth --datadir ~/.goerli attach

You'll be greeted by the command prompt for geth javascript commands.

Welcome to the Geth JavaScript console!

instance: Geth/v1.10.17-stable/linux-amd64/go1.16.13
at block: 6664849 (Tue Apr 05 2022 17:34:43 GMT-0600 (MDT))
datadir: /home/yourname/.goerli
modules: admin:1.0 clique:1.0 debug:1.0 eth:1.0 les:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 vflux:1.0 web3:1.0

To exit, press ctrl-d or type exit
>

To create a new account, run the following command where you'll be asked to enter a passphrase:

personal.newAccount()

After following the prompts, geth will create a keypair within the ~/.goerli/keystore/ directory. You can view the public address of this account at any time by using the following command from the geth console:

personal.listAccounts

At this point, your local machine will have a light copy of the Ethereum Goerli testnet blockchain and a keypair ready to send and receive tokens. Let's begin working with transactions!

Interacting with the Blockchain

A simple task to confirm the functionality of this local testnet setup is to receive currency. Unlike currency on the live, production blockchain, currency on a testnet is often given away freely to permit developers to test and iterate on projects and products.

Providers for free currency - often called "faucets" - may sometimes be unavailable or lacking sufficient funds to distribute tokens freely. At the time of this writing, the faucet mentioned here is functional and has sufficient funds to distribute them to anyone who requests them, but if you encounter problems requesting funds at a later date than when this tutorial was written, you may also seek out alternate sources (remember that they must be operating on the Goerli Ethernet testnet).

Well be using the Goerli Faucet to acquire initial funds to begin experimenting with the blockchain.

From the geth console, find your public address using listAccounts:

personal.listAccounts
["0x..."]

Copy the address, navigate to the Goerli Faucet, paste your address into the Wallet Address field, then click Send Me ETH.

Blocks are minted frequently on this chain, so it's time to check your balance! We'll use the Web3 Python library to interact with the local node.

This guide assumes that you have Python 3 installed on your local machine. Weve provided a repository with sample code as a quick start to getting up to speed with how to use the Web3 library.

Clone the GitHub repository then enter the repository directory:

git clone https://github.com/DopplerUniversity/python-web3
cd python-web3

Create a Python virtual environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Feel free to open web3-app.py and take a look! The Python web3 library makes accessing blockchain information fairly straightforward, so this example code should be a good place to start if you want to experiment later.

Let's verify the script is working by fetching the timestamp of the latest block on the chain:

python web3-app.py latest-block

If you see a timestamp that is within 30 seconds or so, congratulations! You've successfully interacted with your local instance of the testnet blockchain. The next task is to check the balance of the address that your faucet tokens were sent to.

First, confirm that Python is able to see the local account:

python web3-app.py accounts

Your account number should match the address that you previously found within the geth console using personal.listAccounts. Finally, check the account balance.

python web3-app.py balance

The account balance for the address entered in the Ethereum faucet site should be 0.05 ETH. You've checked your account balance from the python web3 library. Well done!

Sending ETH

Until now we've only interacted with the local blockchain with APIs that do not require private account keys. In order to finish this experiment and successfully sign a transaction with your account, we'll practice by sending an amount of ETH back to the Goerli faucet. Doing so will help demonstrate the process as well as replenish funds to future developers on the blockchain!

To send ETH, well use the send_transaction method:

web3.geth.personal.send_transaction(self, transaction, passphrase)

Remember when we created our local Geth account and provided the passphrase?

The passphrase is required here to unlock the private key and sign the transaction in order to validate it as a legitimate transfer of funds out of our account. However, entering a passphrase into a REPL or a shell is a security concern: most REPL and shell histories (like bash or the python interpreter) retain commands that are entered, so it's wise to avoid entering a passphrase manually in plaintext.

What other options exist? One approach is to read secret values - whether it be a passphrase, API key, or otherwise from a file on disk such as .env or JS file but this is a dangerous option as well explain.

Storing secrets in plaintext files is a huge security nightmare are as they can be read by other user processes, and risk being committed during work with tools like git if a developer makes a mistake such as Nat Eliason did when he lost $30,000 by accidentally pushing a file containing his passphrase to GitHub.

Reading the value from an environment variable is a good idea, but that begs the question of how the environment variable containing the passphrase will be securely populated.

This is where a SecretOps platform such as Doppler can help by using the Doppler CLI to inject secrets as environment variables into an application process. This allows an environment variable to be populated with the passphrase securely without the risks of unencrypted local file storage.

To begin, first sign up for a Doppler account. The sign up process will walk through a few helpful tips to get started, and once your account has been configured, proceed to install the Doppler CLI, then authenticate your machine by running:

doppler login

Create a new project called python-web3 which will hold the secrets for this project and keep things organised as well as within their own environment scope:

doppler projects create python-web3
doppler setup --project python-web3 --config dev

Ensure that you run doppler setup in the python-web3 directory as the Doppler CLI scopes secrets access to specific directories. Eventually you should have a setup similar to the one below: with your project directory coupled to the python-web3 project in Doppler.

Run the following command to check you can access secrets from the python-web3 directory:

doppler secrets

 ID NAME DESCRIPTION CREATED AT               

 python-web3 python-web3 2022-05-24T23:40:34.955Z

The next step is to store the keychain passphrase in Doppler where it can be retrieved by the doppler CLI.

Open the python-web3 Doppler Project and create a ETH_PASSPHRASE with your passphrase value, then click Save.

Heading back to the terminal, can verify Doppler saved your passphrase successfully by running:

doppler secrets get ETH_PASSPHRASE

With the passphrase managed by Doppler, we can now run the python script with secrets populated by the doppler run command.

You may inspect the accompanying python code from the cloned repository, but the key lines are those that retrieve your passphrase from the ETH_PASSPHRASE environment variable.

When you invoke the doppler run command, Doppler will set the ETH_PASSPHRASE environment variable for python to use and sign the transaction with the wallet information from the unlocked key. Although the noted address is that of the faucet you originally received your balance from, you may choose an arbitrary amount to send with your command (as long as you have sufficient funds!).

You can find a random account to send funds to from the Goerli top accounts page. Copy one of the to addresses (starting with 0x) and replace {ADDRESS} in the command below to begin the funds transfer:

doppler run -- python web3-app.py send {ADDRESS} 0.01

The transaction once complete will return a unique transaction hash like the following:

HexBytes('0x0d8c2b6924a88cda73c73275051245db7a7e698f419a9d27b8b185dda16a8436')

Your transaction has been submitted to the blockchain! At this point the running geth light node will broadcast this transaction and the transfer of funds will propagate through the network. You may use a few approaches to view the transaction details.

The provided python script supports a command to introspect an arbitrary transaction ID for example, the following command will display the transaction details of the aforementioned transaction ID:

python web3-app.py get-txn 0x0d8c2b6924a88cda73c73275051245db7a7e698f419a9d27b8b185dda16a8436

In addition to viewing details from a local client, using another method to introspect the public ledger is a good idea in order to confirm that the transaction is visible from any client.

For example, you can either use a generally-available web-based block explorer to view transactions associated with a specific address (replace the given hash with yours that is under the variable in your python repl as my_account):

https://goerli.etherscan.io/address/0x858Eb06Bd4dc5BE36Dc5025483a316E1630b35e9

Or you may alternatively view the transaction directly by entering your transaction hash into a URL like the following:

https://goerli.etherscan.io/tx/0x0d8c2b6924a88cda73c73275051245db7a7e698f419a9d27b8b185dda16a8436

In either case you should be able to see the transaction data including the amount and associated gas price.

Congratulations! You just created a transaction on an Ethereum blockchain!

Production Considerations

This guide has covered one of the most basic operations when working with a blockchain - signing and broadcasting a transaction - using Doppler to manage your passphrase securely so crypto keys and passphrases arent inadvertently exposed along with any other sensitive account credentials.

Recall that in this tutorial, we generated a keychain using the geth CLI so the associated public and private key exist only on the local machine.

To write code that could potentially run on any host with a keychain stored remotely, you could elect to store key material in Doppler and instantiate a local keychain based upon the retrieval of that information from a similar Doppler secret.

The web3 library provides the web3.geth.personal.import_raw_key method which accepts a private key and passphrase so that your transaction operations can occur on any machine with a properly setup Doppler environment. The local blockchain can be used for more experiments by virtue of the flexibility of the testnet environment, so you can continue to receive currency from a Goerli testnet faucet and send funds to other addresses.

$20m Series A to build the first SecretOps Platform

Brian Vallelunga — Thu, 28 Apr 2022 01:19:19 +0000

Redefining secrets management through multi-cloud secrets sync and automation at enterprise scale.

We're excited to announce our $20m Series A round led by CRV with participation from existing investors including Google Ventures, Sequoia Capital, and Y Combinator.

40 tech leaders have also joined as angel investors and advisors including GitHub CEO Thomas Dohmke, Datadog CEO Olivier Pomel, Plaid CEO Jean-Denis Greze, Twilio CEO Evan Cooke, Postman CEO Ankit Sobti, and Okta CEO Frederic Kerrest.

By building the first SecretOps Platform, developers and security teams can leave behind the problems caused by .env files, unmanageable secrets sprawl across multiple clouds, and the lack of innovation from traditional secret managers.

The ability to securely store, transmit, and audit secrets has never been more critical as one minor error can lead to catastrophic results.

In a world where putting a single space in the wrong place can literally take down a company’s entire website, Doppler makes it easy to prevent leaks and outages with their developer-focused approach.

Murat Bicer, General Partner, CRV

A SecretOps Platform is the single source of truth for developers and their teams to manage, store, and automatically sync secrets to every major hosting platform, secrets manager, and infrastructure management tooling such as Kubernetes and Terraform.

The reality is that the legacy way of storing secrets in siloed secrets managers and .env files doesn’t make sense anymore.

With Doppler, we can put those days behind us. Just like when GitHub first pioneered the notion of a pull request,—once you see the benefits, there's no turning back.

Brian Vallelunga, Doppler CEO

Fuelling Doppler's growth are customers such as Puma, Hopin, ezCater, Toast, Gather, My Muscle Chef, OnDeck, and many others. More than 15,000 teams trust Doppler to manage their secrets with over 1.5 billion secrets synced every month.

Traditional secrets managers were never built to support a multi-cloud deployment strategy or provide developers with the management features they need as part of the software development lifecycle and deployment process.

But we don’t just need slightly better secrets managers. We need a SecretOps Platform. A platform that satisfies the requirements of both Developers and Security Engineers. Development teams and DevSecOps.

Doppler combines all of the key elements DevOps and Security teams need to control who can see and modify secrets at scale as well as an audit trail, versioning, enterprise grade encryption, secrets rotation and dynamic secrets.

Brian Vallelunga, Doppler CEO

Doppler also seamlessly integrates and syncs secrets to a growing list of infrastructure tools such as Kubernetes and Terraform, platforms such as GitHub Actions and Vercel, and cloud secret managers such as AWS Secrets Manager and Azure Key Vault.

Doppler's Series A is further confirmation that building the first SecretOps Platform is essential to making secrets automation a first-class citizen in the infrastructure management landscape.

About CRV

CRV is a venture capital firm that invests in early-stage startups. Since 1970, the firm has invested in more than 500 startups at their most crucial stages, including Airtable, DoorDash, and Iterable. Founders need more than capital to build a great company. It takes a partner who understands the entrepreneurial journey and knows what it takes to win. From founding to IPO and beyond, CRV is there every step of the way. Founders rely on CRV to be trusted, long-term, committed partners, which has helped make CRV into one of the longest-running venture capital firms in the world. Learn more about CRV and the companies shaping the future at https://www.crv.com.

Why syncing .env files doesn’t scale for secrets management

Ryan Blunden — Wed, 19 Jan 2022 07:59:09 +0000

Learn why using a Universal Secrets Platform is the key to managing environment variables at scale and eliminates the need for syncing .env files.

The benefits of using environment variables to keep secrets out of source code are well established. But are .env files the best method for managing them?

Secrets management has evolved beyond the limited Key-Value storage that .env files provide. However, most developers are either unaware of .env file’s shortcomings or have simply become numb to the pain from the many years of usage and lack of innovation.

This post aims to highlight the risks of continuing to use .env files, why the major cloud vendors and hosting platforms offer a built-in secrets or environment variables store which can be used instead, and how secrets managers such as Doppler and HashiCorp Vault are providing the much-needed management and secrets automation layer on top of encrypted and access controlled secret storage.

A brief history of .env files

The usage of environment variables and .env files for application config and secrets largely began around 2013. It was a long overdue and important step towards more secure secrets management practices.

Libraries such as Python’s dotenv and Node’s dotenv made it easy for developers to use .env files and environment variables for application configuration, giving developers a simple path to removing secrets from their source code for good.

To think that .env file usage has remained practically unchanged for over eight years is quite remarkable. It should come as no surprise then, that it’s time to say goodbye to .env files in exchange for alternatives that better meet the needs of modern application development.

The problems with .env files

Using .env files allowed us to move secrets out of source code. Unfortunately, they introduced a new set of challenges:

Scaling issues related to syncing .env file changes across environments and different cloud providers, increasing the risk of infrastructure misconfiguration and potential downtime.
Easy for .env files to contain syntax errors, requiring additional tools such as dotenv-linter to be added to pro-commit hooks or GitHub checks.
Sharing of unencrypted secrets in .env files over Slack when secrets change or new developers join a team risks breaking the principle of least privilege by exposing secrets to potentially unauthorized users.
Inconsistent format of environment variables can cause issues, e.g. Docker and GitHub require unquoted values while other packages do not.
Patchy and inconsistent support for multi-line secrets such as TLS certificates, SSH keys, JSON, and YAML.
Secrets used in multiple applications must be duplicated in every .env file (instead of dynamic secrets referencing), making updating and rolling credentials tedious and repetitive.
If persisted to disk in plain text, they may be readable by unauthorized users with access to the system and threat actors if file restrictive file permissions aren't used.
Easy to accidentally expose to malicious bots if placed in the webroot of a web server or S3 buckets.
Local development environments break whenever team members forget to share updates that need to be applied to their .env files, e.g. when a feature branch is merged that requires a new secret.

It's clear .env files have serious implications for application security Next, we’ll take a closer look at why the productivity impacts of using .env files may be worse than you think.

The hidden productivity costs from using .env files

Small repetitive problems, such as manually updating .env files across several servers as part of the deployment process, while perhaps initially frustrating and annoying, can easily become just an expected part of the application deployment lifecycle.

While some developers would argue that the papercuts associated with using .env files are minor, one thing we can all agree on is that interruptions can have serious productivity implications for writing code.

According to a recent study, the average lost time per serious interruption is 23 minutes:

"You need to get into the mindset for development and then slowly trace back to where you left off. This can easily take more than 30 minutes." - Gloria Mark, Professor of Informatics California, Irvine.

The cost of misconfiguration errors is not just the time spent fixing an .env file related issue. It's the impact of unexpected context switching and the challenge of getting back into a state of deep work, also known as “flow”.

Why developers have ignored traditional secrets managers

Traditional secrets managers such as Azure Key Vault or AWS Secrets Manager provide encrypted storage and fine-grained access controls, specially designed for storing secrets such as API keys, database credentials, SSH keys, and TLS certificates.

They are incredibly secure, robust, and enterprise-ready. But unfortunately, secret managers such as HashiCorp Vault are built for security teams, not developers.

As a result, they can be complex to implement correctly and often require secret-fetching implementation details to leak into application code—the exact opposite of the benefits afforded by using language-agnostic environment variables.

Even security-minded developers motivated to use a traditional secrets managers have typically given up for one primary reason: Using .env files was much easier.

Instead of environment variables, a vendor-specific SDK, platform integration, or custom application code for fetching secrets from a vendor’s API is often required.

For example, take this AWS Secrets Manager SDK for Node.js sample code for fetching secrets:

// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "<<{{MyRegionName}}>>",
    secretName = "<<{{MySecretName}}>>",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});

// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
// We rethrow the exception by default.

client.getSecretValue({SecretId: secretName}, function(err, data) {
    if (err) {
        if (err.code === 'DecryptionFailureException')
            // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InternalServiceErrorException')
            // An error occurred on the server side.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidParameterException')
            // You provided an invalid value for a parameter.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidRequestException')
            // You provided a parameter value that is not valid for the current state of the resource.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'ResourceNotFoundException')
            // We can't find the resource that you asked for.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
    }
    else {
        // Decrypts secret using the associated KMS CMK.
        // Depending on whether the secret is a string or binary, one of these fields will be populated.
        if ('SecretString' in data) {
            secret = data.SecretString;
        } else {
            let buff = new Buffer(data.SecretBinary, 'base64');
            decodedBinarySecret = buff.toString('ascii');
        }
    }

    // Your code goes here. 
});

It's this level of complexity compared with using environment variables that turns most developers off from the start.

As product teams are incentivized to ship software and new features as fast as possible, migrating to a traditional secrets manager usually only occurs due to regulatory requirements or security mandates.

But is it possible to still use environment variables for modern applications without .env files?

Modern platforms with native environment variable storage

Modern hosting platforms such as Netlify, Vercel, DigitalOcean, Cloudflare Workers, Fly.io, and Railway all come with secure environment variable storage built-in.

This not only shows how easy it is to migrate away from .env files but confirms that environment variables are still the best language and platform-agnostic method for injecting secrets into an application.

Do you need .env files for local development?

It may seem we’re still reliant on .env files for local development if hosting platforms only manage environment variables for applications running on their infrastructure. But this is beginning to change.

Every developer understands that inconsistencies between local and production environments are a recipe for unexpected issues. That’s why a USP provides first-class support for managing secrets in every environment. A trend modern hosting providers are also beginning to follow.

Vercel, for example, offers environment variable storage specifically for local development that is fetched and injected into the Node.js application via the Vercel CLI:

vercel dev

But what about developers using hosting providers without such functionality? This is where a USP such as Doppler fills in the gaps, eliminating the need for manually managed .env files.

Once developers have created a project and installed the Doppler CLI, secrets can be injected as environment variables into any application process:

doppler run -- npm run firebase-local-functions

Developer tooling is rapidly improving to provide a better integrated local development experience that will eliminate the differences between local and production environments and the need for manually managed .env files on developer machines.

A Universal approach to taming secret sprawl

Taming secret sprawl is a growing challenge for every development team and one that is only made worse as the number of .env files increase. We need an entirely new approach to secrets management that goes beyond incremental improvements—A Universal Secrets Platform.

Taking a “Universal” approach means being able to manage and sync secrets to every application on any platform by avoiding the problems associated with siloed secrets and antiquated solutions that don’t scale such as trying to sync dotenv files across platforms.

This can be achieved through a hub-and-spoke model where the USP acts as a single source of truth for secret storage and management with integrations automatically syncing secrets when they change to any external platform, including other secrets managers.

We hope our vision of a Universal Secrets Platform serves as inspiration for other secrets managers to create a more developer-friendly experience in order to make migrating away from .env files a more attractive option to developers.

Summary

We don’t need to sync .env files. We need the developer-specific workflows that a Universal Secrets Platform such as Doppler can provide.

The simplicity of .env files, while attractive at first, is also its greatest weakness. The demands of modern application development and the explosion of microservices across multiple clouds and platforms provide scalability challenges .env files simply can’t address.

The use of .env files was certainly an improvement over hard-coded secrets. But better options now exist for secrets management, and not only will your infrastructure be more secure without .env files, you’ll be more productive without them too.

The Missing Third

Brian Vallelunga — Wed, 20 Oct 2021 06:30:45 +0000

I believe every developer deserves amazing collaboration tools for managing secrets. Using a secrets manager shouldn't come at the cost of a brain aneurysm. In fact, I think a platform designed for developers of all backgrounds can bring moments of joy to your day, promote healthy security hygiene, and boost your overall developer productivity. Doppler was built from the ground up because the Universal Secrets Platform (USP) I needed simply didn't exist.

I am sure at this point you may be asking who are you? What is a secret - sounds mysterious? What is a USP? How does it differ from a traditional secrets manager or even a dotenv file? Lots of questions, I’ll start off with a simple one - who am I.

Hi - I am Brian 👋, an introvert who plays an extrovert in real life. I love building things (mostly through code and legos), have a passion for making weird art, and hanging with friends. I am sharing all this to show I am just another person, another developer, like you. Below are my thoughts on how the current way of managing secrets is broken and a proposed solution through what I call a Universal Secrets Platform (USP).

App = Code + Compute + Secrets

From my experience, running an application typically requires at the minimum code, compute, and secrets. Three pieces of a puzzle that form a living picture. We are lucky to live at a time where amazing collaborative source control platforms exist enabling distributed teams to build together. This paired with CI/CD and cloud-based hosting allows for fully automated deployments that can reach users in most regions at near infinite scale. But what about the other, often deemed less exciting piece to running applications — secrets?

For context, secrets are typically API keys, database urls, certificates, environment variables, and other app configuration such as a port variable. They are often used to grant access to highly sensitive data/services and thus should never be stored in an unencrypted format such as in code or dotenv file.

Responsibility to our users

Secrets are the literal keys to our digital kingdom and deserve to be treated as such. Before Doppler I didn't think much about secrets, as my job was to ship features to customers as fast as possible. Building and shipping is an adrenaline rush, and through that rush, it became easy to lose sight of what I was doing.

I thought of secrets as a means to an end, a required piece of the puzzle that let my code talk to services like Stripe and Twilio. I never stopped to think about the impact it would have if those secrets ever became "not so secret". In one decision, made now or years ago, I could have accidentally leaked all of our users data that had been entrusted with us. I often would store our production Stripe key in an unencrypted dotenv file which granted access to our customer's credit cards and bank accounts. The impact that one secret could have had on their lives if it ever got leaked would have been tremendous.

It's easy to forget that behind the UUID of a user is a real person with a life and loved ones. They trust us with their data, the least we should do is protect. While you might be thinking that a leak would never happen to you, the Twitch source code leak was a brutal reminder that the unexpected can happen at any time, to anyone. I hope this gives you a moment of pause to reframe how you think about the secrets you already have today.

.... a minute later. Pause is over, onwards!

A missing puzzle piece

It would be painful to imagine a world before the likes of GitHub, where you had to take code and share it over email/FTP and then perform surgical merge resolution to adopt the changes. This wouldn’t just happen once, but every time anyone changed code. Then to deploy that newly minted code to production, one person would need to SSH into a server and upload the code manually. Hoping and praying along the way that they didn’t mess up and accidentally bring the service down. These workflows were manual because we didn’t have pull requests, CICD, infrastructure as code, cloud computing, or other fancy automations we benefit from today.

But everything isn’t as automated as it seems, as much of the same problems exist today with secrets. Take dotenv for example… which is a file that holds a list of secrets in an unencrypted (🤦) human-readable format. Each developer and every environment (e.g. staging, production) needs its own dedicated dotenv file that is specific for that machine. This file should never be tracked by source control, leaving developers to the manual, error-prone, and time-consuming dynamics that we previously experienced with code before source control existed. These unencrypted files are then passed around Slack, email, other productivity tools that are not designed to store the literal keys to the kingdom.

Then when updating the ".env" file for production and other environments, the developer has to add the secrets manually. In the case of a medium to large company, they would need to create a task/ticket for the DevOps team who is responsible for managing infrastructure.

Environments must then be updated in a timely manner as the secrets need to be available before the new code that uses them is deployed. This creates a tricky race condition when code is automatically pushed from GitHub while secrets are configured by humans. There is also a greater chance of the service going down when secrets are touched manually as human error can strike in the moment through a seemingly innocuous typo. Secrets need to be perfect, if they are off by even a single character from what they need to be, it can cause serious problems.

Adding or updating a secret in a dotenv file may sound simple, but in reality, it’s much more complex. Changes are often made in groups where some secrets are added, while others are removed or even updated. Imagine distributing all of those changes by hand day in and day out. Sounds like a recipe for a typo.

Automated and collaborative tooling

At this point you may be asking why there isn't a tool that already exists since the conventional way is so painful. Truth be told, I asked the same question before I started Doppler. To my surprise, there are tools that exist but I hadn't heard of them because they weren't built for me. I am a developer, heart and soul, and those tools were built for the security teams. I found the user experience sucked, SDKs complex, on-premise requirements frustrating, and lack of built-in management features left the developer experience out of the equation while ticking all the right boxes for security.

I am a visual person, I want a dashboard. I don’t like repeating high-risk mundane tasks. I want integrations like GitHub where my secrets are automatically deployed alongside my code. Lastly I am human, I forget things and make typos. I want a tool that has my back by catching my mistakes before they become costly. And in the event I do mess up, I should be able to instantly roll back the changes from a click in a dashboard. I want the automation benefits that have revolutionized how we build and ship software applied to the one area that developer tools forgot—secrets.

We as an industry need to define a new devtools category that captures what we as a development community need for managing secrets securely, from individual developers to enterprise organizations.

I propose that we need a Universal Secrets Platform (USP). A single source of truth for developers and their teams to manage, store, and automate secrets syncing to every major hosting platform and cloud provider. Universal is the differentiating prefix and comes with four requirements:

Universally accessible and useable for developers of all skill levels and specialties. From junior developers to the most senior.
Universally empowers least privilege access. Developers and machines should only have access to the secrets they need.
Universally provides a full-featured and easy-to-use dashboard and CLI that takes minutes, not days or weeks to learn.
Universally supports every operating system and infrastructure, from Serverless to Containers, static-sites to Virtual Machines, and everything in-between.

So how does this differ from the secrets managers that exist today? AWS and GCP both have secrets management offerings while new cloud providers like Vercel and Netlify also allow for securely storing secrets. To see the difference, we need to separate storage from management to match reality. The secret “managers” we have today actually only offer simple Key-Value storage and lack support for the workflows developers need in practice. In our new model, we should treat them as storage destinations. The USP becomes the central source of truth, responsible for distributing secrets to the corresponding storage destinations. This also serves as a multi-cloud solution for a modern-day data center.

Let’s take a look at an example through the lens of a project. Straight off, we notice that every environment, including local development, is supported. Not only that, but every developer gets their own isolated secrets store. There is automated testing in place through CircleCI while Netlify is used staging and production deployments. Each one of these environments (development, staging, and production) has at least one destination ranging from a developer’s laptop to CircleCI and Netlify.

A Universal Secrets Platform is the hub where all those secrets are stored, managed, and versioned across environments. When they are changed in the USP, secrets automation updates them in each linked platform (CircleCI and Netlify in this case) with the option of automatically triggering a rebuild or redeploy. This model mirrors what source control platforms provide today. When code is pushed to a branch, GitLab pushes those changes to the destination infrastructure such as AWS.

Measuring success

As an industry, we will have a strong signal that this category is going to be successful when developers treat it like they would a source control platform. As essential as code reviews through pull requests are in today’s software development life cycle, so too will be managing secrets at the cloud and platform level. From the collaborative features enabling distributed teams to work effectively together, to the automations that enable CICD workflows, there is a lot left to do.

Use a Universal Secrets Platform

My hope is that you as a developer walk away from this long-winded article looking to add a Universal Secrets Platform to your toolbox. It's a completely different way of thinking about managing secrets, but just like when GitHub first pioneered the notion of a 'pull request’, once you see the benefits, there's no turning back. Together let's make the internet a more secure place for the projects we work on and the users they serve.

Doppler: How to Set Environment Variables for a Python Django Application using Apache and mod_wsgi in Docker

Ryan Blunden — Tue, 17 Aug 2021 07:00:00 +0000

Using environment variables to configure Django and other Python applications is awesome, but using them with Apache and mod_wsgi in Docker is a tricky thing to get right.

That's why I created this step-by-step tutorial and sample application to put all the info you need in one place.

Although this tutorial is for Docker and Django, the same steps apply, whether you're using a Virtual Machine or a different Python framework.

Prefer just to read the code? Head to the accompanying repository at https://github.com/DopplerUniversity/django-apache-mod-wsgi

Environment Variables, Apache and mod_wsgi

When hosting a Python WSGI compatible framework like Django in Apache with mod_wsgi, the only environment variables populated in the os.environ dictionary are those that exist in the environment of the script that starts Apache. But instead of having to mess with Apache's service manager settings (e.g. systemd or systemctl), there's a better way.

Most Apache distributions provide a shell script specifically for the purpose of setting environment variables that will be made available to modules such as mod_wsgi.

It's then a matter of knowing the location of this shell script as it can be different depending on the Linux distribution. For example:

Debian/Ubuntu: /etc/apache2/envvars
CentOS: /etc/sysconfig/httpd

We'll be using the python:3.9-slim-buster Docker image is Debian based.

Appending App Config and Secrets to the Environment Variables File

Essentially, it boils down to fetching the secrets as key/value pairs and writing them to the envvars file in the typical shell environiment variables format:

export FIRST_NAME="The"
export LAST_NAME="Mandalorion"

But where from and how do we fetch the app config and secrets to populate that file?

As I'm the Developer Advocate for Doppler, I'll start with a Doppler CLI example, but the mechanics of "fetch secrets, then append to file" can easily be adapted.

First, you would need to set up your project in Doppler and you can use the following button to get you started if you want to follow along.

Then use the Doppler CLI inside the Docker container to fetch the secrets (requires a DOPPLER_TOKEN environment variable with a Service Token value):

# Transform JSON key:value pairs into export statements using jq
if [ -n "$DOPPLER_TOKEN" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars using Doppler CLI'
    doppler secrets download --no-file | jq -r $'. | to_entries[] | "export \(.key)=\'\(.value)\'"' >> /etc/apache2/envvars
fi

Notice I've used single quotes, not double quotes around the values?

That's because it gives you the flexibility of storing secrets with double quotes such as JSON in Doppler which you could use for example, to dynamically set Django's ALLOWED_HOSTS setting dynamically for any environment.

ALLOWED_HOSTS = json.loads(os.environ['ALLOWED_HOSTS'])

You could also use a .env file but I wouldn't recommend it and instead, I'd look into using a secrets manager.

But that aside, here is how you could do it using an .env file:

if [ -f "$PWD/.env" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars from .env file'
    cat "$PWD/.env" >> /etc/apache2/envvars
fi

Now that we know how to pass environment variables from Apache to mod_wsgi, let's move onto getting this working in Docker.

Docker Configuration for Apache and mod_wsgi

Let's breakdown the task of configuring a Python Django Application using Apache and mod_wsgi in Docker into three steps:

Custom Start Script
Apache Site Config
Dockerfile

If you only want to see the working code examples, head to the accompanying repository at https://github.com/DopplerUniversity/django-apache-mod-wsgi

As this isn't a Docker or Apache tutorial, I won't be diving too deeply into the Dockerfile or Apache site config file, but if you've got questions, head over to the Doppler community forum and I'll be able to help you there.

1. Custom Start Script

Running your application in Docker is usually a case of setting the CMD, for example:

CMD ["python", "src/app.py"]

But it's trickier here as we first need to append the environment variables to /etc/apache2/envvars before running Apache.

As this requires multiple commands, we'll create a custom script:

#!/bin/bash

# apache-doppler-start

set -e

echo 'ServerName localhost' >> /etc/apache2/apache2.conf # Silence FQDN warning

# Doppler CLI
if [ -n "$DOPPLER_TOKEN" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars from Doppler CLI'
    doppler secrets download --no-file | jq -r $'. | to_entries[] | "export \(.key)=\'\(.value)\'"' >> /etc/apache2/envvars
fi

# Mounted .env file
if [ -f "$PWD/.env" ]; then
    echo '[info]: Appending environment variables to /etc/apache/envvars from .env file'
    cat "$PWD/.env" >> /etc/apache2/envvars
fi

# Run Apache
apache2ctl -D FOREGROUND

2. Apache Site Config

Here is an example Apache site config file for a Django application:

# wsgi.conf
<VirtualHost *:80>
    ServerName django-apache-mod-wsgi
    ServerAlias django-apache-mod-wsgi
    ServerAdmin webmaster@doppler

    # Defining `WSGIDaemonProcess` and `WSGIProcessGroup` triggers daemon mode
    WSGIDaemonProcess django-apache-mod-wsgi processes=2 threads=15 display-name=%{GROUP} python-path=/usr/local/lib/python3.9/site-packages:/usr/src/app    
    WSGIProcessGroup django-apache-mod-wsgi
    WSGIScriptAlias / /usr/src/app/doppler/wsgi.py

    <Directory /usr/src/app/doppler/>
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>

    # Redirect all logging to stdout for Docker
    LogLevel INFO
    ErrorLog /dev/stdout
    TransferLog /dev/stdout
</VirtualHost>

3. Dockerfile

The Dockerfile is reasonably straightforward, installing the Doppler CLI and Apache dependencies before copying the Django source code, custom script, and Apache site config:

FROM python:3.9-slim-buster

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

# Install Doppler CLI and related dependencies
RUN apt-get -qq update && apt-get install -y apt-transport-https ca-certificates curl gnupg jq && \
curl -sLf --retry 3 --tlsv1.2 --proto "=https" 'https://packages.doppler.com/public/cli/gpg.DE2A7741A397C129.key' |  apt-key add - && \
echo "deb https://packages.doppler.com/public/cli/deb/debian any-version main" | tee /etc/apt/sources.list.d/doppler-cli.list && \
apt-get -qq update && apt-get install doppler

# Install Apache and related dependencies
RUN apt-get install --yes apache2 apache2-dev libapache2-mod-wsgi-py3 && \
    apt-get clean && \
    apt-get remove --purge --auto-remove -y && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /usr/src/app

COPY requirements*.txt .
RUN pip install --quiet --no-cache-dir --upgrade pip && \
    pip install --quiet --no-cache-dir -r requirements.txt

# Application source
COPY src/ ./

# Custom CMD script
COPY apache-doppler-start /usr/local/bin/

# Apache site config
COPY wsgi.conf /etc/apache2/sites-enabled/000-default.conf

EXPOSE 80 443

# https://httpd.apache.org/docs/2.4/stopping.html#gracefulstop
STOPSIGNAL SIGWINCH

CMD ["apache-doppler-start"]

With all the pieces in place, we can nowbuild the Docker image (clone the sample repository to follow along):

docker image build -t django-apache-mod-wsgi:latest .

Now we're ready to run the container!

Running the Django Application with Apache and mod_wsgi in Docker

We'll start with a Doppler example, then with an .env file.

With Doppler, you'll first need to set a DOPPLER_TOKEN environment variable to the value of a Service Token. This is what provides read-only access to a specific Doppler config in production environments.

Usually, this would be securely set by your deployment environment (e.g. GitHub Action Secret) but for completeness and simplicity, we'll set it manually below:

export DOPPLER_TOKEN="dp.st.xxxx" # Service token value created from Doppler dashboard

Now run the container:

docker container run \
    -it \
    --init \
    --name doppler-apache-mod-wsgi \
    --rm \
    -p 8080:80 \
    -e DOPPLER_TOKEN="$DOPPLER_TOKEN" \
    django-apache-mod-wsgi

.env File

To run the .env file version, we'll use the sample.env file from the sample repository:

# sample.env
export DJANGO_SETTINGS_MODULE='doppler.settings'
export DEBUG='yes'
export ALLOWED_HOSTS='["*"]'
export SECRET_KEY='bf5e1b31-6ba7-48e2-9175-f2293671e6df'

Then to run the container:

docker container run \
    -it \
    --init \
    --name dotenv-apache-mod-wsgi \
    --rm \
    -v $(pwd)/sample.env:/usr/src/app/.env \
    -p 8080:80 \
    django-apache-mod-wsgi

Summary

Nice work in making it to the end!

Now you know how to configure Python applications hosted with Apache and mod_wsgi running in Docker using environment variables for app configuration and secrets.

Feedback is welcome and you can reach us on Twitter, our Community forum, or send me an email at ryan.blunden@doppler.com.

Why Secrets Management Is NOT Just a Key-Value Store

Ryan Blunden — Tue, 10 Aug 2021 00:28:34 +0000

For those who’ve managed secrets via .env files or have briefly looked at secrets managers such as HashiCorp Vault , it’s easy to get the impression that “secrets management” is simply the storage and retrieval of secrets from a Key-Value data store.

But recurring tasks such as instantly comparing secret values between environments when troubleshooting, or being alerted when a new secret is introduced are crucial features every secrets manager should provide. Without them, development teams are forced to cobble together their own solutions, often in silos on a per-application basis.

The reality is that .env files and traditional secrets managers were never designed to meet the needs of modern application development teams, where microservices and multi-cloud deployments are the new normal.

Essentially, we want the flexibility of a Key-Value secret storage solution but with an operating model and feature set that decreases complexity, increases developer productivity, and standardizes how secrets are managed for applications in every part of the business.

In this post, we’ll explore the most important aspects of secrets management and why Key-Value storage is only the beginning.

Access Control and Permissions

Who needs access to secrets? What should their level of permissions be and which applications and environments should they have access to? Is access provided according to org structure or application ownership?

If access controls aren’t fine-grained enough, you risk violating the principle of least privilege but if too rigid, can impede the ability of teams to embrace a culture of DevOps and “shifting left” to bring security earlier into the application development process.

The following is a starting point for secret and access permission requirements:

Ability to segment parts of the business into separate workplaces so for example, an acquired start-up can onboard to using the secrets manager while remaining entirely separate from the organization’s existing secrets.
Consider how access will be managed across cloud and platform boundaries, e.g. AWS, GCP, and Vercel?
Ability to grant access to secrets for specific applications and environments within each application.
In most cases, developers should only have access to secrets belonging to their applications and only for the appropriate environment (e.g., dev and test). DevSecOps will have greater permission levels so that they can work cross-functionally in any environment.
Temporary secrets access to external entities (e.g. contractors) should be scoped to a subset of application secrets via an auth/service token.
One-off sharing of secrets to external entities (e.g. TLS Certificates to an external firm) should be captured in an activity log and sent via encrypted means using a service such as Doppler Share.
Mandated MFA for secrets manager dashboard access.
Privilege elevation (e.g. developer viewing production secrets) be time-limited and captured in an activity log.
Seamless integration with existing Single Sign-on solutions such as SAML and SCIM to allow default access permission levels to be assigned based on role and/or group membership.

A further consideration is restricting machine access to a CIDR IP address range, preventing leaked service tokens from use outside of trusted corporate and cloud networks.

There’s a lot to consider for access control and permissions and your secrets manager should make it straightforward to meet these requirements.

Versioning and Rollback

Versioning and the ability to roll back changes is a must and shouldn’t come at the cost of reduced performance or additional storage fees.

While most secret managers support versioning and rollback, it’s critical to evaluate the actual process.

Being able to roll back a misconfigured change in a single click with support for triggering an automatic application reload or deploy is something teams should come to expect from a secrets manager, enabling a misconfiguration fix to be applied in seconds.

Secret Documentation

There are wildly different perspectives on documenting code, but the one thing all developers can agree upon is that useful code comments provide context that explains the “why”, not just what the code does.

Such context and comments can be especially valuable for secrets, but where should this be documented?

A reasonable option is a README or inline code comments where the secret is in use, however, the context may be lost if a secret’s value is changed in your secrets manager by someone who hasn’t viewed the code.

Being able to notate secrets directly in the secrets manager reduces the chance of misconfiguration errors that could’ve been easily avoided, had the appropriate comments (and context) be provided.

Local Development

The closer your development environment is to production, the less likely unexpected problems are to arise during deployment, which is why secrets should be accessed the same way in development, as they are in production.

Your secrets manager should treat Development as a proper environment in its own right and be responsible for supplying secrets to developers locally. This saves each developer from manually updating and patching their own environment variables list in their IDE or .env file.

Development environments also present a unique challenge where secret values will often be specific to each developer, e.g. when working on a feature branch that uses a new secret.

Developers need the flexibility to override secrets during development without affecting other developers on the team. This is an essential feature a secrets manager should provide to ensure secrets access remains consistent for every environment.

A Single Unified View for Application Configuration

One of the reasons developers love .env files is that it gives them a single unified view of how an application is configured as they tend to hold both secrets and config data such as API keys, port bindings, and feature flags.

When assessing the transition from .env files to a secrets manager, teams are often confused as to whether secrets and config should be separated. For example, should secrets be migrated to the secrets manager while config continues to exist in .env files?

While secret managers are designed to store sensitive data, it makes sense to store both config and secrets together, as there is no logistical or financial advantage gained by separating them.

If secrets and config are simply displayed as Key-Value pairs in the secrets manager’s dashboard, it’s up to the viewer to define a way to visually organize secrets by application, then further segment into separate environments.

Your secrets manager should provide a simple built-in mechanism to view secrets for a specific application and environment, and storing config as well as secrets in your secret manager ensures you’ll still get a single unified view of how your application is configured.

Reduce Friction to Increase Adoption

While implementing a secrets manager is often a security endeavor, it's better viewed as a productivity one, as every project team can appreciate time and effort saved. And if security posture is improved, well even better!

But if the focus is on security at the cost of productivity (even if short term), gaining widespread adoption will be significantly more difficult, as product teams are incentivized to ship new features, not improve security (unless mandated to do so).

Therefore, the secrets manager that is likely to have the biggest impact in improving security, is the one that provides a nearly frictionless experience for teams to implement and use on a daily basis.

Preventing Misconfiguration Issues

Misconfiguration issues are much easier to deal with if caught before changes are applied in production. To achieve this, developers and DevOps Engineers need access to a secrets activity log that pushes secret events to where they naturally communicate such as a Slack or Microsoft Teams channel.

The secrets management dashboard should also provide indicators of potential problems such as if a secret was added to the staging environment but not production, as well as troubleshooting features such as the ability to instantly compare a secret value across every environment.

A secrets manager must therefore help to prevent known issues associated with application configuration and deployment scenarios to increase production stability and reduce time spent firefighting issues caused by misconfiguration.

Automate Application Reload and Redeployment

The Infrastructure as Code movement put automation at the heart of deployment workflows and an essential feature of any secrets manager should be the option to automatically trigger an application to reload or redeploy if its config or secrets change.

This again highlights the need for secret managers to go beyond simply Key-Value storage, as, as they must know which application to trigger a redeployment for, based on the application and environment the changed secret belongs to.

Secrets Management Goes Beyond Secret Storage

By now, it should be clear that secrets management extends beyond the secure storage of secrets as Key-Value pairs. We’ve touched on processes, best practices, as well as various risks, challenges, and issues that can arise and what a secrets manager should reasonably do to mitigate them.

These relate to key elements of storage, access, visibility, integrations, workflows, and troubleshooting processes for not just secrets management, but application configuration generally.

Doppler is simply the result of building the universal secrets manager we wish had but simply didn’t exist.

Doppler enables you to stop using old ways to store, share, and access secrets via .env files.

It's your centralized source of truth for managing secrets across multiple clouds, platforms, and packaging formats, from cloud-native build packs to containers, serverless, and more.

Forem: Doppler

How we built our Secrets Rotation Engine

Requirements for Doppler’s Rotation Engine

Encryption and Storage

Atomic Operations

Graceful Error Handling

No Public Access Required

The Two-Secret Strategy

Types of Rotation

Rotation State

Two-Phase Commits

Delivering Secret Fields to the Application

Automatic Redeployment

Handling Errors

Closing Thoughts

Doppler Encrypted Secrets Snapshots for High Availability

Build. Encrypt. Deploy. Fallback.

Passphrase

Build

Deploy

Summary

State of Kubernetes Secrets Management in 2022

Side Note: Enable Encryption of Kubernetes Secret Data

Kubernetes Secrets Usage and Challenges

Secrets as Files

Secrets as Environment Variables

Kubernetes Secrets Source of Truth

Secrets Managers and Kubernetes

1. Secrets Manager API or SDK

2. Secrets Agent/Sidecar Injection

3. Kubernetes Secrets Operators

4. Secrets Store Container Storage Interface (CSI)

Auto-Updating Deployments When Secrets Change

Kubernetes Secrets Management Essentials for 2023 and beyond

How the Doppler SecretOps Platform Could Change the Game

Summary

Managing Kubernetes Secrets with the External Secrets Operator and Doppler

Outcomes

Prerequisites

Sample Application

Doppler

External Secrets

Integrating Doppler with Kubernetes

What's Next?

How To Prevent Secrets From Ending Up On Developers' Machines

SecretOps and Local Development

Dynamic Secrets Injection

Ephemeral Secrets Files

Summary

Manually Updating .env Files Isn't DevOps

1. Centralized Environment Variable Management

Why we need SecretOps

2. Dynamically Injected Environment Variables

Platform Injected Environment Variables

CLI Application Runner

3. Docker Container Environment Variables

Docker Compose Environment Variables

Kubernetes Environment Variables

Doppler CLI Created Secrets

Doppler Kubernetes Operator Managed Secrets

Kubernetes Deployments and Environment Variables

Dynamically Created Ephemeral .env Files

Summary

Crash course in Web3 Application Development with Python

A Chain Primer

First Steps

Accounts and Keys

Interacting with the Blockchain

Sending ETH

Production Considerations

Further Reading

$20m Series A to build the first SecretOps Platform

Redefining secrets management through multi-cloud secrets sync and automation at enterprise scale.

About CRV

Why syncing .env files doesn’t scale for secrets management

Learn why using a Universal Secrets Platform is the key to managing environment variables at scale and eliminates the need for syncing .env files.

A brief history of .env files

The problems with .env files

The hidden productivity costs from using .env files

Why developers have ignored traditional secrets managers