Forem: Tyler Langlois

Managing Kubernetes Secrets with the External Secrets Operator and Doppler

Tyler Langlois — Tue, 25 Oct 2022 15:48:28 +0000

API keys, credentials, and other types of sensitive information are the primary way your application calls external services and interacts with the world outside its own runtime. Ensuring that your secrets are secure is one of the most important operational tasks that you need to address, but it's not just rigorous security that's important providing an ergonomic, audited, and maintainable flow for the management of those secrets is crucial as well. If managing credentials is confusing or difficult, mistakes can undo all of the careful work that goes into keeping secrets secure.

Container orchestrators like Kubernetes offer helpful abstractions to address the need for sensitive values with native Secrets. However, a Kubernetes Secret is a somewhat rudimentary object it lacks encryption by default and normally exists as a plain key/value within etcd. By contrast, services like Vault or Doppler are intentionally designed to provide strong guarantees and well-defined access controls. Can we combine the operational power of Kubernetes with the assurance of a fully managed secret provider?

Yes! The External Secrets Operator is a Kubernetes operator that bridges the gap between Kubernetes' native secret support and external systems that provide a canonical source of truth for secret storage. It does this by leveraging custom resources that define how to retrieve external secrets in order to manage the lifecycle of Secret resources in your Kubernetes cluster. In this tutorial, we'll use Doppler as the external secret provider to illustrate using external secrets in a real-world application.

Outcomes

The end goal of this guide will be to leverage well-managed secrets in an application deployed in Kubernetes. To achieve this, we'll use:

The Kubernetes External Secrets Operator
Doppler to store and manage external secrets
minikube or kind for a local Kubernetes API

At the conclusion of this tutorial, you'll understand how to store and manage secrets in Doppler, use those same secrets as native Kubernetes secrets, and ultimately use them in a running application. Let's begin!

Prerequisites

This article assumes that you'll follow along with your own Kubernetes cluster. You may choose to operate in an existing cluster (potentially within a sandboxed namespace), but if you'd like to use a sandbox instead, we suggest using either minikube or kind to create a local Kubernetes installation which will provide a safe environment for testing. In order to provide a clean and pristine environment, this tutorial will use minikube as the Kubernetes target.

In addition to a functional Kubernetes cluster, ensure that the following command-line tools are installed:

helm, which we'll use to install the External Secrets Operator Kubernetes resources. Use the helm quickstart guide to install helm.
The Doppler command line utility doppler using the Doppler CLI Guide documentation.
- You will need to register for a Doppler account to create projects and store external secrets.
kubectl to interact with Kubernetes. kubectl is available for a wide range of operating systems, and the Kubernetes documentation provides comprehensive installation guides for kubectl and other tools as well.
If you choose to use a local Kubernetes sandbox, install minikube and follow the instructions to create a local instance of Kubernetes. This can be as simple as minikube start, but consult the documentation if you need additional assistance.
- minikube bundles a version-compatible installation of kubectl if you choose to use minikube for this exercise. It's a handy time saver.

Once installed, confirm that your environment is ready to go:

kubectl version should confirm that kubectl is present and communicating with the Kubernetes API successfully. If you aren't using minikube, simply use the plain kubectl command for the remainder of the tutorial.

$ minikube kubectl -- version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:38:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

The helm and doppler commands should be available:

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.13"}

$ doppler --version
v3.44.0

With the prerequisites installed, let's proceed with building and running our program.

Sample Application

Before we start creating secrets, let's begin by illustrating the external secrets workflow with a simple example service. The following guide will assume the use of minikube to build and load an application container image, so if your Kubernetes environment is different, you may need to adapt the instructions to work with your specific container registry strategy.

First, perform some initial steps to bootstrap a python Flask application:

$ mkdir -p ~/tmp/secret-sauce
$ cd ~/tmp/secret-sauce
$ git init
$ echo flask > requirements.txt
$ python3 -m venv venv
$ ./venv/bin/pip install -r requirements.txt

Create a file named app.py that contains the following simple web application:

from flask import Flask
from os import environ

app = Flask( __name__ )

sauce = environ.get('SECRET_SAUCE')

@app.route("/")
def index():
    if sauce:
        return f"The secret sauce is: {sauce}!"
    else:
        return "You'll never find my secret sauce."

Define a small Dockerfile that defines how to build a container image for our application:

FROM python:3

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "-m" , "flask", "run", "--host=0.0.0.0"]

You can run this application locally with:

$ ./venv/bin/flask run

Try accessing the running application at http://localhost:5000 to see a response. The root route (/) will render a static string when no secret is present in the environment but will print the secret when the indicated environment variable is defined. Don't print secrets in production! This application is just a demonstration of how to read the value in a real program.

We're ready to load this application into Kubernetes. Assuming that you're following along with minikube, proceed to build the application container in the Kubernetes node under the name "app":

$ minikube image build -t app .
...lots of output...
Successfully tagged app:latest

Create a new Kubernetes Deployment file named deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: webapp
          image: app
          imagePullPolicy: Never
          ports:
            - containerPort: 5000

Take note of a few assumptions in this Deployment:

imagePullPolicy has been set to Never because we're running a locally-built image. By default, Kubernetes would attempt to pull the app image, which doesn't exist. As previously mentioned, if you're following this tutorial in an environment other than minikube, you may need to push the container image to a registry and adjust these settings slightly.
We've exposed port :5000 which we can access later.

Load this Deployment into your running cluster:

$ kubectl apply -f deployment.yaml
deployment.apps/app created

Finally, forward the port in another terminal window in order to access the running application in a simple tunnel.

$ kubectl port-forward deployment/app 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000

Send a request to your Kubernetes application to see it in action:

$ curl http://localhost:5000
You'll never find my secret sauce.

Fantastic! Note that we've received the response that indicates no secret value has been injected into the environment. How can we add a secret to the application and see the secret sauce?

Doppler

By using Doppler, we can achieve a great deal of control over application secrets and manage them at each step of our application's lifecycle, from in our local development environment to within a Kubernetes workload.

If you haven't signed up with Doppler, you can do so now. Once you have an account, proceed to use the login command to set up your account locally:

$ doppler login

Setup Doppler for the demo application by defining a doppler-template.yaml file in the root of the application directory. This YAML file defines a template that we can import to create a new Doppler project easily from the command line:

projects:
  - name: secret-sauce
    description: Kubernetes demo app
    environments:
      - slug: dev
        name: Development
        configs:
          - slug: dev
      - slug: stg
        name: Staging
        configs:
          - slug: stg
      - slug: prd
        name: Production
        configs:
          - slug: prd
    secrets:
      dev:
        SECRET_SAUCE: tartar
      stg:
        SECRET_SAUCE: horseradish
      prd:
        SECRET_SAUCE: tzatziki

Enter the directory in your shell and run the following doppler command in order to bootstrap your Doppler project. This will create a new project called secret-sauce, set up a few different environments, and load an initial secret value for SECRET_SAUCE:

$ doppler import

You'll see output similar to the following:

┌──────────────┬──────────────┬─────────────────────┬──────────────────────────┐
│ ID           │ NAME         │ DESCRIPTION         │ CREATED AT               │
├──────────────┼──────────────┼─────────────────────┼──────────────────────────┤
│ secret-sauce │ secret-sauce │ Kubernetes demo app │ 2022-10-11T22:10:36.567Z │
└──────────────┴──────────────┴─────────────────────┴──────────────────────────┘

View the secrets for this project with doppler secrets:

$ doppler secrets

In addition to some default variables that begin with DOPPLER_, you'll also find the secret value we'd like to inject, SECRET_SAUCE:

┌─────────────────────┬──────────────┐
│ NAME                │ VALUE        │
├─────────────────────┼──────────────┤
│ DOPPLER_CONFIG      │ dev          │
│ DOPPLER_ENVIRONMENT │ dev          │
│ DOPPLER_PROJECT     │ secret-sauce │
│ SECRET_SAUCE        │ tartar       │
└─────────────────────┴──────────────┘

Run a quick test to confirm that our application behaves as expected when a secret is present in its environment variables. To do this, we can use the doppler run command to easily inject the variable into our application.

$ doppler run -- ./venv/bin/flask run

Then issue a request to the listening port to see whether the response has changed:

$ curl http://localhost:5000
The secret sauce is: tartar!

Great! Let's continue on to install the External Secrets Operator.

External Secrets

The External Secrets Operator provides the translation layer between Kubernetes' native secrets and external secrets. The operator leverages custom resources in order to model external secrets, which it then retrieves as necessary and translates into native Kubernetes secrets that your workload can easily consume.

The operator is provided as a Helm chart, so first add the upstream external-secrets Helm repository to access the chart:

$ helm repo add external-secrets https://charts.external-secrets.io
external-secrets has been added to your repositories

Next, install the chart into your Kubernetes cluster. The following command:

Instructs helm to use the external-secrets chart,
passes -n to install the chart into the external-secrets namespace,
creates the namespace as necessary with --create-namespace, and
configures the requisite CRDs as well (--set installCRDs=true)

$ helm install external-secrets \
     external-secrets/external-secrets \
     -n external-secrets \
     --create-namespace \
     --set installCRDs=true

You should see output similar to the following:

NAME: external-secrets
LAST DEPLOYED: Tue Oct 11 12:27:18 2022
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully!

In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets

With this operator installed, we're now ready to set up Doppler to serve as the source for external secrets.

Integrating Doppler with Kubernetes

The first step is to create a service token which serves as the authentication mechanism for the external secrets operator against Doppler. It should be stored inside of a generic Kubernetes secret that the operator will consume.

You can generate and store the token in a single step with some fancy footwork in the shell to avoid copying and pasting the token around. The following command creates a new Doppler token and then immediately loads it into Kubernetes:

$ kubectl create secret generic \
    doppler-token-auth-api \
    --from-literal dopplerToken=$(doppler configs tokens create --config prd doppler-auth-token --plain)
secret/doppler-token-auth-api created

Note that we're passing the --config prd flag to doppler in order to create a token scoped to the prd (production) configuration of our Doppler project. In our application directory, we defaulted to dev, which has its own secrets. With this technique, we're only interacting with development secrets locally while loading production secrets into our container runtime, which keeps the risk of exposure for production secrets minimal.

Next, create a SecretStore CRD that points the operator at your Doppler service token secret. This step sets up Doppler as a source for external secrets that we can call upon later.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: doppler-auth-api
spec:
  provider:
    doppler:
      auth:
        secretRef:
          dopplerToken:
            name: doppler-token-auth-api
            key: dopplerToken

Create the SecretStore in Kubernetes:

$ kubectl apply -f secretstore.yaml
secretstore.external-secrets.io/doppler-auth-api created

We're ready to inject our secret sauce! A new ExternalSecret is the necessary resource that enables us to synchronize one of our Doppler secrets to a related generic Kubernetes secret. Create a new file called externalsecret.yaml with the following YAML:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: secret-sauce
spec:
  secretStoreRef:
    kind: SecretStore
    name: doppler-auth-api

  target:
    name: secret-sauce

  data:
    - secretKey: SECRET_SAUCE
      remoteRef:
        key: SECRET_SAUCE

Load it into Kubernetes:

$ kubectl apply -f externalsecret.yaml
externalsecret.external-secrets.io/secret-sauce created

You can confirm that the secret was loaded by the operator by listing it with kubectl:

$ kubectl get secret secret-sauce
NAME TYPE DATA AGE
secret-sauce Opaque 1 3s

Success! Note that if you do not see the new secret, you can also inspect the External Secret Operator's logs to debug any issues:

$ kubectl -n external-secrets logs -lapp.kubernetes.io/name=external-secrets

Generic secrets can be seen in the Kubernetes dashboard as well. If you're using minikube, you can quickly install and view the Kubernetes dashboard with the following command:

$ minikube dashboard

Your browser will open to the dashboard landing page. You may use the left sidebar to navigate to the "Secrets" link to view your new secret:

The only task left to do is to consume the secret from our application. Here's an updated Deployment that references the newly-created Secret. Note the new env key, which injects the variable SECRET_SAUCE drawn from a secret named secret-sauce under the key SECRET_SAUCE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: webapp
          image: app
          imagePullPolicy: Never
          env:
            - name: SECRET_SAUCE
              valueFrom:
                secretKeyRef:
                  name: secret-sauce
                  key: SECRET_SAUCE
          ports:
            - containerPort: 5000

Apply the updated Deployment:

$ kubectl apply -f deployment.yaml
deployment.apps/app configured

The changed Deployment manifest will recreate any necessary pods, so invoke a new forwarded port once the pods have restarted to forward requests to the new containers:

$ kubectl port-forward deployment/app 5000:5000

Finally, Issue an HTTP request to the running container to see the injected Doppler secret in action:

$ curl http://localhost:5000
The secret sauce is: tzatziki!

Note that the content of this secret differs from the one we saw in our local development environment. The --config prd flag to the doppler command loaded a token with rights to the prd configuration in Doppler, which injects the correct secret for the configuration that the Doppler token has been scoped to.

Congratulations! You've successfully:

Installed the External Secrets Operator into your Kubernetes cluster
Managed an application secret with Doppler
Connected Doppler with the External Secrets Operator
Used a Doppler project secret within a Kubernetes Deployment

What's Next?

There's even more you can do with the External Secrets Operator and Doppler, so if you'd like to learn more, dive into the documentation to learn how to use features like JSON processing, filtering, and more.

If you're actively using Kubernetes secrets stored in etcd today, you should also configure encryption at rest so that your secrets are secure no matter where they're stored. Whether you load them with kubectl or the External Secrets Operator, configuring encryption at rest is still an important step to address every step of your secret management lifecycle.

Crash course in Web3 Application Development with Python

Tyler Langlois — Tue, 31 May 2022 13:00:00 +0000

In the world of cryptocurrency, private keys are king. Mathematical mechanics treat correctly signed transactions as valid regardless of intent, which makes account keys and passphrases a particularly lucrative target for malicious actors. While end-user best practices for securing keys tend to be plentiful, the development side of blockchain work takes place amid a sea of API tokens, key files, and service credentials that can sprawl out of control if not managed correctly.

Ensuring that these sensitive values remain secure is critical and outdated solutions such as .env files are no longer sufficient.

This tutorial will start from the basics of interacting with blockchain APIs from joining as a peer upwards. There are numerous third-party services and APIs that sit atop various blockchains, but establishing secure management practices for primitives such as private keys is fundamental.

No prior experience interacting with blockchain or web3 services is assumed, and we'll use a testnet in order to limit the potential exposure of sensitive information. Doppler will be used to store and retrieve credentials associated with private key material, which is a secure and convenient way to manage secrets without the risks of storing them on disk in unencrypted text files.

Before downloading any programs or generating account keys, there are a few concepts to establish heading into the unique paradigms that accompany working with web3 networks.

While plenty of unique blockchains and products abound in the web3 space, this guide will focus on the Ethereum blockchain, which is one of the more commonly used networks. Like most cryptocurrency blockchains, Ethereum operates in a decentralized way with a vast network of computers that can be interacted with as a peer. However, any sort of operation that requires some proof of ownership, e.g. signing a transaction, does require the authoritative stamp of a private key.

This is the core of how the network operates, which makes the method of communication and management of privileged information a chief concern.

This tutorial will follow a basic outline to work directly with the Ethereum blockchain and its API:

Connecting to the testnet
Generating and securing public keys
Receiving tokens
Sending tokens securely

Let's begin!

A Chain Primer

When choosing how to connect to the Ethereum blockchain, there are two general approaches, each with distinct advantages and disadvantages.

The first is by connecting to a third-party API that serves as a frontend to low-level blockchain network protocols. Services like Infura offer a web-based JSON RPC API, which is widely compatible with a range of libraries and languages and allows developers to interact with the blockchain without downloading the entire ledger, which can take up significant compute resources and disk space (around 500GB at the time of this writing).

Using a third-party API means that your code can run from a variety of environments without being tethered to a running node and the accompanying resource requirements that are necessary. The downside to this approach is reliance upon a third party when one of the core advantages of blockchain technology is decentralisation. Routing communication with the aid of an intermediary party is convenient, but means that clients must trust the provider and are subject to the inherent properties of using a hosted API instead of communicating directly via peer protocols (such as availability and uptime). These tradeoffs are often the right choice, but are tradeoffs nonetheless.

The second approach is to run a native node to communicate to the chosen blockchain directly. As previously mentioned, there are significant computational costs involved, but this means that the client speaks directly to the chosen peer network which brings with it benefits like smaller risk exposure, high availability of the decentralised network, and the ability to self-validate the ledger.

There are additional considerations to take into account when operating a dedicated node, such as choosing whether to download the entire ledger during the initial loading process or instead electing a more lightweight method that incurs a smaller disk usage cost.

This guide will opt for the self-hosted node technique, though in practice, either approach is a reasonable choice.

Finally, a node operator must select which network to connect to when initialising their connection.

Normally, users will choose to connect to the live, "production" chain when interacting with a public ledger, but during local development and testing, a testnet is a better choice. In addition to less traffic (resulting in smaller ledgers and transaction costs), acquiring tokens to send and receive is usually free and does not require any initial funds. This is a significant advantage when sending and receiving tokens without risk of making mistakes that could incur significant consequences.

First Steps

We'll use geth (Go Ethereum) to connect to the Ethereum network, which is a golang implementation that can function not only as a client but a fully featured node as well.

First, follow the instructions for installing geth for your operating system. Once installed, the geth command should be available from the terminal.

geth version
Geth
Version: 1.10.17-stable
Architecture: amd64
Go Version: go1.16.13
Operating System: linux
GOPATH=
GOROOT=/nix/store/j1x3cy8g2cqcr54rg98gw0mdc28jlyc8-go-1.16.13/share/go

The command-line flags passed to the geth CLI dictate which network it will connect to. There are a number of testnets within the Ethereum network to choose from, and this guide will use the Goerli proof-of-authority network as its newer than some of the others, and is not coupled with the computationally-expensive proof-of-work networks.

The --syncmode flag controls the method that geth will use when downloading the historical ledger of the blockchain. On one end of the extreme, the full method will download the entirety of the blockchain ledger which includes all the relevant transactional data. On the other, the light method will only fetch recent block headers and retrieve other information on-demand. When communicating with the production Ethereum network, the full method offers the most comprehensive assurance that the local copy of blockchain data is valid and legitimate. However, in this exploratory case on a testnet, the light method is sufficient.

Begin the geth process in a background terminal on your machine. The initial syncing process will take time, but using the light method on the goerli testnet will not use excessive space (at the time of this writing, the entirety of the chain requires about 500MB of disk space). The following command will store the Goerli chain data in the ~/.goerli directory:

geth --syncmode light --datadir ~/.goerli --goerli
INFO [04-05|15:54:49.916] Starting Geth on Goerli testnet...
...

After some time, the daemon will stop importing historical blocks and the latest blocks will be available.

The logging output from geth will include the age of a block when performing initial synchronisation. When you see blocks being imported with output such as age=4m04w1d, this indicates old blocks being retrieved to construct the local chain. Once older blocks stop appearing, this indicates that the initial synchronisation is complete.

You can continue with the rest of this guide while the blockchain synchronises; full synchronisation will only be necessary once we reach the portion that covers transactions.

Accounts and Keys

At the beginning of this guide, we spoke at length about the importance of keys when interacting with the blockchain.

Public-key cryptography forms the backbone of the cryptographic methods for forming and authenticating transactions on public ledgers. Private keys prove ownership of public addresses and sign transactions to send tokens to other public addresses.

While high-level, user-facing cryptocurrency applications may leverage account information for wallets in the form of traditional username/password combinations, the core values that comprise an "account" on a blockchain like Ethereum is a cryptographic key - tokens belong to an addressed coupled to a private key. The geth CLI includes utilities to generate keys, which it calls an account.

To begin, create a new account. Remember that this "account" is really a locally encrypted file and not an account that can be "logged into" from another machine - if you are using an account tied to a production blockchain, it's critical to ensure that this account is secured by a strong passphrase backed up regularly.

In our case, the local account and keypair will be associated with a testnet, but we'll still use best practices for managing the passphrase by storing it in encrypted secrets store (Doppler) and treat the account as sensitive. Because the running geth process is active, we'll connect to the running daemon via attach instead of using the geth account subcommand.

Enter the geth console:

geth --datadir ~/.goerli attach

You'll be greeted by the command prompt for geth javascript commands.

Welcome to the Geth JavaScript console!

instance: Geth/v1.10.17-stable/linux-amd64/go1.16.13
at block: 6664849 (Tue Apr 05 2022 17:34:43 GMT-0600 (MDT))
datadir: /home/yourname/.goerli
modules: admin:1.0 clique:1.0 debug:1.0 eth:1.0 les:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 vflux:1.0 web3:1.0

To exit, press ctrl-d or type exit
>

To create a new account, run the following command where you'll be asked to enter a passphrase:

personal.newAccount()

After following the prompts, geth will create a keypair within the ~/.goerli/keystore/ directory. You can view the public address of this account at any time by using the following command from the geth console:

personal.listAccounts

At this point, your local machine will have a light copy of the Ethereum Goerli testnet blockchain and a keypair ready to send and receive tokens. Let's begin working with transactions!

Interacting with the Blockchain

A simple task to confirm the functionality of this local testnet setup is to receive currency. Unlike currency on the live, production blockchain, currency on a testnet is often given away freely to permit developers to test and iterate on projects and products.

Providers for free currency - often called "faucets" - may sometimes be unavailable or lacking sufficient funds to distribute tokens freely. At the time of this writing, the faucet mentioned here is functional and has sufficient funds to distribute them to anyone who requests them, but if you encounter problems requesting funds at a later date than when this tutorial was written, you may also seek out alternate sources (remember that they must be operating on the Goerli Ethernet testnet).

Well be using the Goerli Faucet to acquire initial funds to begin experimenting with the blockchain.

From the geth console, find your public address using listAccounts:

personal.listAccounts
["0x..."]

Copy the address, navigate to the Goerli Faucet, paste your address into the Wallet Address field, then click Send Me ETH.

Blocks are minted frequently on this chain, so it's time to check your balance! We'll use the Web3 Python library to interact with the local node.

This guide assumes that you have Python 3 installed on your local machine. Weve provided a repository with sample code as a quick start to getting up to speed with how to use the Web3 library.

Clone the GitHub repository then enter the repository directory:

git clone https://github.com/DopplerUniversity/python-web3
cd python-web3

Create a Python virtual environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Feel free to open web3-app.py and take a look! The Python web3 library makes accessing blockchain information fairly straightforward, so this example code should be a good place to start if you want to experiment later.

Let's verify the script is working by fetching the timestamp of the latest block on the chain:

python web3-app.py latest-block

If you see a timestamp that is within 30 seconds or so, congratulations! You've successfully interacted with your local instance of the testnet blockchain. The next task is to check the balance of the address that your faucet tokens were sent to.

First, confirm that Python is able to see the local account:

python web3-app.py accounts

Your account number should match the address that you previously found within the geth console using personal.listAccounts. Finally, check the account balance.

python web3-app.py balance

The account balance for the address entered in the Ethereum faucet site should be 0.05 ETH. You've checked your account balance from the python web3 library. Well done!

Sending ETH

Until now we've only interacted with the local blockchain with APIs that do not require private account keys. In order to finish this experiment and successfully sign a transaction with your account, we'll practice by sending an amount of ETH back to the Goerli faucet. Doing so will help demonstrate the process as well as replenish funds to future developers on the blockchain!

To send ETH, well use the send_transaction method:

web3.geth.personal.send_transaction(self, transaction, passphrase)

Remember when we created our local Geth account and provided the passphrase?

The passphrase is required here to unlock the private key and sign the transaction in order to validate it as a legitimate transfer of funds out of our account. However, entering a passphrase into a REPL or a shell is a security concern: most REPL and shell histories (like bash or the python interpreter) retain commands that are entered, so it's wise to avoid entering a passphrase manually in plaintext.

What other options exist? One approach is to read secret values - whether it be a passphrase, API key, or otherwise from a file on disk such as .env or JS file but this is a dangerous option as well explain.

Storing secrets in plaintext files is a huge security nightmare are as they can be read by other user processes, and risk being committed during work with tools like git if a developer makes a mistake such as Nat Eliason did when he lost $30,000 by accidentally pushing a file containing his passphrase to GitHub.

Reading the value from an environment variable is a good idea, but that begs the question of how the environment variable containing the passphrase will be securely populated.

This is where a SecretOps platform such as Doppler can help by using the Doppler CLI to inject secrets as environment variables into an application process. This allows an environment variable to be populated with the passphrase securely without the risks of unencrypted local file storage.

To begin, first sign up for a Doppler account. The sign up process will walk through a few helpful tips to get started, and once your account has been configured, proceed to install the Doppler CLI, then authenticate your machine by running:

doppler login

Create a new project called python-web3 which will hold the secrets for this project and keep things organised as well as within their own environment scope:

doppler projects create python-web3
doppler setup --project python-web3 --config dev

Ensure that you run doppler setup in the python-web3 directory as the Doppler CLI scopes secrets access to specific directories. Eventually you should have a setup similar to the one below: with your project directory coupled to the python-web3 project in Doppler.

Run the following command to check you can access secrets from the python-web3 directory:

doppler secrets

 ID NAME DESCRIPTION CREATED AT               

 python-web3 python-web3 2022-05-24T23:40:34.955Z

The next step is to store the keychain passphrase in Doppler where it can be retrieved by the doppler CLI.

Open the python-web3 Doppler Project and create a ETH_PASSPHRASE with your passphrase value, then click Save.

Heading back to the terminal, can verify Doppler saved your passphrase successfully by running:

doppler secrets get ETH_PASSPHRASE

With the passphrase managed by Doppler, we can now run the python script with secrets populated by the doppler run command.

You may inspect the accompanying python code from the cloned repository, but the key lines are those that retrieve your passphrase from the ETH_PASSPHRASE environment variable.

When you invoke the doppler run command, Doppler will set the ETH_PASSPHRASE environment variable for python to use and sign the transaction with the wallet information from the unlocked key. Although the noted address is that of the faucet you originally received your balance from, you may choose an arbitrary amount to send with your command (as long as you have sufficient funds!).

You can find a random account to send funds to from the Goerli top accounts page. Copy one of the to addresses (starting with 0x) and replace {ADDRESS} in the command below to begin the funds transfer:

doppler run -- python web3-app.py send {ADDRESS} 0.01

The transaction once complete will return a unique transaction hash like the following:

HexBytes('0x0d8c2b6924a88cda73c73275051245db7a7e698f419a9d27b8b185dda16a8436')

Your transaction has been submitted to the blockchain! At this point the running geth light node will broadcast this transaction and the transfer of funds will propagate through the network. You may use a few approaches to view the transaction details.

The provided python script supports a command to introspect an arbitrary transaction ID for example, the following command will display the transaction details of the aforementioned transaction ID:

python web3-app.py get-txn 0x0d8c2b6924a88cda73c73275051245db7a7e698f419a9d27b8b185dda16a8436

In addition to viewing details from a local client, using another method to introspect the public ledger is a good idea in order to confirm that the transaction is visible from any client.

For example, you can either use a generally-available web-based block explorer to view transactions associated with a specific address (replace the given hash with yours that is under the variable in your python repl as my_account):

https://goerli.etherscan.io/address/0x858Eb06Bd4dc5BE36Dc5025483a316E1630b35e9

Or you may alternatively view the transaction directly by entering your transaction hash into a URL like the following:

https://goerli.etherscan.io/tx/0x0d8c2b6924a88cda73c73275051245db7a7e698f419a9d27b8b185dda16a8436

In either case you should be able to see the transaction data including the amount and associated gas price.

Congratulations! You just created a transaction on an Ethereum blockchain!

Production Considerations

This guide has covered one of the most basic operations when working with a blockchain - signing and broadcasting a transaction - using Doppler to manage your passphrase securely so crypto keys and passphrases arent inadvertently exposed along with any other sensitive account credentials.

Recall that in this tutorial, we generated a keychain using the geth CLI so the associated public and private key exist only on the local machine.

To write code that could potentially run on any host with a keychain stored remotely, you could elect to store key material in Doppler and instantiate a local keychain based upon the retrieval of that information from a similar Doppler secret.

The web3 library provides the web3.geth.personal.import_raw_key method which accepts a private key and passphrase so that your transaction operations can occur on any machine with a properly setup Doppler environment. The local blockchain can be used for more experiments by virtue of the flexibility of the testnet environment, so you can continue to receive currency from a Goerli testnet faucet and send funds to other addresses.