A Little Bit of a Disaster, A Lot of Motivation: Building a CLI Secret Manager

Saif Mahmud — Thu, 21 Nov 2024 02:23:15 +0000

So, let's talk about a little mishap that turned into a major motivation. My shiny new Macbook Pro M3, fresh out of the box, decided to brick itself after a routine software update. Cue dramatic music. And with it, went all my precious data, including those pesky secrets like API keys, SSH keys, and personal shell scripts. The worst part? A critical production deployment was looming just two days away.

The Domino Effect of Lost Secrets

With my secrets gone, I had to reach out to various teams and departments to obtain new ones. This process, unfortunately, took longer than expected in the corporate world. The delays in acquiring these essential keys ultimately led to the postponement of the deployment.

Why a CLI Secret Manager?

This incident forced me to rethink my secret management strategy. Sure, I could've just zipped up the secrets and tossed them onto Google Drive, OneDrive, or S3. But let's be real, that's a huge security risk:

Unauthorized Access: If someone gains access to your cloud storage account, they could potentially get their hands on your sensitive information.
Data Breaches: Cloud storage providers, while generally secure, aren't immune to data breaches.

Password-protecting the archive is an option, but I'm too lazy to do that. Using a third-party service like LastPass is another possibility, but it requires manual updates and management; again, lazy.

There are also native applications that offer automatic syncing, but let's be honest, the free ones are either poorly maintained or have a bad user experience. And the paid ones? Pfft, please.

Even if there is one that checks out all of my requirements, as a software engineer, I couldn't resist the urge to build my own; something that I could control and customize. So, I decided to build my own CLI secret manager.

What's the Plan?

I'm building a CLI application that will allow me to:

Store: Using strong encryption and secure storage mechanisms.
Retrieve: With simple commands to access the secrets I need.
Manage: Easily add, remove, and update secrets.
Sync: Ensure consistency across multiple machines.
Share: Securely share secrets with others using asymmetric keys.

I've chosen PHP and Laravel Zero as the technology stack for this project. Why PHP, you ask? Well, it's a language I love more than my ex, who I'm still kind of hung up on, but at least I'm not hung up on Python. And Laravel is a rock-solid framework. To make the application accessible to a wider audience, I'll try to create a standalone binary using static-php-cli and box-project.

Stay Tuned for More

I'll be sharing regular updates on the development process, including challenges, solutions, and best practices. I'll also be releasing the project on GitHub once it's complete, so you can contribute or use it yourself.

So, buckle up and join me on this exciting journey as I build a powerful(?) and secure CLI secret manager.

Level Up Your Dev Workflow: Branch-Specific Data Volumes with Docker Compose

Saif Mahmud — Sun, 03 Nov 2024 20:08:15 +0000

Imagine this: you're working on a new feature and you're deep into it, only to be suddenly interrupted by a critical bug in production. You want to switch to a new branch to work on the bug fix, but your local database is still a work-in-progress with changes from your feature branch, hindering you from being able to test any changes for the bug fix.

Yikes! If only there was a way to do this without pulling your hair out 🤔

One approach to solve this problem is using branch-specific volumes in Docker. This blog post will guide you through a script that leverages Docker Compose to spin up services with branch-specific data volumes, streamlining your development process.

With branch-specific data volumes, each branch gets its own isolated data, preventing such conflicts. No more fear of breaking your main development environment or other branches!

Benefits of Using Docker Compose for Development:

Consistent Environments: Docker Compose ensures your development environment is consistent across different machines and setups.
Simplified Setup: Define and manage multiple services in a single docker-compose.yml file.
Faster Development: Quickly spin up and tear down development environments.
Improved Collaboration: Share your development environment with ease.

The Challenge with Default Docker Compose Volume Behavior

A common limitation of Docker Compose is that a single instance of a volume, defined within a Docker Compose file, exists for the entire project. This can be problematic when working with multiple branches, as each branch might require its own isolated data.

The Solution: Dynamic Volume Creation

To overcome these limitations, we'll use a script to dynamically create and manage branch-specific data volumes. Here's how it works:

Detect the Current Branch and Calculate a Hash:

   CURRENT_BRANCH=$(git symbolic-ref --short HEAD)
   CURRENT_BRANCH_HASH=$(echo -n "$CURRENT_BRANCH" | gzip -1 -c | tail -c8 | hexdump -n4 -e '"%08x"')

Define Volume Names Based on the Branch or Hash:

   if [[ "$CURRENT_BRANCH" == "master" ]]; then
       export MYSQL_VOLUME="$APP_NAME-mysql-master-data"
       export REDIS_VOLUME="$APP_NAME-redis-master-data"
       # ... other volumes
   else
       export MYSQL_VOLUME="$APP_NAME-mysql-$CURRENT_BRANCH_HASH-data"
       export REDIS_VOLUME="$APP_NAME-redis-$CURRENT_BRANCH_HASH-data"
       # ... other volumes
   fi

   # Note that the '$APP_NAME' variable has been defined earlier
   # which is basically the project name for the docker compose manifest.

Create Volumes and Copy Data from master if Necessary:

   VOLUMES=(MYSQL_VOLUME REDIS_VOLUME ...)

   for VOLUME in "${VOLUMES[@]}"; do
       if [[ -z "$(docker volume ls -q -f name="${!VOLUME}")" ]]; then
           echo "Creating volume: ${!VOLUME}..."
           docker volume create "${!VOLUME}" > /dev/null

           if [[ "$CURRENT_BRANCH" != "master" ]]; then
               MASTER_VOLUME=$(echo -n "${!VOLUME}" | sed -E "s/(.*?)-$CURRENT_BRANCH_HASH-(.*?)/\1-master-\2/g")

               if [[ -z "$(docker volume ls -q -f name="${MASTER_VOLUME}")" ]]; then
                   echo "Master volume not found: $MASTER_VOLUME"
                   docker volume rm "${!VOLUME}" > /dev/null
                   exit 1
               fi

               echo "Copying data from master volume..."
               docker run --rm -v "$MASTER_VOLUME:/from" -v "${!VOLUME}:/to" busybox sh -c "cd /from; cp -a . /to" > /dev/null
           fi
       fi
   done

Start Docker Compose: You can use the defined volume names as environment variables in your docker-compose.yml file:

   services:
       mysql:
           image: mysql:latest
           volumes:
               - mysql-data:/var/lib/mysql
           environment:
               MYSQL_ROOT_PASSWORD: root
               MYSQL_DATABASE: my_database
       redis:
           image: redis:latest
           volumes:
               - redis-data:/data
       # ... other services
   volumes:
       mysql-data:
           name: ${MYSQL_VOLUME}
           external: true
       redis-data:
           name: ${REDIS_VOLUME}
           external: true
       # ... other volumes

By dynamically creating volumes based on the branch hash, we ensure that each branch has its own isolated data, even if the branch name contains invalid characters for Docker volume names.

Level Up Your Workflow

By incorporating this script and Docker Compose into your development process, you'll enjoy a smoother, more efficient, and less stressful development experience.

For a full working example, check out this GitHub repository: https://github.com/sikhlana/laravel-template/blob/master/bin/run

Forem: Saif Mahmud