Forem: sidpalas

How to Properly Manage Application Secrets (From Beginner to Expert!) 🔐

sidpalas — Tue, 03 Nov 2020 03:44:03 +0000

Where do you fall on the scale? Are there any levels I missed?

Level -2: No authentication
Level -1: All passwords = "password"
Level 0: Hardcode everywhere
Level +1: Move secrets into a config file (and add to .gitignore)
Level +2: Encrypt config file
Level +3: Use secret manager (e.g. AWS Secrets Manager)
Level +4: Dynamic ephemeral credentials (using a tool like Hashicorp Vault)

Port Knocking (Network Security Technique) Explained and Demoed in 5 Minutes!

sidpalas — Tue, 06 Oct 2020 18:28:47 +0000

Remember the secret knock you used to have to determine if someone was allowed into your treehouse as a kid? 👊

There is a digital equivalent called "port knocking" in which a secret sequence of connection requests is used to open up firewall access. 🧱

I just released a quick (<5 min) video in which I explain how it works and provide a working demo!

How to Debug Docker Containers (Python + VSCode)

sidpalas — Mon, 20 Jul 2020 17:14:02 +0000

When containers are working they are amazing and make running your code across different environments seamless...

However, when something goes wrong, the additional layers of abstraction can make debugging them difficult.

Here are three techniques you can use to debug them:

1) Override the entrypoint at runtime and exec into the container
2) Copy files into or out of the container for inspection/comparison
3) Run a remote debugger inside the container and attach to it over the network

I created a video walking through how to use these three techniques:

Leave your own tips in the comments!

DISCUSS: What was your first web app deployment experience like?

sidpalas — Fri, 10 Jul 2020 22:53:15 +0000

Did you deploy to one of the major cloud providers such as AWS, GCP, or Azure?

Did you use a PaaS option like Heroku?

How did it go? Any lessons learned you can share?

I recently had a chance to talk with Dylan Albertazzi, a CS student at Oregon State University, about various options for deploying the web application he has been working on!

We discussed the tradeoffs of options ranging from Heroku to DigitalOcean and how he could go about scaling the application down the road.

[Video] Trying to learn Swift by solving interview prep questions!

sidpalas — Sun, 21 Jun 2020 03:30:09 +0000

I thought it would be fun to try solving an interview algorithm question in swift even though I have never used it before.

It was bit of a struggle, but I managed to get it working eventually... enjoy!

The HIDDEN COST of Machine Learning!

sidpalas — Fri, 12 Jun 2020 23:26:26 +0000

There can be hidden costs associated with developing and deploying machine learning applications that differ from traditional software systems!

It is important to consider these when deciding if ML is right for your particular use case.

Doing Stupid Stuff with GitHub Actions! (FIVE Actions Implemented and Explained)

sidpalas — Sat, 06 Jun 2020 16:58:12 +0000

I decided to have some fun and implement a variety of ridiculous GitHub Actions as a way to learn more about and highlight various features of GitHub Actions.

Here are the actions with timestamp links:

1) Holiday Reminder -- Sending myself a holiday reminder email
2) Recursive Action -- An action that calls itself
3) Exponential Action -- Like the recursive action, but the action calls itself multiple times
4) Smart Lights -- Turning on/off smart home lights with each commit
5) Tic-Tac-Toe -- A game of tic-tac-toe where the compute player plays within an action

Also, the GitHub repo can be found here: https://github.com/sidpalas/stupid-actions

Enjoy!

Using GitHub Actions as a Makeshift Uptime Monitor (Video Tutorial // 3 min)

sidpalas — Sat, 23 May 2020 16:06:11 +0000

With just 15 lines of yaml we can configure GitHub Actions as a simple uptime monitor for a website:

name: Uptime Monitor
on:
  schedule:
    # Run every 10 minutes
    - cron: '*/10 * * * *'
jobs:
  ping_site:
    runs-on: ubuntu-latest
    name: Uptime Check
    steps:
    - name: Ping Site
      uses: srt32/uptime@v0.2
      with:
        url-to-hit: "https://devopsdirective.com/"
        expected-statuses: "200"

Watch the video for the full explanation and walkthrough!

What other non-standard use cases can you think of to do within GitHub actions (or other CI tool)? Leave a comment and let me know!

Creating a Password Protected Website with IAP, App Engine, and CircleCI (Written + Video Guides)

sidpalas — Sat, 16 May 2020 18:52:20 +0000

TL;DR: If you have a static website that needs to be password protected, using Google's Identity Aware Proxy along with App Engine is one of the simplest ways to accomplish this.

If you want to follow along with a site of your own, I have provided a working example in this GitHub repo.

Note: I have also seen S3 + Lambda used to accomplish this

Table of Contents:

The Need
- Other static hosting solutions
- Handling passwords
Solution (IAP + App engine)
Example Site
- Configuring App Engine
- Setting up IAP
- Configuring CI/CD with CircleCI
  - Workflow
  - 1) checkout
  - 2) Install Sphinx
  - 3) Make Docs
  - 4) Deploy Docs
- Next steps
  - Custom Domain
  - Custom OAuth screen

The Need

A company I have been working with (Gauntlet Networks) needed to set up a website containing documentation for the simulation SDK they are building, but wanted to restrict access to only their clients and employees.

Handling passwords

Historically, implementing authentication would require adding significant complexity to our website's architecture. We would need a database of some kind to store the (hashed) passwords and server-side processing to check if a login attempt is valid. We would also probably need to add some form of email confirmation/password reset functionality because people are terrible at remembering their passwords. All of the sudden the one hour task of getting the documentation site set up just ballooned into something that could take weeks to accomplish!

Luckily, we don't actually need to take on all of this complexity ourselves. A standard called OAuth allows us to leverage the authentication system of another entity (for example, Google) to provide "secure delegated access" to our content.

Solution (IAP + App engine)

OAuth by itself would allow us to set up a sign-in flow using an external authentication provider, but for this use case, we want the process to be completely hands off.

This is where Google's Identity Aware Proxy (IAP) comes in. Cloud IAP uses Google Sign-In and GCP's Identity and Access Management (IAM) to handle authentication and authorization to control access to GCP resources. Granting access to the site can then be managed using individual Google accounts and/or Google groups.

Given that Gauntlet was already using Google Cloud Platform, leveraging IAP in conjunction with Google App Engine to host the static site provided a simple, clean solution to satisfy the need.

Example Site

Because a documentation website was the use case that inspired this post, I am using the Sphinx Python documentation generator quick start as the basis for this demo. The static site files will be hosted using App Engine, with CircleCI for CI/CD.

Configuring App Engine

Google has documentation outlining how to use app engine to create a static site by adding an app.yaml file to the codebase that tells App Engine how to interpret URL request paths. Sphinx stores its output in the /docs/build/html directory by default, so I used the following app.yaml:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /
  static_files: docs/build/html/index.html
  upload: docs/build/html/index.html

- url: /(.*)
  static_files: docs/build/html/\1
  upload: docs/build/html/(.*)

With this in place, a running the gcloud app deploy command within the root of the project deploys the application.

Setting up IAP

With the website deployed, the next step was to follow Google's guide to enabling IAP for App Engine.

This involves configuring a few things for the Oauth Consent page (app name, support email, etc...), but the process is relatively painless.

The final step is to grant the IAP-secured Web App User access to all authorized individuals. In the screenshot below, I granted access to allAuthenticatedUsers which allows anyone with a Google account to access it. (Test it out here!)

Configuring CI/CD with CircleCI

At this point, the website is live and access controlled, but in order to minimize future work, it is useful to set up a system to handle continuous integration and deployment.

CircleCi offers simple integration with GitHub and a free tier that will easily handle a small project like this. The first step is to create a CircleCI account and then grant access to the GitHub repo containing the source files using their GitHub marketplace app.

Workflow

To define our workflow, we create a file at .circleci/config.yml containing two parts, a workflow definition and a job definition.

We define the workflow using version 2 of the CircleCI API. This workflow tells CircleCi to run the build_and_deploy job on any push to the master branch.

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build_and_deploy:
        filters:
          branches:
            only:
              - master

Next, we define the build_and_deploy job. Each CircleCI job runs within a Docker image, and because the goal is to deploy to Google App Engine, the google/cloud-sdk provides a good starting point.

version: 2.1
jobs:
  build_and_deploy:     
    docker:
        - image: google/cloud-sdk:slim
    steps:

The job is broken down into four steps:
1) checkout,
2) Sphinx installation,
3) Generating the website
4) Deploying to App Engine

1) checkout

      - checkout

As its name suggests, this step retrieves the source code within the repo and stores it in the working directory.

2) Install Sphinx

      - run:
          name: Install Sphinx
          command: |
            apt install -y python-pip && \
            python3 -m pip install -r requirements.txt

The google/cloud-sdk:slim container image we are using is based on debian, so we can install Sphinx and its dependencies by first installing pip with apt and then using pip to install the requirements.txt file.

3) Make Docs

      - run:
          name: Make Docs
          command: |
            cd docs && make html

The Sphinx quickstart creates a Makefile within the /docs directory which can be used to generate the documentation site. This step changes directory into /docs and executes this make target passing html as the destination for Sphinx to store its output.

Note: html here needs to match the directory used in the App Engine app.yaml config (make html will build the site into docs/build/html where App Engine is configured to find them)

4) Deploy Docs

      - run: 
          name: Deploy Docs
          command: |
            echo ${GCLOUD_SERVICE_KEY} > /tmp/sa_key.json
            gcloud auth activate-service-account --key-file=/tmp/sa_key.json
            rm /tmp/sa_key.json
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
            gcloud --quiet app deploy

The final step in the job is to deploy the site. In order to do this we first need to create a service account (https://console.cloud.google.com/iam-admin/serviceaccounts) and grant it the necessary permissions. Although it seems as though App Engine Deployer would be sufficient, it turns out there are actually a few additional permissions involved with deploying the site. The screenshot below shows the 5 roles I needed to add to achieve a successful deploy.

At this point, I created a key for the service account (.json format) and stored its content as an environment variable in CircleCI. I also stored the Zone and Project ID as environment variables to ensure the deployment was targeting the correct app engine environment.

There was one final "gotcha" having to do with the service account key. My first attempt tried to pass it into the gcloud auth command by piping it in as follows:

echo ${GCLOUD_SERVICE_KEY} | gcloud auth activate-service-account --key-file=-

This caused gcloud to assume the key was a .p12 type and fail.

ERROR: (gcloud.auth.activate-service-account) Missing required argument [ACCOUNT]: An account is required when using .p12 keys

As shown in the full example at the beginning of the step, I was able to solve this by redirecting the echo output to a temporary file containing the json key and pass the filepath to gcloud auth.

That's it! We now have a password protected website, with CI/CD, that falls within the free usage tiers of GCP + CircleCI. If this site had heavy traffic or data egress it could exceed the free tier, but for this internal documentation use case it that is unlikely.

Next steps

There are a couple of additional steps that I will leave as an exercise to the reader.

Custom Domain

By default, App Engine assigns a domain such as https://devops-directive-project.uc.r.appspot.com/ to the website. It is possible to map a custom domain to the site by following this guide from Google.

Custom OAuth screen

The default Oauth consent screen is fairly boring, but can be customized with logo image, application homepage link, etc... from Google Cloud Console

[VIDEO] In-Depth Caddy Webserver Set-Up Walkthrough (localhost → Google Cloud)

sidpalas — Thu, 14 May 2020 03:58:10 +0000

This is a long video, but in 60 minutes it covers a full progression from running a webserver locally all the way to deploying it on Google Cloud Platform!

You will learn about webservers, cloud-based virtual machines, Docker, and more!

I have provided timestamps and some descriptions if you want to find specific portions of the video.

If you find the video useful, a like/subscription is always appreciated!

Timestamps:
1:30 - Workshop start
2:06 - Load file in Chrome

Manually loading a .html file from a web browser will display it, but doesn't allow for anything beyond a single page.

3:23 - Run Python development web server (locally)

A single line of python can be used to run a local webserver:

python3 -m http.server --cgi 8000

4:47 - Access site via internal IP address via another device on the same network

By looking up the local IP address of the computer running the local server (using ifconfig | grep 192) we can connect from another device on the same network.

6:44 - Install Caddy (locally)

Caddy is a webserver that handles https setup automatically!

7:03 - Run Caddy webserver (locally)
8:23 - Configure Caddy using Caddyfile

A Caddyfile is a how we define the configuration for Caddy

12:05 - Set up GCP Compute Engine Virtual Machine with Debian Linux

This effectively gives us a server running in a Google datacenter!
Running a single f1-micro VM falls within the google free tier so it shouldn't cost anything!

13:23 - Connect to VM via SSH
14:31 - Install Caddy (on VM)
16:04 - Run Caddy webserver (on VM)
17:11 - Configure firewall rule for port 80

By default, GCP does not allow access to the virtual machine, so in order to connect, we have to configure a firewall rule.

18:46 - Explaining why we want to use a container

By using Docker, we can ensure that the site will run the same on the Debian virtual machine as on your local system (whichever OS it happens to be running)

22:00 - Install Docker (locally)
23:18 - Run Python Docker image
24:23 - Docker exec into the container

Docker exec ... bash provides a shell session within the container. This is similar to SSH'ing into a remote machine.

26:00 - Run Python development web server (locally, inside container)
27:00 - Set up port forwarding to Python container

By default, the ports of the container are isolated from those of the host system, so they must be explicitly forwarded in order to access.

28:26 - Run Python development web server Take 2
29:27 - Find Caddy container image on DockerHub

DockerHub is like GitHub, but for container images.

30:35 - Run Caddy webserver (locally, inside container)
33:00 - Define Dockerfile

We can make modifications to and build on top of the public image we are using!

34:39 - Build custom Caddy container image
35:34 - Run custom Caddy webserver (locally, inside container)
36:05 - Modify and rebuild custom Caddy container
37:27 - Configure Docker to mount host filesystem

Because the website files are stored outside of the container, configuring Docker to mount the proper directory allows Caddy to serve those files.

39:58 - Run Caddy web server with bind mounts (locally, inside container)

At this point we have everything working how we need it to and are ready to deploy it to the GCP virtual machine!

42:25 - Set up GCP Compute Engine Virtual Machine with Container Optimized OS
45:36 - Copy files to VM using gcloud scp
47:54 - Run Caddy web server with bind mounts (on VM, inside container)
50:22 - Configure domain DNS settings
52:22 - Configure Caddyfile for HTTPS

These days HTTPS should be used for all sites. Google even reduces its search rank of sites that do not!

54:29 - Run Caddy web server with HTTPS (on VM, inside container)
55:36 - Accept Let's Encrypt terms of service
56:37 - Use environment variable to avoid TOS dialog
57:23 - Explaining why we also need to add a bind mount to save HTTPS certificate outside the container

This is necessary because we want to persist the HTTPS certificate even if the container gets restarted. By mounting the host filesystem we can store that file outside the boundaries of the container.

Load Testing Caddy Web Server on a GCP F1-Micro Instance Using K6 (k6.io)

sidpalas — Thu, 14 May 2020 01:50:48 +0000

TL;DR: I used the K6 load testing framework to benchmark the Compute Engine f1-micro and Caddy web server hosting devopsdirective.com. With CloudFlare caching turned off, the server was able to serve an onslaught 800 virtual users continuously reloading the page (while maintaining a median request duration of <400ms), but started dropping requests when increasing the load further.

Originally published @ DevOps Directive

Background

DevOps Directive is a static website generated with Hugo and hosted using Caddy running on an f1-micro GCP Compute Engine instance with Cloudflare in front of it (see The Making of This Site post for details). On a normal day, the site used to get between 1 (thanks Dad!) and 20 visitors, but recently, two articles made it to the front page of Hacker News Link-1 and Link-2 bringing outsized swells in traffic.

In a single hour on March 7th, a total of 1307 people visited this site. Thankfully, even with all of that traffic, CPU usage of the virtual machine never even reached 10% (and the short spikes correspond to redeploying the site with copy edits).

At this point, the setup has proven itself capable of embracing a Hacker News hug without dying, but I wanted to get a sense of what kind of load it can actually handle.

Testing

Site Setup

In order to avoid causing any impact to my actual site, I spun up an identical replica on a separate virtual machine using this script and configured the https://test.devopsdirective.com/ subdomain (which will likely be inactive at the time you are reading this) to resolve to it.

Here is a summary of the configuration:

Compute Engine f1-micro Instance (0.2 vCPU burstable to 1 vCPU for short periods, 0.6GB Memory) running Container Optimized OS (COS)
Caddy (1.0.3) container image with the site content files built directly into the container
Cloudflare configured to proxy traffic and set to the "standard" caching level (I performed tests with caching turned on and caching turned off)

NOTE: I didn't tune/configure the COS image running on the VM, nor specify resource requests in the docker run command.

K6 Performance Testing Framework

To perform the load test I used k6.io, an open source performance testing framework designed for building automated tests with a straightforward javascript config file. It uses the concept of "virtual users" (VUs) which in their words are "glorified, parallel while(true) loops" to load test a site.

Replicating Current Peak

First, I created a script to approximate the load that the two HN posts brought. The peak hour had 1443 page views, or 0.4 pageviews/second. To account for the load not being constant across the entire hour, I rounded this up to 1 pageview/second.

K6 is able use a HAR file to create a representative set of HTTP requests. I used 1 virtual user and adjusted the pause between iterations to achieve just over 1 pageload/second (with a "pageload" corresponding to the batch of HTTP requests). I excluded external requests for things like the Google Analytics script. The full K6 configuration script can be found as a GitHub gist and the resulting output can be seen below:

check_failure_rate.........: 0.00%   ✓ 0   ✗ 138
checks.....................: 100.00% ✓ 414 ✗ 0  
data_received..............: 66 MB   549 kB/s
data_sent..................: 219 kB  1.8 kB/s
group_duration.............: avg=873.64ms min=850.89ms med=864.69ms max=1.13s    p(90)=891.35ms p(95)=908.24ms
http_req_blocked...........: avg=81.77µs  min=156ns    med=446ns    max=133.52ms p(90)=861ns    p(95)=1.42µs  
http_req_connecting........: avg=7.06µs   min=0s       med=0s       max=11.69ms  p(90)=0s       p(95)=0s      
http_req_duration..........: avg=26.89ms  min=12.6ms   med=23.11ms  max=213.29ms p(90)=38.19ms  p(95)=47.38ms 
http_req_receiving.........: avg=726.31µs min=62.01µs  med=242.75µs max=29.96ms  p(90)=1.84ms   p(95)=2.41ms  
http_req_sending...........: avg=52.23µs  min=11.27µs  med=38.27µs  max=4.07ms   p(90)=106.59µs p(95)=115.18µs
http_req_tls_handshaking...: avg=72.85µs  min=0s       med=0s       max=120.64ms p(90)=0s       p(95)=0s      
http_req_waiting...........: avg=26.11ms  min=12.37ms  med=22.51ms  max=212.53ms p(90)=37.05ms  p(95)=45.32ms 
http_reqs..................: 1656    13.799982/s
iteration_duration.........: avg=873.71ms min=850.96ms med=864.75ms max=1.13s    p(90)=891.46ms p(95)=908.31ms
iterations.................: 137     1.141665/s
vus........................: 1       min=1 max=1
vus_max....................: 1       min=1 max=1

Key Takeaways

All requests returned successfully (Status: 200 OK)
The request duration ranged from 13ms to 213ms
The 95th percentile duration was 47ms

Ramping It Up!

With that test as a baseline, I proceeded to run a series of tests, each 60 seconds long, starting with 6 virtual users and increasing the number of VUs with each test. I also reduced the delay between virtual user iterations to 100ms. The most important metric is http_req_duration, which represents is the total request time (http_req_sending + http_req_waiting + http_req_receiving), which I have plotted below for the full set of tests.

Unsuprisingly... a CDN with caching makes a big difference

Up until around 50 VUs, the response time remains flat, with an uncached median of 68ms and a cached median of 31ms.

After 50 VUs, the response times begin to climb in a linear fashion. At 800 VUs the uncached median was 349ms and the cached median was 67ms. As would be expected at these higher loads, most (90+%) of the http_req_duration is spent in the http_req_waiting stage.

The uncached configuration finally gave out during the 1600 virtual user test, with only 414 successful responses, indicating that ~74% of the virtual users never received a response.

Virtual Users and Server Load

It is important to note that while the virtual users run in parallel with each other, they run in serial with themselves. Each individual VU waits until its current pageload is complete before making a new set of requests. As the server slows down under load, this causes the total rate of requests to drop in the more demanding tests. The impact is clearly seen in the total amount of data received during the tests plotted below.

Data received (and pageloads/s) peaks before the more demanding tests

These were the two most informative plots, but all of the data and code to generate plots can be found in a notebook in this GitHub repo. You can load an interactive copy of the notebook using the following binder link:

Link to interactive notebook

Snags Along the Way

I did run into some technical limitations when configuring and executing these tests. Here are the main issues and how I overcame them:

1) Bandwidth Limitations: My home internet was not sufficient to support the load test. Moving to a GCP virtual machine with sufficient bandwidth (Measured @ 900+ Mbps) as the test client running K6 solved this. For the later tests in the cached configuration, this actually still became a limiting factor.

2) Memory Limitations: After moving from my laptop to an n1-standard-1 instance as the testing client, the more demanding tests caused K6 to run out of memory (fatal error: runtime: out of memory). Moving to an n1-standard-8 (30GB memory) solved this.

3) Unix Resource Limits: Because each request group makes multiple HTTP requests, the final test with 1600 target virtual users surpasses the default maximum number of open files allowed by the OS for a single process to manage at once. Using multiple test client VMs in parallel solved this, but increasing the open file limit with ulimit -n <NEW_LARGER_LIMIT> is the approach I ended up using.

(Aside) Total Costs

The total cost to run this experiment was $2.82:

$1.60 for 40.1 GB of network egress
$0.50 for running the f1-micro server for a ~3 days
$0.72 for running the n1-standard-8 testing client for a ~2 hrs

Conclusions

I am continuously amazed at the level of load that even such a tiny virtual machine can handle when serving static content!

Utilizing a service like Cloudflare to help cache and serve content reduces the load on the server significantly. It cut the response times in half under light load and prevented the server from being overwhelmed under heavy load.

I would have liked to record realtime resource (CPU + Memory usage) on the server VM but the GCP cloud monitoring agent isn't compatible with Container Optimized OS, so I settled for the rough 1 min averaged view in the GCP console:

Now we're cooking with gas! (bursting above the 0.2 vCPU limit for a short period)

This test gives me confidence that my current server configuration should be able to handle quite a bit of growth before needing any major overhaul.

In the future, I hope to do similar benchmarking across other hosting options. If someone has a contact at @github or @netlify that could grant me permission to run a test against a Github Pages or Netlify Starter site let me know! Or maybe at @bluehost so I can benchmark some Wordpress installs...