Forem: ✨iMichael✨

Introducing PipFlow

✨iMichael✨ — Sat, 11 Jan 2020 19:37:14 +0000

I recently published a new Python package manager called PipFlow. If you're like me and use Docker for everything (local development + containers in production), you might like it.

Let's take a step back and dive into what problem it solves and at the end I'll tell you why the newer package managers were not a good fit for me. Here are my priors:

I believe in making my local development mirror production NOT make production mirror my local development.
Therefore I do everything in Docker (and k8s)
In Docker, the slimmer the image the better. Also if you can avoid redundant dependencies and build steps, you should.

With that out of the way, let's add some new packages. Requests is cool –– let's install it the old way.

pip install requests

I now have this package in my host operating system, but it's useless there because I use docker locally. So let's get the package and add it to my requirements.txt file.

pip freeze | grep requests >> requirements.txt

My requirements now has requests==2.22.0 in it. Okay cool, but now we need to rebuild our docker image to bake in the new dependency

docker-compose build app

Everything works great now, but let's take a moment to identify some waste: I just installed a package twice to use it. There must be a better way, right? There is:

pipflow add requests

The command above gets the latest requests version and adds it to our requirements file, sorts the file, and rebuilds the docker image – which is the only installation step that matters. The great thing is that we don't need to install pipflow in our Dockerfile, just once in our host operating system.

Pipflow is really a "pip workflow" that just simplifies a few steps into one step. Inspired by the yarn command line API, pipflow also does upgrades for single packages pipflow upgrade <package> and removes packages with pipflow remove <package>. You can also upgrade all packages with pipflow upgrade-all.

So why don't I use one of these newer tools like Poetry or Pipenv? The answer is that these tools solve problems that don't exist in Docker.

Lets read the pipenv README:

You no longer need to use pip and virtualenv separately. They work together

Virtual Environments are not necessary in Docker – Docker already has isolation and sandboxing. Next time you do ADD venv . in Docker, ask yourself what problem you're solving.

What about Poetry and other tools that use lock files, those are useful aren't they?

Nothing is gained by using a secondary lock file in Docker. Requirements txt files are perfectly fine at pinning versions (requirements.txt is your manifest + lockfile already). In fact there is a loss: You have to generate the lock file AND/OR you have to install the tool in your Docker image. Both are wasteful, redundant build steps.

Another thing to remember is that when you do RUN pip install -r requirements.txt in Docker, a new image layer is created – this layer represents all the new packages added to the file system and each layer gets a unique sha256 digest (in effect a lock file mechanism). If you build again and nothing changed in your requirements, the layer is cached. It's a beautiful thing.

Here's what my Dockerfile looks like in a production project that uses pipflow:

FROM python:3.8.0-alpine

EXPOSE 8000

WORKDIR /app

RUN addgroup -S app && adduser -S -G app app

ADD requirements.txt .

RUN pip install -r requirements.txt

ADD . .

USER app

ENTRYPOINT ["scripts/entrypoint.sh"]

You can find the repo on Github and the PyPi package here or just install it with pip install pipflow. The only assumptions are: your Dockerfile/image has pip installed and you have these two lines in your Dockerfile (ideally before you add the rest of the source code for optimal caching).

ADD requirements.txt .

RUN pip install -r requirements.txt

Let me know if you love it, hate it, or ¯\_(ツ)_/¯. It's a young project, so expect a few kinks 😃.

Continuously Deploy Your Frontend using Gitlab and Scotty.JS

✨iMichael✨ — Sun, 05 Aug 2018 07:43:30 +0000

My lovefest for Gitlab continues! Heres a quick post showing you how to continuously deploy your React/Vue/Angular Single Page Applications to S3 and Cloudfront, using just Docker, Gitlab and great tool called Scotty.JS.

First save AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to your Gitlab environment variables (I recommend a special IAM role for this). Then save the files below to your repository and update the placeholder text to what makes sense for your project.

The Dockerfile below performs a multi-stage build that builds the frontend code (Create-React-App) then copies all of it to a directory that Scotty can access.

#Dockerfile
FROM node:8.11.3-slim as node_builder

MAINTAINER @iMichael

WORKDIR /app

COPY /src  /app/src

RUN cd /app/src && \
    npm install && \
    npm run build

FROM node:8.11.3-slim as deployer

RUN groupadd -r react && useradd -r -g react react

RUN npm install scottyjs --global

COPY --from=node_builder /app/src/build /tmp/build

USER react

ENTRYPOINT ["scotty"]

And below is the Gitlab config that ties it all together. Update this with any special logic your team adheres to (like only deploying if all tests pass on the master branch).

#.gitlab-ci.yml
image: docker:latest

services:
  - docker:dind

stages:
- build
- push
- deploy

variables:
  REACT_TAG_NAME: registry.gitlab.com/<user>/<project>/<container-name>:$CI_COMMIT_REF_NAME
  S3_BUCKET: xxxx

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com

build:react:
  artifacts:
    paths:
      - docker-images
  stage: build
  script:
    - mkdir docker-images
    - docker build --pull -t $REACT_TAG_NAME -f path/to/your/Dockerfile .
    - docker save $REACT_TAG_NAME > docker-images/react.tar

push:react:
  stage: push
  script:
    - docker load -i docker-images/react.tar
    - docker push $REACT_TAG_NAME
  only:
    - master

deploy:react:
  stage: deploy
  script:
    - docker run \
      --rm  \
      -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
      -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY  \
      $REACT_TAG_NAME  \
      --source /tmp/build/ \
      --region us-east-1 \
      --bucket $S3_BUCKET \
      --spa

Now every time you push to master on Gitlab (and all tests pass). your code will be deployed to AWS with a Cloudfront URL that you can CNAME to your own domain.

Big thanks to Scotty.js which does all the heavy lifting!

A framework for choosing from multiple job offers.

✨iMichael✨ — Fri, 03 Aug 2018 08:20:52 +0000

"Which job offer should I accept" is a question I see pretty frequently in various Slack groups and hear IRL. The answer of course is always personal and subjective so I thought it would be helpful to write software that helps organize our thoughts in a structured way and eliminate the Decision Fatigue associated with multiple good options.

I've published a very alpha quality version in Python-3.7 @ https://github.com/iMerica/job-offer-selector. Feel free to fork it or use it as inspiration.

The core idea here is:

Rank the categories that are most important to you.
Rank the companies in these categories.
Select the company that performs best using a weighted average.

P.S. Python 3.7's new data class module is pretty awesome. This project relies upon them pretty heavily.

Automagically build and push Docker images to a registry using Gitlab.

✨iMichael✨ — Fri, 03 Aug 2018 07:57:37 +0000

This is an example configuration for how to automatically build Docker images and push to your own Docker registry on Gitlab (free). In this example, I'm building and testing a Docker image that runs Django, but the same basic idea will work for any app.

image: docker:latest

services:
  - docker:dind

stages:
- build
- test
- push

variables:
  DJANGO_TAG_NAME: registry.gitlab.com/<username>/<repo>/<image>:$CI_COMMIT_REF_NAME
  DJANGO_SETTINGS_MODULE: path.to.your.settings
  DATABASE_URL: postgres://postgres:xxxxxxx@postgres/postgres

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com

build:django:
  artifacts:
    paths:
      - docker-images
  stage: build
  script:
    - mkdir docker-images
    - docker build --pull -t $DJANGO_TAG_NAME -f path/to/your/Dockerfile .
    - docker save $DJANGO_TAG_NAME > docker-images/app.tar
  only:
    - master
test:django:
  stage: test
  script:
    - docker load -i docker-images/app.tar
    - docker run --name test-postgres -e POSTGRES_PASSWORD=xxxxxxx -d postgres
    - docker run --rm --link test-postgres:postgres -v $(pwd):/<DOCKER WORKDIR> -e DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE -e DATABASE_URL=$DATABASE_URL $DJANGO_TAG_NAME python manage.py test -v 2

push:django:
  stage: push
  script:
    - docker load -i docker-images/app.tar
    - docker push $DJANGO_TAG_NAME
  only:
    - master

This one trick gives you unlimited CI minutes on Gitlab.

✨iMichael✨ — Fri, 03 Aug 2018 07:49:04 +0000

Gitlab graciously provides 2,000 free minutes per month to all users on Gitlab.com (thanks Gitlab!). I deploy many times a day so it's quite easy to use all 2K minutes in a matter of days.

Here is how to get around that:

Spin up a new compute instance with Docker pre-installed. I recommend Digital Ocean, but Vultr is pretty good too.
Replace the CI_SERVER_URL value with one from your actual server URL. Go to the CI settings for your project and get your registration token under the "Runners" section. Then update REGISTRATION_TOKEN in the yml file.
Ssh into your server and save the code below to a file called docker-compose.yml.
Run docker-compose up -d

version: '2'
services:
  dind:
    restart: always
    privileged: true
    volumes:
    - /var/lib/docker
    image: docker:18.06.0-ce-dind
    command:
    - --storage-driver=overlay2

  runner:
    restart: always
    image: gitlab/gitlab-runner:alpine
    volumes:
    - ./gitlab/runner:/etc/gitlab-runner:Z
    environment:
    - DOCKER_HOST=tcp://dind:2375

  register-runner:
    restart: 'no'
    image: gitlab/gitlab-runner:alpine
    volumes:
    - ./gitlab/runner:/etc/gitlab-runner:Z
    command:
    - register
    - --non-interactived
    - --locked=false
    - --name=Docker Runner
    - --executor=docker
    - --docker-image=docker:18.06.0-ce-dind
    - --docker-volumes=/var/run/docker.sock:/var/run/docker.sock
    environment:
    - CI_SERVER_URL=https://XXXXXXXX.XXXXX
    - REGISTRATION_TOKEN=XXXXXXX