Forem: Vitaliy Yanchuk

Private CI using private docker registry with Drone.io (almost free)

Vitaliy Yanchuk — Mon, 05 Aug 2019 21:48:57 +0000

In previous post I have described how you can set up your own private docker registry for your secret projects, now we want to add CI to this setup.

Currently I host both on single 5$ VPS, which is decent price, considering amount I would need to pay DockerHub for same number of private docker images and for private TravisCI which costs 69$/month for 1 Concurrent job, which they advertise as "IDEAL FOR HOBBY PROJECTS", well not for 69$ for sure.

Prerequisites

Consider you have a domain example.com, and want to have CI on drone.example.com and private registry on registry.example.com

Setting up Drone

Drone requires that you use some external authentication provider, which is actually convenient, I would use GitHub as authentication provider. Also this way drone will import my private repositories on GitHub.

You should check official docs on how to get required credentials if you want to use GitHub as well:
https://docs.drone.io/installation/github/single-machine/

The official doc shows only example how to start Drone with docker run, but I prefer to put it in the docker-compose.yml file, so it is easier to start/restart.

So the combined docker-compose.yml, including Drone and private docker registry would look like:

version: "3"
services:
  drone:
    image: drone/drone
    ports:
      - "1080:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/drone:/data
    environment:
      - DRONE_USER_FILTER=your_user
      - DRONE_GITHUB_SERVER=https://github.com
      - DRONE_GITHUB_CLIENT_ID=<your id>
      - DRONE_GITHUB_CLIENT_SECRET=<your secret>
      - DRONE_RUNNER_CAPACITY=2
      - DRONE_SERVER_HOST=drone.example.com
      - DRONE_SERVER_PROTO=http
      - DRONE_TLS_AUTOCERT=true
  registry:
    restart: always
    image: 'registry:2'
    ports:
      - "5000:5000"
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
    volumes:
      - /root/docker-volumes/registry/registry:/var/lib/registry
      - /root/docker-volumes/auth:/auth

Note important environment variable:
DRONE_USER_FILTER - only this user will be able to use your drone, you should limit it to your user so no other person can use your Drone.
DRONE_SERVER_HOST - the domain you have used to create OAuth authentication

Running specs

In order to make Drone run specs for your repo you need to add special file to your project .drone.yml

kind: pipeline
name: default

steps:
- name: test
  image: registry.example.com/rails_api
  commands:
  - bundle exec rspec
  environment:
    RAILS_ENV: 'test'
    DATABASE_CLEANER_ALLOW_REMOTE_DATABASE_URL: 'true'
    REDIS_URL: redis://redis:6379/
    DATABASE_URL: postgres://postgres:@db/ourcodestyle_test
services:
- name: db
  image: 'postgres:11.1-alpine'
  ports:
  - 5432
  environment:
    POSTGRES_USER: postgres
    POSTGRES_DB: rails_api_test
- name: redis
  image: 'redis:3.2-alpine'
  ports:
  - 6379

image_pull_secrets:
- dockerconfigjson

There is several interesting parts in this file:

image: registry.example.com/rails_api
This way you specify which private docker image you would want your build to start. I have this image with all ruby gems already installed, so it dramatically reduces time before tests start to run.

Also note services section. Drone configuration mimics docker compose syntax so it is easy to configure.

The most important config line is the last one:

image_pull_secrets:
- dockerconfigjson

And it is not well described in the official docs. This setting says how Drone should get credentials for you private repository. You need to have a file /root/.docker/config.json on the machine where you run Drone. I did not have this file in place. You can create this file by yourself, or there is much easier way, just run:

docker login registry.example.com -u testuser

on the machine which would run Drone and it would create that file.

Domain for Drone

In order to share same server, your drone should have its subdomain, and also it is convenient and nice.

It can be done by adding nginx configuration like this:

upstream drone {
  server 127.0.0.1:1080;
}

server {
    server_name drone.example.com;

    access_log /var/log/nginx/drone.example.com.access.log combined;
    error_log  /var/log/nginx/drone.example.com.error.log crit;

    server_name_in_redirect off;

    location / {
      proxy_pass                          http://drone;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
    }

    listen 80;
}

Then don't forget to run CertBot in order to get https

certbot --nginx -d drone.example.com

How to setup private docker registry for your projects to save money

Vitaliy Yanchuk — Wed, 17 Jul 2019 16:03:54 +0000

I got to use docker for my projects which are not open-sourced, some of them are hobby projects or try-outs which I don't want yet to publish, so I need to store them privately.

DockerHub only allows 1 private image to be stored in the Free plan, 5 images for 7$ and 10 for 12$ which is decent price for production projects, but for personal hobby projects may be a lot.

I have a 5$ server on DigitalOcean for personal needs, which I want also to use as a docker registry, and CI for my private projects.

The steps in this guide are not the only one possible, there are many ways to achieve same result, but I tried to make it as quick and as simple as possible, I will describe each configuration why it is required.

Prerequisites

I am using Ubuntu 18.04, so all commands will be related to this system.
You need to have Docker and Docker Compose installed on your host Official docs:
https://docs.docker.com/install/linux/docker-ce/ubuntu/
You need to have an installed nginx.
Examples assume you have a domain: example.com, and that you want to start registry on: registry.example.com

Step 1: Setup nginx

I don't use nginx as docker container as I have also some other static sites on same machine, which I don't want to wrap into containers. And also I find it easier to have nginx outside containers and on the host machine.

ssh to host machine and install nginx, if you don't have it yet

apt install nginx

Then create nginx configuration for your domain:

/etc/nginx/sites-enabled/registry.example.com

upstream docker-registry {
  server 127.0.0.1:5000;
}

server {
    server_name registry.example.com;

    server_name_in_redirect off;

    # disable any limits to avoid HTTP 413 for large image uploads
    client_max_body_size 0;

    # required to avoid HTTP 411: see Issue #1486 (https://github.com/moby/moby/issues/1486)
    chunked_transfer_encoding on;

    location /v2/ {
      # Do not allow connections from docker 1.5 and earlier
      # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }

      proxy_pass                          http://docker-registry;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
    }
}

Because we would use BasicAuth to authenticate to registry we need to setup SSL so that our login/password is not taken by man in the middle attack.
Currently this nginx config does not support SSL which is required to make our registry accessible from outside.
We would use CertBot to update our nginx configuration to support SSL.

Step 2: CertBot

Install CertBot as described here: https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx

Now you can run command:

certbot --nginx -d registry.example.com

It would modify /etc/nginx/sites-enabled/registry.example.com so it supports SSL.

Step 3: Docker Registry

Official doc is here: https://docs.docker.com/registry/deploying/
The official doc is quite long, and describes lots of options for different use cases

Just connect via ssh to your host, go to /root folder.

mkdir docker-volumes
mkdir docker-volumes/registry/
mkdir docker-volumes/registry/registry
mkdir docker-volumes/registry/auth

Create there such docker-compose.yml file:

version: "3"
services:
  registry:
    restart: always
    image: 'registry:2'
    ports:
      - "5000:5000"
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
    volumes:
      - /root/docker-volumes/registry/registry:/var/lib/registry
      - /root/docker-volumes/registry/auth:/auth

Step 4: Authentication

Then we need to restrict access to registry by login/password, we would use BasicAuth. To do so we need to generate htpasswd file which we would put into registry container. Change testuser/testpassword to your own.

docker run --entrypoint htpasswd registry:2 -Bbn testuser testpassword > ./docker-volumes/registry/auth/htpasswd

Step 5: Starting it all up

Now we can start registry, run:

docker-compose up -d

Also restart nginx so it loads new config for registry.example.com


/etc/init.d/nginx restart

Step 6: Verify

In order to check that we configured everything correctly, go to console on your local machine, which you would use to build images (your laptop probably), and try to login:

docker login registry.example.com

Use credentials from Step 4

Now it should be all up. In upcoming posts I will write how to setup your private CI using Drone.io so it can use your private registry in its builds.

How to DRY Apollo GraphQL with QueryComponent

Vitaliy Yanchuk — Tue, 02 Jul 2019 20:19:56 +0000

An official Apollo React guide illustrates an example on how to load a query and show data this way:

import gql from "graphql-tag";
import { Query } from "react-apollo";

const GET_DOGS = gql`
  {
    dogs {
      id
      breed
    }
  }
`;

const Dogs = () => (
  <Query query={GET_DOGS}>
    {({ loading, error, data }) => {
      if (loading) return "Loading...";
      if (error) return `Error! ${error.message}`;

      return (
        <select name="dog" onChange={onDogSelected}>
          {data.dogs.map(dog => (
            <option key={dog.id} value={dog.breed}>
              {dog.breed}
            </option>
          ))}
        </select>
      );
    }}
  </Query>
);

This is nice for the start, but I don't want to copy over all the handling of errors each time in each usage of Query, also I want to write more interesting loader that just text, so having this duplicated code in all the places seems as boilerplate.

So I DRY it with a QueryComponent, so the above example would become:

class Dogs extends QueryComponent {
  query(){
    return `
      {
        dogs {
          id
          breed
        }
      }    
    `
  }

  content(){
    const data = this.state.data
    return (
      <select name="dog">
        {data.dogs.map(dog => (
          <option key={dog.id} value={dog.breed}>
            {dog.breed}
          </option>
        ))}
      </select>
    );    
  }
}

Now I can configure nice loader, and have it same across the site, and make handling errors centralized, and I don't need to import gql every time.

Code for QueryCompoment would be such:

import React from "react";
import PropTypes from "prop-types";
import gql from "graphql-tag";

import Loader from "./Loader";

class QueryComponent extends React.Component {
  constructor(props) {
    super(props);
    this.state = { loading: true, error: null };
  }

  componentDidMount() {
    this.loadQuery({ variables: this.queryVariables() });
  }

  query() {
    throw "query() should be redefined on extended component";
  }

  queryVariables() {
    return {};
  }

  loadQuery({ variables }) {
    const client = this.context.client;
    const query = gql(this.query());

    try {
      client
        .watchQuery({ query, variables }) // performs query as well
        .subscribe(
          ({ data, loading, error, errors, networkStatus }) => {
            // window.console.log("query subscribe fired")
            if (loading) return;
            if (error) {
              this.queryFailed({ error, errors, networkStatus });
            } else {
              this.queryLoaded(data);
            }

            if (this.state.loading) {
              this.setState({ loading: false });
            }
          },
          error => {
            console.log("Query Failed");
            console.dir(error);
            this.queryFailed({ error });
          }
        );
    } catch (error) {
      console.log("query error");
      console.log(error);
      this.setState({
        loading: false,
        error
      });
    }
  }

  loading() {
    return <Loader />;
  }

  queryLoaded(data) {
    this.setState(data);
  }

  queryFailed({ error }) {
    this.setState({
      loading: false,
      error
    });
  }

  handleError(error) {
    const message = error.message;
    return <div>{message}</div>;
  }

  render() {
    const { loading, error } = this.state;
    if (loading) return this.loading();
    if (error) return this.handleError(error);
    return this.content();
  }
}

QueryComponent.contextTypes = {
  client: PropTypes.object
};

export default QueryComponent;

I simplified code a bit for the sake of clarity.

You can check more advanced working example, that also passed query variables, in CodeSandbox:

Come up to style guide consensus with your team using voting

Vitaliy Yanchuk — Sat, 29 Jun 2019 12:59:16 +0000

How to create Style Guide for your team that will suit everyone

OurCodeStyle - create style guide that will best suit your team, or just have fun voting for code style options, find people who have same preferences.

Ruby and JavaScript both have static code analyzers and formatters, like
RuboCop, and ESLint, among most used.

This tools come with a set of rules, and rule options which can be configured based on your preferences, the problem is that people in the team might have different preferences.

Me and my team have been using this tools for a while, and we find some rules useful, some are of ambiguous benefit. For such cases we used Slack bot to vote if we want some rule to be ON or on particular configs of the rule.

But Slack is not very convenient for this, so in my free time I have made a site which contains all the rules from RuboCop and ESLint with their options and params, which can then generate proper .rubocop.yml file based on voting.

But why limit to only own team, I have made it accessible for everyone.

Voting

At above example there is 2 options for a Cop Style/LambdaCall: call, braces.
You can see examples of each option, and vote for your option.

See in action: Style/LambdaCall

Consensus

By default an option wins when it has at least 2 votes more than other options, and at least 3 votes in total (it can be configured for each team separately).
I found it that 50%+1 vote is usually not a good idea, you should configure this for your team needs.

Config Generation

After you have some consensus on options you can generate configuration for linter.
You receive a static link to your generated config, you can either download it and put into project, or add it to your current config as external inheritance source.

Code Style Mates

As you start to vote on options more and more, you can notice that your votes are matching with some other people on your team. To track this better there is a widget in your profile showing people you have most matches, it means your taste on code style is very similar.

Master Style Guide & Your Own

You can use site in 2 ways:

You can create your organization, create style guides in it and vote with your team
Or there is one public organization, called Master, where everyone can vote, this way we can build a collective mind style guide.

Free Form Guide

In addition to RuboCop and ESLint, you can create a free form guide, which you can populate with your own pools, for example if you can solve one problem in several ways and don't know which to choose, just crate an Enum type pool, paste your code blocks and ask your teammates to vote. Syntax highlighting for free form is:

ruby
javascript
python
java
php
go
kotlin
perl
rust
elixir

Please comment if you find it useful or have any issues or proposals