Forem: Jonathan Sundqvist

How to configure github actions for a python package

Jonathan Sundqvist — Sat, 14 Nov 2020 19:00:00 +0000

This is what a build action for running a test matrix in github actions looks like in python. The yaml file is saved in .github/workflows/<your_name>.yml.



name: Running unittests

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.6, 3.7, 3.8]
        django: ["2.2", "3.0", "3.1"]

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements-tests.txt
        if [-f requirements.txt]; then pip install -r requirements.txt; fi
    - name: Linting
      run: flake8
    - name: Run unittests
      env:
        TOX_ENV: py${{ matrix.python-version}}-django${{ matrix.django }}
      run: |
        tox -e $TOX_ENV

In the matrix we define the python versions and in this case also the django versions. django in this case will be set as an environment variable.

Further down I've set a TOX env to set exactly how I expect the tox environment variable to be set. So that is what I'm relying tox to use in the end.

The python package tox-gh-actions may help mapping the python versions with the tox versions. In this case I chose to do it directly with github actions to keep things simpler. One less dependency, one less worry so to speak.

For the full documentation of all the things you can do with github actions there is the official documentation.

Creating a materialized view with SQL procedures

Jonathan Sundqvist — Wed, 04 Dec 2019 00:00:00 +0000

In newer versions of MySQL you can create materialized views directly without any SQL procedures. However I recently had to work with a database that used MySQL 5.6 and in that version it's not possible to create materialized views.

To give some context of what the problem at hand was. There was a huge table with around 1.3 billion entries. In this table there were URLs that had the flag green either set as true or false. The URLs are not unique. Depending on the date a URL could be flagged either as green or not.

What I needed was to get the distinct set of all URLs flagged as green. If a URL got flagged as green and then at a later date got unflagged as green it should not show up in the final set of results.

There isn't any particularly elegant way of creating a performant query to get that data. So I landed in creating a stored procedure to solve the problem.

This stored procedure was then later used to backfill the table called green_presenting.

Let's walk through what's actually happening in this stored procedure.



delimiter //
CREATE PROCEDURE insert_urls(IN url VarChar(255), IN green TinyInt(2), IN id_hp INT(11), IN datum datetime)
BEGIN
    DECLARE hostname VARCHAR(255);
    DECLARE hostwebsite VARCHAR(255);
    DECLARE hostpartner VARCHAR(255);

    IF (green = 0) THEN
        DELETE FROM green_presenting WHERE url = url;
    ELSE
        SELECT name, partner, website INTO hostname, hostpartner, hostwebsite 
        FROM hostingproviders WHERE id = id_hp;

        INSERT INTO green_presenting
        (`modified`, `green`, `hosted_by`, `hosted_by_id`, `hosted_by_website`, `partner`, `url`)
        VALUES (datum, green, hostname, id_hp, hostwebsite, hostpartner, url)
        ON DUPLICATE KEY UPDATE
        modified = datum;
    END IF;
END
//
delimiter ;

Before we start with the signature of the procedure. The very first we need to do is to change the delimiter, from using ; to // that way it won't parse new ; as the end of a statement when we create the procedure.

Let's continue with the signature of the procedure.



CREATE PROCEDURE insert_urls(IN url VarChar(255), IN green TinyInt(2), in id_hp INT(11), IN datum datetime)
BEGIN
    ...
END

The way to define paramaters is to use the keyword IN followed by the parameter name and last the type of the parameter. You can also use the keyword OUT or INOUT. If you don't need to change the parameter in the procedure and have that changed propagated in the database there is no need to use anything else than IN.

A stored procedure always need to begin with BEGIN and end with END



DECLARE hostname VARCHAR(255);

This is how a new variable is created. Which you can then assign something to.



IF (green = 0) THEN
    ...
ELSE
    ...
END IF;

The above is the outline of what an IF statement looks like. Make sure not to forget THEN :).



SELECT name, partner, website INTO hostname, hostpartner, hostwebsite 
FROM hostingproviders WHERE id = id_hp;

This is where we assign the values to the variables we created earlier. It's done with INTO keyword.

That's it!

The above procedure just takes one entry. Since I also needed to backfill all data. I had to create a new procedure that iterates through all entries of the table.

What's noteworthy here is how we create a cursor and loop through it. Otherwise it's pretty much self-explanatory.



delimiter //
CREATE PROCEDURE backfill()
BEGIN
  DECLARE done INT DEFAULT FALSE;

  DECLARE gdatum DATETIME;
  DECLARE ggreen TINYINT(2);
  DECLARE gid_hp INT(11);
  DECLARE gurl VARCHAR(255);
  DECLARE cur CURSOR FOR SELECT datum, green, id_hp, url FROM greencheck where id_hp > 0;
  DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;

  OPEN cur;

  read_loop: LOOP
    FETCH cur INTO gdatum, ggreen, gid_hp, gurl;
    IF done THEN
      LEAVE read_loop;
    END IF;
    call insert_presenting(gurl, ggreen, gid_hp, gdatum);
  END LOOP read_loop;

  CLOSE cur;
END;
//
delimiter ;

The last missing bit would be to create a trigger, that calls the procedure insert_urls. After that we more or less have everything what a materialized view usually does.

Remember Rust with Anki

Jonathan Sundqvist — Fri, 21 Jun 2019 15:00:00 +0000

I've read The Book once before, but it was a long time ago, and at some point I read an o'reilly book about rust too. At the time of reading those books. It felt like the language made sense. I would remember what I read, surely.

A few months go by, and I feel like I want to do something in Rust, get deeper. So I sit down in front of my computer with a blank stare. What is the exact syntax of doing a for loop again in rust? Let alone how do you even make a generic struct that is trait bounded.

Does it sound similar? Well, I decided I would do something about this, so I created an Anki deck which has 212 cards (updated 2019-06-21). The anki deck covers the rust book. The deck won't be that much of use if you haven't read the book. But I'd say it certainly helps you remember once you've read the book.

If you've never heard of Anki; in short it's flash card program which uses spaced repetition to help you remember. So even if I step away from rust at some future point I will remember the syntax, and now that you've also got the deck so will you!

You can download the deck here. If you want to get a notification when I make any changes to the deck. Head over to my own site and register your email, and I'll send you an update when I've added something new.

If you've got any feedback send me a tweet, email or comment on Hacker News :). I'll reply to any comments below too of course!

Troubleshoot why crontab is not working

Jonathan Sundqvist — Thu, 09 May 2019 15:00:00 +0000

Crontab is a particularly finicky unix tool. It's not always the easiest to figure out why on earth your scheduled job is not running as it should.

There are a number of things that could go wrong. The very first thing to do is of course to double check that the crontab syntax is doing what you expect it to do. Crontab guru helps exactly with that.

And as a reminder for what the stars and numbers signify.



# m   h   dom mon dow   username command
# *   *   *   *   *     dokku    command to be executed
# -   -   -   -   -
# |   |   |   |   |
# |   |   |   |   +----- day of week (0 - 6) (Sunday=0)
# |   |   |   +------- month (1 - 12)
# |   |   +--------- day of month (1 - 31)
# |   +----------- hour (0 - 23)
# +----------- min (0 - 59)

The crontab is located at /etc/crontab, however don't go and edit that file. Instead you can create a specific file related to the cronjob in /etc/cron.d. Creating a specific file helps if you are running multiple applications on a host for instance. So you've edited the file and it's still not running.

If you create a new file in /etc/cron.d. Make sure that the file does not contain any dot or else it won't run. It's also important that it has the correct permissions. The correct permissions for the file should be -rw-r--r-- root root. Those permissions are the outcome of the following two commands.



chown root:root yourcronfile
chmod 644 yourcronfile

You might need to restart the cron service for it to pick up the changes you made. You can do that with sudo service cron restart.

You can check the cron logs to make sure that the crontab is working correctly. The logs are by default located in /var/log/syslog. And running the following grep command will get you all the cron logs.



grep cron /var/log/syslog

Even if the job is logged there, any errors won't show up there. So it's helpful to log those separately. If you add the following snippet at the end of your crontab >> /var/log/myjob.log 2>&1, it will do just that. You're bound to thank me later ;).

Your cronjob will fail at some point. A service such as Dead man's snitch will most certainly help you knowing when it failed. The premise is that you'll send a request to their service and if no request has been sent in the last X minutes, it will notify you that your cronjob likely failed and you need to check it out.

Dokku interlude

Having your crontab checked in with git is certainly helpful. If you're using dokku there is the helpful plugin supply-config that does almost that. There is an open PR that amends that gap. Dokku documentation also have some helpful reminders on what to keep in mind when configuring a crontab.

Have you stumbled upon any other ways that crontab could fail?

Deploying Django using Dokku

Jonathan Sundqvist — Fri, 03 May 2019 10:04:12 +0000

This post is also over on my personal blog.

I want to see my site on the web!

That's what I've thought many times, but the thought of deploying things often seems prohibitively complex (unless you go with heroku, which hides most of the complexities at the expense of your wallet). The world of operations is vast and it's sometimes difficult to get a grasp of where to start to achieve what you want; to deploy the site you built.

So this guide should provide that foot in the door while simultanously feeling rewarding with achieving something in the very end, because spending hours without seeing any sort of result is no fun. Let's get started!

Getting started with Dokku

Unless you already have a server where you can develop your project I would suggest that you give scaleway a try. Once you've installed something like debian on your server. The very first thing to do is to configure fail2ban. If you don't your server is bound to be hacked in no time, I learnt the hard way on a digital ocean droplet. You'd be surprised how often someone tries to login to your server.

Now that you're ready to install Dokku follow the official instructions. Once you've executed the bash command for the latest version, set up your public SSH key and added an A record for your server IP address you are ready to deploy your application.

Configuring Dokku for deployment

If you don't yet have a Django project that you would like to deploy I would suggest that you use the following template which dockerizes django. It's a good starting point for any django project and already has sane defaults.

Once you've executed the following commands you'll be able to run django locally. This is also what I'll be using as a base for deploying Django using Dokku.



pip3 install cookiecutter
cookiecutter https://gitlab.com/thiras/cookiecutter-docker-django

SSH into your server and run the following commands



# this is how you will refer to your app when using dokku
dokku apps:create your-django-app

# install the postgres plugin to be able to use postgres with dokku
sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git

# creates a database for your app.
dokku postgres:create djangodb

# link the above database to your app
dokku postgres:link djangodb your-django-app

# configure which domain you'll be using for your app
# I'd recommend using a fully qualified domain
domains:add your-django-app subdomain.yourdomain.com

If you're using the django app from the cookiecutter template you'll notice that it doesn't contain any Procfile. Dokku does require a Procfile. So go ahead and add and commit a file named Procfile containing:

web: gunicorn appname.wsgi:application --log-file

You're now set and can try to deploy your django app!

Debugging deployment issues

Dokku promises that you'll be able to deploy using git. Isn't that a treat. In the dokku documentation it details how you should add the remote to git. These commands are



# if you're using a subdomain
git remote add dokku dokku@yourdomain.com:subdomain

# if you're deploying to the root domain
git remote add dokku dokku@dokku.me:dokku.me

git push dokku master

Problem: You fail to deploy. Because the host isn't recognized.



fatal: 'yourapp' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Solution: If you're facing this problem, perhaps because you're using a subdomain you can change the remote to contain the your public server IP instead. So the remote would look something like this instead dokku@41.151.12.3:subdomain

Problem: It doesn't find a Procfile even though you added one.

Solution: You might have file called .dockerignore. If procfile is named there the container won't pick up your procfile. Removing that should fix that issue.

We try to deploy again, and now it should actually deploy successfully.

Problem: While it successfully deploys the only thing you see is 400 Bad request. When you browse your site. How odd!

Solution: In settings.py you need to change ALLOWED_HOSTS to your own domain. Once you've done this, you can now deploy your site again and that particular problem will go away.

Problem: You try to view the site. All static files such as CSS files returns 404. The site looks horrendous.

Solution: The development server serves static files for you. The gunicorn server does not. There is a management command collectstatic that you need to run every time on deployment.

Create a file called app.json. There you can define commands that should run before deployment. You can read more about it in the dokku documentation



{
  "scripts": {
    "dokku": {
      "predeploy": "python manage.py collectstatic --noinput"
    }
  }
}

This will move your static files to the static folder you defined in django settings.

If you try to deploy again, thinking it is enough, it is not. These files need to be served as well. Nginx normally serves these files. So let's make that happen!

Every deployment Dokku generates an nginx.conf file in ~home/dokku/appname. So we can't really edit that file. However if we take a look at the file it contains the following:



include /home/dokku/appname/nginx.conf.d/\*.conf;

So if we place any conf file in the folder nginx.conf.d it should work. We'll place a file static.conf containing the following:



location /static {
    autoindex on;
    alias   /home/dokku/appname/static;
}

However the above path is not where our static files are generated. They are generated inside our container. Also wouldn't it be nice if our static.conf were in source control. You can accomplish that with the dokku plugin supply-config.

In other words, we need a way to access the static files generated in our container. We can do that by mounting a volume where the static files are located.



dokku docker-options:add your-django-app deploy "-v /home/dokku/appname/static:/app/appname/static"

The path after the colon is where your staticfiles are located after you run the command to collect your static files, and the path before the colon is the same path as in static.conf.

When you deploy this time, even the static files should work. I'll let you in on a secret though. Instead of fiddling with nginx you could also use the django package whitenoise which is probably a more appropriate solution in this case.

But if I told you directly you wouldn't have learnt about all the other Nginx configuration which might be useful for another day.

As a complementary reading, I'd recommend checking out this guide on deploying django using Dokku