Forem: Martin Goyot

Compute Caddy payload using Jinja2 with Ansible

Martin Goyot — Tue, 21 May 2024 18:32:21 +0000

Context

As I may have mentioned in previous articles I run my services using Ansible to deal with the deployment and upgrade of different components. If we generalize the setup, it mostly consists of a reverse proxy (Caddy) listening on 443 and app stacks running under docker on their own network that connects to Caddy's one. Something that looks like that:

Issue

My Ansible playbook is structured in a way where the reverse proxy is spawned first and then each app is deployed one by one, registering its own configuration against Caddy's API in order to be served. This way of structuring it has lead me to a nice place where each service is described as a var of my role which then iterates over each service. The var describes among other things the docker-compose definition, networking, volumes to be backed up, DNS, and the part that gets our attention today: the Caddy API payload. While I love this organization, I have recently grown weary of the time it takes to execute the complete playbook and as such I started looking for culprits. You guessed it, a big bottleneck is indeed the declaration to Caddy's API.

For reference, here is the kind of vars describing the services we are talking about:

---
dns_domain: mydomain.com
applications:
  - dns_entry:
      record: app_a
      value: xx.xx.xx.xx
      state: present
    payload:
      match:
        - host:
          - "app_a.mydomain.com"
      handle:
        - handler: reverse_proxy
          upstreams:
            - dial: "app_a:8080"
      terminal: true

  - dns_entry:
      record: app_b
      value: xx.xx.xx.xx
      state: present
    payload:
      match:
        - host:
          - "app_b.mydomain.com"
      handle:
        - handler: reverse_proxy
          upstreams:
            - dial: "app_b:80"
      terminal: true
    vpn: true

The issue has nothing to do with Caddy itself which is a very nice piece of software, but everything to do with the architecture of the Playbook which falls on me. By pursuing this idea of declaring each service independently I found myself forced to do two things:

Having an ID for each part of the configuration so that I can query it back later when it eventually changes.
Querying the API multiple times for each service.

The need of an ID is no big deal in itself, but I kinda dislike them when they are not really needed, you never know how it is going to evolve and if you're going to end up with conflicts or whatnot. But I'm forced to query the API multiple times during declaration which leads to a lot of unnecessary Ansible's tasks and HTTP calls. In fact, I need to call the API exactly 4 times in the worst case, worst case being that the service already has a configuration registered by Caddy:

Insert the hostname in the TLS automation part of Caddy's configuration.
Get current configuration for service if any.
Delete current configuration if it is different from the new one.
Register the new configuration for the service.

For reference, here is an example of what Caddy's payload looks like in full:

{
   "apps":{
      "http":{
         "servers":{
            "myserver":{
               "listen":[
                  ":443"
               ],
               "routes":[
                  {
                     "handle":[
                        {
                           "handler":"reverse_proxy",
                           "upstreams":[
                              {
                                 "dial":"app_a:8080"
                              }
                           ]
                        }
                     ],
                     "match":[
                        {
                           "host":[
                              "app_a.mydomain.com"
                           ]
                        }
                     ],
                     "terminal":true
                  }
               ]
            },
            "myserver-vpn":{
               "listen":[
                  ":5556"
               ],
               "routes":[
                  {
                     "handle":[
                        {
                           "handler":"reverse_proxy",
                           "upstreams":[
                              {
                                 "dial":"app_b:80"
                              }
                           ]
                        }
                     ],
                     "match":[
                        {
                           "host":[
                              "app_b.mydomain.com"
                           ]
                        }
                     ],
                     "terminal":true
                  }
               ]
            }
         }
      },
      "tls":{
         "automation":{
            "policies":[
               {
                  "issuers":[
                     {
                        "challenges":{
                           "dns":{
                              "provider":{
                                 "api_token":"myapitoken",
                                 "name":"myprovider"
                              }
                           }
                        },
                        "module":"acme"
                     },
                     {
                        "challenges":{
                           "dns":{
                              "provider":{
                                 "api_token":"myapitoken",
                                 "name":"myprovider"
                              }
                           }
                        },
                        "module":"zerossl"
                     }
                  ],
                  "subjects":[
                     "app_a.mydomain.com",
                     "app_b.mydomain.com"
                  ]
               }
            ]
         }
      }
   }
}

And here is what the Ansible playbook looks like:

## main.yml
---
- name: Deploy application
  include_tasks: deploy-application.yml
  loop: "{{ applications }}"
  loop_control:
    loop_var: application

## deploy-application.yml
---
- include_tasks: register-dns.yml
- include_tasks: inject-reverse-proxy-payload.yml
- include_tasks: deploy-compose-stack.yml
- include_tasks: connect-reverse-proxy-to-containers.yml

## inject-reverse-proxy-payload.yml
---
- name: Register hostname in TLS automation
  # Whatever needs to be done here

- name: Get currently loaded configuration for route
  # Whatever needs to be done here

- name: Load configuration for route if needed
  # Whatever needs to be done here, DELETE and POST

Solution

After thinking it through, I realized that it does not make sense with our use case, we could potentially compute the whole payload at once and push it once. This would solve both of my issues: only one call to Caddy's API instead of 4xServices (worst case scenario), and no need to use IDs anymore as I won't be modifying parts of it anymore, if I need to update the configuration I re-send the whole one to Caddy.

In order to implement this, we can leverage on Jinja2 which is the templating engine bundled with Ansible. The payload being "rather big", I prefer isolating it inside a template file instead of directly building it inside the playbook. I can then render the template inside a variable using lookup('template', 'path/to/template.j2').

The templates looks like this:

{% set public_payloads_list = [] %}
{% set private_payloads_list = [] %}
{% for application in applications %}
    {% if 'vpn' in application and application.vpn == true %}
        {{ private_payloads_list.append(application.payload) }}
    {% else %}
        {{ public_payloads_list.append(application.payload) }}
    {% endif %}
{% endfor %}

{% set domain_names = [] %}
{% for application in applications %}
    {{ domain_names.append(application.dns_entry.record + "." + dns_domain) }}
{% endfor %}

{
    "apps": {
        "http": {
            "servers": {
                "{{ inventory_hostname }}": {
                    "listen": [
                        ":443"
                    ],
                    "routes": {{ public_payloads_list | to_json }}
                },
                "{{ inventory_hostname }}-vpn": {
                    "listen": [
                        ":5556"
                    ],
                    "routes": {{ private_payloads_list | to_json }}
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "subjects": {{ domain_names | to_json }},
                        "issuers": [
                            {
                                "challenges": {
                                    "dns": {
                                        "provider": {
                                            "api_token": "{{ provider_dns_secret }}",
                                            "name": "myprovider"
                                        }
                                    }
                                },
                                "module": "acme"
                            },
                            {
                                "challenges": {
                                    "dns": {
                                        "provider": {
                                            "api_token": "{{ provider_dns_secret }}",
                                            "name": "myprovider"
                                        }
                                   }
                                },
                                "module": "zerossl"
                            }
                        ]
                    }
                ]
            }
        }
    }
}

The template starts with two loops, one to render to json and dispatch the service's payload between private_payloads_list and public_payloads_list based on whether or not it is supposed to be access via the VPN, and the other one to collect and rebuild all the hostnames of the services. All thoses lists are then rendered inside the template after going through the to_json filter in order to format them correctly for our payload.

Now that the template is built, we can render it directly inside a variable but not before sending it through a from_json filter so that it fails if anything is broken at this point. Overall, the Ansible playbook now looks like this:

## main.yml
---
- import_tasks: register-dns.yml
- import_tasks: inject-reverse-proxy-payload.yml
- name: Deploy application
  include_tasks: deploy-application.yml
  loop: "{{ applications }}"
  loop_control:
    loop_var: application

## inject-reverse-proxy-payload.yml
---
- name: Inject Caddy payload
  vars:
    caddy_payload: "{{ lookup('template', '../path/to/caddy-payload.json.j2') | from_json }}"
  community.docker.docker_container_exec:
    container: caddy
    command: |
      curl -X POST http://localhost:2019/load
      -H "Content-Type: application/json"
      -d '{{ caddy_payload | to_json }}'

Conclusion

The deployment of my stack is now much faster than it was. Of course there are other areas of my playbook that could still be optimized but this removed the most obvious time consuming part of it. I guess the conlusion is that although I philosophically preferred the previous way of dealing with services registration in Caddy in the sense that it felt more "encapsulated", this new version is much more practical and elegant (no more IDs yay), much so that I tried to apply the same principles to the dns registration part but sadly could not find a way as it seems that the Ansible module for my provider only allows the definition of one domain at a time.

Could we stop prefixing booleans with the verb ?

Martin Goyot — Wed, 07 Feb 2024 16:13:28 +0000

A widespread convention when it comes to Boolean variables naming is to prefix them with a verb. Sometimes the explanation given for this practice is that it helps reading code, other times that this way we directly can spot that this is indeed a boolean. I would like to discuss those two assumptions.

It helps reading code

Maybe this is just me but I like when code reads like instructions. I agree that it can't always be the case but I feel I should strive for it whenever I can. Keeping that in mind we can look at this complex piece of code:

  if isUserLoggedIn {

  }

Here we use the conventional prefixing, and the voice in my head reads: "If is user logged in...".

Now, if we consider this piece of code without prefixing I read "If user is logged in...":

  if userIsLoggedIn {

  }

See, nothing spectacular, this won't change your life but still I feel like the second version is much more flowing than the first one, while prefixing "hurts" the readability. Please do note that we keep the is or has verbs, but at their natural placement, not as prefix.

It helps spotting that it is a Boolean

Let me be honest: this is totally true. Whenever I encounter a variable prefixed with this is or has, I definitely know that this is a Boolean.

What I would like to challenge here, is the need behind this, the use case. Do I really need to know that this is a boolean at this specific point in time. This might be terribly personal, but I distinguish at least two ways of reading code. The first one, which refers to the previous part of this short post, when I want to understand the flow of the code, and the second one, when I'm diving into the code to edit it or spot an bug.

When I'm reading the code in a "I want to understand the flow" way, I don't really need to know that this is a boolean, I need a well named variable that helps me read what is happening, the fact that this is a boolean is a mere implementation detail that I don't need to know about. Moreover, I feel that reading userIsLoggedIn leave very few room to interpret it as something else than a boolean. On the contrary, if I'm diving into the code to edit it or spot a bug, in this case, yes, I need to know. But, if I'm in this mindset, I'm much more into "details mode" and I will definitely check the type of my variable which will be declared in there and this will be the only source of truth for me, not the fact that it starts with is or has.

Conclusion

I guess that all this gibberish is about is that I try to prioritize readability when it comes to writing code, and I feel like this convention can hurt it. Now, if you are in a team where prefixing is a widespread convention, please do follow this rule or challenge it with the team but don't go rogue; being a team player is the one of the most important thing.

How to backup your selfhosted server running under docker-compose

Martin Goyot — Wed, 17 Feb 2021 21:38:18 +0000

When you go down the rabbithole of trying to selfhost your internet life there are two concerns that will arise quite rapidly:

How to be able to respawn everything quickly from scratch if it fails
How to make sure I won't loose my data

The first part is mainly covered by your use of Docker Compose, but the second one can be a bit tedious to think through.

In this article I will propose a backup scenario. Beware, this is in no way a guaranteed approach to backup but rather an idea of one could do this. I in no way endorse any issue that would arise on your side while implementing this scenario.

Description of the Backup Scenario

In this scenario we will try to cover ourselves by following the 321 Rule:

3 copies at least
2 locations
1 off-site

In order to follow this rule we have basically two needs:

A backup tool that we can trust
A way to synchronize the backup repository to another site

For the backup part we will be using Restic, a nice backup tool that encrypts the backups.

restic · Backups done right!

As to synchronizing the backup repository with a off-site location, we will use RClone. Rclone is like RSync but designed to discuss with Cloud backends like BackBlaze, Mega, S3 etc...

Rclone - Rclone syncs your files to cloud storage: Google Drive, S3, Swift, Dropbox, Google Cloud Storage, Azure, Box and many more.

Initialization of the Restic repository

The first thing to do is to initialize our Restic repository. As per usual we are going to use Docker images to use both tools. To do so we are going to bind mount a place of the host file system were we will store the repository. So go where you want to store it and then:

docker run --rm -ti -v $(pwd):/data restic/restic init --repo /data/my-restic-repo

You will be asked for a password for the encryption of the repository, do not loose it, and then you are left with a new folder on your file system: my-restic-repo

Initialization of the RClone configuration

Same as the initialization of the Restic repository, we are going to use a Docker image for this. We will also mount the file system for this. The command to use is:

docker run --rm -ti -v $(pwd):/data rclone/rclone --config=/data/rclone-config config

I will not tell you what to configure for RClone as it highly depends on where you will want to store it, which cloud backend, and so on. You will find the documentation at this address. This said, as a hint, in this article we will use the crypt backend in combination with another cloud backend. So basically it is the crypted remote that will be targeted which himself targets the remote cloud backend I want to use like crypt -> S3.

Configure volumes and exclusions

Now that our tools are configured, we want to set a list of docker volumes that will be backed up. In order to do that, we will simply create a file volumes and list volumes from our docker-compose.yml file that we want to be backed up. We chose to do this because as of today it's not possible to reposition labels on already created docker volumes, but there is work in the way regarding that, and once this is released we will be able to tag the volumes directly in the docker-compose.yml file and just list all docker volumes filtered with this tag, which could be more convenient.

Example:

service1-config
service1-data
service2-config
service2-data
service3-config
service3-data

Now that we listed the volumes to be backed up, we will configure exclusions as there are probably files or folders in those volumes that we don't really want to store (tmp files, transfer files, whatever bulky unimportant files).

In order to do this, we will create an exclusions file and list all exclusions following the Restic exclusion format.

Example:

service1-config/data/transcodes
service1-config/data/transcoding-temp
service2-data/data/SomeUser/files_trashbin

Writing the passwords

In order to provide the encryption passwords both for Restic and for the RClone configuration file, we will write them down in two different files. This is a very simple way to deal with the matter, one could use Docker Secrets, Vault, or whatever to better store those passwords. For the example we will store them individually in a restic-password and rclone-password file.

The backup script

Okay, we are good to take it to the next step. We have our volumes and exclusions set up, the passwords, the RClone config and the Restic repository. Now we need a script. Here is an example bash script that will:

Stop all our containers
Backup the listed volumes
Prune the old backups following a given strategy (see this Restic documentation). We are going to keep 7 daily 4 weekly 12 montly.
Clone the Restic repository to an off-site location with RClone
Start all the containers back

#!/bin/bash

COMPOSE_FILE_PATH=$1
COMPOSE_PROJECT_NAME=$2
RESTIC_REPOSITORY_PATH=$3
VOLUME_LIST_PATH=$4
RESTIC_EXCLUSION_FILE_PATH=$5
RESTIC_PASSWORD_PATH=$6
MACHINE_NAME=$7
RCLONE_CONFIG_PATH=$8
RCLONE_PASSWORD_PATH=$9
RCLONE_REMOTE=${10}

function stop_all_containers {
  echo "Stopping all running containers"
  COMPOSE_HTTP_TIMEOUT=200 docker-compose -f $COMPOSE_FILE_PATH stop
}

function list_all_volumes {
  local result=$1
  local list=""
  local volumes=($(cat $VOLUME_LIST_PATH))
  for volume_name in "${volumes[@]}"
  do
    list="${list} -v ${COMPOSE_PROJECT_NAME}_${volume_name}:/source/${volume_name}"
  done
  eval $result="'$list'"
}

function backup_volumes {
  echo "Retrieving volumes list"
  list_all_volumes volumes_list

  echo "Starting backup"
  docker run --rm \
          -v $RESTIC_REPOSITORY_PATH:/data/restic_repository \
          -v $RESTIC_EXCLUSION_FILE_PATH:/data/excludes \
          -v $RESTIC_PASSWORD_PATH:/data/password \
          $volumes_list \
          restic/restic -r /data/restic_repository backup /source --exclude-file=/data/excludes --password-file=/data/password --host $MACHINE_NAME
}

function start_all_containers {
  echo "Starting back all containers"
  COMPOSE_HTTP_TIMEOUT=200 docker-compose -f $COMPOSE_FILE_PATH start
}

function prune_old_backups {
  echo "Pruning old backups if needed"
  docker run --rm \
          -v $RESTIC_REPOSITORY_PATH:/data/restic_repository \
          -v $RESTIC_PASSWORD_PATH:/data/password \
          restic/restic -r /data/restic_repository forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --password-file=/data/password --host $MACHINE_NAME --prune
}

function clone_restic_repository {
  echo "Syncing the Restic repository on the cloud"
  docker run --rm \
          -v $RCLONE_CONFIG_PATH:/data/config \
          -v $RESTIC_REPOSITORY_PATH:/data/repo \
          -e RCLONE_CONFIG_PASS=$(cat $RCLONE_PASSWORD_PATH) \
          rclone/rclone --config=/data/config --stats-log-level NOTICE --stats 45m sync /data/repo $RCLONE_REMOTE:
}

echo "Starting backup process"
stop_all_containers
backup_volumes
prune_old_backups
clone_restic_repository
start_all_containers
echo "Done"

At this point, we have our entire backup process set up. Just try to run it to see where you could have misconfigured things, it should go through smoothly, if not make the needed adjustments. Once everything is working fine, congratulations, you are now able to backup your server and have the repository backed up off-site!

We can now proceed to the last part!

Automating the backup process

Right now, in order to launch a backup, we have to call our script. But what we really want is it to be run automatically by the server maybe once a day. For this, we are going to use systemd timers' capabilities. Long story short, we are going to create a systemd service that runs the backup script and a timer associated to it to run it daily.

For that, all we need is a service file like that:

///etc/systemd/system/backup.service
[Unit]
Description=Run the backup of the server

[Service]
Type=simple
ExecStart=call/to/script/with/your/arguments
StandardError=journal
User=userThatRunsDockerCompose

And the associated timer:

///etc/systemd/backup.timer
[Unit]
Description=Daily run of backup script

[Timer]
OnCalendar=*-*-* 04:00:00
Persistent=true

[Install]
WantedBy=timers.target

And we are done! All we have to do is to enable and start the service and the timer:

sudo systemctl enable backup.service
sudo systemctl enable backup.timer
sudo systemctl start backup.timer

And you are now done ! Your backup script will run everyday at 4am! If you want further information on how to see the logs and so on, look into systemctl and journalctl you will find all you need, but quick hint: journalctl -u backup.service --since today will give you everything that happened today for your backup.

-- Amike

Add a Play module from your own Nexus Repository

Martin Goyot — Mon, 18 Sep 2017 16:00:00 +0000

Those days I'm mainly working with the Play! Framework for Java. We are in a microservices setup and thus we end up having some common code between the different services. We decided to make Play Modules that we would import in the different projects.

Creating the Module

In order to do so, we created a project with a few utility classes that we published using SBT and our own Nexus respository. The final configuration ended up being something like this:

name := "some-module"
organization := "my.organization"
version := "1.0.0-SNAPSHOT"
scalaVersion := "2.12.2"
lazy val root = (project in file(".")).enablePlugins(PlayJava)

libraryDependencies ++= Seq(
    guice,
    "com.rabbitmq" % "amqp-client" % "4.2.0",
    "org.mockito" % "mockito-core" % "2.10.0" % "test"
)

val nexusHost = System.getenv("NEXUS_HOST")
val nexusUser = System.getenv("NEXUS_USER")
val nexusPassword = System.getenv("NEXUS_PASSWORD")

publishTo := {
    val httpNexus = "http://" + nexusHost
    if (version.value.trim.endsWith("SNAPSHOT"))
        Some("snapshots" at httpNexus + "/repository/maven-snapshots")
    else
        Some("releases" at httpNexus + "/repository/maven-releases")
}

credentials += Credentials("Sonatype Nexus Repository Manager", nexusHost, nexusUser, nexusPassword)

And, then publishing becomes super easy, we only need to run sbt publish and voilà! The module is published inside our own repository.

Using it in your Project

Now comes the second part, actually using your module in your project. For this, theoretically, all you need is to add a Library Dependency:

libraryDependencies ++= Seq(
    "my.organization" % "some-module" % "1.0.0-SNAPSHOT",
    "com.rabbitmq" % "amqp-client" % "4.2.0"
)

But this won't be enough.

First of all, you need to add a resolver to specify the new repository for your module:

resolvers += "My Snapshots" at "https://my.url/repository/maven-snapshots/"

You will also need your credentials to access it:

credentials += Credentials("Sonatype Nexus Repository Manager", "my.url", nexusUser, nexusPassword)

And now you are all excited and you type in sbt clean update... And it fails miserably.

Now, the thing is, SBT is gonna try to find your module at https://my.url/repository/maven-snapshots/my/organization/some-module/1.0.0-SNAPSHOT/.... But if you look carefully in your Nexus the module is actually at /my/organization/some-module_2.12/1.0.0-SNAPSHOT/.... Notice this 2.12? This is your Scala version, and this is actually quite important. So now, you're left with 3 choices.

Remove the cross path by doing crossPaths := false (highly NOT RECOMMENDED )
Specify the Scala version by depending on some-module_2.12 instead of just some-module
Using the %% syntax to let SBT append this Scala version, which is the solution we chose

So here is now our final Library Dependencies declaration:

libraryDependencies ++= Seq(
    "my.organization" %% "some-module" % "1.0.0-SNAPSHOT",
    "com.rabbitmq" % "amqp-client" % "4.2.0"
)

And the complete file for science:

name := """my_service"""
organization := "my.organization"
version := "1.0"
packageName in Universal := "my_service"

lazy val root = (project in file(".")).enablePlugins(PlayJava)

scalaVersion := "2.12.2"

val nexusUser = System.getenv("NEXUS_USER")
val nexusPassword = System.getenv("NEXUS_PASSWORD")

resolvers += "My Snapshots" at "https://my.url/repository/maven-snapshots/"
credentials += Credentials("Sonatype Nexus Repository Manager", "my.url", nexusUser, nexusPassword)

libraryDependencies ++= Seq(
  guice,
  "my.organization" %% "some-module" % "1.0.0-SNAPSHOT",
  "com.rabbitmq" % "amqp-client" % "4.0.2"
)

Please note that the Scala version that is going to be appended is the one minor one of your scalaVersion declaration, so here 2.12.

You're now good to go, you can get back to your fun with Play!.