Forem: Philo Hermans

How to use GitHub Actions build matrix to deploy artifacts to multiple servers

Philo Hermans — Mon, 25 Jan 2021 13:30:00 +0000

In the past, I've always used multiple third-party services to, for example, host and deploy my code to production. I like the idea of minimizing the number of third-party services as they provide overhead, potential security risks, and often additional costs. So I decided to give Github Actions a try to see if I can replicate the famous zero-downtime deployments across multiple servers. GitHub introduced GitHub Actions in 2019, a workflow automation tool that allows GitHub users to use Actions to build their continuous delivery pipelines.

As the article implies, I was able to do so 😄 Before I walk you through every step, I recommend taking a look at the free GitHub Actions: Hello World course if you haven't used GitHub Actions before.

GitHub Action Workflow jobs overview

Each GitHub Action workflow consists of one or multiple jobs which contain one or multiple steps. To deploy our application, we need to create the following jobs:

Create GitHub Action build artifacts for deployment

I want to create one archive containing all the code ready to be deployed on production. You could run commands like npm run production on your production server, but I like to keep my production server mean and clean. This reduces server maintenance overhead, like updating NodeJS across multiple servers.

We want to achieve the following for our Laravel application artifacts:

Install NPM dependencies.
Compile CSS and Javascript assets.
Install Composer dependencies.
Archive our build and remove unnecessary data (e.g., node_modules).
Store our archive so we can deploy it to our servers.

Prepare release on all our servers

We want to make sure our deployments are stable and reliable, meaning we don't want one server to be updated while the second server failed. If the preparation fails on one server, we want the deployment sequence to stop.

We want the release preparation job to do the following:

Ensure we have a directory that holds every release.
Ensure we have a storage directory that shares data between releases.
Ensure we have a current directory that links to the active release.
Extract our build files into our releases directory.

Run optional before hooks

This is an optional feature, but I want to execute specific commands before the release is activated (e.g., chmod directories). So there needs to be a way to configure these so-called before hooks.

Activate the release

Now we are ready to activate our new release without any downtime. We can do this by changing symbolic links; this basically swaps the underlying release, which is linked to our current directory, to a new release inside our releases directory. I'm running PHP FPM on my servers, so I also want to reload PHP FPM to detect the changes.

Run optional after hooks

This is an optional feature as well, but I want to execute specific commands after the release is activated to send a notification that my deployment completed for example. So there needs to be a way to configure these so-called after hooks.

Cleaning up

Given that we are uploading and extracting new releases, we take up more disk spaces after each release. To make sure we don't end up with thousands of releases and a full disk, we need to limit the number of release artifacts living on every server.

Project scaffolding

To get started, you need to create a GitHub repository as testing your workflow requires you to commit and push your workflow yml file. It doesn't matter if your chosen repository is public or private. Feel free to try and implement this with one of your existing projects. I recommend you create a separate branch for this, so your repository stays clean, and you don't have a bunch of angry colleagues looking at you ;)

I'm using a clean Laravel 8.* installation to deploy in this demonstration, which you can download here. Be sure to verify that everything works before you continue.

Finally, create a new workflow file; feel free to give it any name you would like and place it in the .github/workflows directory inside your project.

name: Deploy Application

on:
  push:
    branches: [master]

jobs:
  # Magic

As I mentioned earlier, you will probably end up committing and pushing to GitHub several times. So if you are working along with one of your existing projects, make sure you choose a different branch if you don't want to clutter your commit history.

Our first job: Create deployment Artifacts

Before we have something to deploy, we need to start the build our Laravel application as we would typically do. So let's kick off by checking out our project by using the predefined checkout action by GitHub.

# // code from earlier is ommited for clearity

jobs:
  create-deployment-artifacts:
    name: Create deployment artifacts
    runs-on: ubuntu-latest

  steps:
  - uses: actions/checkout@v2

GitHub will checkout the code from our repository in the container; no further steps necessary. Now that we have our code, we can continue with compiling our assets.

# //
  steps:
  - uses: actions/checkout@v2

  - name: Compile CSS and Javascript
    run: |
      npm install
      npm run prod

Tip: If you use a CDN to deliver your static files, be sure to implement your own solution. The front-end assets are now shared amongst all servers individually, which is not ideal. It may impact your website speed since assets could be loaded from different servers on every request depending on your load-balancer.

We can now continue with our back-end code. Before we can install any composer packages, we need to make sure PHP is installed first. We will use the setup-php action by Shivam Mathur, which makes this a breeze.

# //
  - name: Compile CSS and Javascript
    run: |
      npm ci
      npm run prod

  - name: Configure PHP 8.0
    uses: shivammathur/setup-php@master
    with:
      php-version: 8.0
      extensions: mbstring, ctype, fileinfo, openssl, PDO, bcmath, json, tokenizer, xml

This will configure PHP 8.0 (I've started with PHP 7.4 and updated the article to PHP 8.0) and install the required extensions. If your application requires additional extensions, be sure to add them to the list. We can continue by installing our Composer dependencies.

# //
  - name: Configure PHP 8.0
     uses: shivammathur/setup-php@master
     with:
      php-version: 8.0
      extensions: mbstring, ctype, fileinfo, openssl, PDO, bcmath, json, tokenizer, xml
  - name: Composer install
    run: |
    composer install --no-dev --no-interaction --prefer-dist

Since we are preparing to deploy to production, I've added the --no-dev flag. These additional packages are not required for production. In case you want to run PHPUnit before your deployment, you could either install the dependencies, run PHPUnit, and remove the dependencies or simply create a new workflow. I recommend the latter since it keeps testing and deployment logic separated.

Time to test out what we've got so far! Commit your code changes and push them to GitHub. Once you've done so, visit the Actions page (github.com///actions)

You can click each job to see the execution output. If a job fails, it will show a red cross instead of a checkmark. The execution output will often provide you with the information you need to resolve the issue.

We've successfully compiled our front-end assets and installed our Composer dependencies. Now we need to store the results. GitHub provides an Upload-Artifact helper. This will help us to share the Github Actions artifacts between jobs.

You can upload single files, multiple files, and directories. Since we just want all our files deployed, I prefer to create a TAR archive, so we have a single file to work with. You could also create a ZIP archive, but this will require installing additional software in the build container as Ubuntu doesn't ship with the required libraries.

# //
  - name: Composer install
    run: |
     composer install --no-dev --no-interaction --prefer-dist

  - name: Create deployment artifact
    run: tar -czf app.tar.gz *

This will create a new tar archive called app.tar.gz containing all the files, including the additional build artifacts we've made in the previous steps.

This works just fine, but the archive now contains files we don't need, like the node_modules directory. We only required these to run the npm run production command. Let's fix this by excluding directories from our archive.

# //
- name: Create deployment artifact
  run: tar -czf app.tar.gz --exclude=*.git --exclude=node_modules --exclude=tests *

Our archive will now skip the .git, node_modules, and tests directories from our archive. If you have additional files that are not required to be on your production server, exclude them now. By making the archive smaller, your deployment will be quicker.

I want to change our archive's filename so it's easier to identify which commit our archive contains. GitHub has some global variables you can use in your .yml file, so let's change the name to the commit hash.

# //
- name: Create deployment artifact
  env:
    GITHUB_SHA: ${{ github.sha }}
  run: tar -czf "${GITHUB_SHA}".tar.gz --exclude=*.git --exclude=node_modules *

We use the env option to pass environment variables down into the container. In this case, we define an environment variable called GITHUB_SHA and assign it the commit hash from the current job. If you want to know more about context and expression syntax for GitHub Actions, click here.

We're now ready to upload our build artifacts with the Upload-Artifact helper.

# //
- name: Create deployment artifact
  env:
    GITHUB_SHA: ${{ github.sha }}
  run: tar -czf "${GITHUB_SHA}".tar.gz --exclude=*.git --exclude=node_modules *

- name: Store artifact for distribution
  uses: actions/upload-artifact@v2
  with:
    name: app-build
    path: ${{ github.sha }}.tar.gz

We only need to provide this step with the path to our file. We again use the GitHub Action expression to get the commit hash. Finally, we provide our artifact with a name, which we can use for later reference to download the GitHub Actions artifacts in our deployment job.

Before we can continue with any of the follow-up jobs, we need to prepare something called a GitHub Actions strategy matrix. A strategy creates a build matrix for your jobs. You can define different variations to run each job in.

You can define a matrix of different job configurations. A matrix allows you to create multiple jobs by performing variable substitution in a single job definition.

In our case, servers are the variable in our jobs. We we will use a JSON file inside our repository, so we don't have to switch back and forth between the GitHub UI.

Create a new file called deployment-config.json in the root of your project and add the following contents:

[
  {
    "name": "server-1",
    "ip": "123.456.78.90",
    "username": "web",
    "port": "22",
    "beforeHooks": "",
    "afterHooks": "",
    "path": ""
  }
]

This file contains a list of all our servers we wish to deploy to. We can define the before and after hook, and our Nginx directory path, which will serve our application for each server. For authentication we will use an SSH key which you can store inside a repository secret. In this example I will name the secret SSH_KEY and reference this secret inside our workflow to authenticate the commands we want to execute on our remote server .

You could even host this file somewhere to dynamically populate this file, so if you add more instances to your infrastructure, they will be automatically included in the deployment cycle.

Note: A job matrix can generate a maximum of 256 jobs per workflow run. If you want to deploy to hundreds of servers in a single workflow, you need an alternative. I would recommend going serverless with a solution like Laravel Vapor.

To make this configuration available inside our GitHub Actions workflow, we need to export this data. Using a special syntax, GitHub can identify which data to assign to which variable to reference later on our workflow matrix.

# //
- name: Store artifact for distribution
  uses: actions/upload-artifact@v2
  with:
    name: app-build
    path: ${{ github.sha }}.tar.gz

- name: Export deployment matrix
  id: export-deployment-matrix
  run: |
    JSON="$(cat ./deployment-config.json)"
    JSON="${JSON//'%'/'%25'}"
    JSON="${JSON//$'\n'/'%0A'}"
    JSON="${JSON//$'\r'/'%0D'}"
    echo "::set-output name=deployment-matrix::$JSON"

First, we need to get the JSON from our deployment-config.json by using a simple cat command. Next, we will escape the JSON (only required if your JSON is multiple lines). Finally, we use the ::set-output syntax to tell GitHub which value we want to associate with the deployment-matrix key, ::set-output name= deployment-matrix ::$JSON.

If we want our other jobs to get the output from this job, we need to specify the output reference in our create-deployment-artifacts job.

# //
jobs:
  create-deployment-artifacts:
    name: Create deployment artifacts
    runs-on: ubuntu-latest
    outputs:
      deployment-matrix: ${{ steps.export-deployment-matrix.outputs.deployment-matrix }}
    steps:
      - uses: actions/checkout@v2

# //

We can now reference deployment-matrix in other jobs and get all the required information for each of our servers.

Our second job: Prepare the release on all the servers

Next up, we can continue to prepare our release on every server. Thanks to the deployment matrix configuration we've created, we can cycle through each server to repeat all the steps on every server.

# //
prepare-release-on-servers:
  name: "${{ matrix.server.name }}: Prepare release"
  runs-on: ubuntu-latest
  needs: create-deployment-artifacts
  strategy:
    matrix:
      server: ${{ fromJson(needs.create-deployment-artifacts.outputs.deployment-matrix) }}

As you can see, there are a couple of new parameters. The first new parameter, needs, allows us to make sure a specific step has finished before this job can start. In our case, we want to prepare the release on the servers once the create-deployment-artifacts. If you have multiple step dependencies, you can also pass an array.

Next, we got our strategy parameter, which allows us to define our matrix. In this case, we define a matrix variable named server and assign it to our deployment-matrix we've created in our previous job. By default, the build matrix expects an array and not a JSON string.

As mentioned earlier, GitHub Actions support context and expression syntax to access context information but also to evaluate expressions. This includes a couple of functions to, for example, cast values. To read our server configuration, we need to change our JSON string into a real JSON object using the fromJSON function.

We can retrieve our JSON object through the matrix context. This enables access to the matrix parameters we've configured for the current job. For example, in the code above, you can see we define a variable job name: name: "${{ matrix.server.name }}: Prepare release". In the GitHub UI, this will resolve to "server-1: Prepare release".

We are now ready to continue with the steps of our job, downloading our artifact to our build container, uploading our artifact to the server, extracting our archive, and set up required directories on our server if they don't exist.

# //
prepare-release-on-servers:
  name: "${{ matrix.server.name }}: Prepare release"
  runs-on: ubuntu-latest
  needs: create-deployment-artifacts
  strategy:
    matrix:
      server: ${{ fromJson(needs.create-deployment-artifacts.outputs.deployment-matrix) }}
  steps:
    - uses: actions/download-artifact@v2
      with:
        name: app-build
    - name: Upload
      uses: appleboy/scp-action@master
      with:
        host: ${{ matrix.server.ip }}
        username: ${{ matrix.server.username }}
        key: ${{ secrets.SSH_KEY }}
        port: ${{ matrix.server.port }}
        source: ${{ github.sha }}.tar.gz
        target: ${{ matrix.server.path }}/artifacts

We start by downloading our GitHub Actions build artifact, which is quite simple as GitHub provides a simple action out of the box called "Download Artifacts." We reference the name we've used for uploading our artifact. This will download the artifact into the build container.

Next, we upload the artifact to the server using a third-party SCP action in the GitHub Actions marketplace. This action will copy our file via SSH based on our configuration. Make sure to check out the repository for all available input variables for this action if you, for example, want to use SSH keys for authentication.

The SSH credentials are self-explanatory; we simply reference our JSON object to get the server's connection details and the credentials. The SCP action requires a source input variable; this is the file we want to upload. We use the commit hash context object again to generate the filename ${{ github.sha }}.tar.gz.

Besides the source input variable, we also need to provide the target input variable. I want to keep all the uploaded artifacts in a separate directory. I reference the server path and append the path with the directory name artifacts to achieve this. The final path will be /var/www/html/artifacts.

Let's take a look and make sure everything is working so far. Commit all your changes and push them to GitHub. Visit the actions page of your repository again, and you should see a running action.

GitHub Actions Job Results and Artifacts

Click the server-1: Prepare release job so you can see the output of all the steps. Click on the Upload job to expand the output. Seems to be looking good so far, our GitHub Actions release artifacts are now uploaded to our remote server.

GitHub Actions Job Output

Our final step for this job is to extract our uploaded archive and create a couple of directories if they don't exist. We will use SSH commands to set this up. We will again use a third-party action in the GitHub Actions marketplace called SSH Action. By now, you will be familiar with most of the syntax. The input variables are similar to the previous upload step:

# //
- name: Extract archive and create directories
  uses: appleboy/ssh-action@master
  env:
    GITHUB_SHA: ${{ github.sha }}
  with:
    host: ${{ matrix.server.ip }}
    username: ${{ matrix.server.username }}
    key: ${{ secrets.SSH_KEY }}
    port: ${{ matrix.server.port }}
    envs: GITHUB_SHA
    script: |
      mkdir -p "${{ matrix.server.path }}/releases/${GITHUB_SHA}"
      tar xzf ${{ matrix.server.path }}/artifacts/${GITHUB_SHA}.tar.gz -C "${{ matrix.server.path }}/releases/${GITHUB_SHA}"
      rm -rf ${{ matrix.server.path }}/releases/${GITHUB_SHA}/storage

      mkdir -p ${{ matrix.server.path }}/storage/{app,public,framework,logs}
      mkdir -p ${{ matrix.server.path }}/storage/framework/{cache,sessions,testing,views}
      chmod -R 0777 ${{ matrix.server.path }}/storage

If you use the | character in your script, you can define multiple commands split across multiple lines you want to execute on your server.

# Create a new directory with the commit hash as the directory name
mkdir -p "${{ matrix.server.path }}/releases/${GITHUB_SHA}"

# Extract the tar file into our release directory
tar xzf ${{ matrix.server.path }}/artifacts/${GITHUB_SHA}.tar.gz -C "${{ matrix.server.path }}/releases/${GITHUB_SHA}"

# Create Laravel storage directories and set permissions
mkdir -p ${{ matrix.server.path }}/storage/{app,public,framework,logs}
mkdir -p ${{ matrix.server.path }}/storage/framework/{cache,sessions,testing,views}
chmod -R 0777 ${{ matrix.server.path }}/storage

If you want, you can SSH into one of your servers and verify that the directories exist and the archive is unarchived.

GitHub Actions job result manual verification via SSH

So far, so good! We've made our build, uploaded the results to our server, extracted the archive, and made sure the required directories exist. We are almost there :)

Our third job: Run before hooks

This step is optional, but it's quite useful if you want to execute specific commands before you activate your release. In this example, we are using Laravel, so you might want to run database migrations.

# //
run-before-hooks:
  name: "${{ matrix.server.name }}: Before hook"
  runs-on: ubuntu-latest
  needs: [create-deployment-artifacts, prepare-release-on-servers]
  strategy:
    matrix:
      server: ${{ fromJson(needs.create-deployment-artifacts.outputs.deployment-matrix) }}
  steps:
  - name: Run before hooks
    uses: appleboy/ssh-action@master
    env:
      GITHUB_SHA: ${{ github.sha }}
      RELEASE_PATH: ${{ matrix.server.path }}/releases/${{ github.sha }}
      ACTIVE_RELEASE_PATH: ${{ matrix.server.path }}/current
      STORAGE_PATH: ${{ matrix.server.path }}/storage
      BASE_PATH: ${{ matrix.server.path }}
    with:
      host: ${{ matrix.server.ip }}
      username: ${{ matrix.server.username }}
      key: ${{ secrets.SSH_KEY }}
      port: ${{ matrix.server.port }}
      envs: envs: GITHUB_SHA,RELEASE_PATH,ACTIVE_RELEASE_PATH,STORAGE_PATH,BASE_PATH
      script: |
        ${{ matrix.server.beforeHooks }}

Again, similar job but with a couple of changes. First, we want to make sure the create-deployment-artifacts and the prepare-release-on-servers have been completed by passing an array to the need property.

needs: [create-deployment-artifacts, prepare-release-on-servers]

To make things a bit easier when defining before hooks, I want to use specific environment variables to simplify things. Let's say I want to execute set permissions on the storage directory:

[
  {
    "name": "server-1",
    "beforeHooks": "chmod -R 0777 ${RELEASE_PATH}/storage",
    "afterHooks": "",
    "path": "/var/www/html"
  }
]

To make these environment variables available, you need to explicitly define which variables you want to pass via the envs input variable.

Our fourth job: Activating the release

Time for the most exciting part, if I say so myself, activating our release. We will re-use a big chunk of our previous step with a couple of changes.

# //
activate-release:
  name: "${{ matrix.server.name }}: Activate release"
  runs-on: ubuntu-latest
  needs: [create-deployment-artifacts, prepare-release-on-servers, run-before-hooks]
  strategy:
    matrix:
      server: ${{ fromJson(needs.create-deployment-artifacts.outputs.deployment-matrix) }}
  steps:
    - name: Activate release
      uses: appleboy/ssh-action@master
      env:
        GITHUB_SHA: ${{ github.sha }}
        RELEASE_PATH: ${{ matrix.server.path }}/releases/${{ github.sha }}
        ACTIVE_RELEASE_PATH: ${{ matrix.server.path }}/current
        STORAGE_PATH: ${{ matrix.server.path }}/storage
        BASE_PATH: ${{ matrix.server.path }}
        LARAVEL_ENV: ${{ secrets.LARAVEL_ENV }}
      with:
        host: ${{ matrix.server.ip }}
        username: ${{ matrix.server.username }}
        key: ${{ secrets.SSH_KEY }}
        port: ${{ matrix.server.port }}
        envs: GITHUB_SHA,RELEASE_PATH,ACTIVE_RELEASE_PATH,STORAGE_PATH,BASE_PATH,ENV_PATH,LARAVEL_ENV
        script: |
          printf "%s" "$LARAVEL_ENV" > "${BASE_PATH}/.env"
          ln -s -f ${BASE_PATH}/.env $RELEASE_PATH
          ln -s -f $STORAGE_PATH $RELEASE_PATH
          ln -s -n -f $RELEASE_PATH $ACTIVE_RELEASE_PATH
          service php8.0-fpm reload

Again, we update the need input variable to include all previous steps before running the activate-release job:

needs: [create-deployment-artifacts, prepare-release-on-servers, run-before-hooks]

I've added a new Laravel environment variable, LARAVEL_ENV, which will contain the environment variable for our Laravel application. This variable doesn't contain any data just yet. So let's do this first, you can define key-value secrets per repository via the GitHub UI (repository->settings ->secrets)

GitHub Repository Secrets

Let's take a closer look at our bash script line by line:

# Store the environment data to the /var/www/html/.env file 
printf "%s" "$LARAVEL_ENV" > "${BASE_PATH}/.env"

# Link /var/www/html/.env file to /var/www/html/releases/633be605b03169ef96c2cee1f756852e1ceb2688/.env
ln -s -f ${BASE_PATH}/.env $RELEASE_PATH

# Link /var/www/html/storage directory to /var/www/html/releases/633be605b03169ef96c2cee1f756852e1ceb2688/storage
ln -s -f $STORAGE_PATH $RELEASE_PATH

# Link the release path to the active release path, /var/www/html/current -> /var/www/html/releases/633be605b03169ef96c2cee1f756852e1ceb2688
ln -s -n -f $RELEASE_PATH $ACTIVE_RELEASE_PATH

# Reload php8.0 to detect file changes
service php8.0-fpm reload

Tip: There is a minor delay between the activation and the after hook job. So make sure to include critical commands in the activation job. You could add an extra release hook configuration if you want to define this per server.

Taking a quick look at all the directories via SSH verifies shows that everything is working correctly.

The current directory is pointing to /var/www/html/releases/5b62b9a13...

The current release environment file /var/www/html/current/.env is pointing to /var/www/html/.env and /var/www/html/current/storage is pointing to /var/www/html/storage.

Our Laravel application is up and running with zero downtime! Hurray!

Our fifth job: After hook

Our after hook is exactly the same as our before hook besides a few naming changes:

# //
run-after-hooks:
  name: "${{ matrix.server.name }}: After hook"
  runs-on: ubuntu-latest
  needs: [create-deployment-artifacts, prepare-release-on-servers, run-before-hooks, activate-release]
  strategy:
    matrix:
      server: ${{ fromJson(needs.create-deployment-artifacts.outputs.deployment-matrix) }}
  steps:
    - name: Run after hooks
      uses: appleboy/ssh-action@master
      env:
        GITHUB_SHA: ${{ github.sha }}
        RELEASE_PATH: ${{ matrix.server.path }}/releases/${{ github.sha }}
        ACTIVE_RELEASE_PATH: ${{ matrix.server.path }}/current
        STORAGE_PATH: ${{ matrix.server.path }}/storage
        BASE_PATH: ${{ matrix.server.path }}
      with:
        host: ${{ matrix.server.ip }}
        username: ${{ matrix.server.username }}
        key: ${{ secrets.SSH_KEY }}
        port: ${{ matrix.server.port }}
        envs: GITHUB_SHA,RELEASE_PATH,ACTIVE_RELEASE_PATH,STORAGE_PATH,BASE_PATH
        script: |
          ${{ matrix.server.afterHooks }}

Again the needs input variable is updated to make sure all previous steps have been completed.

Our sixth job: Cleaning up

Every release we upload and extract on our production servers take up space. You don't want to end up having servers meltdown because of their hard drives being full. So let's clean up after each deployment and keep 5 releases.

clean-up:
  name: "${{ matrix.server.name }}: Clean up"
  runs-on: ubuntu-latest
  needs: [create-deployment-artifacts, prepare-release-on-servers, run-before-hooks, activate-release, run-after-hooks]
  strategy:
    matrix:
      server: ${{ fromJson(needs.create-deployment-artifacts.outputs.deployment-matrix) }}
  steps:
    - name: Run after hooks
      uses: appleboy/ssh-action@master
      env:
        RELEASES_PATH: ${{ matrix.server.path }}/releases
        ARTIFACTS_PATH: ${{ matrix.server.path }}/artifacts
      with:
        host: ${{ matrix.server.ip }}
        username: ${{ matrix.server.username }}
        key: ${{ secrets.SSH_KEY }}
        port: ${{ matrix.server.port }}
        envs: RELEASES_PATH
        script: |
          cd $RELEASES_PATH && ls -t -1 | tail -n +6 | xargs rm -rf
          cd $ARTIFACTS_PATH && ls -t -1 | tail -n +6 | xargs rm -rf

Let's take closer look at the commands we are executing:

cd $RELEASES_PATH && ls -t -1 | tail -n +6 | xargs rm -rf
cd $ARTIFACTS_PATH && ls -t -1 | tail -n +6 | xargs rm -rf

This will cd into our release and artifacts directory, list all files in given directory, order the list by timestamp, return the results with the offset of 5 entries, and finally remove given files or folders.

Multi-server deployment

To deploy to multiple servers, you only need to update your deployment-config.json to include the additional servers:

[
  {
    "name": "server-1",
    "ip": "123.456.78.90",
    "username": "web",
    "port": "22",
    "beforeHooks": "",
    "afterHooks": "",
    "path": "/var/www/html"
  },
  {
    "name": "server-2",
    "ip": "901.234.56.78",
    "username": "web",
    "port": "22",
    "beforeHooks": "",
    "afterHooks": "",
    "path": "/var/www/html"
  }  
]

Multi-server configuration

Release activation on multiple servers

That's a wrap! You've successfully made a release artifact from your application build and deployed it to multiple servers without any downtime!

I'd love to hear if you use this workflow to deploy your application, if you are happy with the results, which steps you've implemented or skipped, and the modifications/improvements you've made. Drop me a tweet @Philo01 and talk! 🙌🏼

I'm thinking about releasing a premium video course in the near future where I will show you how to build a real-world application from start to finish using Laravel, Livewire, and Tailwind CSS. I will be covering everything, from registering a domain, setting up a server, writing tests, you name it. Subscribe if you are interested and want to be notified, and you might also get access for free as I will be doing a give away when the course launches.

Electron auto update introduction

Philo Hermans — Sat, 16 Jan 2021 15:16:10 +0000

You’ve just completed building your Electron application; everything is working so far; it’s notarized to work on macOS, and you’ve tested your application on Windows and Linux as well. You shared your application with the world and got some great responses from your community. Eventually, more and more messages appear in your inbox about your app crashing. You discovered a bug in your application that was causing the crashes and quickly fixed it. But how do you get this new version of your application out to your users?

Introducing Electron Auto Update

Electron ships with an auto-update feature so you can quickly ship updates of your product to your users. When I implemented the auto update feature in my first Electron application, I went down the rabbit hole to figure out how the Electron auto-update feature works because it wasn’t that straightforward, in my opinion.

A few things that you should know about Electron auto update:

It only supports macOS and Windows (no support for Linux).
Both the macOS and Windows updater use Squirrel behind the scenes.
The Windows version of Squirrel is looking for maintainers to “reboot” the project and has over 300 issues, meaning you might expect some problems.
You must sign your application on macOS for the auto-updater to work.
You must move your application to the Applications directory on macOS for the auto-updater to work.
On Windows, make sure you don’t update your application on its first run, or your app will throw a very user unfriendly error.

Configuring Electron Auto Update

Implementing Electron Auto Update is relatively easy; it only requires a few lines of code to integrate with your deployment server.

const { app, autoUpdater } = require('electron')
autoUpdater.setFeedURL('[https://dist.unlock.sh/v1/electron/my-app'](https://dist.unlock.sh/v1/electron/my-app'))
autoUpdater.checkForUpdates()

In case you want to check for updates on a regular interval (the code above only executes on startup), you can use setInterval to check every 30 minutes for example:

setInterval(() => {
  autoUpdater.checkForUpdates()
}, 30000)

The update will be downloaded in the background and installed automatically when the application is restarted (most of the time, see troubleshooting for some exceptions).

If you want to make your users aware that a new update has been downloaded and is available to be installed, you can also use autoUpdater.checkForUpdatesAndNotify() instead. The notification will be native to your user’s operating system.

Do you want to learn step by step how to release updates for your public repository? Be sure to check out the section below about Electron auto update for public repositories. I will publish an article in the near future about using a different server for private repositories.

Implementing your Electron Auto Update notification

If you want to use your own in-app update notification instead, you can do this by listening to the update-downloaded event emitted by the auto-updater.

autoUpdater.on('update-downloaded', (event, releaseNotes, releaseName) => {
  //
})

releaseName is only available on Windows.

If you want to force the auto-updater to install your update immediately after the update has been downloaded, you can use autoUpdater.quitAndInstall():

autoUpdater.on('update-downloaded', (event, releaseNotes, releaseName) => {
   autoUpdater.quitAndInstall()
})

Please note that your users might not appreciate this, as work could be lost when your app just quits while they were just filling out a form.

Electron auto update for public repositories

If your code repository is publicly available on GitHub, you can use a free service by Electron to ship your updates. It’s an easy process. Let’s quickly scaffold an application to test this out. I’m using Electron Forge’s starting template; if you want to follow along, execute the following command:

// Yarn
yarn create electron-app auto-update-example

// NPM
npx create-electron-app auto-update-example

To use the public auto-updater, we need to download the NPM dependency, so be sure to install this dependency:

npm install update-electron-app --save

Now just run yarn start or npm start, and your Electron application will build and execute.

We need a public GitHub repository, so head over to github.com/new and create a repository we can use.

To inform the auto-updater about our repository, we need to make sure we define it in our package.json:

{
  "name": "auto-update-example",
  "productName": "auto-update-example",
  "version": "1.0.0",
  "description": "My Electron application description",
  "repository": "[https://github.com/PhiloNL/electron-hello-world](https://github.com/PhiloNL/electron-hello-world)",
  "main": "src/index.js",
  //
}

Let’s open src/index.js and call the updater to check for updates every hour and notify the user when an update is available.

app.on('ready', () => {
    updateApp = require('update-electron-app');

updateApp({
        // repo: 'PhiloNL/electron-hello-world', // defaults to package.json
        updateInterval: '1 hour',
        notifyUser: true
    });
});

Next, we need to publish our app to GitHub. Electron Forge ships with a couple of build-in publishers, including one for GitHub. To install the publisher run the following command:

npm install [@electron](http://twitter.com/electron)-forge/publisher-github

We can define the configuration for different publishers in your package.json file. So let's add our GitHub configuration:

{
  //...
  "main": "src/index.js",
  "config": {
    "forge": {
      "packagerConfig": {},
      "publishers": [
        {
          "name": "[@electron](http://twitter.com/electron)-forge/publisher-github",
          "config": {
            "repository": {
              "owner": "PhiloNL",
              "name": "electron-hello-world"
            }
          }
        }
      ],
      //...
    }
  },
  //...
}

Now let’s publish our application to GitHub by running the publish command. The publish command requires that you set your GitHub personal access token so it can access your account. You can create a personal access token here. Make sure you keep this token secure and don’t share it with anyone.

Please note, from this point forward, you are required to have your application signed and notarized. To learn more about signing and notarizing your application, visit this article.

export GITHUB_TOKEN=<your-token>
yarn run publish

Great, you’ve just pushed version 1.0.0 to GitHub. By default, your release is set to ‘Draft’, awaiting your final approval. So head over to your repository releases and publish your release (github.com/username/repository/releases).

Let’s test if the updater works by publishing a new release. Open src/index.html and make a couple of changes so you can see that the application has been updated.

Next, increment the version number of your application by opening package.json and changing the version number:

{
  "name": "auto-update-example",
  "productName": "auto-update-example",
  "version": "1.0.1",
  // ...
}

Run yarn run publish again and head over to GitHub to publish v1.0.1 of your application. Start v1.0.0 of your application and wait for the notification :)

Click restart, and you will see the new version of your application.

Again, this will work on both macOS and should also work on Windows if you handle the Squirrel events correctly.

Troubleshooting

Since auto-updating happens behind the scenes, you have no idea what is happening, which can be quite frustrating when your application doesn’t update.

To debug what is happening in the background, you can enable the logger by passing it in the update-electron-app constructor.

require('update-electron-app')({
  logger: require('electron-log')
})

You will be able to find the log files in the following locations:

Linux: ~/.config/{app name}/logs/{process type}.log
macOS: /Library/Logs/{app name}/{process type}.log
Windows: %USERPROFILE%\AppData\Roaming{app name}\logs{process type}.log

    [info] Checking for update
    [info] Found version v1.0.1 (url: [https://github.com/PhiloNL/electron-hello-world/releases/download/v1.0.0/auto-update-example-darwin-x64-1.0.0.zip](https://github.com/PhiloNL/electron-hello-world/releases/download/v1.0.0/auto-update-example-darwin-x64-1.0.0.zip))
    [info] Downloading update from [https://github.com/PhiloNL/electron-hello-world/releases/download/v1.0.0/auto-update-example-darwin-x64-1.0.0.zip](https://github.com/PhiloNL/electron-hello-world/releases/download/v1.0.0/auto-update-example-darwin-x64-1.0.0.zip)

Race condition on macOS with Squirrel updater

In some cases, your application might require multiple restarts for the update to work if your user starts the application to quickly after quitting. This can also be the case when using autoUpdater.quitAndInstall(). I've experienced this with Electron Builder, so I'm not sure if this is also the case with Electron Forge. Still, I am assuming that since they all use the Squirrel updater, it affects any application that uses the built-in Electron updater.

After a long search, I finally found this issue and this comment with a possible solution. It’s not ideal, but it fixes the problem.

When you start your application, and the Squirrel updater has found a new update for your application, it will spawn an idle process called ShipIt. This process remains idle until you quit your application. Once your application is closed, the ShipIt process will start extracting your update and replace your application with the latest version. Depending on the size of your application and the user's machine's speed, this may take a moment.

If your application starts too quickly after quitting, meaning before the updater has completed, a new ShipIt instance will replace the process, and the update process restarts. In the meantime, the user might be confused because the app is still running on the same version.

The Gist from the issue above resolves this problem by ensuring the ShipIt process has ended. Let’s break the code down step by step.

const shipItProcesses = await findProcess('name', 'ShipIt');

Look for an active process named ShipIt.

if (shipItProcesses.some(f => f.cmd.includes('com.org.my-app'))) {
  shouldRestartBeforeLaunch = true;
  console.debug('Waiting for auto update to finish');
  setTimeout(makeSureAutoUpdateFinished, 1500);
} else {
 // ...
}

Since a user could be running multiple Electron apps, we want to make sure the ShipIt process belongs to our app com.org.my-app. If this process exists, we wait for the application to start, so the auto-updater has a chance to finish. This check will repeat recursively until the process is gone.

        if (shouldRestartBeforeLaunch) {
          try {
            const Electron = require('electron');
            Electron.app.relaunch();
            Electron.app.exit(0);
          } catch (error) {
            console.error('Failed to restart the app through electron', error);
            process.exit(1);
          }
        } else {
          require('./main');
        }

Next, it will restart the existing app to complete the update process. These multiple restarts will cause your app to bounce a couple of times in the macOS dock, but at least you are sure your user is using the latest version of your application. Finally, it will execute the main code of your Electron application.

That’s it! You’ve successfully used Electron Auto Update together with GitHub to distribute a new version of your application to your user.

Do you want to know more about how to publish updates from private repositories and to license your products? Be sure to subscribe for future articles or give me a follow on **Twitter. I appreciate the support!

Originally published at https://philo.dev on January 13, 2021.

Notarizing your Electron application with Electron Builder

Philo Hermans — Mon, 28 Dec 2020 21:11:19 +0000

Recently I released my first Electron application. When my application was ready to be released, it was required to have it notarized by Apple, or else nobody could use my your application. Although there are quite a few articles on this matter, it still took some trial and error because I ran into a couple of problems that were not mentioned. To save others some time figuring these things out, I've created this walkthrough to get you started.

Electron Builder

I've used Electron Builder together with the Electro Builder Vue CLI to build my first application. According to the official Electron website, it's described as a "complete solution to package and build a ready-for-distribution Electron app that focuses on an integrated experience."

From my experience, Electron Builder works excellent during development, but it seems a bit buggy in regards to making your application available to the general public. I've experienced some issues with notarizing and publishing. So I might give Electron Forge (maintained by Electron) a try for my next application to see how that works compared to Electron Builder (maintained by a third-party).

Apple Developer Program

In order to distribute your Electron app on macOS, you need to participate in Apple's Developer Program, which costs $99 per year. You can sign up at https://developer.apple.com.

Head over to your Developer Portal and click on "Certificates, IDs & Profiles."

Next, head over to "Certificates" and click the blue plus icon to create a new certificate. Depending on your distribution wishes, you need to select a certificate type. In this example, we will go ahead and select "Developer ID Application," which reads, "This certificate is used to code sign your app for distribution outside of the Mac App Store."

Next up, we need to upload a "Certificate Signing Request." You can create this with the Keychain Tool on your Mac. You can find this application in /Applications/Utilities

Next, choose Keychain Access > Certificate Assistant > Request a Certificate from a Certificate Authority. Fill out the form:

User Email Address: your email address
Common Name: anything
CA Email Address: leave empty
Request is: Saved to disk

This will create a new file "CertificateSigningRequest.certSigningRequest". Head back to Apple's Developer Portal again and upload the certificate signing request.

Now download your certificate to your Mac, then double click the .cer file to install in Keychain Access. If you don't do this, you will get some strange errors from Apple when notarizing your application, which doesn't tell you anything useful:

Notarizing nl.philo.artisan-remote found at ./artisan-remote-app/dist_electron/mac/Artisan Remote.app
Error: Apple failed to notarize your application, check the logs for more info

Status Code: 2
Message: Package Invalid
Logs: https://osxapps-ssl.itunes.apple.com/itunes-assets/...

{
  "logFormatVersion": 1,
  "jobId": "cabb3b08-744a-4b94-853c-62fb7908bd25",
  "status": "Invalid",
  "statusSummary": "Archive contains critical validation errors",
  "statusCode": 4000,
  "archiveFilename": "Artisan_Remote.zip",
  "uploadDate": "2020-12-22T22:45:50Z",
  "sha256": "3199b031a2d8819bec4ea52973a363657a8a221d81cc764c3169115cf1dda893",
  "ticketContents": null,
  "issues": [
    {
      "severity": "error",
      "code": null,
      "path": "Artisan_Remote.zip/Artisan Remote.app/Contents/MacOS/Artisan Remote",
      "message": "The binary is not signed.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "Artisan_Remote.zip/Artisan Remote.app/Contents/MacOS/Artisan Remote",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "x86_64"
    },
        {
      "severity": "error",
      "code": null,
      "path": "Artisan_Remote.zip/Artisan Remote.app/Contents/MacOS/Artisan Remote",
      "message": "The executable does not have the hardened runtime enabled.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    //
}

Configure after sign hook

Electron Builder also uses Electron Notarize behind the scenes. For example, this is the afterSignHook.js I'm using:

// "afterSign": "./afterSignHook.js",

require('dotenv').config();
const fs = require('fs');
const path = require('path');
var electron_notarize = require('electron-notarize');

module.exports = async function (params) {
    if (process.platform !== 'darwin') {
        return;
    }

    console.log('afterSign hook triggered', params);

    let appId = 'nl.philo.artisan-remote'

    let appPath = path.join(params.appOutDir, `${params.packager.appInfo.productFilename}.app`);
    if (!fs.existsSync(appPath)) {
        console.log('skip');
        return;
    }

    console.log(`Notarizing ${appId} found at ${appPath}`);

    try {
        await electron_notarize.notarize({
            appBundleId: appId,
            appPath: appPath,
            appleId: process.env.APPLE_ID,
            appleIdPassword: process.env.APPLE_ID_PASSWORD,
        });
    } catch (error) {
        console.error(error);
    }

    console.log(`Done notarizing ${appId}`);
};

Make sure you adjust the appId to your app's name; the convention is country.company.appname. Finally, you need to set your Apple ID and password. It's recommended not to include your credentials in your code, so please use environment variables when possible. You can generate app specific passwords, so you don't need to write your personal password. Apple will provide you with something like edqv-akmn-hasr-tyui.

That should do it. You should now be able to successfully notarize your application with the npm run osx:build command.

At the time of writing Electron Builder still contains a bug that causes the your-app-mac.zip file to have an unsigned copy of your application (the your-app.dmg will work just fine).

Luckily someone wrote a workaround (I've made some adjustments for it to work with my Vue boilerplate). Place this file fixMacDistributionArchive.js in the root of your project.

const path = require('path');
const {execSync} = require('child_process');
const fs = require('fs');
const yaml = require('js-yaml');
const {appBuilderPath} = require('app-builder-bin');

module.exports = function (params) {
    console.log('Verification if MacOS build is present.');

    let macBuild = false;

    params.platformToTargets.forEach((value, platform) => {
        if (platform.name === 'mac') {
            macBuild = value.get('zip').packager;
        }
    });

    if (!macBuild) {
        console.log('No MacOS build is present in platform targets.');
        return;
    }

    console.log('Mac OS build found, creating new archive.');
    execSync(
        `ditto -c -k --sequesterRsrc --keepParent --zlibCompressionLevel 9 "${params.outDir}/mac/${macBuild.appInfo.productFilename}.app" "${params.outDir}/${macBuild.appInfo.productFilename}-${macBuild.appInfo.buildVersion}-mac.zip"`
    );

    console.log('Mac OS build archive has been created.');

    const APP_GENERATED_BINARY_PATH = path.join(params.outDir, `${macBuild.appInfo.productFilename}-${macBuild.appInfo.buildVersion}-mac.zip`);

    try {
        let output = execSync(
            `${appBuilderPath} blockmap --input="${APP_GENERATED_BINARY_PATH}" --output="${params.outDir}/${macBuild.appInfo.productFilename}-${macBuild.appInfo.buildVersion}-mac.zip.blockmap" --compression=gzip`
        );
        let {sha512, size} = JSON.parse(output);

        const ymlPath = path.join(params.outDir, 'latest-mac.yml');
        let ymlData = yaml.safeLoad(fs.readFileSync(ymlPath, 'utf8'));

        ymlData.sha512 = sha512;
        ymlData.files[0].sha512 = sha512;
        ymlData.files[0].size = size;
        let yamlStr = yaml.safeDump(ymlData);

        fs.writeFileSync(ymlPath, yamlStr, 'utf8');
        console.log('Successfully updated YAML file and configurations with blockmap.');
    } catch (e) {
        console.log('Error in updating YAML file and configurations with blockmap.', e);
    }
};

Electron Builder has a couple of hooks you can use to run the fixMacDistributionArchive.js file. Open your vue.config.js file if you are using the Vue CLI and add the afterAllArtifactBuild:

module.exports = {
    configureWebpack: {
        devtool: 'source-map',
        resolve: {
            alias: {
                'vue$': 'vue/dist/vue.esm.js'
            }
        }
    },
    pluginOptions: {
        electronBuilder: {
            nodeIntegration: true,
            builderOptions: {
                mac: {
                    hardenedRuntime: true,
                    entitlements: "./build/entitlements.mac.inherit.plist"
                },
                linux: {
                    target: ["AppImage"]
                },
                publish: ['github'],
                appId: 'nl.codebite.artisan-remote',
                afterSign: './afterSignHook.js',
                afterAllArtifactBuild: './fixMacDistributionArchive.js'
            }
        }
    }
}

All done! You should now be able to share your awesome app with the rest of the world. I'm working on a solution to make it easier for you to distribute your applications. If you are interested make sure you subscribe and be notified at Unlock.

This article was originally published on philo.dev. Head over there if you like this post and want to read others like it.

Introducing Artisan Remote

Philo Hermans — Tue, 22 Dec 2020 17:20:45 +0000

Over the years, I've SSH'ed into dozens of servers to execute Laravel Artisan commands. Creating some bash shortcuts made things a bit easier, luckily. On some occasions, though, clients did not support SSH, and in other cases, they did, but it was a terrible process of back-and-forth emails between clients and hosting providers to get things sorted.

I started to play around with Electron (build cross-platform desktop apps with JavaScript, HTML, and CSS) and had the idea to create something that would fix this issue as my first Electron project. So without further ado, I would like to introduce you to Artisan Remote, the perfect companion for your Laravel applications.

Artisan Remote is a desktop application that allows you to run artisan commands with a simple click and view the output of earlier executed commands across multiple Laravel applications.

Behind the scenes, the application utilizes HTTP endpoints to send and retrieve information from your Laravel applications. To allow Laravel artisan commands to execute commands via HTTP endpoints instead of the command line, I've created a simple package that exposes your artisan commands to the web.

Getting started

Open your terminal, switch to your favorite Laravel application, and install the Artisan Remote composer package:

composer require philo/artisan-remote

Next, you will need to publish the associated configuration file with the vendor:publish command:

php artisan vendor:publish --provider="Philo\ArtisanRemote\ArtisanRemoteServiceProvider"

This command will place a configuration file in your Laravel application's config directory config/artisan-remote.php. Inside the configuration file, you define which commands are available via the HTTP endpoint.

<?php

return [
    'commands'     => [
        \Illuminate\Foundation\Console\UpCommand::class,
        \Illuminate\Foundation\Console\DownCommand::class,
        \Illuminate\Cache\Console\ClearCommand::class,
    ],
    //
];

Given the nature and possible power of commands, you can't execute any without authentication. So Artisan Remote requires you to define at least one API authentication token. You can choose to restrict API tokens only to perform specific commands. For example, you might want to expose the up and down command to your client and give yourself access to execute the queue:restart command:

<?php

return [
    'commands'     => [
        \Illuminate\Foundation\Console\UpCommand::class,
        \Illuminate\Foundation\Console\DownCommand::class,
        \Illuminate\Cache\Console\ClearCommand::class,
    ],
    'auth'         => [
        'c557510d-7f2d-4e69-b600-d1050e5dc896' => [
            \Illuminate\Foundation\Console\UpCommand::class,
            \Illuminate\Foundation\Console\DownCommand::class,
            \Illuminate\Cache\Console\ClearCommand::class,
        ],
        '240d4017-d7e3-4b30-8d74-6cb60816caf2' => ['*'],
    ],
    //
];

Best practices recommend using environment variables for sensitive information. So be sure to adjust your config file accordingly:

        env('MY_ARTISAN_REMOTE_API_KEY') => ['*'],

For those who would like to be able to bring their application back online after running the down command ;) be sure to allow the desktop application to interact with your application when in maintenance mode by listing the API endpoint as an exception.

You can add this exception in the CheckForMaintenanceMode.php middleware in Laravel 7 or PreventRequestsDuringMaintenance.php in Laravel 8.

    /**
     * The URIs that should be reachable while maintenance mode is enabled.
     *
     * @var array
     */
    protected $except = [
        'artisan-remote/*'
    ];

Download and configure the desktop application

Now it's time for the exciting part. Click here to download the desktop application for your operating system and launch it!

To get started, click "Add new Laravel application." and enter your application URL and the API token you've set in the artisan-remote.php configuration file.

That's it! You've just added your first Laravel application to the Artisan Remote desktop application. To execute your first command, click on your application's name; this will present you with an overview of the commands available per your configuration.

Click on the blue play icon on the right side of the command you would like to execute. Clicking the icon will display the available options for the command you've chosen. In case your command has no options, click the "Execute Command" button.

Once the command execution finishes, a new window will pop-up with your command's output. If you want, you can see the history of commands you've run earlier by clicking "Execution history" located in the cog menu.

The Artisan Remote composer package and the desktop application are available for free. The desktop application source code is not available for now as I used Tailwind UI's commercial components. Depending on the amount of interest from people, I might update the design and remove the Tailwind UI components so I can open-source the project.

Feedback is always welcome. If you want to report a bug, submit an idea, go to the repository, and create an issue.

Download for Mac, Windows or Linux.

This article was originally published on philo.dev. Head over there if you like this post and want to read others like it.