Forem: Darragh O'Riordan

Running NextJS in a Docker Container

Darragh O'Riordan — Thu, 13 Apr 2023 17:12:33 +0000

Introduction

This is a guide on how to run a NextJS application in a Docker container.

NextJS is created and maintained by Vercel so it works great on their hosting service. Vercel is an amazing service that allows you to deploy your NextJS application with ease.

However, there are times when you may want to run your application in a Docker container. Using a docker container will allow you to host a NextJS application on Azure, AWS, DigitalOcean, Fly.io or any modern cloud provider.

This guide will show you how to dockerize your NextJS app in minutes.

Prerequisites

You will need the following installed on your machine:

Docker Desktop
NodeJS
A package manager. I use pnpm as my package manager, but you can use npm or yarn. They have similar commands.

Getting Started

You must configure your NextJS application to build a standalone application. This is required for running NextJS in a Docker container.

In your next.config.js file, add the following:

const nextConfig = {
  output: 'standalone',
  // ... other config
}

Configuring Docker

.dockerignore

Add a .dockerignore file to the root of your project. This file will contain the files and folders that you do not want to copy into the Docker container.

I say root of the project here because I build in a monorepo. But you can put this file in the root of your NextJS application if you’re not using a monorepo.

I use this .dockerignore file for my NextJS projects. You can use it as a starting point.

Dockerfile*
.dockerignore
node_modules
npm-debug.log
README.md
.next
.git
.jest_cache
.docker-compose
.vscode
.terraform
.husky

Dockerfile

Add a Dockerfile to the root of your project. This file will contain the instructions for building your Docker image.

This is a multi-stage Dockerfile. This means that we will build the NextJS application in one stage and then copy the built files into a new stage. It results in a smaller final image size for deployment.

STAGE 1: A container with pnpm and python3 is required
STAGE 2: Fetch deps into the pnpm store
STAGE 3: Copy the application code and install all deps from cache into the application
STAGE 4: Build the NextJS app
STAGE 5: Create a clean production image - only take pruned assets

Each stage is stored on the build machine’s cache. This means that if you make a change to your code, only the last 3 stages will be rebuilt. This is a huge time saver.

# STAGE 1: A container with pnpm and python3 is required
FROM node:18-alpine as pnpm_base

WORKDIR /app
# install pnpm
RUN npm i --global --no-update-notifier --no-fund pnpm@7
# install python3 and other deps
RUN apk add --no-cache g++ make py3-pip libc6-compat

# STAGE 2: fetch deps into the pnpm store
# We run pnpm fetch in a separate step to avoid re-fetching deps on every code change
# fetch is a pnpm command that downloads all dependencies to the local store
# You could remove or skip this step if using npm or yarn (but make sure to copy your lock file)
FROM pnpm_base as fetched_deps
WORKDIR /app
# setting production env usually speeds up install for your package manager
ENV NODE_ENV production
# copy the lock file that you use
COPY pnpm-lock.yaml ./
# set the store dir to a folder that is not in the project
RUN pnpm config set store-dir /workdir/.pnpm-store
RUN pnpm fetch

# STAGE 3: Copy the application code and install all deps from cache into the application
FROM fetched_deps as with_all_deps
# I use mono repo so I copy the whole project code (except for ignored things)
COPY . ./
# finally, install all the deps
RUN pnpm install --offline

# STAGE 4: Build the NextJS app
# Here we use pnpm filtering to only build the frontend app
# Then we use pnpm deploy command to prune the dependencies
FROM with_all_deps as builder
RUN pnpm --filter='*frontend' build
RUN pnpm --filter='*frontend' deploy pruned --prod

# STAGE 5: Create a clean production image - only take pruned assets
FROM node:18-alpine AS runner
WORKDIR /app
# We set the NODE_ENV to production to make sure that the NextJS app runs in production mode
ENV NODE_ENV=production
# We add a non-root user to run the app for security reasons
RUN addgroup --system --gid 1001 app
RUN adduser --system --uid 1001 app
USER app

# We copy the built NextJS app assets from the builder stage
# NextJS produces a backend server and a frontend app
COPY --chown=app:app --from=builder /app/apps/frontend/.next/standalone src/
COPY --chown=app:app --from=builder /app/apps/frontend/public src/apps/frontend/public
COPY --chown=app:app --from=builder /app/apps/frontend/.next/static src/apps/frontend/.next/static

# Set the port that the NextJS app will run on
# You should choose a port that is supported by your cloud provider
ENV PORT 5000
# Expose the port to the outside world
EXPOSE 5000

# Finally, we run the NextJS app
CMD ["node", "src/apps/frontend/server.js"]

Building the Docker Image

Now that you have your Dockerfile, you can build the Docker image.

Here we build with a tag of my-frontend-app. You can name it whatever you want. But a named tag makes it easier to run the image later.

docker build -t my-frontend-app .

Running the Docker Image

Now that you have your NextJS Docker image, you can run it.

docker run -p 5000:5000 my-frontend-app

This will run the NextJS app on port 5000. You can now access the NextJS app at http://localhost:5000.

Conclusion

That’s it! You now have a NextJS app running in a Docker container.

You can now deploy this Docker image to any cloud provider that supports Containers (All of them do).

I describe how to run a node container of Dokku in this blog post.

If you have any questions, please leave a comment below.

If you want to see a real world example of NextJs in a Container, check out my NextJS Starter project. It uses Docker to deploy NextJs to Dokku on Digital Ocean.

Building an AI generated game with Stable Diffusion and data from Wikipedia

Darragh O'Riordan — Sun, 27 Nov 2022 17:12:33 +0000

Last week I released a game called Doodle:ai.

In the game you’re shown AI generated images and you have to guess the Wikipedia topic it used to create the game.

Doodle:ai game interface

DoodleAI has had thousands of players since launch and I thought I would share how I built it.

The idea

There has been incredible progress in generative AI models over the last 3 months and I wanted to build something with it.

I had this idea to automatically create Google Doodles based on Wikipedia topics.

As I was building this Google Doodle generator tool, I thought it might be fun to see if I could randomise the output a little and guess what the AI had chosen for a day.

It worked as a game so I took that direction.

The Architecture

The game runs off pre-generated puzzle data.

The backend is a NestJS server which exposes endpoints to queue up puzzle generation jobs and store the output in a Digital Ocean Spaces blob store.

The frontend is a React app which uses the Digital Ocean Spaces store to get the daily puzzle.

The high level process for generating a puzzle

An administrator (Me) queues up a puzzle generation job for a date range. These jobs get put in a Bull queue by a NestJS controller.
The Bull queue workers retrieve and process daily topics from Wikipedia.
The topics are filtered and cleaned up to remove topics that are not suitable for a game.
A random topic is chosen and the worker requests the summaries for Wikipedia links and articles relevant to the topic.
The summaries are combined into an essay and sent to Azure NLP for analysis.
The NLP analysis is filtered and mapped to useful combinations for prompt generation.
Lots of prompts are generated and 4 random prompts are chosen.
The prompts are sent to stable diffusion to generate images.
The images and puzzle data are packaged and sent to Digital Ocean Spaces.
A customer opens the game and the puzzle data is retrieved from Digital Ocean Spaces.

Topic selection

The first step is to select a topic for the game. Wikipedia is a relatively high quality source of well organised trivia.

I use the Wikipedia API to get a list of topics for a given day. I then filter the topics to remove topics that are not suitable for a game.

I originally accessed the wikimedia api directly. The first few games used this data.

But Wikitext is extremely difficult to parse so I have changed to use the wtfWikipedia node library. This returns a structured javascript object and is much easier to work with.

See an example of wikitext below. It’s so hard to parse, thankfully we have open source parsers like wtfWikipedia.

{{Short description|Capital city of Ontario, Canada}}
{{About|the city in Ontario}}
{{Redirect|City of Toronto|the city's government|Municipal government of Toronto}}
{{Pp-move-indef}}
{{Pp-vandalism|small=yes}}
{{Use Canadian English|date=July 2014}}
{{Use mdy dates|date=April 2021}}
{{Infobox settlement
| name = Toronto
| official_name = City of Toronto
| settlement_type = [[List of cities in Ontario|City]] ([[List of municipalities in Ontario#Single-tier municipalities|single-tier]])<!-- Consensus see: [[Wikipedia talk:WikiProject Ontario/Archive 1#City infoboxes: "tier" or "conventional" municipal statuses (or both)?]] -->

Once the Wikipedia pages are parsed, I filter out topics that are not suitable for a game. This is generally topics that I feel will result in images that are too visceral or offensive like mass shootings or terrorist attacks.

The final step is simply choosing a random topic as the correct answer for the game.

Entity Categorisation

The game requires keywords for guesses and keywords for image generation prompts. To create these keywords I build an “essay” from the topic content and any adjacent Wikipedia topics.

  public async getSummariesOfPages(pageNames: string[]): Promise<string[]> {
        const summaries = (await wtf.fetch(pageNames, {
            lang: "en",
            // eslint-disable-next-line @typescript-eslint/naming-convention
            follow_redirects: true,
        })) as WtfWikiDocument[];

        const parsedWikiPages = summaries && summaries.map((x) => x.json());

        this.logger.log(
            "summaries",
            util.inspect(parsedWikiPages[0], false, 8, true)
        );
  // continue to parse and filter the data
  // ...
  // ...
  }

This essay is sent to azure NLP which returns a list of detected entities. This entity list is passed to the next stage.

const client = new TextAnalyticsClient(
  this.config.url,
  new AzureKeyCredential(this.config.apiKey)
)

const results = await client.recognizeEntities([text])
this.logger.log('Text analysis results', results)

// map the results to a list of entities
// ...

Prompt Creation

The difficult part of using AI generation tools is creating prompts.

I have gone through four major refactors of my prompt generation process. I have to provide spoilers for these games to explain the prompts. Sorry about that!

Play all the games first before reading this if you prefer.

1. Simply add all categorised entity keywords to 4 fixed prompts

This method resulted in some very interesting images but maybe 1/3 were usable for the game and they were all quite similar.

Example: "watercolor, 1995 gulf of aqaba earthquake, mercalli intensity scale, gulf, of aqaba earthquake, sinai peninsula, saudi arabia, tsunami"

Play this game here: https://doodleai.darraghoriordan.com/?gamedate=2022-11-22T12:00:00.000Z

2. Split entities randomly across a few random prompts

I tried randomising the keywords and selecting the first half for one prompt and the second half for another prompt and so on. This was actually OK but there were still issues where prompts that were suitable for portraits were used for landscapes and vice versa.

Example: "denny party, seattle,by Josef Thomas, matte painting, trending on artstation HQ"

Play this game here: https://doodleai.darraghoriordan.com/?gamedate=2022-11-13T12:00:00.000Z

3. Select the entities with best confidence scores and randomly join them

I tried only selecting the top 4 entity confidence scores. These were OK but categorisation confidence is not relevance so this often missed good keywords.

Example: "Donald Johanson, Australopithecus afarensis, Lucy (Australopithecus), The Beatles, depth of field, photograph, Polaroid 600 Instant Film"

Play this game here: https://doodleai.darraghoriordan.com/?gamedate=2022-11-24T12:00:00.000Z

4. Split entities in set patterns, use specific lists of prompts suitable for the category

This is the current method I use. It results in mostly relevant images and a good variety of images.

Example: "a hyper realistic professional photographic view picture of Richard Seddon, New Zealand photographic filter unreal engine 5 realistic hyperdetailed 8k ultradetail cinematic concept art volumetric lighting, fantasy artwork, very beautiful scenery, very realistic painting effect, hd, hdr, cinematic 4k wallpaper, 8k, ultra detailed, high resolution, artstation trending on artstation in the style of Albert Dros glowing rich colors powerful imagery nasa footage drone footage drone photography"

Play this game here: https://doodleai.darraghoriordan.com/?gamedate=2022-11-28T12:00:00.000Z

With this prompt generation method I split the entities into some custom groupings and then select a random prompt from a list of prompts suitable for my pseudo-categories. e.g. I make sure Persons are used for portraits and Locations are used for landscapes.

Azure NLP categorisation is pretty good. There is a confidence and often a sub category included for a single entity.

These are the main categories doodle:ai uses.

Person - Names of people.
PersonType - Job types or roles held by a person
Skill - Skills or abilities
Event -Natural - Naturally occurring events.
Event -Sports - Sporting events.
Location - GPE - Geographic and natural features such as rivers, oceans
Location - Structural - Manmade structures. e.g. ships
Organization - Medical
Organization - Stock exchange
Organization - Sports

I link these together to make prompts so things like

Person + Location - GPE Event - Sports + Organisation - Sports

const removeUnwantedCategories = nlpEntities.filter(
  (x) => !unwantedCategories.has(x.category)
)

const geoLocationEntities = removeUnwantedCategories.filter(
  (x) => x.category === 'Location' && x.subCategory === 'GPE'
)
const manMadeEntities = removeUnwantedCategories.filter(
  (x) => x.category === 'Location' && x.subCategory === 'Structural'
)
// more of these
// ...
// then map them to entity combinations and random prompts
const manMadeAtLocations = this.createPrimaryWithRandomSecondaryPrompts(
  manMadeEntities,
  geoLocationEntities,
  locationPromptTemplates
)

//
const generatedPrompts: string[] = [
  ...manMadeAtLocations,
  // and many more of these
  // ...
]
  .sort(() => Math.random() - 0.5)
  .slice(0, requiredNumberOfPrompts) // then grab just 4 and return

return generatedPrompts

Linking entities seems to result in better output so I will likely stick with this method and focus on improving the actual prompts.

Unfortunately, there is no way for me to link things like “person” and “location” accurately in the context of the topic so sometimes there are very weird results due to my random process. e.g. I might end up with “Ghandi, England” where instead “Ghandi, India” would result in a more coherent image.

Another issue is that sometimes the adjacent articles are too far removed from the main topic to be useful but I have no way to tell relevancy or adjacency as a score at the moment. So I randomise the prompts and hope for the best.

I’ll keep tweaking this generation process. It’s fun to see what works and what doesn’t.

There is an incredible amount of randomness in this process.

The topic is chosen at random for a given date
Entities for my pseudo-categories are chosen at random
The order of the combined entities in prompts are random
I create 30-40 prompts per game and choose only 4 of them
Stable diffusion is passed a random seed for each prompt

I have no idea what the output will be until I see it, and I can never recreate a game.

Stable Diffusion

I’m not doing anything very special with Stable Diffusion for Doodle:ai. There are heaps of great articles on how to setup and run stable diffusion on a PC.

One thing I did do is removing the nsfw filter because it’s too eager.

One difficult thing was running the conda environment required for python from a web server.

First I had to wrap the nodeJS spawn call in a Promise.

import { spawn } from 'child_process'

export const spawnPromise = (
  command: string,
  commandArguments: string[],
  cwd: string,
  shell?: boolean
): Promise<string> => {
  return new Promise((resolve, reject) => {
    try {
      // console.log('spawnPromise', cmd, args, cwd, shell)
      const runCommand = spawn(command, commandArguments, {
        cwd,
        shell,
        timeout: 500_000,
        stdio: 'inherit',
        // stdio: ["pipe", "pipe", "inherit"],
      })
      console.log('runcmd connected state:', runCommand.connected)
      console.log('runcmd pid state:', runCommand.pid)
      runCommand?.stdout?.on('data', (data) =>
        // eslint-disable-next-line @typescript-eslint/no-unsafe-argument, @typescript-eslint/no-unsafe-member-access, @typescript-eslint/no-unsafe-call
        console.log(data?.toString() || '')
      )
      runCommand.on('exit', (code) => {
        console.log('spawnPromise exit', code)
        if (code !== 0) {
          reject(new Error(`Command failed with code ${code || 'UNKNOWN'}`))
        }
        resolve('')
      })
      runCommand.on('error', (error) => {
        console.log('spawn error', error)
        throw new Error(error.message)
      })
    } catch (error) {
      reject(error)
    }
  })
}

Then I call the txt2img script combined with conda rather than launching conda as an environment first.

import { spawnPromise } from './SpawnPromise'
import { CoreLoggerService } from '@darraghor/nest-backend-libs'
import { Injectable } from '@nestjs/common'
import os from 'os'
import path from 'path'
import { execSDPromise } from './ExecPromise'
@Injectable()
class StableDiffusionRunner {
  constructor(private readonly logger: CoreLoggerService) {}
  // eslint-disable-next-line @typescript-eslint/require-await
  async run(prompt: string, puzzleDate: Date) {
    this.logger.log('running sdiff', prompt)
    // create random integer seed as string
    const seed = Math.floor(Math.random() * 1_000_000_000).toString()
    return execSDPromise(
      'conda',
      [
        'run',
        '--no-capture-output',
        '-n',
        'ldo',
        'python',
        '-u',
        'optimizedSD/optimized_txt2img.py',
        '--prompt',
        `"${prompt}"`,
        '--n_iter',
        '1',
        '--n_samples',
        '1',
        '--H',
        '512',
        '--W',
        '512',
        '--turbo',
        '--seed',
        seed,

        '--ddim_steps',
        '50',
        '--skip_grid',
        '--outdir',
        `./output/${puzzleDate.getUTCFullYear()}/${
          puzzleDate.getUTCMonth() + 1
        }/${puzzleDate.getUTCDate()}`,
      ],
      path.join(
        os.homedir(),
        'projects',
        'personal-projects',
        'stable-diffusion'
      ),
      '/bin/bash'
    )
  }
}

export default StableDiffusionRunner

I do occasionally run out of memory when generating images if I’m doing other work on the PC - not VRAM, just regular RAM.

I “only” have 16GB. I guess I should have gone for 32GB! If I run generation while I’m not using the PC it works for hours no problem.

The game engine / Frontend Stack

The frontend is based on a stack I use for all my projects.

React
Tailwind CSS
React Query
React Router
Vite
Vitest

All data for the current game is stored in a React Query cache and Local Storage. This means that the game state is persisted across sessions.

There isn’t much technical complexity here. React Query takes most of that away. What a champ of a library!

The frontend for doodle:ai is hosted on Netlify.

I also have an instance of Vercel’s og-image-generator running on vercel for this project.

This allows me to create an og-image using the first image for a puzzle for a given day.

The Backend Server

The backend host is simply a Digital Ocean Spaces instance. Spaces are Digital Ocean’s blob store that matches AWS S3 APIs (you can use AWS clients to work with Spaces).

I have a CDN in front of the game files to increase performance and reduce cost.

The backend for generating puzzles is actually a NestJS web application. It might seem strange to run this from a web app instead of a CLI tool but I have a bunch of prebuilt libraries for NestJS that make it very, very fast for me to develop new projects with.

Because stable diffusion requires an NVidia GPU, I run the puzzle generation on my gaming PC with 16GB RAM, an SSD and an Nvidia 2080ti 11GB VRAM.

It takes 2-3 minutes to create a single puzzle. This means around 18 hours for a year of puzzles. Because I’m tweaking the algorithms I only generate a few weeks in advance at the moment. I’ll probably generate a year in advance once I’m happy with the algorithms.

Optimised stable diffusion models could probably do a puzzle in 30 seconds and this is something I’m investigating. It would be easy to move the workers (NestJS application) to a cloud GPU instance but these GPU-enabled Cloud VMs are quite pricey.

Having a server is also useful for cron jobs associated with the game’s Twitter account - @PuzzleAI.

For this role the server is running on the same $5 Digital Ocean droplet that I use for all my projects.

The generation APIs are not exposed on this server (no endpoints are public). So it just sits there running cron.

There is a standard conda environment on my machine which has all the dependencies for image generation using stable diffusion.

Summary

I’m really happy with how Doodle:ai has turned out. It’s been a lot of fun to work on and I’ve learned a lot.

The game stayed on the front page of Hacker News for a full day and I’ve had a lot of positive feedback from people who have played it.

The game provides a nice platform, and a reason for me to keep learning and improving my knowledge of tools like Stable Diffusion.

Play the game at doodleai.darraghoriordan.com and let me know if you like it!

Easiest way to optimise images for web

Darragh O'Riordan — Fri, 25 Nov 2022 17:12:33 +0000

Here is how I optimise all pngs and jpgs in a folder for publishing to the web.

To get the tools install with brew or whatever you use.

brew install imagemagick optipng

Then run the following commands to produce optimised images for web sites. This will often reduce the size of the image by 50% or more with no obvious loss of quality for web.

# for jpgs
mogrify -strip -interlace Plane -sampling-factor 4:2:0 -quality 85% *.jpg

# For pngs and bitmaps
optipng *

DORA metric monitoring for your team in just 15 minutes with Apache Dev Lake

Darragh O'Riordan — Sun, 16 Oct 2022 17:12:33 +0000

DORA (DevOps Research and Assessment) metrics are an excellent way for engineering organisations to measure and improve their performance.

Up until now, monitoring the DORA metrics across Github, Jira, Azure Devops etc required custom tooling or a tedious manual process.

With Apache Dev Lake you can get beautiful reporting for DORA metrics on your local machine in as little as 15 minutes (honestly!).

From Google Sheets to Grafana

DORA Metrics

DORA metrics are 4 dimensions for improving your engineering team. The metrics came out of a report by Google in 2020. They’re designed so they balance velocity and stability.

Deployment Frequency
Lead time for changes
Median Time to Restore Service
Change Failure Rate

These metrics are explained in detail in books (here) and blogs (here) so I won’t go in to any detail on what they are here.

I have a note at the end of this article on updates to DORA since 2020 and the newer Microsoft SPACE framework.

The old way of tracking DORA metrics

Most teams will have various support systems like Github, Azure DevOps, Jira etc.

Extracting the metrics required for DORA across so many developer tools is difficult to automate without custom software. The time cost is too much, especially for small teams.

When I first started tracking DORA I used a spreadsheet and did everything manually.

It wasn’t pretty and it was a lot of work but I was able to see a noticeable increase in deployed work when deployment frequency increased.

Custom spreadsheet of DORA metrics

Tracking DORA metrics in 2022: Apache Dev Lake

Apache Dev Lake is a project from the Apache Foundation specifically for tracking engineering performance metrics.

The project is new and it is in the incubation stage at the Apache Foundation.

Dev Lake is a set of services in containers - a database, a data pipeline, a grafana instance with pre-built reports and a configuration web application.

The dev lake team provide helm charts and docker compose files for easy setup of the infrastructure.

The benefits of tracking your DORA metrics in Dev Lake are:

It connects to most major platforms out of the box (Jira, Github, DevOps)
Comes with pre built Grafana charts for DORA and many other metrics
Free and you can easily run it locally to get started
Your data or your client’s data never leaves your organisation

Having the data processed and stored locally is a huge benefit for many organisations. It means you don’t have to add an additional supplier to get started with DORA metrics.

Visit the dev lake site for more details.

Setting up a local instance of Apache Dev Lake

You should have docker desktop installed before continuing.

Get the docker compose and .env files from the releases page - https://github.com/apache/incubator-devlake/releases/latest and place them in a folder
Open a terminal in that folder and mv .env.example .env
docker compose up
That’s it! Wait for the 4 services to start
You should be able to access the configuration site at https://localhost:4000

The Dev Lake homepage

There are some sites already configured on the instance in the image above. Yours will be empty for now.

Now you can add any connections to your engineering tools like Jira or Github.

Adding connections to common dev tools

Github is very popular so let’s use that to demonstrate setting up a tool connection.

Click on Data Connections > Github
Give the connection a name you’ll remember
The endpoint is https://api.github.com/ unless you have a special enterprise account - then you’ll need to use your own endpoint
Get a personal access token from Github and paste it in the token field (https://github.com/settings/tokens)

The PAT permissions you need are:

repo:status
repo_deployment
public_repo
repo:invite
read:user
read:org

I set the github rate limit to 5000 requests per hour but you can leave that as default if you like.

Adding a Github connection

Configuring projects in Dev Lake

Now that you have a connection to your dev tools you can configure projects. In Dev Lake a project is called a “Blueprint”.

I’ll add one of my open source projects (https://github.com/darraghoriordan/eslint-plugin-nestjs-typed) to show how it works.

Click on Blueprints > Add Blueprint in the main left-hand menu
Give it a name and select the Github connection you created earlier

Add blueprint screen - summary

Click next and enter the name of the repo(s) associated with your project

Add blueprint screen - repository

Click next and on the summary screen click on the “add transformation” link. You have to tell Dev Lake how to identify deploys and incidents for your setup.

Add blueprint screen - optional transformation

On the add transformation screen you have to configure the tag you use on a Github issue that identifies an incident and the Github actions Job that identifies a deploy.

Add blueprint screen - adding transformation

For the deployment section make sure that you’re getting the Job name and not the workflow name.

To get the job name on Github, click into an old run of your workflow and you’ll see the job name on the left hand side.

Github actions settings for transformation

Click on “save and run now” and you should be done configuring Dev Lake. Wait for the blueprint data collection run to complete.

You can open the Blueprints page anytime and click on the graph “squiggle” icon to open the blueprint overview for your project.

Eslint for NestJs Blueprint overview

The status overview allows you to see the status of the data pipeline for your project.

Configuring Sync Settings for Local

Because this is a local instance you can go to settings and change the sync frequency to “Manual”.

This isn’t required but makes sense for local instances.

Then you can use the blueprint status page and the “Run Now” button to manually get the latest data when you need to.

Viewing the DORA metrics

Ok so now that you have all the data downloaded, filtered and transformed into the DORA metrics you can view them in Grafana.

Click on the “Dashboard” menu button to open Grafana.

Then click on the magnifying glass icon in the left-hand menu to open the search bar.

Search for “DORA” and you should see a dashboard called “DORA”.

DORA dashboard on Grafana

The image is the metrics for my small open source project. I update it once a month maybe. The users do report issues occasionally so I guess now I’ll have to fix them in a bit quicker!

You can filter to a specific Github Repo if you added more than one.

Along with DORA there are 10-20 other pre-built reports you can access. Here’s one for Github stats.

Github engineering report on Grafana

SaaS Platforms for DORA Metrics

It’s important to mention that there are some excellent platforms for tracking DORA and more metrics if you want more mature, managed applications with support.

Multitudes

Multitudes tracks DORA but supplements engineering metrics with around well-being and collaboration.

More details on the multitudes site.

Multitudes marketing page

Linear B

LinearB tracks DORA metrics and helps you fix workflow issues.

More details on the LinearB site.

LinearB marketing page

Developments in Engineering and DevOps Metrics since DORA 2020

There have been developments since these metrics were first published in 2020 and it’s a rapidly changing, exciting space.

The DevOps report of 2022 changed the categorisation to “Low, Medium, High”, removing the “elite” name.

They also added a 5th dimension to DORA for reliability.

Microsoft released their SPACE metrics paper in 2021 and arguably they are much more human focused which is great.

It’s worth reading the microsoft docs for SPACE and the Future of Work Report - https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/

Conclusion

It’s easier than ever to track your DORA metrics. You can use a local instance of Apache Dev Lake or a SaaS platform like Multitudes or LinearB.

If you’re interested in tracking DORA metrics in your organisation, I’d recommend starting with a local instance of Dev Lake. It’s free and you can get started in minutes.

Like anything, don’t over do it with these metrics. They’re just a tool to help you improve your engineering practices. They shouldn’t be used as a KPI.

A summary of NDC Sydney 2022 Developer Conference

Darragh O'Riordan — Sat, 15 Oct 2022 17:12:33 +0000

NDC Sydney is a developer conference in Sydney. This was my first in-person conference in 3 years because of Covid!

I try to take notes for each of the talks and here they are! I’ll try to add the relevant videos to talks if they’re released.

Remember that these notes are just my notes. They’re paraphrased and summarised, the speakers would have provided must more clarity and went into more detail!

Day 1

Securing the future

Laura Bell - CEO of Safestack

The tools and supply chain that we’re building into all our products are getting very complex. The worst future attacks are likely to be through supply chain rather than direct.
OWASP hasn’t changed too much since 2001 considering how our tooling and processes have changed
Mstic - Data relations and connection analysis software - https://github.com/microsoft/mstic
https://Safestack.io/free-plan

Mayday - Lessons from aviation disasters

Adele Carpenter - Engineer at Trifork Amsterdam

Aviation can’t afford mistakes especially repeated mistakes
The root cause analysis they do is insane
Open communication for the entire software development process is critical

Building inclusion into orgs

Sammy Herbert - Head of Dev Exp at SixPivot

I didn’t take too many notes here because I was listening! Possibly the best talk of the first day!

AFR “best places to work” is the best award to do if you’re going to do one
18% of Australian sw engineers are women
50% of women leave engineering by 35
Consider having readmes for people, list the important questions first!
A big issue is glue work, Management (and teams)need to value glue work

Day 2

Rocking as an azure architect

Adam Cogan - Chief Architect SSW

Heavy use of diagrams - use the architecture center as a starting point - https://learn.microsoft.com/en-us/azure/architecture/
There is a developer cli for setting up local env. Use it to save time.
Use IaC - bicep - there’s an extension for vs code and visual studio now.
Be concious of cloud cost from the start - watch out for the login on the pricing calculator on azure, it’s the weird link in the middle of the page
Prepaying for a year saves 30%, 2 years could save 50%
Setup cost analysis and alerts - billing menu > cost analysis - avoid “denial of wallet” attack. You must create budgets.
Review the client’s existing on prem cost using the TCO calculator - https://azure.microsoft.com/en-au/pricing/tco/calculator/
Learning for anyone - Good pdf here: https://azure.microsoft.com/en-us/resources/whitepapers/developer-guide-to-azure/
Must have someone actually responsible for the cloud architecture (not all the team, have someone accountable)

Delivering the worlds first flying car race

Dom Raniszewski - Principal Consultant at Telstra Purple

Lesson 1 - you can’t force your excitement on others
Lesson 2 - you should give people space to deal with uncertainty
Fixed and growth mindsets - https://fs.blog/carol-dweck-mindset/
To deal with a very uncertain project you should at least instil and encourage these three traits in the team - courage, commitment and passion

Overdue: When Agile and DevOps meet real world deadlines

Brenton Race - CTO at Blueshift

Steve Mconnell - more effective agile book - https://www.amazon.com/More-Effective-Agile-Roadmap-Software/dp/1733518215
There are agile boundaries in your org - Agile likely stops at product development teams. VPs etc are not working in agile.
“understand the context that you’re in, and only apply principals that make sense in that context”
Structure team(s) to maximise focus - reduce communication overhead - isolate issue types e.g. you can create shield teams (support team that protect the teams doing feature work) - Dar note: Seen this before, can lead to resentment
Expect change, use it to readjust to maximise business value.
Track and communicate progress to trigger change conversations as early as possible
Clarify what you’re being asked for - is the question “can you estimate this please?” or is it “can you come up with a plan to deliver an outcome by a certain date?”
“set aside time for team and INDIVIDUALS to improve”
software estimation book - steve mconnell - https://www.amazon.com.au/Software-Estimation-Demystifying-Developer-Practices-ebook/dp/B00JDMPOVQ
https://brentonrace.com

Next decade of software

Richard Campbell - Consultant, podcaster, author

We’re literally running out of physical atoms (physical space) to improve processing power , so we’re going to have to get more efficient.
It hasn’t been an issue for developers for a while, but with the way the economy is going developers should make sure that they understand how their company makes money, and if the role they are doing is helping that. If you’re on a project that doesn’t affect revenue, change your job or your company.
You can probably bet on the browser lasting as a secure runtime environment. Most people trust it, the tools are getting better. WASM opens the browser as a runtime to all languages.
Rapid application development tools (Nocode and low-code) will enable subject matter experts to create software. This is a fantastic outcome. We will have to write software to allow these types of apps to integrate with our and other apps.
AI will become more ubiquitous and will enhance our tooling. It probably wont replace us, but will allow us to provide more value to customers
The always with you compute device (phone) has clearly won, we are cyborgs but it’s an external augmentation. it’s likely that AR will enhance or replace this. There are devices out there but they’re not in consumer space yet. Hololens costs USD $3500 and around $1000 / month in IoT costs. If you compare this to blackberry’s in 1999 it’s a similar trajectory where blackberrys and their portable email were “industrial” or business-only devices, not consumer.

The Visible Developer: Why You Shouldn’t Blend In

Heather Downing - Principal Dev Advocate at Okta

Figure out what you want from the career
Do things and tell people - help people outside of your team or job - write the things you do down somewhere so that you remember at the performance review type thing you do in your company.
Working hard and doing a good a job in your current job is insufficient
Externalise you thought leadership - You should share your knowledge outside of you organisation. Leading by example externally demonstrates your ability (internally also!).
You should network! - Give first to as many people as you can. We’re a big-small industry.
Network in person, “your network is your net-worth”.
Be your honest self on social media - it’s ok to share some frustrations and things - it makes you relatable. “Post with purpose” - before posting social media think “what purpose does this provide” - “what will this do for my personal brand” e.g. if you’re super interested in a topic you should post about it!
Speak like a human :D - e.g. don’t always shit-post - “is this a better topic for your therapist or the internet?”
Be moderately responsive - to grow a personal brand you do need to respond occasionally
If you’re not online, you’re out of sight
Visibility is good but consistency is great - post regularly, schedule posts if you can
make sure you have a dedicated website dedicated to your works - if you don’t like designing hire someone who can! Wix or squarespace is better because you can just pay and let a good designer work directly on it.
Attitude matters - skills can be taught but attitude can’t. Life isn’t “fair”. Some things are out of your control.

Habits of well known developers …

They pay attention to what others value, identify the goals of the people around you in the organisation. Show interest, learn about that, go above your paygrade, ask questions above your paygrade, how does your company make money! Take action on an important thing without being asked first.
Choose the right place for YOUR values. “If your friend / child / anyone you care about were looking for a job, would you recommend where you currently work to them?”. if not, why are you there?
Well known devs focus on the big picture - is what you’re working on important?
They are comfortable being uncomfortable - you have to be able to work on things that are new to you or stretching your capabilities. if you’re not deliberately stretching your capabilities you might get stuck in a rut.
They like their colleagues. Even the jerks in the company. They go out of their way to put the human first, if you don’t like someone build trust anyway. Everyone needs to feel valued, even the jerks. “Empathy is being concerned about the human, not just their output”. Figure out why the jerk says what they say and does what they do
The know how to become a trusted voice? Questions and listen. speak little, listen much and apply what you hear. Be positive and respectful.

Day 3

Why the web is this way

Steve Sanderson - Chief Architect at Microsoft

This was a great history of web dev langs, browsers, frameworks. I didn’t take notes because I’m old enough to have used everything he showed 👴. It was cool that he actually ran all the old bits of software!

Building Credibility with crowd natives

Shane Baldacchino - Chief Architect at Microsoft

What are cloud native orgs? - they move fast, they fail fast, they know their stuff (hence the high salaries)
They use data and output to feed innovation - Challenger Vs Champion for models - https://www.ibm.com/docs/es/sc-and-ds/8.0.0?topic=steps-champion-challenger-overview
Architectures are always changing - At the moment the landscape is changing faster than it ever has before. Being able to iterate and change is the only way to stay competitive.
(Stack overflow dev survey 2022) 62% of developers are in the industry 9 years or less. They don’t care about VMs. They don’t care about DNS, memory etc. They just want an endpoint and they want the cloud to handle the rest.
Everything is becoming full stack
Open source is the present and the future.
Architectures you have to be across: search indexes (lucerne), event streaming, event driven architecture
Use a workflow management platform for stateful micro services - e.g. apache airflow
Machine learning - Know it, or at least understand the difference between training and inference
L7 routing is the new norm - short lived compute, ephemeral ports, we can’t have traditional networking. envoy, istio, sidecars, service discovery.
“Culture eats tech choices for breakfast” - conway’s law, crazy company = crazy code. centralized command and control wont work any more - centralised operations is not conducive to speed. centralised operations increases business risk. centralised operations encourages a hero mentality.
if you have complex systems things will fail and they will fail differently all the time. Decentralise the operations of technology and bring it closer to the creators (and make them accountable for running their environment, and the cost of their environment)
no change freezes - it’s a sign of immaturity, deploying more frequently reduces risk. Your best competitors do not stop releasing.

How to say no (and still influence people)

Sasha Hilton - National Head of Digital at Telstra Purple

The context - you get an unwelcome request

Common concerns about saying no

people think I’m lazy, people think I’m incompetent, someone gets upset, the requester has more power than me, could i get fired?
conflict avoidance
perfectionism
power imbalances - there are real and imagined power imbalances, but if you feel like there is one, it will affect you anyway.

Benefits of saying no

your time is precious, learning is a gift you can give to other people (by not doing the work yourself), you’re reducing the risk of burnout

How to get better

Read Never Split the Difference - Chris Voss for expert tactics - https://www.amazon.com/Never-Split-Difference-Negotiating-Depended/dp/0062407805
Build trust, ask questions, build rapport, negotiate.
be pre-emptive about saying no e.g. use a contract.
set boundaries - have a mantra (“turning the ring” ) , keep a resentment channel for yourself - write down what makes you resentful.
practice saying no, get used to saying it.
Fawning - fight, flight, freeze and … fawn - the please and appease response
Ways to create calm - (mindfulness helps) notice the feeling, label it
engage your physical sense “3 things you can see, 2 things you can touch, 1 thing you can smell”
engage you pre-frontal cortex, count backwards from 1013 * 12 or something that makes you think
engage your motor cortex - wiggle your toes, write some thing down
parents have some creative ways of saying no - park things in the future, blame it on a higher power, empathise but ignore request
“that’s just not possible”
“let me know if i should reshuffle priorities”
“i’m not comfortable with that sorry”
“as a rule, i don’t…”
“sorry ths isn’t the answer you’re looking for”
“i can’t right now but i could next month/after the project”

Empowering Minds of All Kinds: Neurodiversity in Tech

Ryan Oakly

The proposed stats around things like autism and aspergers should be thought of as a minimum because there is so much under-diagnosis, especially among older folks that wouldn’t even think about different being a condition (or don’ ask, don’t tell)

ally is a verb not a noun
some people will never want to tell you their condition, people are are afraid of judgement

Environment

Have areas of low light
Have areas in co-working spaces that are slightly separated or blocked off
have quieter spaces
allow use of headphones or ear muffs
be flexible around preferred work areas

Job design

give the option to provide written or verbal instructions
give detailed instructions if desired
adjust job design to remove tasks people struggle with
adjust recruitment e.g. “do you want me to leave you alone for 5 mins so you can do this task”
have policies and discussions
microsoft had specific recruitment strategy since 2016
you can’t decide who innovation comes from

The Art of Effective Feedback

Marcus Bristol - Engineering Manager at PushPay

Book - “Thanks for the feedback” - Douglas Stone and Sheila Heen - https://www.amazon.com/Thanks-Feedback-Science-Receiving-Well/dp/0143127136
feedback is a core ingredient for satisfaction in life

Three common types

Tell and sell - “sports and coaching”
Tell and listen - “this is what i’ve been seeing, why it’s not , tell me about it”
Problem solving - “coaching, steer the convo but let the person figure it out”

Giving feedback

You have to care, and show it!
Evaluate behaviours, not the person
Everyone is different so adjust feedback and mechanism to the person
Understand motivations
Listen and empathise
Give positive feedback / positive reinforcement regularly too - feedback isn’t just for bad behaviour
Provide examples of good behaviour
toastmasters is a great place to learn feedback

Summary

It was great being at a conference again. A nice mix of talks, and a good mix of people. I’m looking forward to next year already.

4 Surprising uses for GitHub as a cloud datastore

Darragh O'Riordan — Fri, 14 Oct 2022 17:12:33 +0000

GitHub is a great place to store code. But it’s also a great place to store any data that you need backed up, versioned, accessible from anywhere and easy to use.

In this post I’ll show you some surprising uses for GitHub as a datastore like…

as a comments system for a website
as an operating system configuration management tool
as a datastore for personal data management
as a software update delivery platform  ## Is it a good idea to use GitHub as a datastore?

The tools I’m going to show you are all free and are built in a way to use Guthub as their primary datastore.

This is a “parasitic” use of Github and the usage could be banned in the future but they seem to be supportive of these kinds of uses as long as you don’t abuse them too much.

There are examples of accounts that have heavily used the github apis but because they provide utility to the tech community they are generally left alone.

Check out this post about how Github worked with CocoaPods when they were maxing out 5 CPUs on a server and using terabytes of Github’s data: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 - the Github team went out of their way to help them!

Limitations of GitHub as a datastore

One of the only limitations in using GitHub as a datastore is the file size limitation. The maximum file size is 100MB. This is fine for most use cases. But if you need to store large files you can use a service like S3 or Azure Blob Storage.

The seconds major limitation is the api request limit. On Github you’re limited to 1,000 requests per hour per repository. Personal tools are unlikely to hit this limit. But if you’re building a tool that is used by thousands of people Github may not be suitable.

Github as a comments system for a website

This blog uses utterences as a comments system. You can see it if you scroll right down to the bottom of the page (I prefer if people use twitter so I hide it).

Showing comments on my blog

Utterences is a free and open source comments system that uses GitHub issues as its backend. That’s right, Github issues! Quite clever use of that api and you don’t have to pay for a service like Disqus.

Showing comments on github

You can see exactly how I set utterences and github up on my Gatsby blog in this post: How to add utterences to your gatsby blog

In the time since utterences was built, Github added a community product called “Discussions” with a full api. So someone took utterences and made it work on the discussions API.

This is probably a much better api to use for comments on a blog. So if you’re thinking about using Github as a blog comments system today, use Giscus.

Get Giscus here.

Github as an operating system configuration management tool

If you’re a mac or linux user you have “dotfiles” that are used to configure how your operating system works for you. The dotfiles live in your home directory and they’re called dotfiles because they usually start with a ..

A file or folder that starts with a . is usually hidden by default in apps like Finder on Mac or ls in a terminal. This is handy because you don’t want to see all the files that are used to configure your operating system. They don’t change very often and would clutter up your view.

Use Command + Shift + . to show hidden files in Finder on Mac or ls -a on cli.

Showing hidden files

Now if all of your configuration for how the computer works, looks and feels to use are stored in files - what happens if you need to reinstall your operating system? Or if you have 2 computers and you want to sync your configuration between them? You’ll lose all your configuration and have to start from scratch.

This is where dotfiles come in. You can store your configuration in a git repository and then you can sync it between computers. You can also use it to backup your configuration and restore it if you need to reinstall your operating system.

Homeshick is a tool that uses Github as a datastore for your dotfiles. You can make a change to dotfiles on one system, commit it to Github and then homeshick will periodically check for changes and sync them to your other systems.

Showing homeshick in action

You can see how I configure homeshick and github in this post: Consistent modern shell tooling on MacOS and Windows WSL

Github as a datastore for personal data management

If you’re a developer and you take notes you might be interested in Obsidian - It’s like OneNote or Evernote but it’s based on markdown files. Obsidian has really nice tagging and linking between notes built in.

Showing obsidian

Obsidian has a healthy plugin ecosystem and heaps of functionality like automatic importing of kindle notes, a trello board replacement, daily journaling helpers and more.

Showing obsidian

Obsidian is available for Mac, iOS, Windows and Android. To keep your markdown notes synchronised across all of your devices you need a service. Obsidian themselves provide a paid synch-service but there is an Obsidian Git plugin that uses Github as a datastore!

You can read more details about how I configure and use Obsidian on Github in this post: Obsidian - A free personal data management system on windows, mac and mobile

Github as a software update delivery platform

I’ve recently created a tool called Local Dev Tools. The application is a toolkit for developers so they don’t have to paste things like JWTs or JSON into random websites to convert or decode them.

Local Dev Tools my first native application in maybe 10 years and expectations for applications have changed quite a bit since the last time I compiled a native app.

Customers expect automatic updates to be applied when they run the app. The infrastructure required to host and deliver updates is a lot of work!

Update functionality is provided by the app stores for android, windows and mac but if you’re in the stores then you’re limited in monetization strategies, you have to adhere to their policies and have to use their store fronts.

While I’m also deploying the apps to the various app stores, I decided to use Github as a datastore for my updates for anyone who purchases the application from Gumroad.

If you use Github actions you can create a workflow that will build your application and upload it to Github as a release. Then you can use the Github API to check for new releases and download them.

Nuts is a simple tool that monitors a Github release stream for updates and turns it into an RSS file that’s compatible with electron updater (and squirrel delivery).

Showing nuts in action

So once you have semantic releases publishing your assets on Github you can run a nuts server and use it to deliver updates to your application.

You can get nuts at https://github.com/GitbookIO/nuts

Not surprising but worth mentioning - Github as a host for your personal website

Just in case you haven’t heard of it. Github pages is a product provided by Github that allows you to host a static website for free as an alternative to Netlify. You can use it to host a blog, a portfolio or a landing page for your business.

The benefits are a very close integration with Github and the ability to use Github actions to build and deploy your website.

Check out their product page here: https://pages.github.com/

You can deploy Gatsby sites to github pages

https://www.gatsbyjs.com/docs/how-to/previews-deploys-hosting/how-gatsby-works-with-github-pages/

Obsidian - A free personal data management system on windows, mac and mobile

Darragh O'Riordan — Thu, 13 Oct 2022 17:12:33 +0000

Obsidian is a free personal data management system based on markdown notes. Because it’s based on markdown it’s perfect for developers or other power users.

Obsidian has a healthy plugin ecosystem for extending the base functionality. There are plugins for mind mapping, kanban boards, importing amazon kindle highlights, daily notes and using git as a backend.

Obsidian has a powerful search engine that allows you to search for notes by their content, tags, and even the content of the notes they link to. This linking technology is what makes Obsidian so powerful.

Installing obsidian

You can grab obsidian mobile versions from the relevant app store for your device.

On PC or Mac you can download the desktop version from the obsidian website.

Setting up a GitHub Repo Backend

The first thing you should do is set up a GitHub repo as a backend for your notes. This will allow you to sync your notes across devices and also back them up to the cloud (Github).

Create a github repo on github itself. You will likely want this to be private. You can do this by clicking the + button in the top right of the github website and selecting New repository.

Obsidian desktop git plugin can use all the features of a GitHub client. But the mobile version is limited to http only. So you will need to create a personal access token to use with the mobile version.

Creating a personal access token

To create a personal access token go to your github settings and select Developer settings from the left hand menu. Then select Personal access tokens from the left hand menu. Then click Generate new token.

Copy the token and save it somewhere safe. You will need it later.

Adding a .gitignore file

Use the following .gitignore file to ignore the files that obsidian creates. Add this to the repo using your favorite git client.

# Dependency directories

node_modules/
jspm_packages/

# This file is used to keep track of last auto backup/pull

.obsidian-git-data

# exclude workspace cache

.obsidian/workspace

# plugins - we use different plugins on mobile and desktop so we don't want to sync them

.obsidian/plugins
.obsidian/community-plugins.json

# Exclude "bad" names

null\*

# Add below lines to exclude Mac stuff

.trash/
.DS_Store

Setting up the desktop version of Obsidian

Clone the repo you created earlier to your local machine using your usual git tool.

When you start Obsidian it will prompt you to set up the vault location. Set it to the repo you created.

Install the git plugin from the community plugins menu and configure it to use . as the repo location.

I use the following plugins on desktop

Advanced Tables
Calendar
Kanban
Git
Periodic Notes
Tasks
Mindmap
Kindle Highlights
Diagrams
Advanced Slides

Setting up the mobile version of Obsidian

Remember that it’s better to not check in the plugins folder into obsidian if you’re going to use it on mobile. Some plugins just don’t make any sense on a small screen device.

To configure Obsidian Git for a mobile device requires a bit of extra work. You need to create a personal access token on GitHub and then use that token to authenticate Obsidian Git.

Install obsidian from the app store of your mobile device
install the Obsidian Git plugin from the community plugin page
Configure the Obsidian Git plugin - Add your username and Github secret access key
Set a “Custom Base Path” folder for your git repo - remember this is the folder that will be synced to github
Now clone the repo - go to the actions menu > choose “clone”
Set the full GH url for repo (remember it must be the https address)
Set the sub directory you added as the Custom Base Path
Now change the settings for Obsidian Git plugin - set auto pull every 5 mins, add the device name (e.g. “Mobile”)to the “Commit message auto message” and “hostname” placeholders, turn ON “pull on startup”, you must add an author name and an author email.

Because my mobile Obsidian is primarily used for reading and taking notes I use the following plugins

Periodic notes
calendar
Ttasks
Kanban.

Conclusion

There are often merge conflicts on the mobile version of Obsidian. I’ve often had to delete and rebuild the whole data set because resolving merge conflicts on mobile is a pain. But it’s a small price to pay for the power (and cost) of Obsidian.

If you really enjoy the tool you can pay Obsidian for a first-class built in synchronization platform that they support and build directly in to the application.

It’s worth reading the docs for Obsidian to get the most out of it. Especially around linking. And take the time to explore all the commands in the command palette. It’s a powerful tool!

How add comments to your Gatsby site with utterances

Darragh O'Riordan — Sat, 01 Oct 2022 17:12:33 +0000

If you want to add comments to your blog quickly there is a nice tool called utterances that uses a GitHub Repo as the api.

Here is how I setup utterances for my gatsby blog in typescript.

Utterances

Utterances is a javascript tool that uses a GitHub repo as the store and api. This makes it ideal for static sites.

People will have to log in to their GitHub accounts to post a comment so spam shouldn’t be too much of an issue. This does mean that Utterances is mostly suited to a site with a developer audience.

Because data is stored in a GitHub repo, you actually own it! This is much nicer than tools like Disqus.

Setup on Github

You have to have a public GitHub repo. My blog content is private so I set up a specific repo for blog comments.

The you install the app on that repo by visiting

https://github.com/apps/utterances

Click “install” and then select your comments GitHub repository.

Adding the UI to your site

You have to inject a script where ever you want the control. I created a component for this. You can see how each of the settings work below.

import React, { useEffect } from 'react'
export const Comments = () => {
  const commentsInjectionRoot: React.RefObject<HTMLDivElement> =
    React.createRef()

  useEffect(() => {
    if (commentsInjectionRoot.current?.children.length === 0) {
      const scriptEl = document.createElement('script')
      scriptEl.setAttribute('src', 'https://utteranc.es/client.js')
      scriptEl.setAttribute('crossorigin', 'anonymous')
      scriptEl.setAttribute('async', 'true')
      scriptEl.setAttribute(
        'repo',
        'darraghoriordan/darragh-oriordan-com-comments'
      )
      scriptEl.setAttribute('issue-term', 'pathname')
      scriptEl.setAttribute('theme', 'github-light')
      commentsInjectionRoot.current?.appendChild(scriptEl)
    }
  }, [])

  return (
    <div className="container pt-8">
      <h1 className="mt-0 mb-0 text-3xl font-normal leading-normal">
        Comments
      </h1>
      <hr className="my-0" />
      <div ref={commentsInjectionRoot} />
    </div>
  )
}

Then where ever you want the comments add the component.

<Comments />

comments ui

Summary

Utterances is an easy way to add comments to a developer site and you own all the data so it’s low risk.

Give it a try today! - https://utteranc.es/

How to use SSH with Git and ssh-agent on Windows

Darragh O'Riordan — Sat, 24 Sep 2022 17:12:33 +0000

I needed to run git natively in windows (no wsl) for a recent project. I use ssh certificates with passphrases to authenticate with my git provider.

Ssh requires the certificate passphrase every time you use a connection. It's annoying typing this passphrase in to terminal when using a git command.

The Problem

On most *nix systems there is an ssh-agent installed that will store your pass phrases so you don't have to enter them when using Git with ssh.

Ssh-agent is harder to configure on windows because some of the default settings and paths are different to *nix systems.

I didn’t want to use Git for Windows because it uses GitBash. I couldn’t use WSL because I wanted git to work on any terminal in windows.

These are the steps I had to research to use Git on Windows with the built in Windows ssh-agent.

Note: You must be an administrator to perform the required actions.

Open ssl on Windows

If you use Windows 10 or higher there is a built-in openssl instance. You can turn it on in the Optional Features settings pane.

Microsoft provide more instructions here: https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse?tabs=gui

Follow the instructions to install it if you don’t have it.

A note on certificates

I’ll assume that you have ssh certificates available and any ssh aliases are set in the config file

The default location for the config file on windows is

$HOME\.ssh\config

You should create that file if you need ssh aliases. You can read more about this in my article on ssh for git accounts - https://www.darraghoriordan.com/2021/05/04/configure-multiple-github-accounts-one-computer/

Enabling Ssh agent

Open a powershell terminal as administrator and run the following to have ssh-agent available.

# Have ssh agent start automatically
Get-Service ssh-agent | Set-Service -StartupType Automatic

# Start ssh agent now
Start-Service ssh-agent

# Should work successfully
Get-Service ssh-agent

Configure git to use Windows ssh

# tell git to use ssh.exe
git config --global core.sshCommand "'C:\Windows\System32\OpenSSH\ssh.exe'"

Load keys into ssh agent

Copy your keys into a folder that ssh-agent can access. Anywhere in the $HOME/.ssh should be ok.

Then add the key to ssh-agent. You will be prompted for a password and ssh agent will remember it for you.

ssh-add "C:\Users\darragh\.ssh\authorized_keys\darraghPersonalGithub"

Should you work for a consultancy or a product company?

Darragh O'Riordan — Sun, 11 Sep 2022 17:12:33 +0000

I've recently crossed over 2 years in a software consultancy. Previously I spent around 10 years working in product companies.

As you can imagine there are some differences between them and I wanted to write down some of the things I've noticed.

Note: These are 100% my thoughts, and don't represent any of my employer's, past or present! I'll hopefully update this with more thoughts in a couple of years as I learn more and my views inevitably change.

Types of Consulting

First it's important to mention that there are a few different types of software consulting:

Freelancers - who often work alone and find their own work.
Augmentation - where a person or team might join an existing team for a short time to help accelerate some development
Bespoke product development - where a team will identify product-market fit or problem-solution fit and build the solution

In this article I'm talking about a bespoke product development consultancy because that's what I do!

In a product development consultancy you usually work on a team with designers, product experts and engineers of various specialities. Team size is typically 5–10. You might sometimes do work by yourself but there are better outcomes from teams so that's what we prefer.

You will usually work on projects, i.e. a specific goal and end date. The projects tend to be one of two main categories

iterating on new problems quickly to find product-market fit or problem-solution fit
upgrading an existing system because it can't be changed quickly and safely in its current state.

Using New Tech

In my consultancy work I've used leading-edge tech, especially the greenfield projects. Even if you're working with legacy code in consulting, it's usually because you're transforming the legacy solution to a more modern stack.

Notice that I'm deliberately not saying "bleeding-edge" because we usually don't want to use unstable, un-tested technology in client solutions.

I don't want to name any specific tech examples because they will be out of date in 5 minutes but we're often already using items in the "Adopt" ring of the thoughtworks tech radars - https://www.thoughtworks.com/radar.

In consulting work I've found that there's more appetite to select the best language or tool for the job even if it's not exactly what all the existing stack is built in.

New tools are sometimes slower to get adopted in product organisations because of the inertia in a large existing code base. However the advantage of working in a product org with a legacy product is that the decisions are often already made for you so you can just get started on solving problems.

All new features in an existing product will probably be built in the existing stack. Many companies have templates or "golden samples" for new services.

Rates of learning

As a consultant you get very comfortable with learning quickly. Not only learning technologies but every project is a new team, a new industry and a new business strategy to internalise.

I'm not saying you don't have to learn a lot in a product company, but the breadth of "new" in every project when consulting is definitely more pronounced.

One of the primary benefits of always working on new things is that you learn to over come imposter syndrome very quickly. You know that you know nothing about this new business, but you've been in this situation many times before. You're used to the ambiguity, you've solved problems for other clients when starting from scratch previously and you know you can learn quickly.

To support the learning required consultancies will spend heavily on education and training

cloud certifications
MOOCs
access to conferences
hiring external experts to quickly train
investing in new tools like AI code generation to accelerate learning

A disadvantage to consulting on shorter engagements is that you don't get the depth of expertise in whatever industry segment and context that a product company might operate in.

Spending 4-5 years in a product org, solving a specific problem in a single industry gives you the time to try many approaches and produce comprehensive solutions across the whole problem space compared to a typical consulting engagement.

Variety in your work

There's so much variety in consulting! I've worked on maybe 8 projects in the last 2 years. The projects were in government, developer experience tools, construction, sporting tech, financial services, property tech, crypto and healthcare industries. My team has worked on many other projects in different industries that I've also had to have a passing knowledge of.

All of those industries have different user bases, markets, compliance standards and business models. You will have the opportunity to work on apps with millions of users and brand new apps looking for product market fit. Local companies and global companies. The possibilities in consulting are endless really.

I love learning about how different businesses work so this suits me well. I can't think of an advantage working at a product company might have here. (But ask me again in another 2 years time and I might be sick of jumping around so much!)

Similar to the rate of learning, working at a product company for a few years means that you'll have much more depth in whatever context your product is in compared to consulting.

Company Culture

Strong daily culture is an area where I think working at a product company is more advantageous.

In a product company you typically have long-lived teams working on products that they have long-term ownership over. The work they're doing is directly driving the goals of the organisation that they're part of, and so they feel closer to the mission and culture of that company.

In consulting you will spend a lot of your day working in your client's culture which takes you away from your org. In consulting your work directly drives the goals of the client's organisation. Of course your work does indirectly make the consultancy successful, but there is a gap there between the two.

This disjointed feeling and ambiguity as you move between the cultures on projects is noticeable to me. Missions and principles are difficult to maintain when most of your working day is in an organisation that might be quite different.

Consultancy companies are well aware of this and do the extra work compensate. Consultancies invest heavily in culture building. It has to be more deliberate and consistent compared to a product company.

For example, we do weekly catchups, weekly games in-office and online, team lunches, regular project check-ins, surveys, monthly events - you name it we probably do it to help retain our culture. Every consultancy works hard at this problem.

The Gnar have a nice article about their efforts here: https://dev.to/thegnarco/exploding-consulting-myths-culture-1n56
Thoughtworks are the content masters and have a whole blog here: https://www.thoughtworks.com/en-au/insights/culture
A few different leaders give their thoughts here: https://dynamicbusiness.com/featured/lets-talk-culture-2.html

The final point here is that for shorter consulting projects you miss out on the benefits of long-lived teams and the trust that forms when working together on problems for a long time. Not all consulting work is short term like this, but likely more than in a product organisation.

On the flip side there are huge benefits from observing other cultures as an outsider, you can be very objective and learn from the good aspects of another org's culture. But you also have to make sure you protect yourself from anything that's too divergent from your own.

If you're into missions then sometimes the client mission is more attractive than your consulting company! I've loved working on projects where I've helped people start new businesses or helped people get fitter. The client's devotion to solving these kinds of problems is infectious!

Ownership

I think most people who work in software feel a certain level of ownership in the work they do. Building software products is a craft rather than an assembly line. This sense of ownership and pride in my work is something I don't think I will ever be able to shake.

Whenever I built something in a product organisation I knew had to maintain it for as long as I stayed at the company. It was rewarding knowing that I would have ownership on the work for a long time.

Ownership is far more fleeting in consulting. While you're working on something you're 100% committed to it, but if you're successful you have to hand the work over to the client and that's the last you'll see it.

So I feel sad now when I have to hand the work back to the client after building something for 6 months. This is one of the things I miss the most about working in a product organisation!

Delivery focus

Software is expensive to build, it doesn't matter if you're consulting or in a product company.

Most good consulting companies will work closely with clients to prioritise and reduce scope up-front to avoid missing deadlines. But the client is usually paying directly for your time so there's more attention when things get delayed.

Because delivery time pressure is higher than in a product organisation you have to monitor mental health and burnout for yourself and everyone around you as a result. No software company can afford burnout in their teams so there is emphasis on well-being in any good software company.

I'm not saying that there are no deadlines in product companies, there definitely are! But there is usually revenue that the business generates that is not explicitly the time you spend on the work. Consulting is a different model and a different mentality.

A benefit of such delivery focus is that you get really great at identifying project risks. You also reduce waste wherever you can like subjective coding arguments. You develop strong pragmatism and you have an extreme focus on solving only the client's problem.

Responsibility and Self-reliance

Ok so this point will be very organisation dependent. But I feel that in the consulting work there is more responsibility and self-reliance expected of everyone, from a brand new developer to a senior architect.

You're always representing the consultancy to the customer so you can never ignore issues and you have to follow up on what ever you promise to do. The client is paying a high fee for your expertise so they have high expectations.

As a consultant there is an expectation that you're an expert at a given role and will produce high quality outcomes, there is little room for mistakes and rework. You'll often be in discussions with the client by yourself and you regularly have to advocate for outcomes that client's might not initially agree with.

Advocacy skills like this are priceless in your career and the sooner you learn them and the more you practice the better.

The equivalent at a product company might be regularity working with customers directly, fixing their problems and presenting it back to them for critique. There's definitely product company teams that do this, but I would say that in most cases it's not part of the job for a developer, at least in my experience.

The benefit in product companies is that you don't have to do this so much, it is stressful at times!

Summary

If you're a new developer the breadth of experience you get at a consultancy can really accelerate your career.

The exposure to multiple industries, tech stacks, market sizes and everything else will make you an excellent contributor to projects. You'll never have to worry if you can "hit the ground running" like you see in so many job adverts.

If you're an experienced product developer then working at a consultancy will give you a degree of freedom and responsibility that might surprise you. You'll never know what problem will be coming next and how exactly you'll lead a team to solve it - but it is up to you to get it done!

In a product company the ability to target a specific industry, a culture you admire or an interesting tech stack and work on that with focus for a few years is unique. It's not something you're likely to get at a consultancy. A product company is the best place to find stability, depth and focus like that.

I've enjoyed my time at product companies and my time in consulting. They are quite different as you'd expect. I'd happily do either again.

I hope some of the points in this article give you some insight into trying one or the other!

What are you're thoughts on consulting Vs product companies? - let me know on twitter - @darraghor!

Caching

Darragh O'Riordan — Thu, 08 Sep 2022 00:00:00 +0000

With most web applications you can drastically increase performance by using caching for data that’s frequently read across network boundaries. This lesson will explore some common caching techniques, you’ll learn how some common tools and libraries provide caching for us.

While caching helps with performance it can also cause some surprises and bugs in applications and i’ll discuss some of those too.

Database course index

This is part of a full course on persistence in postgres with typeorm and sql!

There is a github repo to go with this course. See part 2 for instructions.

Caching

If you look at data flow in an app you can see how latency is reduced when a cache is used. The example below shows how a read might be cached.

Caching is used outside of networked applications also. Computer processors make heavy use of caching for example. Its orders of magnitude faster to read from RAM Vs a hard drive, same improvement reading from a processor cache Vs computer RAM.

Caching will usually make the cost of your application infrastructure cheaper. You can keep most data on a cheap storage system and only put current data on a faster storage system.

Caching helps to smooth out unpredictable access patterns. The effect of even very short caching can be significant.

If you have something that is read heavy but changes often and you cache it for just one second, you can guarantee that any downstream infrastructure can only get 60 requests/minute. Any hardware these days could easily handle 60rpm. Most people that use the system wont notice a delay less than 1 second for new data.

In the diagram above there are 500k r/pm to some infrastructure without a cache. This would be pretty tough for a decent RDBMS to handle but Redis could handle that cheaply. In the second diagram you can see the effect of putting a short cache in front of the RDBMS - RDBMS requests are flat and predictable.

Common caching scenarios

Here are some tools you probably use every day that use caching

DNS

The domain name system that runs the internet is based on local caches. Records propagate around the internet with Time to Live set. Your browser first asks the cache in your computer how to find google.com, if that record is stale then the browser goes to your ISP for the DNS record. If your ISP doesn’t have a record then the browser goes to one of the root nameservers for the internet.

Having a local cache for common DNS records like google.com results in much faster browser experience.

React Query

The react query library is an excellent data retrieval library for client applications. React query has built in read-through caching. Each request your application makes is stored in a local cache. React query will reuse the data in the local cache to quickly display data to the user. Then it will refresh the data in the background to reduce the appearance of spinners.

Apps with data stores

Most applications store data somewhere, often a database. Most applications are read heavy so we can increase performance and reduce the cost of the data store by utilising a cache between the application and the data store.

Common caching patterns

There are a bunch of very common caching patterns. These three are the main ones I see when building web applications.

Cache aside

Cache aside is very common. The application is aware of the cache and acts as the coordinator.

Assuming a cache miss scenario:

The application checks the cache for data.
Cache returns empty response
Application gets the data from the database
Application stores the data for the next cache retrieval

Read through caching and Write behind caching

In read through and write behind caching the application only interacts with the cache. The cache layer then communicates with the database layer directly. The application is never aware of the database layer.

Assuming a cache read miss:

The application queries the cache
The cache has no data so it queries the datastore

The cache stores the result and returns it to the client application

Assuming a cache write

The application writes data to the cache
The cache writes the data to the database

These two patterns are not always used together. e.g. you might always read through the cache but write directly to the database. It’s just easier to talk about them in one section because the principal is so similar.

Slonik Postgres client can provide an example of read through caching. It allows you to configure interceptors. The interceptor are middleware that run on your query pipeline.

You can then apply hints like @cache-ttl 10 to the interceptors in your sql query. See the documentation at https://github.com/stockholmux/slonik-redis-cache

const
import { sql, createPool } from 'slonik'
import redis from 'redis'
import redisCache from 'slonik-redis-cache'

const client = redis.createClient({
  /* connection options */
})

const pool = createPool('postgres://localhost:5432/applicationDb', {
  interceptors: [redisCache(client)],
})

pool.connect(async (connection) => {
  let results = await connection.query(
    sql`SELECT * FROM aTable -- @cache-ttl 10`
  )
  console.log(results)
})

An example of write through pattern can be described using Redis Enterprise with Redis Gears (https://docs.redis.com/latest/modules/redisgears/).

Redis Gears allows you to set up Redis to run Python and Java with hooks into the Redis runtime. The hook for Write Through Caching will propagate Redis writes to your datastore.

Cache Invalidation

If your datastore has data that isn’t in the cache yet then you have a stale cache. Whether this is an issue is highly dependent on the data and your application. At some stage you will want to invalidate this data so that the cached record gets replaced with the new record.

There are excellent articles online describing how to solve this issue so i'll just give a brief overview here. Check out https://redis.com/blog/three-ways-to-maintain-cache-consistency/ for a deep dive into cache consistency and cache invalidation.

The following are the common methods used to invalidate cached data. One important thing to note is that if you use write-through then your cache should always be up to date.

Time to live

Time to live invalidation is extremely common. You ask the cache to automatically expire an item after a set time period, meaning your data layer will have to go to the source to refresh the item after some set time period.

Cache on write

For data with a high read rate and that changes infrequently, you can simply update the cache each time you write to your datastore. This invalidation strategy becomes expensive as write frequency increases.

Cache Eviction

Least recently used is an eviction method. This focuses on freeing resources, rather than keeping data fresh, by deleting from the cache data that isn’t getting read very often.

There are other common eviction mechanisms

Most Recently Used
Least Recently User
First in, First out
Last in, first out

Their usage will be highly dependent on your application.

Issues to watch our for with caching

Caching is notorious in software development for causing difficult to debug issues. For example if you're debugging an issue and the data doesn't make sense, check for caching somewhere. Caching is a form of state in these scenarios.

These are some of the things I look for when caching is present in a system.

Inappropriate load for caching

If your database is not under significant load adding a database can have a negative effect because the latency between app -> Cache -> DB can be more than a more traditional app -> DB not under load

Caching too eagerly

If you cache many things that are not read very often then you will fill up your cache. This will make it more expensive to run and having so many records will slow down the cache.

Incorrect configuration

If you cache data for too long or you don’t clear the cached data at the right time then you might return stale data to a customer.

Conclusion

You’ll come across caching on most apps. Tweaking the caching strategies and invalidation strategies that are used will help you to improve user experience and cost of your infrastructure.

Caching introduces complexity that has to be managed carefully.

How to run Monica personal CRM on Dokku

Darragh O'Riordan — Mon, 29 Aug 2022 00:00:00 +0000

I left my home country right after university and I worked and lived in a few countries since then. I’ve met lots of amazing people but I’ve always struggled to remember contact details and important dates for everyone.

As we all get busier with life time moves faster and faster and sometimes I realise that it’s months since I checked in on people. I have forgotten the last things we talked about and what’s important in everyone’s lives.

I needed a personal CRM.

Monica personal CRM

Monica is a privacy focused relationship management tool. You run it on your own server so no-one has access to the data but you. There are no social networking features and it doesn’t parse or interpret your data in any way.

Monica CRM (from Monica HQ)

(Click for larger screenshot)

If you’re old like me you might have used Facebook for things like birthdays in the decade 2008-2018ish but most of us have deleted their data on there, or are ready to.

Monica replaces that functionality in a privacy-focused way. It removes that final blocker and allows you to delete your social media accounts.

Dokku self hosting

Dokku is a heroku-like PaaS platform that you can run on your own server. It’s perfect for a solo developer to run multiple applications on a VM and it makes handling security, networking, 0-downtime deploying, let’s encrypt https etc. a breeze!

Heroku has just announced that they’re removing their free plan so now is a fantastic time to try Dokku on a $5 Digital Ocean droplet

I host 3 apps on my Dokku instance on digital ocean, they all have different databases and urls. The only infrastructure work I have to do is log in once a quarter and update apt packages and Dokku itself.

I have an article at https://www.darraghoriordan.com/2021/12/29/run-node-app-postgres-dokku-digital-ocean/ that describes how to setup a Dokku instance. You can follow that to setup Dokku and then come back here. There is also excellent documentation on digital ocean describing how to setup Dokku.

You will need a url configured if you want to set up tls for the Monica app (you should do this).

So for the next steps I assume you have an instance of Dokku running on a server and you’re ssh’d into it.

The following steps describe how to set up Monica on your Dokku instance.

1. Add a swap file to your Dokku host

Note: These commands are run on your Dokku server

The Monica build process takes a bit of memory. If you’re on a cheap server like a $4 digital ocean droplet you might get errors that the build failed with no details. This is likely an out of memory issue and digital ocean doesn’t configure a swap for you by default.

First you should check if you have a swap file set. If the swapon show command returns nothing then you don’t have a swap and you should add one.

# check if a swap file is set
sudo swapon --show

# check how much free space you have
free -h

Run these commands to setup your swap file on ubuntu. I set a 1GB swap file on my droplet.

# create the allocation
sudo fallocate -l 1G /swapfile

# change permissions for swap
sudo chmod 600 /swapfile

# make it a swap file
sudo mkswap /swapfile

# turn it on
sudo swapon /swapfile

Now add the configuration to fstab so there is still a swap on your server after any reboots.

sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=10

The digital ocean article recommends turning down the default swappiness because this is a server not a desktop. Sounds like a great idea so let’s do that.

sudo nano /etc/sysctl.conf
ADD: vm.swappiness=10

Digital ocean have a details article on setting a swap file here.

2. Add Monica as a Dokku application

Note: These commands are run on your Dokku server

First thing we need is a Dokku application

dokku apps:create monica-crm

3. Add a database for Monica

Next we need a database. Monica is built around MySQL but I prefer to use MariaDB instead of MySQL. First we install the MariaDB plugin and then create and link a new database instance.

# install the MariaDB plugin
dokku plugin:install https://github.com/dokku/dokku-mariadb.git mariadb

# create the database instance
dokku mariadb:create monica-crm-db

# link the Dokku database to the Dokku application
dokku mariadb:link monica-crm-db monica-crm

4. Add networking for Dokku app

Next we need to setup some networking for our application. I’m assuming you setup a domain when configuring your Dokku instance, and that you’ve already configured letsencrypt plugin.

# set a domain for the app (the default will be app-name.yourdomain.com e.g. monica-crm.yourdomain.com)
dokku domains:set monica-crm monica.yourdomain.com

# set a config variable for email address. This is required by lets encrypt
dokku config:set --no-restart monica-crm DOKKU_LETSENCRYPT_EMAIL=your.email@somedomain.com

# now register a new domain with letsencrypt and install the certificate
dokku letsencrypt:enable monica-crm

5. Add Monica configuration settings

Before you run Monica in a production-like environment you will have to configure some settings for the application. These are set with environment variables.

# APP_URL is used in emails sent by monica so that links work as expected
# APP_DISABLE_SIGNUP blocks anyone from signing up to your instance. We will set this to true later!
# APP_KEY is a security key and you should generate a random set of 32 alphanumeric characters
# APP_ENV tells the application to use tls if set to production - we want this because we have tls with lets encrypt
dokku config:set --no-restart monica-crm APP_URL=https://monica.yourdomain.com APP_DISABLE_SIGNUP=false APP_KEY=MUvCHANGETORANDOMALPHAmbG APP_ENV=production

6. Configure email sending for Monica

One of the nice features of Monica are reminder emails for important events or any regular catchups you want to have. Monica needs some email provider so that it can send your emails.

This is optional but very worthwhile.

I have an SMTP email service that I pay for, so that’s what I use. You can use other providers like mailgun or AWS SES. There are more details on the Monica documentation site on how to configure other email sending providers.

Email settings are also environment variables.

# Set the various environment variables for email. These are fairly self-explanatory.
dokku config:set --no-restart monica-crm MAIL_MAILER=smtp MAIL_HOST=smtp.mysmtpprovider.com MAIL_PORT=587 MAIL_USERNAME=myusername@mysmtpprovider.com MAIL_PASSWORD=mysmtppassword MAIL_ENCRYPTION=tls

7. Fork and clone the Monica application

NOTE: These commands are run on your local machine!

We fork Monica on Github or using gh cli, then clone our copy Monica to a local folder. This is so that we can make some changes to the code for Dokku.

# if you have gh cli you can fork (and clone - it will ask you) from your terminal
gh repo fork monicahq/monica

# clone Monica - change the username (skip if you already cloned)
git clone git@github.com:mygithubusername/monica.git

Monica is built to run directly from the repository on Heroku but we must make minor edits for Dokku.

First file is a new file .buildpacks in the root with this content list the buildpacks to use.

https://github.com/heroku/heroku-buildpack-php
https://github.com/heroku/heroku-buildpack-nodejs

Next is an edit to the existing app.json file where we add a cron.

"cron": [
     {
         "command": "php artisan schedule:run",
         "schedule": "*/10 * * * *"
     }
 ],

Now commit and push those 2 changes to your forked Monica instance.

8. Build Monica on Dokku for the first time!

Note: These commands are run on your local machine

# you have to add a remote to the local repository that points to your dokku server
git remote add dokku dokku@the_ip_or_hostname_of_your_dokku_server:monica-crm

# and push the code! This will take a while because it kicks off a build of the application
git push dokku main:master

If that worked as expected it will print lines at the end like

======> Application deployed:
        http://monica.yourdomain.com
        https://monica.yourdomain.com

Go to the url! Monica will ask you to set up your account.

Monica login screen

9. A final security clean up for Monica

Remember the APP_DISABLE_SIGNUP variable from before. Now that you have set up your account you probably don’t want anyone else signing up for an account on your server. So let’s shut off that possibility.

On your Dokku server:

dokku config:set monica-crm APP_DISABLE_SIGNUP=true

Summary

I’ve just started using Monica and enjoying it so far. It has a simple interface which is just perfect. Time will tell if it helps me as a personal CRM!

Dokku provides a nice alternative to Heroku’s free tier. Even though it’s not free, it can be run on a very cheap host like a $5 Digital Ocean droplet.

Once you have setup your Dokku instance you can add new apps very quickly and Dokku makes it easy for a solo-developer to manage multiple applications. I recommend checking it out!