Forem: François

My new programming language is English

François — Wed, 08 Apr 2026 00:00:00 +0000

My new programming language is not TypeScript, Python, Go, or even Rust. It is English.

I spend most of my time explaining to an agent what I want, why I want it, and what constraints matter. I give exact descriptions when needed, plus links to the right docs. Sometimes those docs are full of typos, and that is OK. Then I review what comes out. For a huge part of my day-to-day engineering work, this is simply faster.

And if we are honest, in most systems, the Pareto rule applies.

For 80% of what we build, I do not care that much whether the agent used map or a for loop, OOP or functional style, or some "very elegant" abstraction. Is it an API endpoint? A DB query? A CRUD flow? Fine. What I care about is the output. Is it correct? Is it safe? Can we ship it?

It is also very likely I will not be the one maintaining that code. Another agent probably will.

Now, this only works if your engineering basics are strong. Actually, they need to be stronger than before. Tribal knowledge does not work with agents starting fresh on each run.

Because "English is my new programming language" does not mean "standards do not matter anymore." It means standards and guardrails matter so much more.

You need CI/CD that blocks bad changes. You need tests: unit, integration, end-to-end. You need gradual rollouts, smoke tests, monitoring, alerting. You need pull request review. You need context awareness on the scope of the change and the criticality of what you touch.

And locally, you need fast feedback loops: linter, formatter, fast builds. Shift-left has never been this true. If feedback is fast, the agent iterates fast and can run its loop locally.

Agents are like super smart interns.

Very capable. Very fast. Sometimes impressive. Still not autonomous in the way people fantasize about. I would trust a good intern to implement a lot when motivated. I would absolutely not skip reviewing their changes. Same here.

If the intern breaks production, blame the game, not the player. Same for code created by an agent, in my opinion.

Review is still a thing. Maybe the thing nowadays. We spend more time writing specs and reviewing PRs, and I still do not think we have reached a turning point where we can skip checking generated code for production-scale projects.

When things go wrong, you need to know the structure of the codebase and where to look. Maybe this is just engineering curiosity, maybe it is risk management, probably both. But the day prod is down and Claude plus OpenAI are down too, you still need to fix it yourself.

During review, I am not only checking low-level style comments like "you used let instead of const" or "careful, you mutate state here."

What I really want to check is whether we are using current best practices. Is this API using the new decorator the team agreed on? Is ownership clear? Is observability in place? Are we respecting patterns we know scale in production? Is this internal and low risk, or is this critical path code where every generated line deserves deeper review?

And when I find myself repeating these rules, questions, and comments, that is usually a signal: write them down.

That is why skills (like Claude Skills) are a big deal.

Skills are how standards get distributed: how to create a service, define ownership, model a domain, and speak business language in the codebase. Less tribal memory, less repeating yourself in PR comments, more consistency by default.

Same story for documentation.

To code well in this era, good documentation is not "nice to have." It is core infrastructure. API specs, READMEs, architecture docs, sharp examples, and links to the few pages that actually matter. What I think is core to good AI-driven development:

README.md (for your developers too!) - Here I would expect to find the getting started instructions, a summary of the project, and links to relevant docs.
AGENTS.md - Keep it very short here. Only give the big lines (main tech, quick repo structure). I like Matt Pocock's approach here: https://www.aihero.dev/a-complete-guide-to-agents-md

"The ideal AGENTS.md is small, focused, and points elsewhere. It gives the agent just enough context to start working, with breadcrumbs to more detailed guidance." Matt Pocock.

Skills - Basic AI skills on how to work with your repo and your libs: what you expect for commits, pull requests, testing, and ways of working with specific libs. If you work with Cloudflare, add their skills to your repo! That'll enable anyone working with the repo to produce less garbage.
Type-safe schemas - If you use end-to-end type-safe schemas to build your clients and backend APIs, agents usually work better with that than with API specs. But you should still generate OpenAPI specs from the schemas, for instance. And make it clear with a skill that this is automated.
Tests - Tests are documentation. Get powerful smoke/e2e tests and you'll save yourself a lot of headaches when shipping faster. Regression is your biggest enemy here; you don't want to break existing flows by adding new features. Again, it's nothing new in our SDLC world.

Without that, you spend the same amount of time re-explaining your system to the agent as you would with a junior engineer. Again and again and again.

The thin line now is deciding what should be a skill, what should be a README, and what should just be a code comment.

Explaining a task to an agent feels a lot like leaving instructions to myself for one month later (I am pretty sure you know that comment you left somewhere in the codebase and you stumble into it from time to time, thanking your past self for clearly describing what the issue was 😅).

I know I will forget context. The agent never had it. So I need clear explanations of what should be done, how to do it, and where the source of truth lives.

It is funny when you think about it. We used to say good engineering practices were documentation, tests, and clear code comments. Now we are saying it again like it is brand new.

In the end, using AI agents feels a lot like scaling your engineering team. In the past, if your team grew x10, you needed more process, more test coverage, and better documentation. Working with agents feels very similar, just faster.

So yeah, my new programming language is English.

Not because code disappeared into a black box. Because the highest leverage moved one level up: from typing instructions for computers to writing intent and constraints for systems that generate implementation.

And in that world, the teams that win are not the teams that type fastest. They are the teams that specify clearly, review rigorously, and document like context is always missing.

I wanted the best GitHub notifications in Slack, so I built it

François — Tue, 17 Mar 2026 00:30:00 +0000

Two things really annoyed me with GitHub notifications.

First: when I merge my own PR and break CI/CD on main, I am not alerted.

Second: existing notification solutions often spam me with bot comments and status updates (Vercel, Cloudflare, Wiz, CodeRabbit, and more), while human review signal gets buried under these notifications.

That is why I built GitNotifier: GitHub notifications in your Slack DMs.

TL;DR: it alerts you when your PR breaks main, on comments written by human on your PRs, you can mute bots, etc ;) (and more! keep reading eheh)

The blind spot: post-merge CI failures on main

Most GitHub notifications are PR-centric. That sounds fine until you realize what happens after merge.

If your PR gets merged, and then CI/CD fails on main, you often miss it. The branch is red, production might be blocked, and you discover it too late.

I even wrote a workaround years ago using GitHub Actions and Slack webhooks:
Sending slack notifications with github actions. It's quite manual because you need to edit your GitHub Actions in all your repos and configure the right tokens... but it did the job for a while.

It still felt like duct tape. I wanted something that works out of the box.

Bots now talk more than humans

On many PRs, I get more bot comments than human comments.

You know the pattern:

Vercel preview updates
Cloudflare checks
Wiz security notes
Bito.ai suggestions or CodeRabbit AI review comments
and more

None of these are inherently bad. I actually like reading them (sometimes) when I review a PR. The problem is they should not be notifications. Kind of like with a service level objective (SLO): you only want alerts on symptoms, not for every small CPU burst in your VMs.

In 2026, reviewing code often takes longer than writing code (thanks Claude, ChatGPT & co). If the most important part of the workflow is human review, then human comments should have the highest signal.

Here is a graph from the GitNotifier analytics page to show this clearly in the repos I work with: bot comment volume often exceeds human comment volume (crazy, right?).

Existing tools did not really solve it for me

I tried a lot of options. Most had one or more of these issues:

Too expensive for what they do
Hidden inside enterprise bundles
Focused on employee monitoring or performance tracking
Poor UX (some honestly felt vibe-coded)
Limited customization

I "just" want good GitHub notifications in my DMs, not a full-blown team performance tracking system costing me a house per month. Notifications that are actually useful in daily engineering work.

Most PR reviews involve 2 people, so why spam a channel?

A PR review is usually between:

the author
the reviewer
(a review agent? but they are noisy 😁)

Sometimes 3 people. More than that is rare.

So sending every update to a shared Slack channel is often the wrong default. It creates ambient noise, trains people to mute channels, and increases alert fatigue for everyone. The fewer messages, the better.

IMHO, the right destination for most PR events, such as comments, is personal Slack DMs, not public channels.

Shared channels should stay for shared context. Direct actions should go to directly involved people.

What I wanted instead

I built GitNotifier around a few principles:

Signal over volume
Human-first notifications
Strong customization
DM-first delivery
Actionable alerts, not passive spam

In other words: fewer notifications and better notifications.

Do things from Slack, do not just get pinged there

Another frustration: every notification forces context switching.

I wanted to stay in Slack and continue working, so GitNotifier lets me do things directly from Slack:

react to comments
reply to comments
merge PRs that are ready

The goal is simple: reduce tab-jumping, keep momentum. As a big fan of Raycast, I want to be more ✨productive✨ when I use a tool.

DX matters: setup should be boring

Dev tools die when setup is painful.

The flow I wanted:

Admin installs 2 apps (GitHub and Slack)
Users join in 1 click from Slack

That is it.

No heavy rollout project. No setup overhead. No long onboarding docs. Just connect and start receiving useful notifications.

What changed for me

After using GitNotifier daily:

I miss fewer critical CI issues
I focus faster on human review threads
PR handling is faster because actions happen in Slack, and PRs get merged faster.

I did not need more notifications. I needed better ones 😁

If you want to give it a try, check out GitNotifier

If you are dealing with the same GitHub + Slack notification fatigue, I would love your feedback!

Stop committing with the wrong email: Multiple Git Configs for GitHub

François — Wed, 20 Aug 2025 00:00:00 +0000

As a developer working on both personal and professional projects, you might find yourself needing to use different SSH keys and email addresses when committing to GitHub. This is especially common when you want to keep your work and personal contributions separate, or when you need to use different SSH keys for different organizations.

In this guide, I'll show you how to set up multiple Git configurations using separate config files, allowing you to automatically use the right SSH key and email based on which directory you're working in.

This guide assumes you sort your git directories by folder. Personally, I keep all my private repos in ~/Projects/perso and all my work repos in ~/Projects/<company_name>. To simplify this post, let's use ~/perso and ~/pro.

Note that I'll be focusing on mac and some commands may vary between OS.

The Problem

By default, Git uses a single global configuration (~/.gitconfig) for all repositories. You might have setup this once when you got your laptop with git config --global user.email ... for instance. This means:

All commits use the same email address
All SSH connections use the same key
You can't easily switch between personal and professional identities (you could configure the git config per repo, but it comes really cumbersome and error prone)

The Solution: Multiple Git Config Files

We'll create separate Git configuration files for different contexts and use Git's includeIf directive to automatically load the right config based on your current directory.

Step 1: Create Your SSH Keys (optional)

If you already have several SSH keys, you can skip this section 🙏

To generate separate SSH keys for personal and professional use we'll use ed25519 keys (have a look here for details about why). Feel free to use RSA if you like verbosity.

# Personal SSH key
ssh-keygen -t ed25519 -C "your-personal-email@example.com" -f ~/.ssh/id_ed25519_perso

# Professional SSH key
ssh-keygen -t ed25519 -C "your-work-email@company.com" -f ~/.ssh/id_ed25519_pro

When prompted, you can either:

Press Enter to create keys without a passphrase (less secure but more convenient)
Enter a strong passphrase for additional security (if you do so, I'd recommend to add them to your keychain on mac.)

Step 2: Add Keys to SSH Agent

Add both keys to your SSH agent:

ssh-add ~/.ssh/id_ed25519_perso
ssh-add ~/.ssh/id_ed25519_pro

You can read more about it in the github docs.

Step 3: Configure SSH for Different Hosts

Create or update your ~/.ssh/config file:

# Personal GitHub
Host github-perso
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_perso
    # If you use a passkey, you might need more options like UseKeychain, AddKeysToAgent and IdentityAgent.

# Professional GitHub
Host github-pro
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_pro

Step 4: Set Up Git Config Files

By default Git will use ~/.gitconfig. When you're in a directory that matches the gitdir pattern:

~/perso/ → will load ~/.gitconfig-perso config override.
~/pro/ → will load ~/.gitconfig-pro config override.

Main Git Config (`~/.gitconfig`)

This is your global configuration that applies to all repositories:

[user]
    name = Your Name
    email = your-personal-email@example.com

[core]
    editor = vim

[init]
    defaultBranch = main

# Include personal config for personal projects
[includeIf "gitdir:~/perso/"]
    path = ~/.gitconfig-perso

# Include professional config for work projects
[includeIf "gitdir:~/pro/"]
    path = ~/.gitconfig-pro

Personal Git Config (`~/.gitconfig-perso`)

[core]
  sshCommand = "ssh -i ~/.ssh/id_ed25519_perso -F /dev/null"

Professional Git Config (`~/.gitconfig-pro`)

[core]
  sshCommand = "ssh -i ~/.ssh/id_ed25519_pro -F /dev/null"

[user]
    name = A very profesional name
    email = "my-profesional@email.com"
    signingkey = /Users/<your_user>/.ssh/id_ed25519_pro.pub

Step 5: Add SSH Keys to GitHub

Copy your personal & professional public key:

pbcopy < ~/.ssh/id_ed25519_perso.pub
pbcopy < ~/.ssh/id_ed25519_pro.pub

Add both keys to your GitHub account:

Go to GitHub → Settings → SSH and GPG keys
Click "New SSH key"
Paste each key and give them descriptive names
(optional) Add both your keys as Authentication keys and Signing keys. This will help you to sign your commits using your SSH key.

Step 6: Set Up Commit Signing (Optional but Recommended)

To verify that your commits are authentic, you can sign them using your SSH keys. This provides cryptographic proof that the commit came from you. It also looks nice in the github UI ☺️

Configure Git for SSH Signing

Update your Git configs to use SSH signing:

Main config (~/.gitconfig):

[gpg]
    format = ssh
[commit]
    gpgsign = true

Bonus (hide your email) 🥷

In some cases, you might want to hide your email on Github. To do so, you can head to https://github.com/settings/emails and enable "Keep my email addresses private".

By using the random email github generates, here 11248623+Lp-Francois@users.noreply.github.com for my account, Github will hide your personal email.

To use it with your git config, put it in this block:

[user]
    name = Francois Le Pape
    email = "11248623+Lp-Francois@users.noreply.github.com" # email found in https://github.com/settings/emails"
    signingkey = /Users/francois/.ssh/id_ed25519_perso.pub

Conclusion

With this setup, you now have a clean separation between your personal and professional Git identities. No more accidentally committing with the wrong email or SSH key!

The beauty of this approach is that it's completely automatic - Git will detect which directory you're in and apply the appropriate configuration. You can focus on coding instead of remembering to switch Git configs manually.

Hope this helps ✨

Stop using "GitOps" to sell your products

François — Tue, 20 Aug 2024 00:00:00 +0000

Hello there, and welcome to the first rant on this blog. No, deploying infrastructure from Git doesn't mean you are doing "GitOps". It's called Infrastructure-as-Code on Git.

GitOps is IaC, but IaC is not necessarily GitOps.

So what is GitOps?

Well, there is a specification, a standard, named OpenGitOps that defines it the following way (the main focus of this post is on the 4th principle):

1. Declarative

A system managed by GitOps must have its desired state expressed declaratively.

Well, there is not much to add, it shouldn't be a for-loop in a bash script that uses an API, but just plain declarative configuration code.

2. Versioned and Immutable

Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.

That's the Git part of GitOps! Although it started with Git (and that's where the name comes from), now you can find another state store like OCI repositories.

3. Pulled Automatically

Software agents automatically pull the desired state declarations from the source.

The "software agents" are the tools like Flux, ArgoCD, etc. that will pull the desired state from the source (Git or OCI repository). It shouldn't be manual.

4. Continuously Reconciled

Software agents continuously observe actual system state and attempt to apply the desired state.

That's to me THE point where people get GitOps wrong (or on purpose wrong, to sell you more buzzwords).

If the state of the system changes, you should expect the software agent to reconcile the state to the desired state. In other words, what is in Git should be reflected in the Kubernetes Cluster all the time.

Adding a CI/CD pipeline to your repository that runs a kubectl apply is not GitOps. GitOps is to have a tool checking for drift detection and re-applying the desired state when needed.

Let's take an example: The new intern - let's name him Kevin - gets production access, and for debug purposes sets the replica number of a Kubernetes deployment to 0 instead of 3. Well, the GitOps software agent should see that the state of the cluster is different from the desired state and re-apply the desired state, causing the deployment to be back to the desired state: 3. it's a very powerful feature that brings production robustness to manual errors and drift detection!

Do you want to see a super powerful and strong GitOps agent? Here is mine:

➜  ~ while true; \
  do kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/snapshot-initial-v1.31/content/en/examples/pods/simple-pod.yaml; \
  sleep 15; \
done
pod/nginx created
pod/nginx unchanged
pod/nginx unchanged
pod/nginx unchanged

It's declarative, versioned and immutable in Git (Pod specs are in YAML here). It's pulled from Git, and it's continuously reconciled every 15 seconds. It's GitOps!

Final notes

This post is written by a "DevOps" engineer, so don't pay too much attention to buzzwords after all. I hope I shed some light on the GitOps principles at the cost of a clickbait title (and maybe a few Twitter dramas).

Sources

OpenGitOps principles - https://opengitops.dev/
4 Core Principles of GitOps - The new stack - https://thenewstack.io/4-core-principles-of-gitops/
How We Are Moving from GitOps to Kubernetes Resource Model in 5G Core - https://www.youtube.com/watch?v=crmTnB6Zwt8&t=302s
Flux Beyond Git: Harnessing the Power of OCI - Stefan Prodan & Hidde Beydals, Weaveworks - https://www.youtube.com/watch?v=gKR95Kmc5ac&t=1215s (Using OCI repository instead of Git)
and many others on the internet!

:wq

How to increase AWS ELB keep-alive timeout when deployed from Traefik

François — Wed, 08 May 2024 00:00:00 +0000

Let's see together how to change the Load balancer settings when created from an ingress controller like Traefik on Kubernetes.

The default timeout of the AWS Elastic Load Balancer (ELB) is 60 seconds. For some use cases (like long LLM calls) this might be too short and you might want to increase it.

When deploying Traefik on Kubernetes, it might be hard to find the right configuration to increase the keep-alive timeout on the ELB, as Traefik is the one creating the ELB.

Well, after some searches it is pretty easy to change it. You need to add the right annotation on the Kubernetes Service with the type Load Balancer.

Here is the documentation with available annotations: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/

With the Traefik Helm chart, it is as simple as updating this values.yaml file:

service:
  #    https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#resource-attributes
  annotations:
    # 2 minutes
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '120'

Some examples of configuration used with nginx ingress controller: https://github.com/zephyrproject-rtos/infrastructure/blob/4b767ceec69f725614ded3dcb715c8ebffeadc83/terraform/zephyr-aws-blueprints/helm_values/nginx-values.yaml#L4

And that's it!

NodeJS Best Practices: Redacting Secrets from Your Pino Logs

François — Sat, 30 Mar 2024 00:00:00 +0000

Learn to mitigate potential security issues by redacting sensitive data (like user credentials or secrets) from your logs using Pino logger for Node.js. Consider this a crucial part of your logging strategy.

In today's world where data breaches are increasingly common, secure logging is not just important - it's absolutely essential. One area that developers often overlook is the logging of sensitive data, like user credentials or secrets. Such data leaks can lead to serious security vulnerabilities and violations of GDPR compliance if you're not careful.

This article will guide you through redacting such sensitive data from your logs when using Pino, a performant and lightweight logger for Node.js, aiding your application in maintaining GDPR compliance and reducing the risk of privileging escalation.

Pino

When instantiating a pino instance, you usually start with a similar piece of code:

// You can find the Gist here: https://gist.github.com/Lp-Francois/3d1b36907de8283a8a3eedca31c9cfc3

// file:`example.js`
// install packages: `npm i pino-http --save`
'use strict'

const pino = require('pino-http')
const http = require('http')
const server = http.createServer(handle)

const logger = pino({
  base: undefined, // removes pid and hostname from the logs
})

function handle (req, res) {
  logger(req, res)
  req.log.info('my request')
  res.end('hello world')
}

server.listen(3000, () => console.log('[i] running on http://localhost:3000'))

and when running it, then sending requests to it, you can

# terminal window 1
$ node example.js
# terminal window 2
$ curl http://localhost:3000 -H "authorization: bearer my-super-secret" -H "Content-Type: application/json"

# logs in terminal 1:
[i] running on http://localhost:3000
{"level":30,"time":1711808000315,"req":{"id":1,"method":"GET","url":"/","headers":{"host":"localhost:3000","user-agent":"curl/8.4.0","accept":"*/*","authorization":"bearer my-super-secret","content-type":"application/json"},"remoteAddress":"::1","remotePort":63547},"msg":"my request"}
{"level":30,"time":1711808000322,"req":{"id":1,"method":"GET","url":"/","headers":{"host":"localhost:3000","user-agent":"curl/8.4.0","accept":"*/*","authorization":"bearer my-super-secret","content-type":"application/json"},"remoteAddress":"::1","remotePort":63547},"res":{"statusCode":200,"headers":{}},"responseTime":7,"msg":"request completed"}

Notice the authorization header containing the secret in the logs 😢: ..."accept":"*/*","authorization":"bearer my-super-secret"...

Redaction

Pino has a "hidden" (hard to find) documentation on redacting secrets.

It is possible to pass to your pino logger an array containing paths to keys holding sensible data.

To remove our authorization header from the req object, we just have to append this to our config: redact: ['req.headers.authorization'].

Here is the full example:

'use strict'

const pino = require('pino-http')
const http = require('http')
const server = http.createServer(handle)

const logger = pino({
  base: undefined,
  // ⚠️ The next line is the important one:
  redact: [
    'req.headers.authorization',
  ]
})

function handle (req, res) {
  logger(req, res)
  req.log.info('my request')
  res.end('hello world')
}

server.listen(3000, () => console.log('running on http://localhost:3000'))

Outputing the following logs:

[i] running on http://localhost:3000
{"level":30,"time":1711824390585,"req":{"id":1,"method":"GET","url":"/","headers":{"host":"localhost:3000","user-agent":"curl/8.4.0","accept":"*/*","authorization":"[Redacted]","content-type":"application/json"},"remoteAddress":"::1","remotePort":58055},"msg":"my request"}
{"level":30,"time":1711824390594,"req":{"id":1,"method":"GET","url":"/","headers":{"host":"localhost:3000","user-agent":"curl/8.4.0","accept":"*/*","authorization":"[Redacted]","content-type":"application/json"},"remoteAddress":"::1","remotePort":58055},"res":{"statusCode":200,"headers":{}},"responseTime":9,"msg":"request completed"}

Notice the authorization header now has a [Redacted] value!

Note the keys are case sensitive! If you want to add to the redact list an uppercase header like req.headers.CREDENTIALS be careful as it would ignore the lowercase credentials header.

It is also an option to drop both value and key (redact.remove), or change the value of the redacted field to something else with the censor option (I'll let you read the documentation for that 😉).

I didn't find many resources online that covered this tip specifically, so I hope this insight proves valuable to you!

Happy safe logging with Pino!

Good bye sleep() and delay() with NodeJS, hello setTimeout()

François — Mon, 27 Nov 2023 00:00:00 +0000

Stop to use home-made utilities sleep() and delay(), and use the native nodeJS timers API: setTimeout().

I have seen and added many times in nodeJS codebase the following snippet:

const sleep = (ms) =>
  new Promise((resolve) => setTimeout(resolve, ms));

Well, today I learned that nodeJS has a native API for that: setTimeout().

// https://nodejs.org/api/timers.html#timerspromisessettimeoutdelay-value-options
import { setTimeout } from 'node:timers/promises';

await setTimeout(2000); // sleep 2s

So time to stop to use home-made utilities sleep() and delay(), and use the native nodeJS timers API!

How to use OpenTelemetry to expose custom Prometheus metrics from nodeJS applications

François — Wed, 08 Nov 2023 00:00:00 +0000

A standard way of exposing metrics for a nodeJS application is to use the prom-client package, which has everything one would need. But we are in 2023, almost 2024 and it is time to use industry standards, and here I am talking about OpenTelemetry. Let’s answer the question together on how to use open-telemetry JS to expose custom metrics.

I don’t think there is a need to present the OpenTelemetry project anymore. 2nd CNCF project with highest velocity at the time of writing, vendor-neutral and open-source.

Example nodeJS project

The following commands help to setup a nodeJS project with some OpenTelemetry packages. You might notice when using open telemetry package that the list in package.json grows quite fast. But be assured, most of them are lightweight. For instance @opentelemetry/exporter-prometheus is 24.4
kB minified and gzipped.

mkdir otel-prom cd otel-prom
npm init -y

touch index.js

npm install --save @opentelemetry/exporter-prometheus @opentelemetry/api @opentelemetry/sdk-metrics

# Note: I am using the following versions:
# "@opentelemetry/api": "^1.7.0",
# "@opentelemetry/exporter-prometheus": "^0.45.0",
# "@opentelemetry/sdk-metrics": "^1.18.0",

@opentelemetry/sdk-metrics and @opentelemetry/exporter-prometheus are required to register metrics and export them in prometheus format.
@opentelemetry/api is optional, and contain some useful utilities & interfaces.
You could even add @opentelemetry/sdk-node to get some native nodejs metrics.

Note: The Otel packages shown in this blog are currently considered experimental packages under active development. New releases may include breaking changes.

Then proceed to edit the file index.js with the following content:

const { ValueType } = require("@opentelemetry/api");
const { PrometheusExporter } = require("@opentelemetry/exporter-prometheus");
const { MeterProvider } = require("@opentelemetry/sdk-metrics");
const http = require("http");

// Mock a potential database call to query data
const queryDatabaseStuff = async () => new Promise((resolve) =>
    setTimeout(resolve(Math.floor(Math.random() * 100)), 50)
  );

const startMetricsExporter = () => {
  console.log(`starting prometheus metrics server`);

  // you can choose in the options the port, and if you want to start a webserver
  const exporter = new PrometheusExporter({
    port: 9100,
  });

  const meterProvider = new MeterProvider();
  meterProvider.addMetricReader(exporter);
  const meter = meterProvider.getMeter("prometheus");

  // create the gauge
  const outdatedDataCountGauge = meter.createObservableGauge("outdated_data_count", {
    valueType: ValueType.INT,
    description: "outdated data count",
  });

  // callbacks are executed when the /metrics endpoint is hit
  outdatedDataCountGauge.addCallback(async (result) => {
    const outdatedDataCount = await queryDatabaseStuff();

    result.observe(outdatedDataCount);
  });
};

// simulate a normal webserver, it could be fastify, express, etc.
// Just to illustrate you could run the exporter along a classic webserver
const startNormalWebServer = () => {
  console.log(`starting normal web server`);

  http
    .createServer(function (req, res) {
      res.write("Hello World!");
      res.end();
    })
    .listen(8089);
};

startMetricsExporter();
startNormalWebServer();

Run the application with node index.js to see 2 web servers. You can go to localhost:8089 to see the normal web server, and localhost:9100/metrics to see the metrics endpoint.

Here are more detailed explanations of what the code is doing:

queryDatabaseStuff() is to mock a DB call that would probably fetch some data count, like outdated data.

meter.createObservableGauge then records a Gauge, where you can pass a description and the expected value type. The part not very well documented in OTel is the addCallback function, which allow you to run a function when the metrics endpoint is called.

Be aware this approach is optimal for relatively light database queries. In situations where your query examines multiple databases, executing complex aggregations that can slow down the system, a more effective strategy is to run the query at predetermined intervals in the background. The recent results are stored and merely retrieved during the callback, rather than executing a comprehensive query each time. This tactic essentially minimizes the time-consuming database operations during the metrics' collection and maintains a responsive and efficient metrics endpoint.

About the options passed to PrometheusExporter, I would advise keeping the default which is to start a web server on a different port. Therefore you are sure you won’t expose the metrics by mistake to the outside world.

and... with Kubernetes? ☸️

If you use Kubernetes as your underlying platform, you can then have a service exposing 2 ports, with only one of them being for the ingress of the application web server, and the other one for the ServiceMonitor watching the metrics web server.

---
apiVersion: apps/v1
kind: Deployment
# skip the boring part
...
        ports:
          - containerPort: 8089
            name: http
            protocol: TCP
          - containerPort: 9100
            name: metrics
            protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    app: my-web-app
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8089
    - name: metrics
      port: 9100
      protocol: TCP
      targetPort: metrics
  selector:
    app: my-web-app
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-servicemonitor
  labels:
    app: my-web-app
spec:
  endpoints:
    - interval: 30s
      port: metrics
  jobLabel: ''
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
        app:  my-web-app

It makes it way easier to prevent exposing the metrics endpoint to the outside, you then don't have to juggle with ingress path routing to exclude /metrics.

Conclusion

OpenTelemetry is a great project, and I am glad to see it growing so fast, but it can be uneasy to get familiar with the Otel terms. I hope this blog post will help you as documentation is still missing for observability with nodeJS in general.

Hope this post helps, and won’t be deprecated too fast!

How to stream big JSON files with low-memory footprint in nodeJS

François — Thu, 17 Aug 2023 00:00:00 +0000

For production-grade applications, actions like reading a big file in memory is not ideal as they might cause OOM crash (Out Of Memory), and require a lot of resources for sometimes a really short duration. It would be a waste of resources and money to have a big server to only handle rare cases, and run at 5% of the capacity the rest of the time.

Legend: Example of a container with average memory consumption between 50 mb to 150 mb, when processing files it spikes to 4GB of RAM, leading to OOM crash.

Most of the time, especially with applications running on Kubernetes (like the one from the screenshot), a best practice is to stream data instead of loading it in memory. Here is how you can do it using Stream Transformers, a less-known Node.js feature.

Streaming big objects with nodejs

Generate data

Generate dummy data with this one-liner to paste in a terminal, to simulate a big JSON file.

node << EOF
require('node:fs').writeFile('dummy.json', JSON.stringify(
  new Array(process.argv[2] || 50).fill({ id: 0, name: 'dummy' })
), _ => console.log('done'))
EOF

JSON stream NPM Module

To help with processing JSON, we can use a tiny library like stream-json.

As written on the GitHub README:

stream-json is a micro-library of node.js stream components with minimal dependencies for creating custom data processors oriented on processing huge JSON files while requiring a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual primitive data items (keys, strings, and numbers) can be streamed piece-wise.

Install the json stream module:

npm install stream-json

Code example

const { pipeline, Transform } = require("node:stream");
const fs = require("node:fs");

const { streamArray } = require("stream-json/streamers/streamArray");
const Batch = require("stream-json/utils/Batch");
const { parser } = require("stream-json");

const logMemory = () =>
  console.log(
    `Memory usage: ${
      Math.round((process.memoryUsage().heapUsed / 1024 / 1024) * 100) / 100
    } MB`
  );

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const customTransformer = new Transform({
  objectMode: true, // to accept arrays
  transform: async (chunk, encoding, cb) => {
    logMemory();
    console.log("chunk length: ", chunk.length);
    await sleep(500);
    cb();
  },
});

pipeline(
  // read from file
  fs.createReadStream("dummy.json"),
  // parse JSON
  parser(),
  // takes an array of objects and produces a stream of its components
  streamArray(),
  // batches items into arrays
  new Batch({ batchSize: 2 }),
  customTransformer,
  (err) => {
    if (err) {
      console.error("Pipeline failed", err);
    } else {
      console.log("Pipeline succeeded");
    }
  }
);

and it should give an output similar to:

➜  node-stream-demo node index.js
Memory usage: 4.6 MB
chunk length:  2
Memory usage: 4.64 MB
chunk length:  2
[...]
chunk length:  2
Memory usage: 4.32 MB
chunk length:  2
Memory usage: 4.33 MB
chunk length:  2
Memory usage: 4.33 MB
chunk length:  2
Pipeline succeeded

The 2 interesting parts are in the pipeline() function:

new Batch({ batchSize: 2 }) grouping data 2 by 2, very convenient to process data in batches.
customTransformer to do async operations on the batches, like inserting data in a database by X.

Hope the example helps and provides ideas on how to use Node Stream Transformers!

How to debug RDS database from a Kubernetes cluster

François — Sat, 05 Aug 2023 00:00:00 +0000

When running Kubernetes workflows, most people tend to keep the stateless applications like applications on the cluster, and the stateful applications like databases outside of the cluster, on the same VPC. This is a good practice, but it makes debugging a bit more complicated.

Note: I use AWS as an example, but this can be applied to any cloud provider with managed database & K8s using private networks

I assume your database is private and only accessible from a VPC. Maybe for some obscure reasons, you still need to access this database to debug something (Staging database of course cough cough). Well, it might be a bit annoying.

One way would be to spawn an EC2 instance inside the same VPC, install a bunch of tools like netcat, mysql, postgres and debug from the VM. Not very convenient, you might left resources running and it's not very reproducible.

What I would advise, is to use a "reverse-proxy" container inside your cluster. Being on the same VPC as your managed database, you will be able to access it, and this container port can be port-forwarded to your machine. Of course, all the following expect you to have enough privileges to create a pod inside your cluster.

----------        -------        ------------
| Laptop |  <-->  | Pod |  <-->  | Database |
----------        -------        ------------

You will need 2 terminal windows:

One to create the pod opening the connection to the database
One to port-forward the pod port to your local machine

DB_PORT=5432
DB_HOST=64eddd8e4b61-postgresql.993191f72591.eu-central-1.rds.amazonaws.com

kubectl run psql-tunnel -it --image=alpine/socat --tty --rm --expose=true --port=$DB_PORT tcp-listen:$DB_PORT,fork,reuseaddr tcp-connect:$DB_HOST:$DB_PORT

Here is an explanation of each argument that might be useful to understand what is happening:

socat: This image allows to establish two bidirectional byte streams and transfers data between them, perfect for our use case!
--rm: Removes the container automatically once it exits. (It helps in cleanup and avoids leaving unused containers.)
tcp-listen:$DB_PORT,fork,reuseaddr: Configures socat to listen on TCP port 5432 inside the container. The fork option allows handling multiple connections, and reuseaddr allows reusing the address.
tcp-connect:$DB_HOST:$DB_PORT: Specifies the destination address and port where the incoming connections to 5432 inside the container will be forwarded. It connects to the host 64eddd8e4b61-postgresql.993191f72591.eu-central-1.rds.amazonaws.com on port 5432.

and the second command to port-forward the container port to localhost:

kubectl port-forward psql-tunnel 54323:5432

You can now use any GUI to connect to the DB and visualize what is happening (TablePlus, DBeaver, PhpMyAdmin, etc)!

Sending slack notifications with github actions

François — Sun, 02 Jul 2023 00:00:00 +0000

This is a small post about sending neat slack notifications from Github Actions. If it looks good, you get developers to click on the notification, trust me 😄

Here are the steps:

Create a new slack app from https://api.slack.com/apps, named Github actions and with a cool github logo
In Incoming webhooks, create a webhook, and save the webhook as a Github Actions repository secret with the name SLACK_NOTIFICATIONS_TOKEN
Add the newly created app to your channel: Channel settings > Integrations > Apps > Add apps. You should see a message on the channel: John Doe added an integration to this channel: Github Actions
From the same setting page, get the Channel ID, you will need to set it in the YAML.
Tweak your existing YAML manifest in .github/workflows to add the following snippet:

  # [...]
  deploy:
    name: 'Deployment'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      # - name: Deploy
      #   run: deploy.sh
      # [...]

      - name: notify slack
        id: slack
        uses: slackapi/slack-github-action@v1.24.0
        # https://api.slack.com/apps/<your-app-id>/incoming-webhooks
        if: ${{ failure() }}
        with:
          channel-id: 'F04D4MXFY3R' # slack channel #alerts-test
          payload: |
            {
              "text": "GitHub Action failed",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Production deployment check failed* :fire_engine:\n High error rate detected after deployment! Rolling back..."
                  }
                },
                {
                  "type": "actions",
                  "elements": [
                    {
                      "type": "button",
                      "text": {
                        "type": "plain_text",
                        "text": ":github: Failed action"
                      },
                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                    },
                    {
                      "type": "button",
                      "text": {
                        "type": "plain_text",
                        "text": ":grafana: Grafana dashboard"
                      },
                      "url": "https://<your_organization>.grafana.net/d/my-app-dashboard?from=now-1h&to=now"
                    }
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_NOTIFICATIONS_TOKEN }}
          SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

Feel free to customize the "mrkdwn" part as you want, you can learn more about formatting here: https://api.slack.com/reference/surfaces/formatting.

In the buttons, I like to add the failed action that sent the notification, and a link to the production app grafana dashboard.

💡 Note 1: You can notice I use custom slack emojis to ✨beautify✨ the slack message: github and grafana.
Check this link on how to add custom emojis if you don't know yet: https://slack.com/help/articles/206870177-Add-custom-emoji-and-aliases-to-your-workspace
You can go to https://slackmojis.com/ to find the Github and the Grafana ones.

💡 Note 2: You can use if: ${{ failure() }} on the step level to only send the notification when the previous step (the deployment) failed :D

2026 updates: I got tired of hacky solutions so I built my own tool for getting the best GitHub notifications in your Slack DMs. Give it a try! gitnotifier.com.

How to add a MySQL Planetscale database as a grafana cloud datasource

François — Fri, 19 May 2023 00:00:00 +0000

The grafana cloud free plan is amazing for most of personal projects! I use it a lot on https://fixtheops.dev, and recently I wanted to display some data from my platnetscale mysql database to visualise the user growth. Platnetscale free plan being amazing as well, I highly advise the platform (https://planetscale.com/).

First you need to add a new datasource, in our case: mysql.

Then head over to Planetscale, and create a new password with read-only rights.

This part is important, as from grafana you can run DROP commands.

Then in the datasource add the host, user, password. As Planetscale is using SSL, make sure to toggle "With CA cert" option.

In authentication details, you need to provide the TLS/SSL root certificate. Here is how to obtain it:

# if you have pbcopy on mac:
curl https://letsencrypt.org/certs/isrgrootx1.pem | pbcopy

# otherwise, copy paste what you get from the curl command
curl https://letsencrypt.org/certs/isrgrootx1.pem

How it should look like:

And here you are! All good to build beautiful dashboards for free ✨

Forem: François

My new programming language is English

My new programming language is not TypeScript, Python, Go, or even Rust. It is English.

Agents are like super smart interns.

I wanted the best GitHub notifications in Slack, so I built it

The blind spot: post-merge CI failures on main

Bots now talk more than humans

Existing tools did not really solve it for me

Most PR reviews involve 2 people, so why spam a channel?

What I wanted instead

Do things from Slack, do not just get pinged there

DX matters: setup should be boring

What changed for me

Stop committing with the wrong email: Multiple Git Configs for GitHub

The Problem

The Solution: Multiple Git Config Files

Step 1: Create Your SSH Keys (optional)

Step 2: Add Keys to SSH Agent

Step 3: Configure SSH for Different Hosts

Step 4: Set Up Git Config Files

Main Git Config (~/.gitconfig)

Personal Git Config (~/.gitconfig-perso)

Professional Git Config (~/.gitconfig-pro)

Step 5: Add SSH Keys to GitHub

Step 6: Set Up Commit Signing (Optional but Recommended)

Configure Git for SSH Signing

Bonus (hide your email) 🥷

Conclusion

Stop using "GitOps" to sell your products

1. Declarative

2. Versioned and Immutable

3. Pulled Automatically

4. Continuously Reconciled

Final notes

Sources

How to increase AWS ELB keep-alive timeout when deployed from Traefik

NodeJS Best Practices: Redacting Secrets from Your Pino Logs

Pino

Redaction

Good bye sleep() and delay() with NodeJS, hello setTimeout()

How to use OpenTelemetry to expose custom Prometheus metrics from nodeJS applications

Example nodeJS project

and... with Kubernetes? ☸️

Conclusion

How to stream big JSON files with low-memory footprint in nodeJS

Streaming big objects with nodejs

Generate data

JSON stream NPM Module

Code example

How to debug RDS database from a Kubernetes cluster

Sending slack notifications with github actions

How to add a MySQL Planetscale database as a grafana cloud datasource

Main Git Config (`~/.gitconfig`)

Personal Git Config (`~/.gitconfig-perso`)

Professional Git Config (`~/.gitconfig-pro`)