Forem: Ivan Pesenti

Have you ever run Robocopy on Linux?

Ivan Pesenti — Mon, 24 Nov 2025 11:24:06 +0000

Cross-platform is a term we're used to hear several times per day. It means a component can run on different platforms without any significant modifications. In the context of software, it means the same piece of software can be run on different operating systems without worrying about the underlying platform that runs it.

Of course, there are some situations where this doesn't apply, but let's assume this scenario doesn't exist.

We can summarize the cross-platform concept with the motto: "Build once, run everywhere".

If this motto rang a bell in your head, you're in the right place, and you should go ahead with the reading. In fact, this is one of the fundamental pillars of Docker.

Today, I'll show how I put in practice this concept in the last pet project I did.

Windows Robocopy... On Linux

Disclaimer: I wasn't able to run Windows Robocopy on my machine running Ubuntu 24.04 LTS. Robocopy is a Windows-based tool. In Linux, there are alternatives like cp or rsync, among others. Being able to run Robocopy on Linux is beyond cross-platform; it's more like magic.

The idea behind this project has been to develop, build, run, and test a Windows-native application on an incompatible machine, which was Linux-based.

The final application will be consumed directly in your terminal, exposing a Command-Line-Interface (in short CLI) API.

The Journey

In a nutshell, the journey of this application has consisted of:

Development of the core feature
Set up the CI/CD process to build and release the application
Manual run of the binary on a Windows machine for testing purposes
Automatic testing of the application on a Windows machine

The most appealing part turned out to be the automatic testing of the application (point 4).

In today's adventure, my travel companions will be #Go, #Docker, and the #dockerSDK. In case you're not familiar with the DockerSDK, it's a smart way to programmatically interact with the Docker Daemon from your Go code. Thanks to this pet project, I managed to familiarize myself with it.

Disclaimer: if you're eager to see the code, you can check it out in my GitHub profile.

Now, let's unveil ten uh-oh moments I came across.

1. `dockur/windows` image

The dockur/windows image discovery has been mindblowing. While surfing the Internet, I randomly came across this Docker image that allows you to run Windows... in a container!

Visit the Dockur GitHub page for more details. Over there, you can also find a super nice introductory video to quickly ramp up.

If you're curious about how this image works under the hood, refer to their documentation, where you should find every technical detail.

In the documentation, there's a handy docker-compose.yml ready to use. It resembles something like this:

services:
  windows:
    image: dockurr/windows
    container_name: windows
    environment:
      VERSION: "11l"
      KEYBOARD: "it-IT"
      REGION: "en-US"
    devices:
      - /dev/kvm
      - /dev/net/tun
    cap_add:
      - NET_ADMIN
    ports:
      - 8006:8006
      - 5985:5985
      - 3389:3389/tcp
      - 3389:3389/udp
    volumes:
      - ./windows:/storage
      - ./testdata:/shared
    restart: always
    stop_grace_period: 2m

This is an extremely powerful tool, but it comes with considerations.

First Run

If you're familiar with Windows-based operating systems, you'd also be familiar with their setup process... The first time you run this image, you are walked through the initialization process where you can customize settings.

The last thing you want is to have to go, yet another time, through this lengthy process. To bypass this, you can use a Docker volume:

    volumes:
      - ./windows:/storage

For convenience, I used the windows folder, located in the root of my project.

TL;DR: Embrace a bit of patience and run this container.

Purposes

This Windows container is used for two purposes:

performing manual testing
performing automatic testing

The first consists of manually connecting to the Windows Desktop instance (you can easily do that in your browser), performing the action, and checking the results. The latter is done in the end-to-end test that leverages the capabilities of a tool called Windows Remote Management (WinRM).

Having this properly set up was mandatory for the successful project.

Automated Testing Process

The diagram below depicts the automated testing process our application should undergo to be considered healthy:

The process is straightforward:

Building the Docker image starting from the source code
Creation of a container with the final executable binary (called extractor)
Copying out the binary from the extractor container
Copying the binary to the running Windows container
Testing the binary on the Windows container

Now, let's start exploring the capabilities offered by the Docker SDK.

2. The Docker Client

The Docker SDK wraps the HTTP calls done against the Docker Daemon. These are the same calls done when interacting with Docker via the Docker CLI or the Docker Desktop environment. To consume these APIs, a Docker Client must be created in our code. That's where the client.Client struct shines.

The client.Client exposed by the Docker Module provides you with a handle to interact.

We initialized the Docker client in the robocopy_test.go file:

func setupDockerClient(ctx context.Context) (err error) {
 dockerClient, err = client.NewClientWithOpts(client.FromEnv)
 if err != nil {
  return err
 }
 dockerClient.NegotiateAPIVersion(ctx)
 return nil
}

Please note dockerClient.NegotiateAPIVersion(ctx) is needed to match the version used by the API.

All the subsequent code snippets are taken from the robocopy_test.go file.

3. List Containers

Listing the containers on your host can be done via the ContainerList method, as you can see below:

func getWindowsContainerID(ctx context.Context, dockerClient *client.Client) (*string, error) {
 containers, err := dockerClient.ContainerList(ctx, container.ListOptions{})
 if err != nil {
  return nil, err
 }
 for _, c := range containers {
  if c.Image == windowsImageName {
   return &c.ID, nil
  }
 }
 return nil, fmt.Errorf("no running Windows container found")
}

4. Building a Docker image

One of the most interesting parts. The code looks like this:

func buildDockerImageForRobocopyBinary(ctx context.Context) error {
 // create tarball from source code
 buf := new(bytes.Buffer)
 tw := tar.NewWriter(buf)
 defer tw.Close()
 if err := createTarball(".", tw); err != nil {
  return err
 }
 twReader := bytes.NewReader(buf.Bytes())
 imageBuildRes, err := dockerClient.ImageBuild(ctx, twReader, build.ImageBuildOptions{
  Tags:   []string{"test-my-robocopy"},
  Remove: true,
 })
 if err != nil {
  return err
 }
 defer imageBuildRes.Body.Close()
 if _, err := io.Copy(os.Stdout, imageBuildRes.Body); err != nil {
  return err
 }
 return nil
}

The process consists of two steps:

creation of a tar archive with the application's source code
running the ImageBuild method by passing:
- a tar reader
- the image build options

The unspecified options resolve with the default values. Please be sure to have the Dockerfile placed in the root of the project

I encourage you to dig into the createTarball function to learn more about the creation of the tar archive.

5, 6, 7 Containers Management

Here, there are three things tied together. Let me share the code, and then I'll walk you through:

func copyBinaryToWindowsContainer(ctx context.Context) error {
 // creation of the extractor container
 containerRes, err := dockerClient.ContainerCreate(ctx, &container.Config{
  Image: "test-my-robocopy",
  Cmd:   []string{"echo", "hello"},
 }, nil, nil, nil, "extractor")
 if err != nil {
  return err
 }
 extractorContainerID = containerRes.ID
 // pull out the binary from the extractor container
 reader, _, err := dockerClient.CopyFromContainer(ctx, containerRes.ID, "/go-robocopy.exe")
 if err != nil {
  return err
 }
 defer reader.Close()
 // copy binary to the target Windows Container
 return dockerClient.CopyToContainer(ctx, *windowsContainerID, "./shared", reader, container.CopyToContainerOptions{})
}

The process is:

creating the extractor container based on the test-my-robocopy image we tagged before
copying from the extractor container the binary produced (/go-robocopy.exe)
copying to the windows container the executable binary

8, 9 Images Management

Now, we're entering the cleanup process. The goal is to get rid of the image built for the test. First, the code:

func removeDockerImagesByRepoTags(ctx context.Context, repo, tag string) error {
 filters := filters.NewArgs(filters.Arg("label", fmt.Sprintf("repo=%v", repo)), filters.Arg("label", fmt.Sprintf("tag=%v", tag)))
 images, err := dockerClient.ImageList(ctx, image.ListOptions{Filters: filters})
 if err != nil {
  return err
 }
 for _, img := range images {
  if _, err := dockerClient.ImageRemove(ctx, img.ID, image.RemoveOptions{}); err != nil {
   return err
  }
 }
 return nil
}

The process is:

building the filter since we want to filter images on two criteria:
- repo where we passed in the value test-my-robocopy
- tag where we passed in the value latest since we did not specify any tag for the image (bad practice: always specify tags)
retrieval of matching Docker images
Docker images removal

10. Container Removal

Last, we remove the extractor container with this trivial cleanup function:

func removeContainerByID(ctx context.Context, containerID string) error {
 return dockerClient.ContainerRemove(ctx, containerID, container.RemoveOptions{})
}

Untold Stories

If you navigate the GitHub Repository, you'll notice a lot of stuff that has not been covered. These can be topics for future posts. An extract of the list can be:

Git Hooks
GitHub Actions
Goreleaser
tar archive in Go
Windows Remote Management (WinRM)
working with CLI-apps and flags in Go

If you want me to cover any of these topics, feel free to reach out.

Outro

I had a lot of fun with this project. Especially, I could have adopted any unneeded complexity without having to worry about the old-good patterns, best practices, and guidelines. After all, this is the beauty of useless pet projects.

Thanks for your attention, folks! If you have any questions, doubts, feedback, or comments, I'm available to listen and discuss. If you want me to cover some specific concepts, please reach out.

The Return of the Living Dead: A Docker Spin-Off 🎦

Ivan Pesenti — Mon, 09 Jun 2025 10:50:06 +0000

Zombie fighting has always been one of the most appealing subjects in cinema. Defeating a zombie with a fancy weapon and preventing it from infecting other human beings provides pure adrenaline. Usually, the hero wins, everyone is happy, good triumphs over evil, and so on... Hold on! Can you imagine if zombies had also infected the last survivor of our species, and no one was left to protect humanity's hope? 👻

Have you ever seen a film like this before? I don't think so.

In this blog post, you'll be the savior, and the unwanted running containers will be the zombies to get rid of. Enough cinema, let's switch back to Go 🐭

In the Previous Episodes ⬅️

In the previous episode, we explored how to run an application that depends on a Graphite instance. We ran it via docker compose and also used it for integration tests by leveraging the capabilities of testcontainers-go.

Check out my blog post if you missed it.

The docker-compose.yml file we used was:

services:
  webserver:
    build: "."
    container_name: webserver
    restart: always
    environment:
      - GRAPHITE_HOSTNAME=graphite
      - GRAPHITE_PLAINTEXT_PORT=2003
    ports:
      - 8080:8080
    depends_on:
      graphite:
        condition: service_healthy
    networks:
      - todo-network

  graphite:
    image: graphiteapp/graphite-statsd
    container_name: graphite
    restart: always
    ports:
      - 80:80
      - 2003-2004:2003-2004
      - 2023-2024:2023-2024
      - 8125:8125/udp
      - 8126:8126
    healthcheck:
      test: ["CMD-SHELL", "netstat -an | grep -q 2003"]
      interval: 10s
      retries: 3
      start_period: 30s
      timeout: 10s
    networks:
      - todo-network

networks:
  todo-network:
    driver: bridge

Please refer to the previous post if you want a deeper understanding or if you just want a refresh.

Living Dead 🧟‍♀️🧟‍♂️

What has originated the Living Dead? When did it all begin?

To follow along with the tutorial, you must have Docker CLI and Docker Compose installed on your machine. If you're on Windows/MacOS, you likely already have everything needed. If you're on Linux, just confirm you have everything installed.

To understand the issue, let's reproduce it first:

Start the compose stack with the command docker-compose up -d
Confirm you can see the two running containers with the command docker ps. If you look at the NAMES column of the output (the last column), you should see respectively the names graphite and webserver
Run the command go test ./tests -tags=integration to start our integration tests and... 💣💣

The error should resemble something like:

--- FAIL: TestGetTodos (0.86s)
    get_todos_test.go:17: 
                Error Trace:    $HOME/graphite-poc/tests/container.go:24
                                                        $HOME/graphite-poc/tests/get_todos_test.go:17
                Error:          Received unexpected error:
                                compose up: Error response from daemon: Conflict. The container name "/graphite" is already in use by container "ceb36087cd8f58a82bdc07d755c36b7e1d043b5e4a670a37cf44c1947f6c7826". You have to remove (or rename) that container to be able to reuse that name.
                Test:           TestGetTodos
FAIL
FAIL    github.com/ossan-dev/graphitepoc/tests  7.805s
FAIL

DISCLAIMER. The upcoming solution terminates the containers based on their name. This could be disruptive if done without a mindful process. It's safe to do on your machine. Before shipping this code to production, please adopt the needed adjustments.

Our Crusade ⚔️

To defeat the Living Dead (i.e., the undesired running containers), we need to go through these stages: understand the problem, identify the code that caused it, and fix it.

Back to our cinema analogy, I've never seen a hero winning just because it defeated hundreds of thousands of zombies. He must find a smart way to tackle the problem at the root. So there we go.

Look at the Error 🔎

The issue is clearly reported: we're trying to spin up a container called graphite, but there's already a namesake container running. We have a conflict on the container name, similar to when you're trying to violate a UNIQUE CONSTRAINT you might have on a column in a database. It also proposes two options for dealing with the old container:

Renaming
Removing

I would go with the second option since I don't want to keep memory allocated, but, more importantly, I want to free the TCP ports allocated to the conflicting container.

Identifying the Root Cause

Now, by leveraging the information contained in the error log, let's find the file that has caused the issue.

The culprit is the spawnWebServerContainer function, contained in the tests/container.go file.

Let's see its content:

func spawnWebServerContainer(t *testing.T) {
 t.Helper()
 os.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true")
 compose, err := tc.NewDockerComposeWith(tc.WithStackFiles("../docker-compose.yml"))
 require.NoError(t, err)
 t.Cleanup(func() {
  require.NoError(t, compose.Down(context.Background(), tc.RemoveOrphans(true), tc.RemoveImagesLocal))
 })
 ctx, cancel := context.WithCancel(context.Background())
 t.Cleanup(cancel)
 err = compose.
  Up(ctx, tc.Wait(true))
 require.NoError(t, err)
}

Digging into it, when we invoke the Up method, we're blindly attempting to spin up the containers defined in the docker-compose.yml file. In case we don't have any previous namesake containers running, everything will be fine. If we're not in this situation (remember the docker-compose up -d command I asked you to run before), our test code will crash by the aforementioned error ⛑️.

Beating the Living Dead 🦸

There are several ways to fix this situation. I'm going to present the solution I feel best suited.

The `reRunContainersAfterConflict() Func

I introduced a new function to run the containers again in case of conflict.

Be aware of "in case of a conflict" since we won't blindly retry to run the containers upon every received error. Container startup could fail for an endless number of reasons beyond the conflicting names.

The function code is as follows:

go func reRunContainersAfterConflict(ctx context.Context, composeReq tc.ComposeStack) error { dockerClient, err := client.NewClientWithOpts() if err != nil { return err } dockerClient.NegotiateAPIVersion(ctx) containers := composeReq.Services() for _, c := range containers { err = dockerClient.ContainerStop(ctx, c, container.StopOptions{}) if err != nil { return err } err = dockerClient.ContainerRemove(ctx, c, container.RemoveOptions{}) if err != nil { return err } } return composeReq. Up(ctx, tc.Wait(true)) }

Let me walk you through the code:

client.NewClientWithOpts() instantiates a new client that wraps the calls sent to the Docker Daemon. It's provided by the official Docker client, github.com/docker/docker/client
dockerClient.NegotiateAPIVersion(ctx) is a "self-healing" 💊 call that fix the potential mismatch we have between the Client version and the API version. You could see something like:

`text
--- FAIL: TestGetTodos (0.89s)
get_todos_test.go:17:
Error Trace: $HOME$/graphite-poc/tests/container.go:49
$HOME/graphite-poc/tests/get_todos_test.go:17
Error: Received unexpected error:
Error response from daemon: client version 1.48 is too new. Maximum supported API version is 1.47
Test: TestGetTodos

FAIL
FAIL github.com/ossan-dev/graphitepoc/tests 8.436s
`
composeReq.Services() lists the containers we have defined in our stack. So we know the containers to terminate
dockerClient.ContainerStop and dockerClient.ContainerRemove are idempotent calls to respectively stop and terminate the unwanted containers

Let's see the logic contained in the function's caller.

The Caller 📲

The caller function is spawnWebServerContainer (you can see its code above).
We had to do a small code change:

go err = compose. Up(ctx, tc.Wait(true)) if err != nil && errdefs.IsConflict(err) { require.NoError(t, reRunContainersAfterConflict(ctx, compose)) return } require.NoError(t, err)

When an error happens, we use the IsConflict function provided by the github.com/docker/docker/errdefS package. This provides us with insights into the type of error, which is crucial for the decision-making process.

Final Episode 🎦

Let's see if the changes helped us to overcome the issue. With the containers up and running (if they aren't running, bring them with docker-compose up -d), we run the command:

text go test ./tests -tags=integration

The output is:

text ok github.com/ossan-dev/graphitepoc/tests 43.035s

Hoora! 🎖️ We finally won against the Living Dead. Time to celebrate 🎉

I hope you learned something today. Maybe, the next time you will have to terminate containers manually, you'll leverage this code.

FYI: I had a lot of fun with the cinema comparison, even if I'm not a cinema person.

Thanks for your attention, folks! If you have any questions, doubts, feedback, or comments, I'm available to listen and discuss. If you want me to cover some specific concepts, please reach out.

Take it easy with Graphite and Docker 🐳

Ivan Pesenti — Wed, 28 May 2025 10:31:23 +0000

I've been stuck recently at work while trying to write an end-to-end test against a web server that was exposing its capabilities via REST APIs. The issue was me trying to make the system under test write some metrics in a specific Graphite instance, used to collect all the metrics emitted in a cloud environment. Ideally, the desired workflow was:

Run a test invoking an HTTP endpoint with an invalid request
The HTTP endpoint processes the request and sends back the response to the client
Contextually, the HTTP endpoint should have also written a metric to the Graphite server to record the failing request
The test, as part of his assertions, should have also checked whether the metric has been emitted or not

Nothing too fancy, you might think... But I was not able to get back a reliable result. I was not satisfied with the result, so that's why I decided to use some time to dig into this 🔎.

TLDR the root cause 🌳

There were a couple of things worth noting in this approach. First and foremost, this check isn't meaningful in the context of the end-to-end test. This should check the correct behavior exposed by the System Under Test and not its internals like emitting a metric or writing a log entry.

The second problem was the query used to retrieve the desired metric. Especially the from query parameter in the Graphite /render API, played a crucial role in getting the expected result (I'll cover that later).

Another issue was the lack of knowledge of the Graphite Server's configuration.

The list could go ahead, but I prefer to stop here to preserve my developer reputation 😆 .

Graphite

The first thing to do was to improve my Graphite knowledge to understand its internals and fully control it 🪄. There you go. Below is a bulleted list with some of the concepts you might need to be aware of:

Graphite is a time-series database or TSDB. Graphite Documentation
In case you're not familiar with TSDB, please refer to this documentation It's vital that you understand how it works before proceeding with this blog post.
There are three components in the Graphite infrastructure:
1. Carbon: the backend of Graphite listening for time-series data. It can ingest data via different protocols. We'll use the plaintext protocol
2. Whisper: text-based file used to store data points received by Carbon
3. Graphite-Web: UI application used to render graphs and dashboards
There are two kinds of metrics that Graphite natively supports:
1. Counter: only-increasing metric
2. Gauge: a picture of a value at a specific time. This is also used to create Timers (histograms)
The metrics are organized in a hierarchical way (with the . used to qualify/organize the metrics). Good naming is fundamental to organizing and fetching metrics properly, by also supporting the usage of wildcard characters. Give a look at this amazing blog post for further details

There's more to cover, but I don't want to be too much off-topic with the blog post.

Docker Come to the Rescue 🛡️

I decided to create a small project to experiment and deep dive into Graphite (yes, you can blame me).

You can already guess my saviors: Docker & testcontainers-go. These two guys saved my day.

Spoiler: it was not the first and it won't be the last 😍

Now, let's dirty our hands.

If you get lost, you can find the full repo at my GitHub account.

I used the typical TODO application since I didn't want to consume extra cognitive load to understand a more complex app. Thus, allowing me to focus only on the technologies I needed to experiment with.

System Under Test 🏋

The app is a simple web server exposing two HTTP: GET routes via REST. Below, I share the most significant files used.

The HTTP Handler 🕸️

The code is in the internal/todos/todos.go file. Here, I share the code for the GetTodoByID handler:

func (t *TodoHandler) GetTodoByID(w http.ResponseWriter, r *http.Request) {
 rawID := r.URL.Query().Get("id")
 if rawID == "" {
  metrics.WriteMetricWithPlaintext(t.GraphiteConn, "webserver.get_todo_by_id.errors.missing_id", 1.0)
  w.WriteHeader(http.StatusBadRequest)
  w.Write([]byte("please provide a TODO ID"))
  return
 }
 id, err := strconv.Atoi(rawID)
 if err != nil {
  metrics.WriteMetricWithPlaintext(t.GraphiteConn, "webserver.get_todo_by_id.errors.invalid_id", 1.0)
  w.WriteHeader(http.StatusBadRequest)
  w.Write([]byte("please provide a numeric TODO ID"))
  return
 }
 for _, v := range todos {
  if v.ID == id {
   data, err := json.MarshalIndent(v, "", "\t")
   if err != nil {
    metrics.WriteMetricWithPlaintext(t.GraphiteConn, "webserver.get_todo_by_id.errors.invalid_format", 1.0)
    w.WriteHeader(http.StatusInternalServerError)
    return
   }
   metrics.WriteMetricWithPlaintext(t.GraphiteConn, "webserver.get_todo_by_id.success", 1.0)
   w.Write(data)
   return
  }
 }
 metrics.WriteMetricWithPlaintext(t.GraphiteConn, "webserver.get_todo_by_id.errors.not_found", 1.0)
 w.WriteHeader(http.StatusNotFound)
 w.Write([]byte("todo not found"))
}

Please look at how I shaped the metrics naming. Keep the names consistent, and it will be much easier to retrieve them and not mess things up.

Metrics Sending 📩

The source code is in the file internal/metrics/manager.go. The content is:

func WriteMetricWithPlaintext(graphiteConn net.Conn, name string, value float64) {
 if _, err := fmt.Fprintf(graphiteConn, "%s %f %d\n", name, value, time.Now().Unix()); err != nil {
  fmt.Println("error while wrapping metrics to Graphite:", err.Error())
 }
}

We're sending metrics via the plaintext protocol. The message must adhere to the following string template %s %f %d\n where:

%s is the metric name like webserver.get_todo_by_id.success
%f is the metric value in float64 like 1.0
%d is the timestamp in Unix format like 1748413179

A sample of the message is like webserver.get_todo_by_id.success 1.0 1748413179, followed by a \n character.

The conn parameter is a simple net/TCP, instantiated in the init() function of the cmd/webserver/main.go file:

func init() {
 graphiteHost := config.GetEnvOrDefault("GRAPHITE_HOSTNAME", "graphite")
 graphitePort := config.GetEnvOrDefault("GRAPHITE_PORT", "2003")
 conn, err := net.Dial("tcp", net.JoinHostPort(graphiteHost, graphitePort))
 if err != nil {
  panic(err)
 }
 todoHandler = todos.NewTodoHandler(conn)
 if todoHandler == nil {
  panic("could not start the application")
 }
}

graphite is the name of the Graphite container we're going to use. Let's see how we can power up our simple yet effective application.

Containerize the Web Server 🎁

The Dockerfile is pretty basic, so I won't spend time covering it.

FROM golang:1.24-alpine AS build

WORKDIR /app

COPY go.mod go.sum ./

RUN go mod tidy && go mod download
RUN go mod verify

COPY . .

RUN go build -o webserver cmd/webserver/main.go

FROM alpine

COPY --from=build /app/webserver /webserver

EXPOSE 8080

CMD [ "./webserver" ]

What's most interesting to cover is the Docker Compose file we will be using to start the two containers at once.

Power Up 🔋

To coordinate the startup of the containers, we will use the docker-compose.yml file. The content is below:

services:
  webserver:
    build: "."
    container_name: webserver
    restart: always
    environment:
      - GRAPHITE_HOSTNAME=graphite
      - GRAPHITE_PLAINTEXT_PORT=2003
    ports:
      - 8080:8080
    depends_on:
      graphite:
        condition: service_healthy
    networks:
      - todo-network

  graphite:
    image: graphiteapp/graphite-statsd
    container_name: graphite
    restart: always
    ports:
      - 80:80
      - 2003-2004:2003-2004
      - 2023-2024:2023-2024
      - 8125:8125/udp
      - 8126:8126
    healthcheck:
      test: ["CMD-SHELL", "netstat -an | grep -q 2003"]
      interval: 10s
      retries: 3
      start_period: 30s
      timeout: 10s
    networks:
      - todo-network

networks:
  todo-network:
    driver: bridge

Pay attention to the following key points:

the todo-network was created to make communication between containers possible
the environment value used in the webserver service to refer to the graphite service
the depends_on condition with service_healthy value defined in the webserver service
the ports mapped in the graphite service (you can map only the ones you need)
the healthcheck defined in the graphite service that links to the depends_on condition defined above

Now, let's see if we can overcome the initial issue of being able to test the correct Graphite metrics emission.

The Test Code 🪖

Here, the big player is testcontainers-go. If you're curious and want to learn more about it, take a look at the documentation.

I'm a huge fan of this package, and I believe it's something you must try in your next project. Let's engage in a discussion if you want to find out more about how I use it in my projects.

With this package, I'm able to spawn a fresh new webserver and graphite containers on each test run. This helps to correctly assess metrics. It provides better isolation and control of what's happening with the Graphite container.

The code I used to interact with the Docker containers is contained in the tests/container.go file:

package tests

import (
 "context"
 "os"
 "testing"

 "github.com/stretchr/testify/require"
 tc "github.com/testcontainers/testcontainers-go/modules/compose"
)

func spawnWebServerContainer(t *testing.T) {
 t.Helper()
 os.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true")
 compose, err := tc.NewDockerComposeWith(tc.WithStackFiles("../docker-compose.yml"))
 require.NoError(t, err)
 t.Cleanup(func() {
  require.NoError(t, compose.Down(context.Background(), tc.RemoveOrphans(true), tc.RemoveImagesLocal))
 })
 ctx, cancel := context.WithCancel(context.Background())
 t.Cleanup(cancel)
 err = compose.
  Up(ctx, tc.Wait(true))
 require.NoError(t, err)
}

It uses the docker-compose.yml to spin up the containers we need in our Integration Tests. It will also add the cleanup code.

Test the HTTP Handler

The test code for the GetTodoByID handler resides in the tests/get_todo_by_id_test.go file.
An extract of its content is:

func TestGetTodoByID(t *testing.T) {
 spawnWebServerContainer(t)
 client := http.Client{}
 // ... success scenario omitted for brevity
 t.Run("Invalid ID", func(t *testing.T) {
  r, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://127.0.0.1:8080/todo?id=abc", nil)
  require.NoError(t, err)
  res, err := client.Do(r)
  require.NoError(t, err)
  require.Equal(t, http.StatusBadRequest, res.StatusCode)

  baseUrl, err := url.Parse("http://127.0.0.1:80/render")
  require.NoError(t, err)
  params := url.Values{}
  params.Add("target", "webserver.get_todo_by_id.errors.invalid_id")
  params.Add("from", "-5min")
  params.Add("format", "json")
  baseUrl.RawQuery = params.Encode()
  require.NoError(t, err)
  r, err = http.NewRequestWithContext(context.Background(), http.MethodGet, baseUrl.String(), nil)
  require.NoError(t, err)
  require.EventuallyWithT(t, func(collect *assert.CollectT) {
   isMetricEmitted, err := isMetricEmitted(client, r, "webserver.get_todo_by_id.errors.invalid_id", 1)
   require.NoError(collect, err)
   require.True(collect, isMetricEmitted)
  }, time.Second*30, time.Second*3, "metric not emitted enough times")
 })

 t.Run("Missing ID", func(t *testing.T) {
  r, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://127.0.0.1:8080/todo?id=", nil)
  require.NoError(t, err)
  res, err := client.Do(r)
  require.NoError(t, err)
  require.Equal(t, http.StatusBadRequest, res.StatusCode)

  baseUrl, err := url.Parse("http://127.0.0.1:80/render")
  require.NoError(t, err)
  params := url.Values{}
  params.Add("target", "webserver.get_todo_by_id.errors.missing_id")
  params.Add("from", "-5min")
  params.Add("format", "json")
  baseUrl.RawQuery = params.Encode()
  require.NoError(t, err)
  r, err = http.NewRequestWithContext(context.Background(), http.MethodGet, baseUrl.String(), nil)
  require.NoError(t, err)
  require.EventuallyWithT(t, func(collect *assert.CollectT) {
   isMetricEmitted, err := isMetricEmitted(client, r, "webserver.get_todo_by_id.errors.missing_id", 1)
   require.NoError(collect, err)
   require.True(collect, isMetricEmitted)
  }, time.Second*30, time.Second*3, "metric not emitted enough times")
 })
}

This code is not test-ready. The focus is on:

the Graphite request to get back the raw metrics. We're targeting the /render API with a bunch of values:
- target is the name of the metric
- from is self-explanatory. It could have been omitted, and, in this case, it would have defaulted to 24 hours. This value is used to adjust the precision of the retrieved data points. Setting it too high or too low could filter out the data points we need
- format could have been several other formats such as csv, raw, png, json, and so on
the isMetricEmitted function is used to issue the HTTP request to Graphite. More details on it below

Let's see the code interacting with Graphite.

The Metrics Checker

The code is contained in the tests/metrics.go file:

type graphiteDataPoints []struct {
 Target string `json:"target"`
 Tags   struct {
  Name string `json:"name"`
 } `json:"tags"`
 Datapoints [][2]any `json:"datapoints"`
}

func isMetricEmitted(client http.Client, req *http.Request, metricName string, expectedNumberOfTimes int) (bool, error) {
 res, err := client.Do(req)
 if err != nil {
  return false, err
 }
 defer res.Body.Close()
 if res.StatusCode != http.StatusOK {
  return false, fmt.Errorf("expected status code to be 200OK, got %d", res.StatusCode)
 }
 var graphiteDataPoints graphiteDataPoints
 err = json.NewDecoder(res.Body).Decode(&graphiteDataPoints)
 if err != nil {
  return false, err
 }
 actualNumberOfTimes := 0
 for _, v := range graphiteDataPoints {
  if v.Tags.Name == metricName {
   for _, vv := range v.Datapoints {
    if vv[0] != nil {
     actualNumberOfTimes++
    }
   }
  }
 }
 if actualNumberOfTimes >= expectedNumberOfTimes {
  return true, nil
 }
 return false, fmt.Errorf("metric: %s emitted %d time(s) out of %d", metricName, actualNumberOfTimes, expectedNumberOfTimes)
}

This is only a regular HTTP request sending.

The trickiest part has been how to successfully build the HTTP request to send out.

Has Everything Worked as Expected? ⁉️

To ensure we decently ~~wasted~~ invested time, let's run the tests. Use the command:

go test ./tests -tags=integration

And you should have back an output like:

ok      github.com/ossan-dev/graphitepoc/tests  45.259s

Our tests have successfully run. I hope you have learned something new today!

Thanks for the attention, folks! If you've got any questions, doubts, feedback, or comments, I'm available to listen and speak together. If you want me to cover some specific concepts, please reach me.

Time to drop the pen and grab a deserved coffee ☕

Leverage Your Test Suite With testcontainers-go & docker-compose

Ivan Pesenti — Fri, 04 Oct 2024 07:29:05 +0000

Welcome back, folks! Today, we will cover the end-to-end tests in an intriguing blog post. If you've never written these kinds of tests or if you strive to improve them, keep reading as I'll walk you through this exciting journey. By the end of the article, you'll know how to empower the usage of the testcontainers-go package to let your test suite shine.

The Premise 📓

Before moving ahead, let's set the boundaries for this blog post since we will cover several concepts, tools, and techniques.

The Survival List 🗡️

Since we'll touch on several topics throughout the rest of the blog post, I feel it's a good idea to put them together here.

The tools I present throughout this blog post are a mix of tools I know well and some I used for the first time. Try not to use these tools without thinking, but evaluate them based on your scenario.

We're going to rely on:

The Go programming language
Docker
The testcontainers-go package with the compose module
The ginkgo testing framework and the gomega assertion package

To avoid bloating the reading, I won't cover every aspect and facet of the topics presented here. I will put the relevant documentation URLs where needed.

The Scenario 📈

Let's assume we need to write end-to-end tests on a project we don't own. In my case, I want to write end-to-end tests on a project written with the Java programming language. Since I didn't know how to code in Java, my testing option was only end-to-end tests. The service I had to test was a set of REST APIs. The solution has been obvious: exercise the endpoints by issuing HTTP requests.

It allows testing the exposed features like a black box. We only have to deal with the public surface: what we send to the server and what we get back from it. Nothing more, nothing less.

We care about the api/accounts endpoint that lists the bank accounts in our database (a MySQL instance). We're going to issue these two requests:

HTTP Method	Address	Expected Status Code
GET	`api/accounts?iban=IT10474608000005006107XXXXX`	`200 StatusOK`
GET	`api/accounts?iban=abc`	`400 StatusBadRequest`

Now, you should have a clearer idea of our goal. So, let's jump into the test code.

Let's Have Fun 💻

In this section, I present all the relevant code we need to write for the dreaded end-to-end tests.

The `docker-compose.yml` file

Since we don't bother with the source code, the starting point is the docker-compose.yml file. The relevant code is:

services:
  mysqldb:
    image: "mysql:8.0"
    container_name: mysqldb
    restart: always
    ports:
      - 3307:3306
    networks:
      - springapimysql-net
    environment:
      MYSQL_DATABASE: transfers_db
      MYSQL_USER: bulk_user
      MYSQL_PASSWORD: root
      MYSQL_ROOT_PASSWORD: root
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

  api_service:
    build: .
    container_name: api_service
    restart: always
    ports:
      - 8080:8080
    networks:
      - springapimysql-net
    environment:
      - spring.datasource.url=jdbc:mysql://mysqldb:3306/transfers_db
      - spring.datasource.username=bulk_user
      - spring.datasource.password=root
    depends_on:
      mysqldb:
        condition: service_healthy
    volumes:
      - .m2:/root/.m2

networks:
  springapimysql-net:

The file content is pretty straightforward. We can summarize the things defined in the following list:

The mysqldb service doesn't deserve any further explanations
The api_service service is the system we're testing
The springapimysql-net network hosts the two services defined above

For further Docker Compose reference, you may have a look here. Now, let's see the end-to-end test code.

The `ginkgo` Testing Framework

The ginkgo testing framework helps us in building the test suite. It's entirely written in Go. Furthermore, it provides a CLI utility to set up and run the tests. Since we will use it later, let's download it from here. You can download it in two ways:

By using the go install command (if you've installed Go on your system)
By downloading the compiled binary (useful if you don't have Go installed on your system)

To check whether you have a working utility on your machine, you can run the command ginkgo version (at the time of writing, I have the version 2.20.2).

Please note that the ginkgo command is not mandatory to run the tests. You can still run the tests without this utility by sticking to the go test command.

However, I strongly suggest downloading it since we will use it to generate some boilerplate code.

Lay the Foundation with Ginkgo

Located in the root directory, let's create a folder called end2end to host our tests. Within that folder, initialize a Go module by issuing the command go mod init path/to/your/module.

Now, it's time to run the command ginkgo bootstrap. It should generate a new file called end2end_suite_test.go. This file triggers the test suite we'll define in a bit.

This approach is similar to the one with the testify/suite package. It enforces the code modularity and robustness since the definition and running phases are separated.

Now, let's add the tests to our suite. To generate the file where our tests will live, run another ginkgo command: ginkgo generate accounts. This time, the file accounts_test.go pops out. For now, let's leave it as is and switch to the terminal. We fix the missing packages by running the Go command go mod tidy to download the missing dependencies locally on our machine.

The `end2end_suite_test.go` file

Let's start with the entry point of the test suite. The content of the file looks neat:

//go:build integration

package end2end

import (
 "testing"

 . "github.com/onsi/ginkgo/v2"
 . "github.com/onsi/gomega"
)

func TestEnd2End(t *testing.T) {
 RegisterFailHandler(Fail)
 RunSpecs(t, "End2End Suite")
}

The only unusual thing might be the dot-import within the import section. You can read more about it in the documentation here.

Whoop! A wild `testcontainers` appears 🐳

At some points, we need some magic to get to the next testing level. It happened to be testcontainers-go. For the sake of this demo, we use the compose module (for further reference, please refer to here).

This tool can run the compose file we saw earlier and execute the end-to-end tests against the running containers.

This is an extract of the testcontainers-go capabilities. If you want to learn more, please refer to the doc or reach out. I'll be happy to walk you through its stunning features.

This package allows running the end-to-end suite with a single command. It's a more consistent and atomic way to run these tests. It allows me to:

Start the containers before the suite
Run the tests relying on these containers
Teardown of containers after the suite and cleanup of the resources used

Having the code written this way can help you avoid the hassle of dealing with docker cli commands and makefiles.

The `accounts_test.go` file

Now, let's look at the code where our tests live.

//go:build integration

package end2end

import (
 "context"
 "net/http"
 "os"

 . "github.com/onsi/ginkgo/v2"
 . "github.com/onsi/gomega"
 tc "github.com/testcontainers/testcontainers-go/modules/compose"
 "github.com/testcontainers/testcontainers-go/wait"
)

var _ = Describe("accounts", Ordered, func() {
 BeforeAll(func() {
 os.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true")
 composeReq, err := tc.NewDockerComposeWith(tc.WithStackFiles("../docker-compose.yml"))
  Expect(err).Should(BeNil())
  DeferCleanup(func() {
   Expect(composeReq.Down(context.Background(), tc.RemoveOrphans(true), tc.RemoveImagesLocal)).Should(BeNil())
  })
 ctx, cancel := context.WithCancel(context.Background())
  DeferCleanup(cancel)
 composeErr := composeReq.
   WaitForService("api_service", wait.ForListeningPort("8080/tcp")).
   Up(ctx, tc.Wait(true))
  Expect(composeErr).Should(BeNil())
 })

 Describe("retrieving accounts", func() {
  Context("HTTP request is valid", func() {
   It("return accounts", func() {
 client := http.Client{}
 r, err := http.NewRequest(http.MethodGet, "http://127.0.0.1:8080/api/accounts?iban=IT10474608000005006107XXXXX", nil)
 res, err := client.Do(r)
    Expect(err).Should(BeNil())
    Expect(res).To(HaveHTTPStatus(http.StatusOK))
   })
  })

  Context("HTTP request is NOT valid", func() {
   It("err with invalid IBAN", func() {
 client := http.Client{}
 r, err := http.NewRequest(http.MethodGet, "http://127.0.0.1:8080/api/accounts?iban=abcd", nil)
    Expect(err).Should(BeNil())
 res, err := client.Do(r)
    Expect(err).Should(BeNil())
    Expect(res).To(HaveHTTPStatus(http.StatusBadRequest))
   })
  })
 })
})

At first glimpse, it might seem hard to digest. To keep things easier, let's break it down into smaller parts.

The `Describe` container node

The Describe container node is nothing but a wrapper to hold the relevant code for our suite. Everything must live within it. It's part of the scaffolded code: var _ = Describe("accounts", Ordered, func() {}. Within the {}, you should put all of the relevant code. To enforce the usage of setup nodes (like BeforeAll), we must define the Describe container as Ordered.

Do not worry if you forgot to add it since the Go compiler will complain.

Let's move on.

The `BeforeAll` setup node

This node allows us to extract the common setup logic. This code portion executes once and before the tests within the suite. Let's recap what's doing:

Set the environment variable TESTCONTAINERS_RYUK_DISABLED to true. You can learn about the configuration here. If you're curious about Ryuk, you may want to look at this
Create a *tc.DockerCompose variable based on the docker-compose.yml file we provided
Defer the function invocation to terminate containers and cleanup of the resources
Start the compose stack and wait for the container called api_service to be up and ready to listen on the 8080/tcp port

I simplified the test code since I don't want to make this blog post even longer 😆.

Finally, the tests! 🥁

The test functions live within a Describe Container Node. You can find out how ginkgo handles the test specifications by referring to here. The Describe node allows you to group and organize tests based on their scope. You can nest this node inside other Describe ones.

The more you nest the Describe node, the more you narrow the test scope.

Then, we have the Context Container Node that qualifies the parent Describe. It qualifies the circumstances under which the tests are valid. Finally, we have the It section, the Spec Subject. It's the actual test we're performing and is the leaf level of the hierarchy tree. The test code is self-explanatory, so I'll jump to the section where we run the tests.

3, 2, 1... 🚀

Congrats 🏆 We managed to get here. Now, we only miss the test-running operations. In the blink of an eye, we'll get our test execution report printed onto the terminal.

Let's switch to the terminal and run the command ginkgo --tags=integration -v. After a while, you'll see the output printed on the terminal.

Closing Notes 🤝

I know there are a lot of things condensed into this blog post. My goal has been to provide insights and approaches on how to write a good testing suite. You may want to adapt the presented tools, packages, and techniques to other kinds of tests or use cases.

Before leaving, I'd like to underline another beauty of the compose module of the testcontainers-go package.

If you stick to the configuration I provided, you're sure to use the latest Docker images, and you can avoid hours of troubleshooting due to outdated image usage. It's analogous to the command docker compose build --no-cache && docker compose up. You'll thank me 😹

Gorm: Sneak Peek of Custom Data Types

Ivan Pesenti — Fri, 13 Sep 2024 10:35:41 +0000

Welcome back, folks 👀! Today, we discuss a specific use case we might face when moving data back and forth from/to the database. First, let me set the boundaries for today's challenge. To stick to a real-life example, let's borrow some concepts from the U.S. Army 🪖. Our deal is to write a small software to save and read the officers with the grades they have achieved in their careers.

Gorm's Custom Data Types

Our software needs to handle the army officers with their respective grades. At first look, it might seem easy, and we probably don't need any Custom Data Type here. However, to show off this feature, let's use a non-conventional way to represent the data. Thanks to this, we're asked to define a custom mapping between Go structs and DB relations. Furthermore, we must define a specific logic to parse the data. Let's expand on this by looking at the program's targets 🌏.

Use Case to Handle

To ease things down, let's use a drawing to depict the relationships between the code and the SQL objects:

Let's focus on each container one by one.

The Go Structs 🧱

Here, we defined two structs. The Grade struct holds a non-exhaustive list of military grades 🎖️. This struct won't be a table in the database. Conversely, the Officer struct contains the ID, the name, and a pointer to the Grade struct, indicating which grades have been achieved by the officer so far.

Whenever we write an officer to the DB, the column grades_achieved must contain an array of strings filled in with the grades achieved (the ones with true in the Grade struct).

The DB Relations 🙌

Regarding the SQL objects, we have only the officers table. The id and name columns are self-explanatory. Then, we have the grades_achieved column that holds the officer's grades in a collection of strings.

Whenever we decode an officer from the database, we parse the grades_achieved column and create a matching "instance" of the Grade struct.

You might have noticed that the behavior is not the standard one. We must make some arrangements to fulfill it in the desired way.

Here, the models' layout is overcomplicated on purpose. Please stick to more straightforward solutions whenever possible.

Custom Data Types

Gorm provides us with Custom Data Types. They give us great flexibility in defining the retrieval and saving to/from the database. We must implement two interfaces: Scanner and Valuer 🤝. The former specifies a custom behavior to apply when fetching data from the DB. The latter indicates how to write values in the database. Both help us in achieving the non-conventional mapping logic we need.

The functions' signatures we must implement are Scan(value interface{}) error and Value() (driver.Value, error). Now, let's look at the code.

The Code

The code for this example lives in two files: the domain/models.go and the main.go. Let's start with the first one, dealing with the models (translated as structs in Go).

The `domain/models.go` file

First, let me present the code for this file:

package models

import (
 "database/sql/driver"
 "slices"
 "strings"
)

type Grade struct {
 Lieutenant bool
 Captain    bool
 Colonel    bool
 General    bool
}

type Officer struct {
 ID             uint64 `gorm:"primaryKey"`
 Name           string
 GradesAchieved *Grade `gorm:"type:varchar[]"`
}

func (g *Grade) Scan(value interface{}) error {
 // we should have utilized the "comma, ok" idiom
 valueRaw := value.(string)
 valueRaw = strings.Replace(strings.Replace(valueRaw, "{", "", -1), "}", "", -1)
 grades := strings.Split(valueRaw, ",")
 if slices.Contains(grades, "lieutenant") {
 g.Lieutenant = true
 }
 if slices.Contains(grades, "captain") {
 g.Captain = true
 }
 if slices.Contains(grades, "colonel") {
 g.Colonel = true
 }
 if slices.Contains(grades, "general") {
 g.General = true
 }
 return nil
}

func (g Grade) Value() (driver.Value, error) {
 grades := make([]string, 0, 4)
 if g.Lieutenant {
 grades = append(grades, "lieutenant")
 }
 if g.Captain {
 grades = append(grades, "captain")
 }
 if g.Colonel {
 grades = append(grades, "colonel")
 }
 if g.General {
 grades = append(grades, "general")
 }
 return grades, nil
}

Now, let's highlight the relevant parts of it 🔎:

The Grade struct only lists the grades we forecasted in our software
The Officer struct defines the characteristics of the entity. This entity is a relation in the DB. We applied two Gorm notations:
1. gorm:"primaryKey" on the ID field to define it as the primary key of our relation
2. gorm:"type:varchar[]" to map the field GradesAchieved as an array of varchar in the DB. Otherwise, it translates as a separate DB table or additional columns in the officers table
The Grade struct implements the Scan function. Here, we get the raw value, we adjust it, we set some fields on the g variable, and we return
The Grade struct also implements the Value function as a value receiver type (we don't need to change the receiver this time, we don't use the * reference). We return the value to write in the column grades_achieved of the officers table

Thanks to these two methods, we can control how to send and retrieve the type Grade during DB interactions. Now, let's look at the main.go file.

The `main.go` file 🚦

Here, we prepare the DB connection, migrate the objects to relations (ORM stands for Object Relation Mapping), and insert and fetch records to test the logic. Below is the code:

package main

import (
 "encoding/json"
 "fmt"
 "os"

 "gormcustomdatatype/models"

 "gorm.io/driver/postgres"
 "gorm.io/gorm"
)

func seedDB(db *gorm.DB, file string) error {
 data, err := os.ReadFile(file)
 if err != nil {
  return err
 }
 if err := db.Exec(string(data)).Error; err != nil {
  return err
 }
 return nil
}

// docker run -d -p 54322:5432 -e POSTGRES_PASSWORD=postgres postgres
func main() {
 dsn := "host=localhost port=54322 user=postgres password=postgres dbname=postgres sslmode=disable"
 db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
 if err != nil {
 fmt.Fprintf(os.Stderr, "could not connect to DB: %v", err)
  return
 }
 db.AutoMigrate(&models.Officer{})
 defer func() {
 db.Migrator().DropTable(&models.Officer{})
 }()
 if err := seedDB(db, "data.sql"); err != nil {
 fmt.Fprintf(os.Stderr, "failed to seed DB: %v", err)
  return
 }
 // print all the officers
 var officers []models.Officer
 if err := db.Find(&officers).Error; err != nil {
 fmt.Fprintf(os.Stderr, "could not get the officers from the DB: %v", err)
  return
 }
 data, _ := json.MarshalIndent(officers, "", "\t")
 fmt.Fprintln(os.Stdout, string(data))

 // add a new officer
 db.Create(&models.Officer{
 Name: "Monkey D. Garp",
 GradesAchieved: &models.Grade{
 Lieutenant: true,
 Captain:    true,
 Colonel:    true,
 General:    true,
  },
 })
 var garpTheHero models.Officer
 if err := db.First(&garpTheHero, 4).Error; err != nil {
 fmt.Fprintf(os.Stderr, "failed to get officer from the DB: %v", err)
  return
 }
 data, _ = json.MarshalIndent(&garpTheHero, "", "\t")
 fmt.Fprintln(os.Stdout, string(data))
}

Now, let's see the relevant sections of this file. First, we define the seedDB function to add dummy data in the DB. The data lives in the data.sql file with the following content:

INSERT INTO public.officers
(id, "name", grades_achieved)
VALUES(nextval('officers_id_seq'::regclass), 'john doe', '{captain,lieutenant}'),
(nextval('officers_id_seq'::regclass), 'gerard butler', '{general}'),
(nextval('officers_id_seq'::regclass), 'chuck norris', '{lieutenant,captain,colonel}');

The main() function starts by setting up a DB connection. For this demo, we used PostgreSQL. Then, we ensure the officers table exists in the database and is up-to-date with the newest version of the models.Officer struct. Since this program is a sample, we did two additional things:

Removal of the table at the end of the main() function (when the program terminates, we would like to remove the table as well)
Seeding of some dummy data

Lastly, to ensure that everything works as expected, we do a couple of things:

Fetching all the records in the DB
Adding (and fetching back) a new officer

That's it for this file. Now, let's test our work 📸.

The Truth Moment

Before running the code, please ensure that a PostgreSQL instance is running on your machine. With Docker 🐋, you can run this command:

docker run -d -p 54322:5432 -e POSTGRES_PASSWORD=postgres postgres

Now, we can safely run our application by issuing the command: go run . 🚀

The output is:

[
        {
                "ID": 1,
                "Name": "john doe",
                "GradesAchieved": {
                        "Lieutenant": true,
                        "Captain": true,
                        "Colonel": false,
                        "General": false
                }
        },
        {
                "ID": 2,
                "Name": "gerard butler",
                "GradesAchieved": {
                        "Lieutenant": false,
                        "Captain": false,
                        "Colonel": false,
                        "General": true
                }
        },
        {
                "ID": 3,
                "Name": "chuck norris",
                "GradesAchieved": {
                        "Lieutenant": true,
                        "Captain": true,
                        "Colonel": true,
                        "General": false
                }
        }
]
{
        "ID": 4,
        "Name": "Monkey D. Garp",
        "GradesAchieved": {
                "Lieutenant": true,
                "Captain": true,
                "Colonel": true,
                "General": true
        }
}

Voilà! Everything works as expected. We can re-run the code several times and always have the same output.

That's a Wrap

I hope you enjoyed this blog post regarding Gorm and the Custom Data Types. I always recommend you stick to the most straightforward approach. Opt for this only if you eventually need it. This approach adds flexibility in exchange for making the code more complex and less robust (a tiny change in the structs' definitions might lead to errors and extra work needed).

Keep this in mind. If you stick to conventions, you can be less verbose throughout your codebase.

That's a great quote to end this blog post.
If you realize that Custom Data Types are needed, this blog post should be a good starting point to present you with a working solution.

Please let me know your feelings and thoughts. Any feedback is always appreciated! If you're interested in a specific topic, reach out, and I'll shortlist it. Until next time, stay safe, and see you soon!

Go Gin Templates: Break Them Down

Ivan Pesenti — Fri, 02 Aug 2024 13:28:17 +0000

Today Problem

Welcome back, folks! Today's problem ❌ is how to embed one HTML template into another. In this way, we can foster the reusability of the code and make it less redundant. Let's pretend to have two HTML pages to render. These are identical unless for the middle part of the page. Why don't we have a header, a footer, and two bodies with only the specific page's information? You already figured it out since it's a typical use case. It's worth to cover. Let's see how we can achieve it with Go and the Gin web framework.

What is HTML Templating?

Before getting straight into the code, let's talk a little about HTML templating. It allows you to blend HTML tags with values provided later. Gin manages the HTML templating by empowering the usage of the package html/template that belongs to Go's Standard Library. You can read more here. Now, let's start presenting the solution 🚀.

The solution

First, let's look at the HTML templates, and then we'll see how to invoke them in the code and how to run some tests. Let's present the final HTML pages we'd like to have:

As you can see, the headers and footers are identical. However, the "core" parts differ based on the requested page.

HTML Templates

All the templates lay in the templates folder. The files are:

header.tmpl
page1.tmpl
page2.tmpl
footer.tmpl

First, let's cover the static ones.

The `header.tmpl` file

Its content is:

{{define "header" }}
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>{{ .title }}</title>
</head>
<body>
 <h1>BASELINE</h1>
 <div id="content">
{{end}}

Here, there are a couple of things to pay attention to. All the template's code resides between the start and end definitions of the template. In this case, the boundaries are {{define "header" }} and {{end}}. Then, we use a parameter called title within the <title> tag that can be assigned whenever we invoke this template from the child templates. More on that later.

The `footer.tmpl` file

Here, the content is even easier:

{{define "footer"}}
</div>
<p>FooterInfo</p>
</body>
</html>
{{end}}

The footer template is the final part of all the pages that will always be consistent.

The `page1.tmpl` file

Now, the spicy part 🌶️. The HTML part that changes lives in the page1.tmpl file. The content is as follows:

{{template "header" . }}
<h2>Welcome to Page 1</h2>
<p>This is the content of Page 1.</p>
{{template "footer"}}

Here, there are a couple of things worth mentioning. First, we add the reference to the header template by using the line {{template "header" . }}.

There's a 1:1 relationship between the names used in the header.tmpl and page1.tmpl files (e.g. in both we used header).

The . means to pass along over the pipeline all the parameters. Thanks to this way, we can pass the value Page 1 IS HERE to the page1.tmpl file that, in turn, will pass it to the header template that is the one using it.

Then, we type the specific content for this page and reference the footer template to spread consistency around 🎇.

Please be aware that I skipped the code for the page2.tmpl file since it's almost identical to the one just seen.

Now, let's take a look at the Go code.

The Go Code

Here, we have to tie everything up and be able to respond to the HTTP endpoints. For the sake of the demo, the whole code is in the main.go file. First, let me present the code, and then I'll walk you through all the relevant sections of it.

package main

import (
 "fmt"
 "net/http"
 "os"

 "github.com/gin-gonic/gin"
)

func main() {
 gin.SetMode(gin.DebugMode)
 r := gin.Default()

 // load templates
 r.LoadHTMLGlob("templates/*.tmpl")

 r.GET("/page1", func(c *gin.Context) {
 c.HTML(http.StatusOK, "page1.tmpl", gin.H{
   // this value goes to the "header.tmpl" file
   "title": "Page 1 IS HERE",
  })
 })

 r.GET("/page2", func(c *gin.Context) {
 c.HTML(http.StatusOK, "page2.tmpl", gin.H{
   "title": "Page 2 IS HERE",
  })
 })

 if err := r.Run(":8080"); err != nil {
 fmt.Fprintf(os.Stderr, "cannot run the srv: %v\n", err)
 os.Exit(1)
 }
}

The first noticeable thing is the r.LoadHTMLGlob("templates/*.tmpl") call. It loads all the matching templates in a specific directory. For completeness, you could also have used r.LoadHTMLFiles() to specify the files you want to load.

In real-life projects, do not hard-code the templates' filesystem path in your code.

Then, we instrument the router to answer GET HTTP requests that hit the /page1 and /page2 addresses. Within the HandlerFunc body, we used the method c.HTML that allows us to set:

The HTTP Response Status Code
The template to use for this kind of request
Optional parameters we may want to pass to the template to make it more dynamic

We used the type gin.H for parameters to have great flexibility since it's based on the underlying type map[string]any.
Finally, we launch our server and catch 🔎 the eventual error.

Give It a Try

Now, let's make sure that everything works as expected. Issue go run ., and checks that the server started without problems. To test it out, we're going to use the curl command (you may need to install it on your machine. More on it here):

curl http://localhost:8080/page1

The output printed in the shell should be this one:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Page 1 IS HERE</title>
</head>
<body>
    <h1>BASELINE</h1>
    <div id="content">

<h2>Welcome to Page 1</h2>
<p>This is the content of Page 1.</p>

</div>
<p>FooterInfo</p>
</body>
</html>

You can also try it directly from the web browser of your choice. It doesn't matter, you'll have the same result.

That's a Wrap

This blog post is not exhaustive about HTML templating. I encourage you to have a look at these resources:

Any feedback is highly appreciated.

Before leaving, I strongly invite you to reach out if you are interested in some topics and want me to address them. I swear I shortlist them and do my best to deliver helpful content.

Thank you very much for the support, and see you in the next one 👋

Unit Tests & Mocking: the Bread and the Butter

Ivan Pesenti — Fri, 12 Jul 2024 08:36:20 +0000

Premise

Welcome back, folks 🤝 After a while, I finally took the time to start this new series of blog posts around computer science, programming, and Go. This series will be focused on different things we can deal with when dealing with our daily work.

How you should read it

Before getting into it, let me share how this blog post (and the upcoming ones) is meant to be read. This blog post targets a specific subject 📍. Therefore, I suggest you read other resources to gain a broader overview. It aims to be your starting point to further dig into the topic. For sure, I'll manage to share resources whenever necessary. Finally, there won't be any GitHub repositories since the code will be pretty focused and not part of a project. Now, you're ready to embark on the journey with unit testing and Go.

Unit Tests & Mocking

The goal is to be able to write a unit test for a struct that holds a dependency toward another one. Unit testing is a technique in which you test a specific Unit by mocking its dependencies. Let's use an example to better depict it. We're going to write a test for the billing package. This could be one of the several packages in your codebase. For the sake of the demo, everything has been written to the billing.go file (do not do this in production). Within this file, we defined these types:

The Invoice struct model, holding the CreatedAt and Amount fields
The Archiver interface represents the dependency 💉 our UUT relies on
The InvoiceManager struct is our Unit Under Test model
The Store struct implementing the Archiver interface

Before showing the code, let me share this drawing to let you understand better the actors:

The complete source code looks like this:

package billing

import (
 "errors"
 "fmt"
 "os"
 "time"

 "github.com/google/uuid"
)

type Invoice struct {
 CreatedAt time.Time
 Amount    float64
}

type InvoiceManager struct {
 Archiver Archiver
}

func (i *InvoiceManager) RecordInvoice(invoice Invoice) (err error) {
 id, err := i.Archiver.Archive(invoice)
 if err != nil {
  return err
 }
 fmt.Fprintf(os.Stdout, "recorded invoice with id: %s\n", id)
 return nil
}

type Archiver interface {
 Archive(invoice Invoice) (id string, err error)
}

type Store struct{}

func (s *Store) Archive(invoice Invoice) (id string, err error) {
 if invoice.Amount < 0 {
  return "", errors.New("amount cannot be less than 0")
 }
 // logic omitted for brevity
 return uuid.NewString(), nil
}

The Unit Under Test 🤠

With the previous code in mind, our task is to write a test for the method RecordInvoice based on the InvoiceManager receiver type. The function can take two paths:

The happy path is when the i.Archiver.Archive invocation doesn't return any error
The "unhappy" path is when it gives back an error

As good developers, we are asked to write two unit tests. However, we'll write only the happy path test since we would like to focus more on the steps to get there. After all, everybody knows how to write unit tests for this small piece of code.

First things first: mocking

The InvoiceManager struct depends on the Archiver interface. Let's see how we can mock it. Here, we have two options: hand-writing it or automatically generating it. We opt for the latter even if the project's size is trivial.

Be aware that, in real-life projects, this method can save you a consistent amount of time.

Mockery tool

We'll take advantage of this tool, which can be downloaded here. Before proceeding, make sure you have correctly installed it on your machine. To confirm it, you can run this command in your terminal:

mockery --version

In your terminal, navigate to the root folder of your project and run the following command:

mockery --dir billing/ --name Archiver --filename archiver.go

This command specifies three parameters:

--dir is the directory where to look for interfaces to mock: our code is contained within the billing folder
--name is the name of the interface to mock: Archiver is the identifier for our interface
--filename is the filename for the mock file: we used archiver.go to keep the naming convention consistent

When you run the command, you'll notice a new mock folder has been created. You'll find the archiver.go mock file inside it. By default, the mockery tool creates a new folder (and a new package called mock) to hold all the mocks.

If you're not good with this approach, you can override it when running the tool.

Based on my experience, I think the default behavior might be applied in almost all cases. You might also notice that the compiler started to complain. This is due to missing packages in your project. The fix is easy. You should just run the command go mod tidy where your go.mod file is located. Then, double-check the errors have gone away, and you'll be ready to use the mocks.

The test code

Let's see how we can exploit the scaffolded mock in our unit tests. By the books, a unit test should have three stages: Arrange, Act, and Assert (aka AAA paradigm). We'll cover each of these in the subsequent sections. First, I'm going to show you the code, and then I'll walk you through all the relevant parts of it:

package billing_test

import (
 "testing"
 "time"

 "github.com/ossan-dev/unittestmock/billing"
 "github.com/ossan-dev/unittestmock/mocks"
 "github.com/stretchr/testify/assert"
)

func TestRecordInvoice(t *testing.T) {
 // Arrange
 invoice := billing.Invoice{CreatedAt: time.Now(), Amount: 66.50}
 store := mocks.NewArchiver(t)
 store.On("Archive", invoice).Return("16668b88-34a0-4a25-b1da-6a1875072802", nil).Once()
 uut := &billing.InvoiceManager{
 Archiver: store,
 }
 // Act
 err := uut.RecordInvoice(invoice)
 // Assert
 assert.NoError(t, err)
 store.AssertExpectations(t)
}

Arrange 🧱

Here, we've to invoke the function NewArchiver from the mocks package to get a new instance of our mock. Then, we set it up by using three methods:

The On method specifies to which invocation this mock has to reply (also with what arguments)
The Return method specifies which values to return from the mock when invoked
The Once specifies how many times to return values from this mock

Lastly, we instantiate our UUT by passing in the store mock as the Archiver interface. We can safely proceed.

Act 👊

Within this stage, we invoke the method RecordInvoice defined on the uut variable. No further explanations are needed here.

Assert 🤞

In this final stage, we have to check two things:

The uut variable gives back whatever we expect. In this case, we expect a nil error
The store mock behaves as expected

The second point means that the method Archive has been invoked with the expected arguments, the correct number of times, and so on.

Run Test

Now, we can safely run our test. To run it, we invoke the command:

go test -v ./billing

And that's the outcome on my machine:

=== RUN   TestRecordInvoice
recorded invoice with id: 16668b88-34a0-4a25-b1da-6a1875072802
--- PASS: TestRecordInvoice (0.00s)
PASS
ok      github.com/ossan-dev/unittestmock/billing       0.004s

That's a Wrap

I hope you found this blog post helpful. As you may imagine, there are several things to cover regarding unit testing, mocking, etc. Any feedback is highly appreciated.

Before leaving, I strongly invite you to reach out if you are interested in some topics and want me to address them. I swear I shortlist them and do my best to deliver helpful content.

Thank you very much for the support, and see you in the next one 👋

Go API tutorial: the begin

Ivan Pesenti — Fri, 15 Apr 2022 13:51:59 +0000

Welcome back guys 👋 It's been a while since my last post written for our amazing community. A lot of cool things have happened during this time (including my transfer to another company). Due to this now, my daily work is focused on Go programming language. I want to say that I'm not yet a well-rounded programmer with this language but I'll do my best to build a very interesting series about the thing that I love most: write secure, testable and successful software.

I'm really excited for this series and I hope it's the same for you, dear reader 😃

Preamble

As I said above I'm not the most skilled person at Golang on the Earth, so if u find any mistakes or something that can be improved or explained better let me know and I'll take the necessary actions to fix it.

Another thing that I want to state is that I strive to have an incremental approach throughout this tutorial, so every post will do only one thing (like the Single Responsibility Principle which is one of my favorite principle when it comes to programming 🤖).

This tutorial is not written on stones so, if u'd like to see something feel free to reach me and I can introduce your request in the series.

The big picture 🌎

The final goal of this tutorial is to build a REST API with Go programming language. This API will be a TODO API (I know, I know... very few imagination 🙄). But during our journey we're going to introduce a lot of cool stuff and obviously everything will be backed up by Unit Tests which is an evergreen topic to cover IMO.

First step 🔰

This post will cover just the basics of how to setup a Golang project and some basic HTTP stuff, such as exposing a dummy HTTP endpoint with a simple GET method. This will be the foundation for the next parts 🏰.

Let's start 🚀

Without having much more delay we can start having fun together!

Prerequisites

To follow this tutorial u must have these programs and tools on your machine:

Visual Studio Code (or any other IDE)
Go installed. U can download from here
staticcheck tool. It's a tool for linting which can perform some checks on your code about simplifications, styling rules, performance issues and so on. U can find more about it here. It's not mandatory but I strongly recommend it
gofmt is a package that can assist u in formatting the code. I suggest you to install globally on your machine from this link. Again, not mandatory but suggested!
Postman (or any other API client that you're comfortable with)

Setup 🛠️

Let's create a new folder for this tutorial with some meaningful name (such as todo-api). Open this folder with your IDE and inside the integrated terminal issue these two commands: mkdir src and cd src/.

Next step is to initialize a Golang package. In order to do this we need this command go mod init todoapi. Thanks to this now we're also able to manage the module dependencies by importing go packages from the internet 🌏. If everything goes fine you will see a new file called go.mod below the "src/" folder. The content should be something similar to this:

module todoapi

go 1.18

🔴 IMPORTANT: 1.18 means the minimum version needed for compiling this Golang program.

Main

The main file is the entry point of our program and its main function will be the first called by the running OS when it will execute our code

🧐NOTE: Golang is a compiled programming language that means that it will produce an executable file to run.

Now you can create the "main.go" under the "/src" folder. Fill in with this simple code to check that everything is fine:

package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

Confirm that everything is working by issuing in the CLI this command go run . or by pressing F5 (start debugging) in your IDE. You should see the string Hello, World! printed in the console 👋.

Routing 🔀

As you now, web api is all about receiving an HTTP request and providing an HTTP response. When an HTTP request comes in we need some mechanism to decide which controllers (or handlers, these two names can be used interchangeably but I prefer to stick with the former) have to manage this kind of request and provide a relevant response.

This mechanism is called Routing. This one has to instrumented on how to manage the requests so we need a way of mapping each accepted route to a relative controller within a one-to-one relationship 🚧.

In order to setup the router u need to perform the following steps:

Create a "router/" folder below "src/"
Create a "router.go" under this folder
Fill the router.go file with this code:
```
package router

import (
    "net/http"

    "github.com/gorilla/mux"
)

func SetupRoutes(router *mux.Router) {
    router.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("pong"))
    }).Methods(http.MethodGet)
}
```
This code will instrument the HTTP server to manage the requests made to "/ping" 🏓 endpoint with the GET HTTP method and it provides an inline handler that will only respond with a dummy plain text "pong" 🏓 and 200 Status OK.

An handler is a function that has the following two parameters in the method signature: ResponseWriter and Request. The former is the actual response that will be sent back to the client 🔙 and so you've to manipulate it (body, status code, headers, etc) while the latter is the HTTP request that comes in 🔜 and it holds all of the info such as request payload, route, querystring params which are accessible within our endpoint code.

There are a lot of packages that can be used for routing, my preferred one is the one from Gorilla 🦍. More details here.
In the CLI issue the following command to install this library go get github.com/gorilla/mux
Check if this dependency was added in go.mod file

Golang tools

staticcheck

Now it's time to lint your code 👨‍🏫. Open up your CLI and issue this command staticcheck ./....

gofmt

After linting it's time for formatting 💠. In the CLI issue go fmt ./... and you will see the files that have been formatted according to Golang formatting rules.

⚠️ WARNING: u need to make sure that you're located where your main.go file is when running these commands otherwise you end up in having a lot of issues due to the fact that either the tools doesn't recognize it as a valid Golang pkg or nothing happened at all.

Wiring all together ⛓️

Now, switch back to main.go and make use of newly created stuff through this code:

package main

import (
    "fmt"
    "net/http"

    "todoapi/router"

    "github.com/gorilla/mux"
)

func main() {
    fmt.Println("starting up HTTP server...")

    r := mux.NewRouter()

    router.SetupRoutes(r)

    err := http.ListenAndServe(":8000", r)
    if err != nil {
        panic(err)
    }
}

Here we're going to instantiate a new router with a method provided by Gorilla package. After that we instrument it with the setup method in our router.go file.

The last part is responsible for starting up an HTTP server that will accept requests on port 8000 and it will manage them according to r that is our router BTW.

Before go ahead run go mod tidy in your terminal to make sure that the dependencies are fine.

Test 🤓

For this time (and only for this time) we're going to do some manual tests. Don't worry from the next post we're gonna to use unit tests 😉.

First of all launch our Go package with go run . or with F5. Next, open Postman and prepare a simple GET request pointing to this URL: "http://localhost:8000/ping". Issue the request and you should get back a "pong" response with 200 HTTP StatusOK 👌.
👀 NOTES: to interrupt the running server you can press CTRL + C on your keyboard.

Success 🐎

We have successfully finished up this first tutorial which is just a warm-up for the series. Now you have the basics about how to spin up an HTTP server in Golang and instrument it to respond to the requests.

What's next ⏭️

The next article will be about our controller 🛂. So we're going to define our first controller that will interact with HTTP request and HTTP response and we will introduce the unit test that it's a huge topic to deal with ⛰️.

If this sounds interesting for you don't miss if for any reason 😁.

I hope you enjoy this post and find it useful. If you have any questions or you want to spot me some errors I really appreciate it and I'll make my best to follow up. If you enjoy it and would like to sustain me consider giving a like and sharing on your favorite socials. If u want u can add me on your socials this makes me very very happy!

Stay safe and see you soon! 😎

Support two authentication providers with .NET 5

Ivan Pesenti — Mon, 18 Oct 2021 07:33:26 +0000

Hi folks 👋🏻! I'm super happy to have you here for the last episode of the series about Authentication in .NET 5.

This post will be our icing on the cake 🍰.

🔴IMPORTANT❗🔴: this post heavily relies on stuff made in the previous two episodes. So I strongly encourage you to checkout the links above and then jump back here again.

As always, if u get in trouble in following this tutorial u can check the final solution in GitHub at this link.

Rewind the tape 🔙

Up to now, we have built out what follows:

A Google Firebase proj (create with Google Developer console) with a test user who can sign-in with username and password. You can open the Google console with this link
Built a web api proj that makes use of token-based authentication with Jwt format
Connected our web api with the Firebase proj so users can access protected resources (after a successful sign-in on Firebase)
Implemented our custom sign-in mechanism by exposing an endpoint to issue a Jwt token to users

👀NOTE👀: if you've not followed the previous posts but you have a similar solution to the one presented here (maybe with different auth providers) the following approach still works for you even if you probably have to make some adjustments.

Final challenge 🎯

During this post we're going to change our web api in order to support two authentication providers at the same time. What this means is that a user can choice how to sign-in against our web api. The user could authenticates himself against Google Firebase or against our custom endpoint. If the sign-in phase is successful the user will got back a valid Jwt token which can be used to access our protected resources. As before, if the sign-in is not successful he'll got back an error response and cannot access our super-secure resources 😎.

Requirements

To follow this post on your machine you must have installed the following:

NET 5 Runtime. You can download from here
Visual Studio Code (you can use another IDE if you wish). Download can be found here
Postman (you can use another program to consume REST-api if you wish). Download can be found here
C# extension for VSCode (powered by Omnisharp)

Let's start 🚀

To complete the task we've to carry about two sections: the settings and authentication's registration. After these tasks we'll do the final test.

Check settings 🔧

First open up the "appsettings.json" file and check that you have an identical structure like me (obviously replace the value with yours):

"Jwt": {
    "Firebase": {
      "ValidIssuer": "https://securetoken.google.com/auth-series",
      "ValidAudience": "auth-series"
    },
    "AuthDemo": {
      "Key": "This is where you should specify your secret key, which is used to sign and verify Jwt tokens.",
      "ValidIssuer": "localhost:5001",
      "ValidAudience": "localhost:5001"
    }
  }

⚠️WARNING⚠️: in a real-world app you must not store these sensitive data in a non-secure location such as appsettings.json. Moreover the key should not be so easy to guess 😋.

Here you see that we're going to support two authentication providers: "Firebase" and "AuthDemo" (that is our custom endpoint).

Providers Registration 🎙

The next file you have to open is "Startup.cs" and look at the ConfigureServices() method.

After the Firebase registration you have to adjust the code in a way that looks similar to the one below:

    ...
    services.AddTransient<ITokenService, TokenService>();

    // firebase auth
    services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer("Firebase", opt =>
    {
        opt.Authority = Configuration["Jwt:Firebase:ValidIssuer"];
        opt.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ValidIssuer = Configuration["Jwt:Firebase:ValidIssuer"],
            ValidAudience = Configuration["Jwt:Firebase:ValidAudience"]
        };
    });

    // auth demo
    services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer("AuthDemo", opt =>
    {
        opt.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ValidIssuer = Configuration["Jwt:AuthDemo:ValidIssuer"],
            ValidAudience = Configuration["Jwt:AuthDemo:ValidAudience"],
            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(Configuration["Jwt:AuthDemo:Key"]))
        };
    });

The main change in this code is that we're going to use another overload of the method AddJwtBearer. This overload accepts as the first parameter the authentication schema name which is used to identify the providers uniquely.

🧐 NOTE 🧐: you should not read the settings in this way as it's not strongly-typed and so it's a more error-prone method. Consider using the Options pattern. More on this here.

Add policy to manage multiple schemas 📜

The last step left is to add a policy to our web api that allow it to manage multiple authentication schemas at the same time. This policy must be written right below the code of the previous section. So open up "Startup.cs" and locate the "ConfigureServices()" method and write the code:

services.AddAuthorization(opt =>
{
    opt.DefaultPolicy = new AuthorizationPolicyBuilder()
    .AddAuthenticationSchemes("Firebase", "AuthDemo")
    .RequireAuthenticatedUser()
    .Build();
});

This code snippet is self-explanatory. Its purposes are to register the schemas declared above as you should notice from the command .AddAuthenticationSchemes("Firebase", "AuthDemo") and to require the users to be authenticated in order to access our resources.

Final Test 🐲

We've finished the coding phase so we can finally give a try to our work ⚒. In your terminal go into the folder where it's contained the .csproj file.

Issue a dotnet run command and wait for the web api to start properly.

Postman

To test our software we need of three requests (that you can build by following the recipes from the previous posts). Below you can find the three requests with their titles and purposes:

FirebaseSignIn: used to sign-in a user in Firebase platform
AuthDemoSignIn: used to sign-in a user against our custom endpoint
WeatherForecast: used to access our restricted resource

Foremost, try to execute the WeatherForecast without authentication configured:

You should get back a 401 Unauthorized error.

Now execute the FirebaseSignIn request and copy the "idToken" value returned. Then switch back to the WeatherForecast request and change the authorization type to "Bearer Token" and paste in the token as you can see below:

Execute again this request and check if you receive a 200 OK response together with the requested data.

Now open the AuthDemoSignIn request and execute it to get back the Jwt token. Copy the returned token.
Switch back to WeatherForecast request and replace the Firebase token with the latter one. Execute again and now it should works as expected 🌟!

Final thoughts 💭

Now we reached the end 🏁. During this series you learned a bunch of things about authentication 🔐 within .NET 5. To show off these capabilities we make use of web api project template but the same still applies to other projs such as MVC or Single Page Application.

Supporting multiple authentication providers could be a transient phase such as when you're migrating from one auth provider to another or permanent as you would like to let users decide which authentication provider utilize.

In both cases this series will provide you some guidelines about the steps you have to follow.

Greetings 👋🏻

Now, it's time to say goodbye 😄.

🔵IMPORTANT🔵: remember that authentication is one CRUCIAL aspect of software development, so don't joke with it as you'll pay the consequences for sure 🤕.

I hope you enjoy this post and find it useful. If you have any questions or you want to spot me some errors I really appreciate it and I'll make my best to follow up. If you enjoy it and would like to sustain me consider giving a like and sharing on your favorite socials. If u want u can add me on your socials this makes me very very happy!

Stay safe and see you soon! 😎

Custom JWT Authentication .NET 5

Ivan Pesenti — Mon, 27 Sep 2021 16:13:13 +0000

Hi guys! Welcome back to my Authentication series. It's a pleasure for me to have you here 😁.

🔴IMPORTANT❗🔴: this is the second post of a series so, if you missed the previous one, I strongly suggest you to check it out from the link above.

As always, if u get in trouble in following this tutorial u can check the final solution in GitHub at this link.

Quick recap of the previous episode 🔙

In the previous blog post we achieved what follows:

Develop a web api with VSCode and test it through Postman
Use Firebase Admin Sdk to manage our Firebase proj directly from the web api
Use token-based authentication in our web api with the format of JWT
Set up logic to verify the JWT token validity

What's next ⏭

In this episode we're going to write an authentication endpoint which will be responsible for issuing JWT tokens to the users that will provide valid sign-in credentials.

This means that our web api is in charge of both issue the JWT token and verify it.

As before, if a user doesn't provide a valid JWT token, he can't access our restricted resource (in this case the WeatherForecast endpoint).

Preamble

This will be a demo application to show off a way to implement authentication in .NET 5. You can take away these concepts and adapt them to your real-world requirements.

🔵IMPORTANT🔵: I'm not going to follow every best practices to save time but I'll do my best to spot things that are not "real-world ready" to use.

👀NOTE👀: if you don't care about Firebase you can follow directly this blog post without having to deal with the previous one.

Let's start 🚀

Without much delay, let's jump into the coding part 💻.

Prerequisites

To follow this tutorial you must install on your machine some tools and programs:

NET 5 Runtime. You can download from here
Visual Studio Code (you can use another IDE if you wish). Download can be found here
Postman (you can use another program to consume REST-api if you wish). Download can be found here
C# extension for VSCode (powered by Omnisharp)

Prepare configs

The first thing we've to do is to add keys about JWT authentication in our web api. Please be sure to add the following code in appsettings.json:

"AuthDemo": {
    "Key": "This is where you should specify your secret key, which is used to sign and verify Jwt tokens.",
    "ValidIssuer": "localhost:5001",
    "ValidAudience": "localhost:5001"
}

This section must be inside a parent node called "Jwt".

⚠️WARNING⚠️: in a real-world app you must not store these sensitive data in a non-secure location such as appsettings.json.

User model

Create a "Models" folder within your web api proj. After that create a class called UserModel.cs with the following code:

using System.ComponentModel.DataAnnotations;

namespace AuthSeries.Models
{
    public class UserModel
    {
        [Required]
        public string Email { get; set; }
        [Required]
        public string Password { get; set; }
    }
}

This class will hold the sign-in credentials of the users that will be passed to our authentication endpoint.

🔎HINT🔎: we've used a "Required" data annotation that will enforce the properties to be present in the input model.

Implement token service feature

Create a "Services" folder. In this folder you're going to create two files.

ITokenService.cs

using AuthSeries.Models;

namespace AuthSeries.Services
{
    public interface ITokenService
    {
        string BuildToken(string key, string issuer, UserModel userModel);
    }
}

This interface will be implemented by our concrete implementation that we're going to use in our controller to issue a JWT token.

TokenService.cs

using System;
using System.IdentityModel.Tokens.Jwt;
using System.Security.Claims;
using System.Text;
using AuthSeries.Models;
using Microsoft.IdentityModel.Tokens;

namespace AuthSeries.Services
{
    public class TokenService : ITokenService
    {
        private const double EXP_DURATION_MINUTES = 30;

        public string BuildToken(string key, string issuer, UserModel userModel)
        {
            // TODO: put real-world logic to evaluate sign-in credetials
            // ...
            var claims = new[]
            {
                new Claim(ClaimTypes.Name, userModel.Email),
                new Claim(ClaimTypes.NameIdentifier, Guid.NewGuid().ToString())
            };

            var securityKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(key));
            var credentials = new SigningCredentials(securityKey, SecurityAlgorithms.HmacSha256Signature);
            var tokenDescriptor = new JwtSecurityToken(issuer: issuer, audience: issuer, claims, expires: DateTime.Now.AddMinutes(EXP_DURATION_MINUTES), signingCredentials: credentials);
            return new JwtSecurityTokenHandler().WriteToken(tokenDescriptor);
        }
    }
}

This class is in charge of issuing a JWT token by taking in a secret key, a valid issuer and the user's sign-in credentials.

⚠️WARNING⚠️: this code is not real-world as we must check the sign-in credentials against a valid source such as a database, a txt file, a call to an external system and so on (take a look at the comment in the code).

Users endpoint

Now, it's time to take advantage of what we've done up to now. We create the endpoint that will be called from the client to get back a valid JWT token issued with the logic above.

In the "Controllers" folder create a file called UsersController.cs with the following code:

using AuthSeries.Models;
using AuthSeries.Services;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;

namespace AuthSeries.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class UsersController : ControllerBase
    {
        private readonly IConfiguration configuration;
        private readonly ITokenService tokenService;

        public UsersController(IConfiguration configuration, ITokenService tokenService)
        {
            this.configuration = configuration;
            this.tokenService = tokenService;
        }

        [HttpPost]
        [Route("sign-in")]
        public IActionResult Post(UserModel userModel)
        {
            return Ok(tokenService.BuildToken(configuration["Jwt:AuthDemo:Key"], configuration["Jwt:AuthDemo:ValidIssuer"], userModel));
        }
    }
}

This class simply have a constructor with two dependencies (more on "ITokenService" soon) and expose a single action that will return a valid JWT token to a user.

🧐NOTE🧐: you should not read the settings in this way as it's not strongly-typed and so it's a more error-prone method. Consider using the Options pattern. More on this here.

Connects the dots 🧩

One of the major changes that we must do is inside the "Startup.cs" in the "ConfigureServices()" method. The method will look like this:

public void ConfigureServices(IServiceCollection services)
{

    services.AddControllers();
    services.AddSwaggerGen(c =>
    {
        c.SwaggerDoc("v1", new OpenApiInfo { Title = "AuthSeries", Version = "v1" });
    });

    // Auth.Demo section

    // here we register our service
    services.AddTransient<ITokenService, TokenService>();

    // here we specify our authentication settings to validate the JWT token
    services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(opt =>
    {
        opt.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ValidIssuer = Configuration["Jwt:AuthDemo:ValidIssuer"],
            ValidAudience = Configuration["Jwt:AuthDemo:ValidAudience"],
            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(Configuration["Jwt:AuthDemo:Key"]))
        };
    });
}

Here there are two important things that you must be aware of:

The TokenService registration. We've registered our TokenService in the IOC container provided by .NET. The service lifetime used is transient that is every time we need of an instance of ITokenService, a new fresh one is returned to us from the built-in IOC container. More about service lifetime here.
The JWT token parameters used to verify a token. 🔎NOTE🔎: this code is not real-world ready for a couple of reasons. First, the secret key must not be saved in appsettings.json and must not be so easy to guess. Last, but not least, you should not read the keys in this way but it would be better if you take advantage of Options pattern for example.

If you started directly from this post you must add the following two Nuget packages by issuing these commands (⚠️WARNING⚠️: be sure to run these commands inside the proj folder):

dotnet add package Microsoft.AspNetCore.Authentication
dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer

Final touch 🍓

If you've started from this post you must also do two brief final steps in "WeatherForecastController.cs":

Decorate the "Get()" method with this attribute [Authorize]
Add this using statement: using Microsoft.AspNetCore.Authorization; at the top of the file

Moment of truth 👨🏻‍🏫

Now the last thing left is to give a try to our work.
Issue a dotnet run in your preferred terminal and wait for the application to start.

Bring up Postman and try a simple request at the endpoint https://localhost:5001/weatherforecast. This should result in a 401 unauthorized error.

As expected, we must sign in ourselves before try to hit the WeatherForecast endpoint.

Create a new Postman request with the following parameters:

type: POST
url: https://localhost:5001/users/sign-in
body:

{
    "email": "test@test.com",
    "password": "password"
}

When you run this request you get back a JWT token in the output console. Copy it to your clipboard and switch back to the WeatherForecast request.

In the Authorization tab select:

Type: Bearer Token
Token: the token in your clipboard

Finally repeat the test again... et voilà 😍. You should get back a 200 OK response along with the requested data.

Let's recap

Congrats 🏆❗ You successfully did this blog post. By following this tutorial you're able to issue your own JWT token with your custom logic.

The final step 🐱‍🏍

In the next blog post (that will be the last of this series), we're going to merge these two posts by allowing the users to authenticate either via Firebase or via our custom endpoint. Stay tuned 👑!

Stay safe and see you soon! 😎

Firebase Authentication .NET 5

Ivan Pesenti — Fri, 27 Aug 2021 12:38:02 +0000

Security has always been one of my major concerns when developing applications, especially in web development where our resources can be consumed from everywhere (and potentially from anyone). Be able to restrict the resources' access to only certain people is fundamental.

Nowadays data loss, identity theft and other cyber threats are constantly increasing. So, in order to reduce this probability that could lead (in the worst scenario) to a business failure, we must write secure and reliable code in our daily development life.
If all of this is a also for you a concern I strongly encourage u to read carefully this blog post (that will be the first of a series of three).

The big picture 🌍

Throughout this series we're going to build a web api project that supports two authentication providers: Google Firebase and our web api itself. The user, in order to access the web api resource, must be authenticated either against the former or against the latter authentication providers. If this is the case the user will receive a 200 OK response together with the requested data. Otherwise, it will receive a 401 Unauthorized response with no data. The authentication mechanism would be the token authentication in JWT format.

🧐NOTE🧐: if u need to learn more about Google Firebase u can checkout this link.

To maintain the blog post leaner I decided to split this building process into three parts.

The first piece

In this blog post we're gonna to create a web api project, create some requests to test it, configure stuff related to Firebase in our web api proj, use the Firebase console to manage our proj and implement the Firebase authentication mechanism in our web api.

Preamble

This will be a demo application to show off a way to implement authentication in .NET. U can take away these concepts and adapt them to your real-world requirements.

🔴IMPORTANT❗🔴: I'm not going to follow every best practices to save time but I'll do my best to spot things that are not "real-world ready" to use.

As always, if u get in trouble in following this tutorial u can check the final solution in GitHub at this link.

Let's start 🚀

After these necessary thoughts we can start the fun part, what everybody are waiting for... the coding!

Prerequisites

To follow this tutorial u must install on your machine some tools and programs:

NET 5 Runtime. U can download from here
Visual Studio Code (u can use another IDE if u wish). Download can be found here
Postman (u can use another program to consume REST-api if u wish). Download can be found here
Google Firebase project with a user registered.

⚠️WARNING⚠️: u must be sure that Email/Password Sign-in method is enabled. When u create a Firebase project it's not enabled by default. The correct config is the same as in the below image:

Setup web api proj

First, we must create a web api proj. I'll do these kind of stuff from dotnet CLI. So open up a terminal (or use the one integrated in VSCode) and issue:

dotnet new webapi -n "AuthSeries"

When VSCode will prompt u to restore dependencies give a yes and go ahead.

Test it

Now to be sure that everything looks fine before start to edit we're going to give it a first try.

Change directory with cd AuthSeries and run the web api with the command dotnet run. By default the web api will be launched on port 5001 for HTTPS protocol.

Open up Postman and create a request with the following params:

HTTP method: GET
URL: https://localhost:5001/weatherforecast

Hit "Send" and u should receive dummy data generated by our web api.

Firebase stuff

Open your Firebase project within the Google Firebase Console.

🧐NOTE🧐: the project's URL should be something like 'https://console.firebase.google.com/u/0/xxxxxxx'.

Create Web App

Click on the little gear next to "Project overview" and select "Project settings". In "General" tab scroll toward the bottom and click the icon in the following image:

Give a meaningful name, I choose "auth-series-app" to discriminate from the Firebase proj called "auth-series" and click "Register App". After that select "Continue to console".

Create Service Account Key

In order to manage programmatically the Firebase proj from the web api codebase through the Firebase Admin SDK we must generate a secret private key that we'll be used to identify our web api into Firebase scope.

Go to "Service accounts", "Firebase Admin Sdk" and click "Create service account".

The following window will appear:

Click "Generate key" and a JSON file will be downloaded on your machine. Keep on hand this file as we're going to use it immediately.

Wire up Firebase and Web Api ⛓

Now, let's connect the dots! We've to create a bridge between Firebase and our web api proj.

Firebase registration inside our Web Api

Change directory (if you're not already there) with cd AuthSeries and issue mkdir Firebase. Copy the JSON file downloaded above into this folder. Issue the following command in CLI to add a NuGet package:

dotnet add package FirebaseAdmin

Open the file Startup.cs and add this using FirebaseAdmin; below the other using statements.
Finally add the following code to the ConfigureServices() method of Startup.cs class file:

FirebaseApp.Create(new AppOptions
{
    Credential = GoogleCredential.FromFile(@"C:\Projects\SampleProjects\auth-series\auth-series\AuthSeries\Firebase\auth-series-firebase-adminsdk-rk7k4-4dc58434f2.json")
});

🔵IMPORTANT🔵: u must replace my path with your. In production u should not hard-coded info in this way. U could use environment variable, .NET User Secrets and so on.

Add firebase authentication in IOC container

Up to now, we've only registered the Firebase proj inside our web api.

The next thing is to tell to our web api that we'd like to use Firebase as authentication provider.

To achieve it we must change some files.

appsettings.json

"Jwt": {
    "Firebase": {
      "ValidIssuer": "https://securetoken.google.com/auth-series",
      "ValidAudience": "auth-series"
    }
}

⚠️WARNING⚠️: u must replace "auth-series" with the project id u choose for your Firebase project (u can find it in "Project settings > General" in Firebase Console).

Startup.cs

Before editing this file u have to install two NuGet packages. Issue in the terminal these two commands:

dotnet add package Microsoft.AspNetCore.Authentication
dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer

After that be sure to append these two using statements at the top of the file:

using Microsoft.IdentityModel.Tokens;
using Microsoft.AspNetCore.Authentication.JwtBearer;

Now u are ready to edit the two methods of the Startup.cs.

ConfigureServices method

Under the Firebase app's registration (the code added above) place the following code to set parameters for JWT token provided by Google Firebase.

// firebase auth
services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(opt =>
{
    opt.Authority = Configuration["Jwt:Firebase:ValidIssuer"];
    opt.TokenValidationParameters = new TokenValidationParameters
    {
        ValidateIssuer = true,
        ValidateAudience = true,
        ValidateLifetime = true,
        ValidateIssuerSigningKey = true,
        ValidIssuer = Configuration["Jwt:Firebase:ValidIssuer"],
        ValidAudience = Configuration["Jwt:Firebase:ValidAudience"]
    };
});

🔎HINT🔎: in order to make the code type safer u should consider using the options pattern and bind settings to POCO classes. Moreover u have to place these sensitive data in a more secure place.

Configure method

With the following change you're about to add the authentication middleware in the middlewares' pipeline.

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();

WeatherForecastController.cs

The final thing to change is the endpoint that will be secured. Add using Microsoft.AspNetCore.Authorization; at the top of the file.

Above the signature of Get method add [Authorize] to restrict access to this action only to authenticated users.

Final test

Now, let's test our work. In order to prove that we're right we need of two requests: one to sign-in a user in Firebase and one to query our endpoint.

Firebase request

Before creating the request u must get the web API key from Firebase project. As before, u have to navigate "Project settings > General" and copy the web API key as u can see below:

In postman create a request with the following params:

HTTP method: POST
URL: https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?
params:
- key: <your secret web API key>

body:

{
    "email": "test@test.com",
    "password": "password",
    "returnSecureToken": true
}

Execute the request and copy the idToken that u get back in the output console.

WeatherForecast request

Reopen the request used above
Go in "Authorization" tab
Select "Bearer Token" as type
Paste in the token copied before
Hit send and u'll get the requested resource

Summary

Congratulations 🏆! U successfully did the first part of this series. Now you're able to restrict 🔐 access to your resource to only authenticated users. Moreover, u are able to integrate Google Firebase as authentication provider in your web api with a couple of simple steps.

What's next 🐱‍🏍

In the next blog post we're going to generate our JWT token directly within our web api instead of using Firebase. We're going to create an endpoint to sign-in users by issuing a JWT token.

If this sounds exciting for you, don't miss it.

Stay safe and see you soon! 😎

Playing with Azure Functions: Mission 3

Ivan Pesenti — Mon, 19 Jul 2021 12:19:48 +0000

Hi champions! Welcome back to my Azure Function series. First, if u are here with me I must celebrate u 👏🏻 as u have been successfully carried out the previous missions. If this is not the case, please be sure to checkout the other two posts.

Even if it's not mandatory to checkout them, I strongly suggest u to do because in this last mission we're going to take our Azure Function built in the previous missions and make some changes on top of it. Moreover, this post will assume that u have a basic knowledge about the topic (don't worry u can gain it by reading the previous posts and u'll learn the basic concepts with a bit of fun 😂).

Today we're gonna to talk about UPGRADING our Azure Function from .NET Core 3.1 to the desired .NET 5.0.

So, stay tuned!

NOTE: If u get in trouble in following the tutorial u can find all the code in "end" folder of my GitHub repo that u can find here.

Our previous achievements 🥳

If u have completed the MISSION 1 and 2 u now have a working solution with the following features:

A proj called "azure-function-entities" that holds the models for our application. It has a class named "Output.cs" made up by props that are logged by our Azure Function. Its target framework is .NET Standard 2.1.
A proj called "azure-function-managers" responsible for holding the services for our application. It has only a service named "GreetingsService" with its interface "IGreetingsService". Its target framework is .NET Standard 2.1.
A proj called "azure-function" with our Timer-Triggered Azure Function that on each second divisible by 5 will log messages to our console. It used models and services provided by the projs listed above. Its target framework is .NET Core 3.1 (LTS).

Let' start 👨🏻‍🚀👩🏻‍🚀

Final mission

Be ready ⚔! Our journey is going to finish but this mission will be the hardest. Through this mission we're going to upgrade our .NET Core 3.1 Azure Function that has been developed with in-process programming model to a NET 5.0 Azure Function that will run in an isolated worker process. This migration is divided into three minor migrations: one for each projs in our solution.

Now it's time to stop to speak and start to develop 💻.

Initial setup 👷🏻‍♂️

.NET 5 SDK

First, it's mandatory to be sure that our development environment can support projs targeting the NET 5.0 framework. If u haven't developed yet a NET 5.0 application probably u should download the related SDK. U can download it from here. After u have successfully installed it u can try to issue this command in a shell:

dotnet --version

If the number u got is starting with 5, u're good to go.

Renaming phase

📖HINT📖: imo you should do these steps with VSCode closed and open it later.
As we're going to keep the previous working function, we opt to duplicate the "start" folder and make the migration upon this newly created folder.

So, copy-paste the "start" folder and rename it to "end". Using the terminal go into "end" folder and u'll find a file called "start.sln". Rename it to "end.sln" to be consistent with parent directory name.

⚠️WARNING⚠️: also check the "end.sln" in order to be sure that no more refs pointing to "start" are left in it. If this is not the case, be sure to adapt them according to the new file system structure.

Now issue this command to bring up VSCode:

code . -r

azure-function-entities proj

Open the file "azure-function-entities.csproj" and change the target framework as below:

<TargetFramework>net5.0</TargetFramework>

After that go into "models" folder and open up the "Output.cs" class file. Change the actual implementation with the following code:

namespace azure_function_entities.models
{
    public record Output(string Message, string ApiKey);
}

🔎NOTE🔎: this is a new language syntax introduced by C# 9.0. It's called record and u can learn more about them here.

Compile this project singularly to be sure that it's still working as expected.

azure-function-managers proj

Open the file "azure-function-managers.csproj" and change the target framework as below:

<TargetFramework>net5.0</TargetFramework>

Compile this project singularly to be sure that it's still working as expected.

azure-function proj

In this project we're going to make the most substantial changes in our application. So, in order to keep things neat and clear, I decided to dedicate a little section to each affected file.

"azure-function.csproj"

First, edit the target framework as shown below:

<TargetFramework>net5.0</TargetFramework>

🔴IMPORTANT🔴: right below this setting add this one that is mandatory to successfully do the migration:

<OutputType>Exe</OutputType>

The last change is to upgrade the section about Nuget Package: replace your section with this one:

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.0.3" OutputItemType="Analyzer"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.3.0"/>
    <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="5.0.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.0.1" />
  </ItemGroup>

When u save, VS Code will prompt u to restore the dependencies of csproj file, confirm and wait for the operation to finish.

🔎NOTE🔎: once the operation finished a bunch of error messages and warning will start arising. Don't worry, we'll fix them in a minute 🐱‍👤.

"local.settings.json"

In order to change the Azure Function host from "in-process" to "isolated" we need to update the "FUNCTION_WORKER_RUNTIME" setting as follow:

"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated"

U can learn more about the isolated process for Azure Function here.

"Startup.cs" ❌

With this new programming model, we've full control over the configuration of the working process. We can register our application services, define the configuration of the worker process and so on. All this bunch of stuff will be defined in the next file we can deal with. We'll go step-by-step in the next paragraph.

With this in mind we can safely delete this file from our proj.

"Program.cs" 🗼

This is the place where we can do all of the necessary setup of our Azure Function. Of course, we've to create this file in our proj.

🕵🏻‍♂️NOTE🕵🏻‍♂️: in our case, this file is missing because we're upgrading from an older version of Azure Function. If u start a NET 5.0 proj from scratch, this file will be included in the auto-generated files.

Add a file named "Program.cs" in the "azure-function" proj and paste the following code:

using azure_function_managers.services;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System.Reflection;

namespace azure_function
{
    public class Program
    {
        public static void Main()
        {
            var host = new HostBuilder()
            .ConfigureFunctionsWorkerDefaults()
            .ConfigureAppConfiguration(options => options.AddUserSecrets(assembly: Assembly.GetExecutingAssembly(), optional: true, reloadOnChange: true))
            .ConfigureServices(services => services.AddScoped<IGreetingsService, GreetingsService>())
            .Build();

            host.Run();
        }
    }
}

⚠️WARNING⚠️: always double check the namespaces as their are one of the primary source of errors in this copy-paste operations.
The purpose of this code is to configure, build and run the worker process for our Azure Function. More in depth u can see that:

The host builder is setup with default configs
User secrets are added as configuration providers for our host process
The "Greetings" service is added to the built-in container with a service lifetime of scoped Finally the host is built and launched.

"TimerTriggerFunc.cs"

This file is where the Run method is defined. Here u can see the input/output bindings of the function, the trigger that fires it up and the code to execute. There are three main changes to do in order to get the job done (I'll explain in a bit):

using System;
using azure_function_entities.models;
using azure_function_managers.services;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;

namespace azure_function
{
    public class TimerTriggerFunc
    {
        private readonly IGreetingsService _greetingsService;
        private readonly IConfiguration _configuration;

        public TimerTriggerFunc(IGreetingsService greetingsService, IConfiguration configuration)
        {
            _greetingsService = greetingsService;
            _configuration = configuration;
        }

        [Function("TimerTriggerFunc")]
        public void Run([TimerTrigger("%TimerCron%")] TimerInfo myTimer, FunctionContext context)
        {
            var logger = context.GetLogger<TimerTriggerFunc>();
            logger.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
            Output output = new(_greetingsService.SayHello("ivan"), _configuration.GetValue<string>("ApiKey"));
            logger.LogInformation($"Message: {output.Message}");
            logger.LogInformation($"ApiKey: {output.ApiKey}");
        }

        public class TimerInfo
        {
            public ScheduleStatus ScheduleStatus { get; set; }
            public bool IsPastDue { get; set; }
        }

        public class ScheduleStatus
        {
            public DateTime Last { get; set; }
            public DateTime Next { get; set; }
            public DateTime LastUpdated { get; set; }
        }
    }
}

The first thing to notice is the new using statements at the top of the file. The most important is using Microsoft.Azure.Functions.Worker; that allow us to use "isolated" process features.

Second the new signature of the Run method: instead of ILogger now the function expects a FunctionContext type that represents the execution context of our function. From it we can get a logger to use in our method instead of the one provided by "in-process" model & .NET Core 3.1.

🔎NOTE🔎: in the third line of the body of Run method we used two new features of C# 9.0: the record types (link above) and the Target-typed new expression (more on it here).

Last but not least we must define the TimerInfo and ScheduleStatus types that now are no more provided by the package Microsoft.Azure.WebJobs.

🔵IMPORTANT🔵: it's always a best practice to define the types in their own class files but for the sake of simplicity I put the types definition inside the "TimerTriggerFunc.cs" file.

Kind remainder before the battle ⚔

If u have not started the Azure Storage Emulator a popup will appear like this one:

Go in the start menu and type in "Azure Storage Emulator" and open the program. In the command prompt u should see the confirmation "The storage emulator was successfully started.".

Win or lose? 🏁

Now the moment of truth... Press F5 and check if our Azure Function works or not.

The function still runs even if it's been upgraded to the new features.

Final thoughts 💭

I think that this is an important achievements for us. Microsoft will follow this direction so at certain point in time will be mandatory to conform to the "isolated-process" programming model. Just for ref I'll share u an image that shows off the Azure Function Roadmap:

This image is self-explanatory: sooner or later u will be about migrating your old-fashioned Azure Function to the fresh-new way of doing things.

Outro + Greetings 👋🏻

The intent of this post is to make your life easier in this migration process that u may encountered in your development life.

Congratulations for reaching this final step and achieve it. It's been a pleasure to fight with u. I hope u find this series helpful and funny. I made my best to make this content as valuable as possible.

Stay safe and see you soon! 😎

Forem: Ivan Pesenti

Have you ever run Robocopy on Linux?

Windows Robocopy... On Linux

The Journey

1. dockur/windows image

First Run

Purposes

Automated Testing Process

2. The Docker Client

3. List Containers

4. Building a Docker image

5, 6, 7 Containers Management

8, 9 Images Management

10. Container Removal

Untold Stories

Outro

The Return of the Living Dead: A Docker Spin-Off 🎦

In the Previous Episodes ⬅️

Living Dead 🧟‍♀️🧟‍♂️

Our Crusade ⚔️

Look at the Error 🔎

Identifying the Root Cause

Beating the Living Dead 🦸

The `reRunContainersAfterConflict() Func

The Caller 📲

Final Episode 🎦

Take it easy with Graphite and Docker 🐳

TLDR the root cause 🌳

Graphite

Docker Come to the Rescue 🛡️

System Under Test 🏋

The HTTP Handler 🕸️

Metrics Sending 📩

Containerize the Web Server 🎁

Power Up 🔋

The Test Code 🪖

Test the HTTP Handler

The Metrics Checker

Has Everything Worked as Expected? ⁉️

Leverage Your Test Suite With testcontainers-go & docker-compose

The Premise 📓

The Survival List 🗡️

The Scenario 📈

Let's Have Fun 💻

The docker-compose.yml file

The ginkgo Testing Framework

Lay the Foundation with Ginkgo

The end2end_suite_test.go file

Whoop! A wild testcontainers appears 🐳

The accounts_test.go file

The Describe container node

The BeforeAll setup node

Finally, the tests! 🥁

3, 2, 1... 🚀

Closing Notes 🤝

Gorm: Sneak Peek of Custom Data Types

Gorm's Custom Data Types

Use Case to Handle

The Go Structs 🧱

The DB Relations 🙌

Custom Data Types

The Code

The domain/models.go file

The main.go file 🚦

The Truth Moment

That's a Wrap

Go Gin Templates: Break Them Down

Today Problem

What is HTML Templating?

The solution

HTML Templates

The header.tmpl file

The footer.tmpl file

The page1.tmpl file

The Go Code

Give It a Try

That's a Wrap

Unit Tests & Mocking: the Bread and the Butter

Premise

How you should read it

1. `dockur/windows` image

The `docker-compose.yml` file

The `ginkgo` Testing Framework

The `end2end_suite_test.go` file

Whoop! A wild `testcontainers` appears 🐳

The `accounts_test.go` file

The `Describe` container node

The `BeforeAll` setup node

The `domain/models.go` file

The `main.go` file 🚦

The `header.tmpl` file

The `footer.tmpl` file

The `page1.tmpl` file