Forem: Aldo Vázquez

Set up a multi-container app with docker compose

Aldo Vázquez — Sat, 11 Feb 2023 22:10:15 +0000

Usually, we use Docker to build a whole application environment in different machines, this can be on an AWS instance, a VM, or even on your localhost.

Of course, you can write a giant Dockerfile to run all the components of your application, database, backend, web server configuration, even a cache service, this is a solution, with some disadvantages:

Building the image will take so long
If any component requires an update, the whole image will be rebuilt
Debugging any component will be a nightmare

Well, the other option is to build many Dockerfiles to actually manage multiple containers, this solution of course solves all the issues above, since we can work with each component individually.

Yet not everything is perfect, now we’ll need to handle many containers manually, run docker run ... for each component, creating a network to connect all our containers, setup many environment variables, and passing aliases to each container to communicate them is quite another nightmare.

How do we fix this? Maybe creating an init.sh file with each docker run ... instruction is a suitable approach, but there is another solution offered by Docker to handle this kind of situations, it is called docker compose.

`docker compose`

Compose is a tool for defining and running multi-container applications. It allows us to define all the services in a single file called docker-compose.yml file and then start, or stop, all of them with a single command.

Installation

Most of the newer versions of Docker Desktop already comes with docker compose command, although, you can always check the installation instructions at their official GitHub repository.

Post’s code

For this docker compose tutorial, I will use a simple REST API written in Go with a MySQL database. Since the app’s functionality is not in our scope I won’t explain it, yet if you have any question related to the example app, please don’t hesitate to reach me out by email or on Twitter.

You can find all the code used in this post here

`docker-compose.yml`

In the repository for this post you will find a file named docker-compose.yml this file will be the “entry point” for our application, let’s take a look into it.

version: "3.7"

services:
  db:
    build: ./db
    platform: linux/x86_64
    container_name: todos-db
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_DATABASE=todos
    image: db:1.0
    volumes:
      - type: volume
        source: db-volume
        target: /data/mysql

  app:
    depends_on:
      - db
    build: ./app
    container_name: todos-api
    environment:
      - DB_URI=root:root@tcp(db:3306)/todos?parseTime=true
    image: api:1.0
    ports:
      - "80:8080"

volumes:
    db-volume:
        name: db-volume

The docker-compose.yml schema requires the docker compose schema version at the top of the file, the latest version is 3.8 but any 3 version will work for this project in particular, if you are eager to learn more about the changes between versions, the change-log is published here.

Immediately, we need to define our services, each of them require a name, for example, my database service is named db this names also works as aliases within the network.

Let’s take a closer look at db service.

db:
  build: ./db
  platform: linux/x86_64
  container_name: todos-db
  environment:
    - MYSQL_ROOT_PASSWORD=root
    - MYSQL_DATABASE=todos
  image: db:1.0
  volumes:
    - type: volume
      source: db-volume
      target: /data/mysql

Ok, now field by field:

build specifies which Dockerfile to use to create this service, in the project’s repository, we have a db folder, and within in we have a Dockerfile with some customizations for our database
platform is not actually necessary, but a good tip in case you are working with an Apple Silicon Processor.
container_name as you might have guessed, specifies the name of the container created, is equivalent to docker run -n flag
environment allows us to specify environment variables used by the container, is equivalent to docker run -e flag
image is for specifying the tag for this image. Is equivalent to docker build -t flag.
volumes is a list of the docker volumes being used by this container

Pretty straightforward, right? There are many more fields that can be specified, docker compose allows customizing each aspect of the containers, this configuration is a pretty simple, but solid, one.

But we have another service, app is for defining my REST API application container, let’s take a look at it.

app:
  depends_on:
    - db
  build: ./app
  container_name: todos-api
  environment:
    - DB_URI=root:root@tcp(db:3306)/todos?parseTime=true
  image: api:1.0
  ports:
    - "80:8080"

And again, let’s explore each field:

depends_on is a quite useful field, it allows us to make our container wait until the service specified here is up, since we depend on a database to store To-dos, it is important to wait until the db service is up.
build again, receives the context to build a new image, within the app folder we have a Dockerfile to build the API
container_name as you might have guessed, again, is for naming our container
environment allows us to specify environment variables used in our containers, this time we only use one variable, but as you can see, the URI points to db:3306 remember when I said this names also works as aliases within the network? docker compose creates a network with each service connected to it; therefore, we can point each service using its alias.
image again is for tagging our images
ports is a list of the ports exposed by this container and the mapping to the host’s actual ports, is equivalent to docker run -p flag

`docker compose up`

Ok, but now what? As I said at the beginning of this post, docker compose allows us to run all our containers with a single command, this command will build all the images (if they don't exist) for our application and also run all the required containers.

But that’s not it, as I mentioned, it also creates a new docker network and attach all the containers to it, same thing for all the volumes specified in the volumes field at docker-compose.yml.

Let’s then run it using the -d flag to run our containers in the background.

docker compose up -d

If the images don’t exist, all the building log will be displayed; otherwise, the output should be something similar to:

[+] Running 2/0
 ⠿ Container todos-db   Running                                                                                                                   0.0s
 ⠿ Container todos-api  Running

As you can see, the *.container_name field is already reflected here.

Then if you list your images you should see something like:

$ docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED             SIZE
api          1.0       b5c4cf711443   49 minutes ago      18.1MB
db           1.0       53c16ac987e5   About an hour ago   517MB

Again, the field *.image was used to name our images and add the version of the image to it.

But let’s see our containers! The output will be something similar to:

$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                  NAMES
806258079a07   api:1.0   "./server"               4 minutes ago   Up 4 minutes   0.0.0.0:80->8080/tcp   todos-api
d01357663cea   db:1.0    "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes   3306/tcp, 33060/tcp    todos-db

Where you can see the port mapping that we used in the app.ports field the name in *.container_name fields and even the *.image name.

Finally, let’s take a look at our Docker networks!

$ docker network ls
NETWORK ID     NAME                    DRIVER    SCOPE
f8d14f7ddf0b   dockercompose_default   bridge    local

There is a new network, being used for this specific project, and we didn’t need to do nothing! It was automatically created and all the containers are already attached to it.

By the way, you can test the API with these commands:

# Create a new TODO
curl -X POST http://127.0.0.1/todos -H "Content-Type: application/json"  -d '{"description": "Write a comment!"}'

# Get TODOs
curl -X GET "http://127.0.0.1/todos"
curl -X GET "http://127.0.0.1/todos/1"

# Mark as complete
curl -X PUT "http://127.0.0.1/todos/1?completed=1"

# Delete a TODO
curl -X DELETE "http://127.0.0.1/todos/1"

Conclusion

docker compose is a great tool, even it is not the best option to be used in production environments, is really useful to replicate different environments in different hosts, quick, easy to read and easy to share. I encourage you to build your next project using a docker-compose.yml file!

Note from the OP

Hi! Long time no see, I recently moved so I was very very busy, but I’m back with many ideas and drafts for new posts!

Thank you so much for all your support and reading my content, I’ll really appreciate any feedback or suggestion!

Lighter Docker images

Aldo Vázquez — Fri, 27 Jan 2023 04:26:52 +0000

Sometimes, using a custom Dockerfile results in massive images, and, yes, they are a bunch of layers that will reproduce an isolated environment, their huge size makes sense.

Today, I am going to show you a way to save many space with a simple and cool Docker trick.

Dockerfiles

When it comes to create a custom image, you will need to write a Dockerfile, as you might know, each sentence in your file represents a new layer, let’s see a quick example.

FROM ubuntu:latest # Pulls latest ubuntu image

RUN apt update # Updates apt repositories
RUN apt install figlet # Installs figlet

CMD ["figlet", "-c", "Hello from dockerized ubuntu!"]
 # Runs!

If you try to build and run the previous Dockerfile

docker build -t hello .
docker run hello

The output will be something like

Well, the output is pretty, but let’s see how much space does my hello image use.

docker image ls

The output is

REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
hello        latest    3d0b79dcdd1f   10 minutes ago   109MB

109MB to print a hello world in my console using Ubuntu, this is actually huge, now, think about how much space a real-life application Docker image will use.

Single-stage Dockerfile

For this example, we will use a super simple Go REST API, the code is already done, and you can find it here.
Since the scope of this blog post is how to reduce the size of your Docker images, I will not explain the Go code, but I promise to create a post about that in the future.

In the repository for this post you will find two Dockerfiles, let’s take a look at Dockerfile first

FROM golang:latest

WORKDIR /server
COPY . .

RUN go mod download
RUN go build -o server .

CMD ["./server"]

The image built from this Dockerfile will

Pull the latest golang Docker image
Create a directory named /server and change the context to it
Copy all the files in the current directory into our container
Execute go mod download to install all the project’s dependency
Execute go build -o server . To build the app and rename the bin to server
Finally, execute the server binary every time we run a container from this image

Quite straightforward, right? Let’s build the image and take a look at the size, for this you can run the following commands.

docker build -t GoServer .
docker image ls

The output should be something similar to

REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
hello        latest    3d0b79dcdd1f   12 minutes ago   109MB
GoServer     latest    34e3ad5acd86   21 minutes ago   1.02GB

1.02GB is being used to isolate my server environment, it can be shocking, but let’s take a look at what it contains.

All my project dependencies
All Go dependencies
A lightweight Linux distro
The code of my app
The final binary of my app

So, it actually stores many things that we only use once, for example, since I already compiled my project, maybe storing all the dependencies is actually unnecessary, just as all the code of my app, and all the Go dependencies.

Then, we can get rid of some items of the above list and keep it like

~~All my project dependencies~~
~~All Go dependencies~~
A lightweight Linux distro
~~The code of my app~~
The final binary of my app

But, how do we achieve this?

For this, Docker actually has a solution, a different strategy of building Dockerfiles, splitting them in STAGES to reduce the quantity of layers.

Multi-stage Dockerfiles

In the project’s repository you have another Dockerfile, called Dockerfile.multi the extension .multi is not actually necessary, I just used it to differentiate it from the previous Dockerfile, let’s take a look at it

# Build stage
FROM golang:latest AS builder

WORKDIR /build
COPY . .
RUN go mod download

RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o server .

# Run stage
FROM alpine:latest

WORKDIR /server
COPY --from=builder /build/server .

CMD ["./server"]

It might look really complex, since it uses FROM twice, but it is actually smaller and easier to read (with practice).

What will this Dockerfile do when we build the image?

Pull latest golang Docker image and give it the builder alias
Create a directory named /build and change the context to it
Copy all the files in the current directory into our container
Execute go mod download to install all the project’s dependency
Execute go build -o server . To build the app and rename the bin to server, with some GO FLAGS enabled to make our binary executable in different environments.

Ok, we will take a break here, at this point in the Dockerfile

We copied the code
Installed the project dependencies
Compiled our project Many of the files that we want to get rid of were used in this First stage, named builder.

Ok, ready? Let's see the Second stage.

Pull latest alpine Docker image, a lightweight linux distro
Create a directory named /server and change the context to it
Copy the final binary server extracted from the previous builder stage, we do this using the from= key argument
Finally, execute the server binary

It was a longer road, but it worths it, let’s see the space used by our images using the same two commands as before, with a couple of modifications.

docker build -t GoLight -f Dockerfile.multi .
docker image ls

And, the output is

REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
hello        latest    3d0b79dcdd1f   12 minutes ago   109MB
GoServer     latest    34e3ad5acd86   21 minutes ago   1.02GB
GoLight      latest    309467e3d24e   6 minutes ago    17.5MB

It worked! Now our server image uses only 17.5MB even less than the hello image, but why?

Multistage Dockerfiles allows us to extract files and data from specific points on the creation of our images.

Therefore, we can get rid of many unnecessary things and save a lot of disk space without sacrificing functionality.

Don’t believe me? Let’s take a brief look at the execution of a container using each image.

We will use the heavier image first

docker run -p 8080:8080 GoServer:latest

This container then, creates a web server and starts to listen at :8080, currently our API only has one endpoint, so let’s perform a GET request to http://127.0.0.1:8080/songs

curl -X GET http://127.0.0.1:8080/songs

And the output should be the next one

[
    {
        "id": 1,
        "name": "Alabaster",
        "artist": "Foals",
        "album": "Total Life Forever"
    },
    {
        "id": 2,
        "name": "Bravery",
        "artist": "Human Tetris",
        "album": "River Pt. 1"
    },
    {
        "id": 3,
        "name": "Lately",
        "artist": "Metronomy",
        "album": "Metronomy Forever"
    },
    {
        "id": 4,
        "name": "Paranoid Android",
        "artist": "Radiohead",
        "album": "OK Computer"
    }
]

Ok! The heavy container is working properly and using 1.02GB of my disk, now let’s try the same using the lighter image (Remember to stop the current container)

docker run -p 8080:8080 GoLight:latest

This command will create a container but using the lighter image this time, the behavior is the same; therefore, we can perform the same GET request.

curl -X GET http://127.0.0.1:8080/songs

And the output is the same as before, give it a try!

Conclusion

Docker is a great tool and widely used in the market these days, although it has some disadvantages.

Of course, massive images are still lighter than virtual machines, but we can use Docker in a better way to exploit its full potential.

Multistage Dockerfiles work better with compiled languages, since you can use a stage for build and compile and another to actually execute the binary, yet this is not a limitation.

Please stay tuned for more Docker posts!

Writing a sequence class in Python

Aldo Vázquez — Sat, 21 Jan 2023 23:45:16 +0000

Python data model

Why `len` and not `list.length`?

This is a recurrent question when it comes to writing Python code, and if you are familiar with OOP, you probably asked the same question before.

To talk about this, we should take a look into special methods some other languages, like Ruby, call them magic methods.

These methods are not intended to be invoked directly by the user; instead, the interpreter invokes them.

As a result, we can deduce that the truth of the matter is len(sequence) built-in function calls sequence.__len__() behind the scenes.

That being said, our question remains unanswered, but actually, Python has a pretty good explanation for this behavior.

A user defined __len__ method to calculate the length of a sequence might be very custom, and vary from user to user.

But __len__ method of Python built-in types, like list, tuple, str or any other sequence is more complicated, and faster, than that.

Since Python is written in C, invoking len function with a built-in sequence type, Python, rather than iterating over the sequence and count how many items are inside it, looks for a C data structure field called ob_size, which is way faster.

This is because, __len__ method of the built-in sequences is written to do so.

We can conclude, Python is quite flexible; therefore, we are free to create any custom type, customizing each special method of our classes as much as we need.

In this article, I will show you how defining a couple of special methods, you will be able to create classes which behaves as any built-in Python sequence.

Writing a Playlist class

To show you, how to create a Playlist class which will behave as a Python builtin sequence we will create a new Song namedtuple , after all, what is a Playlist without songs?

Why a `namedtuple` and not a `class`?

Since my Song is only a “vessel” that don’t require any special logic and its data will remain the same during all the program execution, a namedtuple is a better option to use, they also have a nice human-readable string representation, which is easier to read when we use print.

from collections import namedtuple

song_attrs = ("name", "album", "artist")
Song = namedtuple("Song", song_attrs)

Ok, now our Song type is done, then, let’s start with the Playlist class.

from typing import List

class Playlist:
    def __init__(self):
        self.__songs: List[Song] = []

So far, we created a Playlist class which has a songs private attribute, this attribute is specifically a list of Song instances.

The next step is to implement a __len__ and a __getitem__ methods as well, Python does not use interfaces, but it actually requires implementing at least those two methods in order to considere your class as a sequence.

from typing import List

class Playlist:
    def __init__(self):
        self.__songs: List[Song] = []

    @property
    def songs(self) -> List[Song]:
        return self.__songs

    def __len__(self) -> int:
        return len(self.songs)

    def __getitem__(self, index: int) -> Song:
        return self.songs[index]

    def add_song(self, song: Song):
        self.songs.append(song)

In the above’s example, we defined a property which is actually optional, this is an equivalent of a getter, the advantage of using properties is, since we didn’t define a setter trying to set the songs property will raise an exception rather than allowing the user to override the full attribute, making it “read-only”.

On the other hand, we defined a __len__ method, which actually returns the length of the list of songs, then we can now use the next snippet.

my_playlist = Playlist()

album = "Soft Sounds"
artist = "Delta Sleep"

names = "Strongthany,Dotwork,Camp adventure,Sans Soleil"

for name in names.split(","):
    my_playlist.add_song(Song(name, album, artist))

len(my_playlist)
# Out: 4

But we defined a __getitem__ as well, this method, “unlocks” the sequence[index] syntax; therefore, we can access any of our songs by index.

my_playlist[0].name
# Out: Strongthany

my_playlist[-1]
# Out: Song(name='Sans Soleil', album='Soft Sounds', artist='Delta Sleep')

But that’s not all! It also “unlocks” slicing making us able to use the following syntax:

my_playlist[1:3]
# Out: [Song(name='Dotwork', album='Soft Sounds', artist='Delta Sleep'), Song(name='Camp adventure', album='Soft Sounds', artist='Delta Sleep')]

my_playlist[::-1]
# Out: [Song(name='Sans Soleil', album='Soft Sounds', artist='Delta Sleep'), Song(name='Camp adventure', album='Soft Sounds', artist='Delta Sleep'), Song(name='Dotwork', album='Soft Sounds', artist='Delta Sleep'), Song(name='Strongthany', album='Soft Sounds', artist='Delta Sleep')]

This ability, allows us to iterate over our sequence using a for sentence, as if the Playlist instances were built-in list instances, this is because, we actually delegated all the sequence implementation to our __songs attribute.

for song in my_playlist:
    print(song.name, "by ", song.artist)
# Out: 
# Strongthany by  Delta Sleep
# Dotwork by  Delta Sleep
# Camp adventure by  Delta Sleep
# Sans Soleil by  Delta Sleep

Although, no, it is not all yet! Many of the Python built-in modules, require a sequence to work, and, since our Playlist class is technically a sequence, we can use them with our instances.

for song in reversed(my_playlist):
    print(song.name, "by ", song.artist)
# Out: 
# Sans Soleil by  Delta Sleep
# Camp adventure by  Delta Sleep
# Dotwork by  Delta Sleep
# Strongthany by  Delta Sleep

from random import choice

choice(my_playlist)
# Out: Song(name='Dotwork', album='Soft Sounds', artist='Delta Sleep')

Conclusion

Python data model provides much flexibility, I haven’t worked (yet) with other language that actually allows you to emulate its built-in types, Go slices are useful, yes, but you cannot create a custom structure that fulfills your program necessities and behave as a built-in slice.

As you might have guessed, yes, you can emulate any other built-in type in Python.

For this article, I based myself on “Fluent Python, 2nd Edition” a great book written by Luciano Ramalho one of my biggest influences to become a Pythonista, please take a look into the book! You’ll learn many new things, just as I did.

Monitoring Linux instances with Prometheus and Grafana

Aldo Vázquez — Fri, 20 Jan 2023 23:52:29 +0000

If you want to know how is everything going in your servers, you must monitor them, yeah, sometimes read htop or similar commands output can be hard to understand, there are a lot of alternatives, and today I am going to show you how to set up a monitoring suite to always know what is going on your servers, linux instances, and more.

In this tutorial, I will show you how to mount a complete, flexible, and scalable monitoring system, using two great (and free) tools.

What are we going to use?

Prometheus is a powerful, open-source and easy-to-use monitoring solution, it is going to help us to collect and fetch a lot of information from your Linux instances.
Grafana on the other hand, is the most popular technology to create monitoring dashboards in a fast and easy way, also it has a lot of plugins and integrations, therefore, if it exists, you can create a Grafana panel for it.

Let's start!

Installing node_exporter

First of all, we will need to install a Prometheus light-weight daemon in all your servers that you want to monitor, this daemon is node_exporter.
Let's install it.
Connect to your server

$ ssh user@target-instance

Download the node_exporter files

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
Uncompress and rename the folder
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
$ mv node_exporter-1.0.1.linux-amd64 ne_files

Create a new user

$ sudo useradd --no-create-home -s /bin/false node_exporter
Change the node_exporter binary ownership to your new user
$ sudo chown node_exporter:node_exporter ./ne_files/node_exporter

Let's add node_exporter to your local binaries

$ sudo mv ./ne_files/node_exporter /usr/local/bin/

Now we are adding node_exporter as a new service
Create a service file for node_exporter with

$ sudo vim /etc/systemd/system/node_exporter.service

Then, add this to the new file, do not forget to save your changes.

[Unit]
Description=node_exporter
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Reload the services daemon to see your new service

$ sudo systemctl daemon-reload

Start it!

$ sudo systemctl start node_exporter

Ok, but how do I know it is working?
You can use:

$ sudo systemctl status node_exporter

Or go to your favorite browser and navigate to
http://<target-instance>:9100/metrics

If everything is working, you should see something similar to

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000108492
go_gc_duration_seconds{quantile="0.25"} 0.000108492
go_gc_duration_seconds{quantile="0.5"} 0.000108492
go_gc_duration_seconds{quantile="0.75"} 0.000108492
go_gc_duration_seconds{quantile="1"} 0.000108492
go_gc_duration_seconds_sum 0.000108492
go_gc_duration_seconds_count 1
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.846384e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.689864e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445623e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter

Remember, you need to repeat this process within every Linux instance that you want to monitor.

Installing Prometheus

Once all your target instances are running node_exporter, we will install Prometheus to scrape and understand your server's metrics.
You can install it either in your localhost or on another server. In this tutorial, we are installing Prometheus in a new server that will monitor everything. Don't worry if you want to install the monitoring suite in your localhost; both processes are very similar.

Connect to your monitoring server

$ ssh user@<monitoring-server>

Download the Prometheus files from here

$ wget https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz

Now, let's untar and rename the Prometheus files

$ tar xvf prometheus-2.23.0.linux-amd64.tar.gz
$ mv prometheus-2.23.0.linux-amd64 p_files

Create a new user to use Prometheus

$ sudo useradd --no-create-home -s /bin/false prometheus

Change the Prometheus binaries ownership

$ sudo chown prometheus:prometheus p_files/prometheus
$ sudo chown prometheus:prometheus p_files/promtool

Move Prometheus binaries to your local binaries

$ sudo mv p_files/prometheus /usr/local/bin
$ sudo mv p_files/promtool /usr/local/bin

Create folders to store Prometheus configuration

$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus

Do not forget to change the directories ownership

$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus

Copy configurations to your new folders

$ sudo cp -r p_files/consoles /etc/prometheus
$ sudo cp -r p_files/console_libraries /etc/prometheus

As you might guess, yes, you have to change the directories ownership

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Create a prometheus.yml file within /etc/prometheus folder

$ sudo vim /etc/prometheus/prometheus.yml

Add this to prometheus.yml

global:
 scrape_interval: 10s
scrape_configs:
 - job_name: 'prometheus'
 scrape_interval: 5s
 static_configs:
 - targets: ['<target-instance-1>:9090', '<target-instance-2:9100>']

This is a basic Prometheus configuration file. You must add every instance with the node_exporter daemon running in the "targets" array.
Once the prometheus.yml file is ready, change its ownership

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Now, let's create a Prometheus service file

$ sudo vim /etc/systemd/system/prometheus.service

Add the following configuration to prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
 --config.file /etc/prometheus/prometheus.yml \
 --storage.tsdb.path /var/lib/prometheus/ \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

Reload the service daemon

$ sudo systemctl daemon-reload

It's time to run your server!

$ sudo systemctl start prometheus

Check your service status

$ sudo systemctl status prometheus

Now, let's check the Prometheus Web UI

Open your favorite browser and navigate to
http://<monitoring-server-ip>:9090/
Now, you should see something like this,

Let's run a simple query to see if Prometheus is retrieving your instances metrics correctly.
Copy the following query into Prometheus Expression input, then click Execute.

sum by (mode, instance) (irate(node_cpu_seconds_total{job="Prometheus",mode="user"}[30s]))*100/4

Then, you should see a similar output

As you can see, Prometheus also lists all the instances you added in the targets array before.

Installing Grafana

We have installed node_exporter in your instance to export its metrics.
Then we installed Prometheus on a different server to interpret the instance's metrics. Now, we will install Grafana to create dashboards and charts to make the information easier to read.
The Grafana installation is pretty short. If you want (or need) to install it using the standalone binaries, following the previous steps should give you an idea.
Grafana has a "ready to download and install" deb package; using the following snippet should install Grafana as a new service.

$ sudo apt-get install -y adduser libfontconfig1
$ wget https://dl.grafana.com/oss/release/grafana_7.3.4_amd64.deb
$ sudo dpkg -i grafana_7.3.4_amd64.deb

Once Grafana installation is done, run the service with

$ sudo systemctl start grafana-server

Now open a new tab on your browser and navigate to
http://<monitoring-server>:3000
You should see a login screen.

The default username and password are always "admin" after you log in, Grafana will ask you to change your admin password. Remember to use a strong password, please.

Now, let's go to setup a new Data Source to fetch information from our Prometheus service.

Click the gear icon on the left bar, then click the Data Sources section.

Then click the Add data source button

As you might guess, yes, we will add a new Prometheus data source

You should see a similar screen

Set the HTTP URL to use your Prometheus service, by default it is set to localhost, which is true in this case, because Grafana and Prometheus are running in the same host.

Then scroll down and click on "Save and test"

Now, navigate to Manage dashboard sections, click the dashboard button on the left bar, then click on "Manage" button.

In this screen, click on Import

You should get this screen, as you can see, there are some ways to import dashboards, click on Upload JSON file.

You can find a lot of examples on internet or, if you prefer, you can create your own custom dashboard. For this tutorial we are using this dashboard, developed by Grafana Labs.
Download the JSON file and select it at the file explorer, then this window should be displayed.
Note: You can find a lot of dashboards here.

Just keep going with the dashboard settings, remember to choose the right Prometheus dashboard at the "Prometheus" dropdown, then, click on import.

And now, your dashboard is ready!
At the upper left corner, is a "Host" dropdown, there you can select any of your instances added to the Prometheus configuration file.
Yes, those are a lot of charts, Grafana allows you to remove or add as many as you want, so, get fun creating your own monitoring dashboard!
Thanks for reading, this is my first tutorial/blog ever! I will appreciate any feedback or comment.

First steps with Ansible.

Aldo Vázquez — Fri, 20 Jan 2023 23:26:17 +0000

Ansible is one of the most recent tools released by Red Hat and one of the best choices for Developers and DevOps engineers to automate almost everything inside a new server. It is free and open-source, and trust me, configuring remote servers has never been so easy.

In this tutorial, I intend to show you how to install and start to use this powerful tool.

Installing Ansible

First, you will need an Ansible controller host to manage all your nodes. This host could be your localhost, don't worry, Ansible will work no matter which operating system you use. I am using Ubuntu, but the installation steps for any other os are almost the same.

Let's install Ansible using apt.

$ sudo apt install ansible

Once the installation finished, run

$ ansible --version

If the package installation was successful, you should see something like this.

Ansible nodes

Now your ansible controller is working, pretty easy right?
Let's set up your nodes, open the /etc/ansible/hosts file with your preferred editor, you should see this template

# This is the default ansible 'hosts' file.
#
# It should live in /etc/ansible/hosts
#
#   - Comments begin with the '#' character
#   - Blank lines are ignored
#   - Groups of hosts are delimited by [header] elements
#   - You can enter hostnames or ip addresses
#   - A hostname/ip can be a member of multiple groups
# Ex 1: Ungrouped hosts, specify before any group headers.
#green.example.com
#blue.example.com
#192.168.100.1
#192.168.100.10
# Ex 2: A collection of hosts belonging to the 'webservers' group#[webservers]
#alpha.example.org
#beta.example.org
#192.168.1.100
#192.168.1.110
# If you have multiple hosts following a pattern you can specify
# them like this:
#www[001:006].example.com# Ex 3: A collection of database servers in the 'dbservers' group
#[dbservers]
#
#db01.intranet.mydomain.net
#db02.intranet.mydomain.net
#10.25.1.56
#10.25.1.57# Here's another example of host ranges, this time there are no
# leading 0s:#db-[99:101]-node.example.com

You can add your hosts at the bottom of the file or delete all the content and replace it with your hosts. The only rule is to follow the syntax below.

# Linux host
# host_alias ansible_host=<host_ip_or_url> ansible_connection=ssh ansible_user=<host_user> ansible_ssh_pass=<ssh_password_for_user>
# Windows host
# host_alias ansible_host=<host_ip_or_url> ansible_connection=winrm ansible_user=<host_user> ansible_password=<password_for_user>
# I am adding two servers, both of them are Linux servers
web1 ansible_host=my.webserver.com ansible_connection=ssh ansible_user=root ansible_ssh_pass=Passw0rd
db1 ansible_host=my.databaseserver.com ansible_connection=ssh ansible_user=root ansible_ssh_server=Passw0rd

Now, add some groups for your servers. Groups let you run commands and scripts on multiple hosts simultaneously.

[web_nodes]
web1 ansible_host=my.webserver.com ansible_connection=ssh ansible_user=root ansible_ssh_pass=Passw0rd
[db_nodes]
db1 ansible_host=my.databaseserver.com ansible_connection=ssh ansible_user=root ansible_ssh_server=Passw0rd

Save your changes and close the file.

Ansible depends on sshpass package, install it running the following command

$ sudo apt install sshpass

The only remaining step is to connect to your hosts through ssh; this is for adding your host's fingerprints to the ~/.ssh/known_hosts file.

$ ssh root@my.webserver.com
$ ssh root@my.databaseserver.com

Now, try your hosts by running:

$ # Ansible has an 'all' group which contains all your defined hosts
$ ansible all -m ping

If everything is going well, you will see a similar output

Running playbooks

Your hosts are ready to be used through Ansible. Please create a new folder in your home directory.

$ mdkir ~/ansible-tutorial
$ cd ~/ansible-tutorial

Ansible's scripts are just YAML files. If you have worked with YAML before, this step will be straightforward.
Open a new playbook.yml file and add the following lines

- name: My playbook
  hosts: web1
  tasks:
  - name: Runnig date
    command: date

name is an alias for the playbook
hosts defines the hosts to run the playbook, can be a group
tasks is an array that holds the actions or modules to use

Save your changes and close the file, now run it with
$ ansible-playbook playbook.yml
You should see a similar output as below

A simple, real-life example

You learned some new things so far

Install and configure an Ansible controller
Add a new playbook
Run it on your hosts

Now it is time to learn how we can use Ansible in a real-life scenario.
For this example, you are going to install and run an Nginx webserver.
Let's create a new playbook

$ vim nginx-playbook.yml

To start to write this playbook, add a name and the host where the Nginx should be running (remember the identation)

- name: 'Install and deploy nginx'
  hosts: web1

Now, add the tasks, in this case you will need to define two actions.
Install the latest Nginx version
Start Nginx server

tasks:
  - name: 'Install Nginx latest version'
    apt:
      name: nginx
      state: latest

This task looks for Nginx on the target host. If Ansible finds Nginx's latest version, the installation won't happen.

- name: 'Run Nginx'
    service:
      name: nginx
      state: started

And this task checks if the Nginx server is already running. If it is not, then the step is skipped.
Your complete nginx-playbook.yml should look like this

- name: 'Install and deploy nginx'
  hosts: web1
  tasks:
    - name: 'Install Nginx latest version'
      apt:
        name: nginx
        state: latest
    - name: 'Run Nginx'
      service:
        name: nginx
        state: started

Save your changes, and now, try it!
Since we are installing a package in your host, you will need to add the '-b' flag to the playbook command.

$ ansible-playbook nginx-playbook.yml -b 
$ # -b or --become grants root privileges to ansible user.

And finally let's check the browser!

Conclusion

Ansible is a great ally; it makes it easy to install and deploy things in all your servers. Sure, this is just an introductory tutorial, but this tool has one module for everything you need to automate. You can find all always available here.

Thanks for reading. Every feedback, comment, or sharing is highly appreciated!

Forem: Aldo Vázquez

Set up a multi-container app with docker compose

docker compose

Installation

Post’s code

docker-compose.yml

docker compose up

Conclusion

Note from the OP

Lighter Docker images

Dockerfiles

Single-stage Dockerfile

Multi-stage Dockerfiles

Conclusion

Writing a sequence class in Python

Python data model

Why len and not list.length?

Writing a Playlist class

Why a namedtuple and not a class?

Conclusion

Monitoring Linux instances with Prometheus and Grafana

What are we going to use?

Installing node_exporter

Installing Prometheus

Installing Grafana

First steps with Ansible.

Installing Ansible

Ansible nodes

Running playbooks

A simple, real-life example

Conclusion

`docker compose`

`docker-compose.yml`

`docker compose up`

Why `len` and not `list.length`?

Why a `namedtuple` and not a `class`?