Forem: Juan Luis Sanz

Notes: Distributed Systems in One Lesson by Tim Berglund

Juan Luis Sanz — Tue, 13 Sep 2022 15:54:00 +0000

Here are some notes on this original video from Tim Berglund. Check it out!

As far as the client goes, a distributed system is a set of computers that act as a single one.

Characteristics

They operate concurrently
Fail independently
They don't share a global clock

Three topics

1) Storage

Start off with a single relational database, you usually get more reads than writes

Read replication

To mitigate the excessive load a DB might be exposed to when the reads become too much, you can use read replication. One master node for writes and several for reads. This is fine until the master node saturates (usually on writes, as reads can scale)

More info on read replication here

Sharding

Sharding. You split the read replication scheme by a key (say First name, so from A-F is one shart, from F-N, then N-Z)

✅ Solves	❌ Creates problem
- You now have more room to distribute your writes	- Consistency is broken
	- Data model is broken (no `JOIN` across shards)

To solve latency issues, you may consider denormalizing your DB (aka removing relations between tables). Please don't.

More info on sharding here

Consistent hashing

Say you have a key/value database. You hash the key and store it according to the shards you have determined. In this example

Each shard indexes a 1/8th of the content, indexed by the hash's first number. If I want to store

{key: "Tim", value: "Americano"}

I'd hash the key first

{hashedKey: "9F72", value: "Americano"}

find the corresponding node (in this case A000 since A000 > 9F72 > 8000) and store it there.

Replication

We'd need to ensure there are no problems with system failure (aka one node going down), you can replicate the data in other nodes (say in the next two nodes as well).
❌ The problem that comes with this is inconsistency, since you are now writing to more than one node. What if you want to write to 3 nodes but one is down? It'll be inconsistent when it's back online.

Consistency or R+W>N

If the number of nodes I write + the number of nodes I read is greater than the number of replicas, you have strong consistency

CAP theorem

Consistent: When I read something I want it to be the latest version of it.
Available
Partition tolerant: If a part of the system falls, the entire system must hold. The theorem basically states that sometimes you must give up one of the three parts in real life applications. Example: If a partition goes down and a read request comes in, you can either:
a) Give the best answer you can, although it may not be consistent
b) Sacrifice availability and return an error to ensure that no inconsistent reply is sent.

2) Computation

Distributed computation is basically like moving from a single threaded computation to a multithreaded one, but several times worse.

Imagine you have a bunch of data sitting in a lot of computers and you need to do stuff to it.

Old way: MapReduce

Pushes all your data through two functions

Map
Reduce

Example: Say you have the sentence

I want to eat two burgers with fries
and drink two sodas with ice

Map

When you push it throug a map function, it basically makes a "stateless" count of each word (since it won't have any memory yet, so all numbers will effectively be 1)

{I, 1}
{want, 1}
{to, 1}
{eat, 1}
{two, 1}
{burgers, 1}
{with, 1}
{and, 1}
{drink, 1}
{two, 1}
{sodas, 1}
{with, 1}
{ice, 1}

Shuffle

Since this is a distributed computation, sometimes you may need to move these words across the network, write them next to each other and so on.

Reduce

The reduce function in this case will count properly all usages of a single word

{I, 1}
{want, 1}
{to, 1}
{eat, 1}
{two, 2}
{burgers, 1}
{with, 2}
{and, 1}
{drink, 1}
{sodas, 1}
{ice, 1}

We have now managed to split the work of map and the work of reducing among many computers.

More info on map-reduce here

Spark

map-reduce becomes transform-action. Same scatter/gather paradigm.
Spark is also storage agnostic, compared to Hadoop.

Kafka

In kafka everything is a stream, in real time.

3) Messaging

Systems can talk to each other instead of talking to each other via shared databases.
Messages are consumed by subscribers
Created by producers
Organized into topics
Processed by brokers
Usually persistent in the long term, but you can change the retention period to whatever you want.

❌ Problems you may encounter:

A topic gets too big (huge retention period, messages are huge, read and writes are too fast...)
Reliability
Guarantee delivery even if a computer is down

Apache Kafka

⚠ When scaling, you lose system-wide order. You still have partition-wide order.

Producers will create events and randomly assign it to partitions (not really randomly, but for the sake of the argument let's assume so). These partitions will have a consumer on the other side processing whichever event is first on the queue.

The great thing is that, if after the event is created you want to store it, you can! Anyone can subscribe to the topic and from the event generation, you may have a consumer that stores on a MySQL DB the information and another that does some form of real-time processing.

More info on kafka here

Done!

I hope these short notes help you! Don't forget to check the original video and share it!

Have a wonderful day!

Install letsecrypt certificates on all your docker containers

Juan Luis Sanz — Wed, 09 Feb 2022 18:27:00 +0000

When dealing with containers running on a same server, you may sometimes run into the following problem:

How do I give each of the containers its own domain/subdomain? and how do I do it securely?

First of all, this problem can be addressed in several ways, and neither is more correct than another. I will simply show you a quick and easy way to do this using:

Docker
Traefik
Letsencrypt

and doing so automatically.

Setup

Let's pretend we want to setup two services:

Service	domain	port
Grafana	grafana.mydomain.com	80, 443
InfluxDB	influx.mydomain.com	8086

⚠ This setup is just for testing purposes: any docker container will work with this method (wordpress, mariaDB, mongoDB, ELK stack, Minecraft server, custom images...)

Grafana: an amazing monitoring tool which can display data from an impressive array of sources, including influxDB. Let's say we want to monitor some service we have setup elsewhere!
InfluxDB: a time-based DB, in which we can store time events (for example, the temperature of the CPU every n seconds or the number of IO operations on our drives)

How do I give each container its own domain/subdomain?

Well that depends on the setup you are running:

a) I run it all from a server with a public, static IP like a VPS

Easy! just head over to your DNS settings, and create an A record for each domain/subdomain. In this case both will point to the same IP address.

A slightly less repetitive approach would be to point one of the domain to the IP address via an A record and any other domains point towards that domain via CNAME records. That way if the IP address changes, you only have to do it once!

b) I run it from a public IP, but it changes every now and then

This scenario would apply if for instance you want to host from your home and the public IP address your ISP assigns you changes periodically. To circumvent this there are several possibilities which I won't go into detail, but they're easy to follow. IT boils down to having your server at home communicating with your DNS provider, telling them your public IP each time it changes, so the DNS entry can be changed accordingly.

Don't worry no need to do this by hand, just look this up online for a quickstart.

Use a service like duckdns
Cloudfare dynamic IP service

How would my server know to which container to point to, if all DNS entries resolve to the same IP address?

That's where traefik comes in!

Step 1) Create a docker-compose.yml file for traefik

Here a small template, change parameters accordingly!

version: "3"

services:
  traefik:
    image: traefik
    container_name: traefik
    ports:
      - "80:80"         # HTTP port for grafana
      - "443:443"       # HTTPS port for grafana
      - "8086:8086"     # Exposed port for influxDB

      # expose port below only if you need access to the Traefik API
      #- "8080:8080"
    command:
      #- "--log.level=DEBUG"
      #- "--api=true"
      - "--providers.docker=true"

      - "--entryPoints.port443.address=:443"
      - "--entryPoints.port80.address=:80"
      - "--entryPoints.port8086.address=:8086"

      - "--certificatesResolvers.le-ssl.acme.tlsChallenge=true"
      - "--certificatesResolvers.le-ssl.acme.email=myemail@mydomain.com"
      - "--certificatesResolvers.le-ssl.acme.storage=/letsencrypt/acme.json"
    volumes:
      - traefik-data:/letsencrypt/
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always

volumes:
  traefik-data:

Simple, isn't it? if you run it with

docker-compose up

you will notice not a lot has changed, and that's because we need to set the subdomains themselves on the container definitions.

Step 2) Add each container's particular configuration

So, for example, grafana and influxDB could look like so

version: "3"

services:
  influx:
    image: influxdb:1.7 # or any other recent version
    labels:
      # SSL endpoint
      - "traefik.http.routers.influx-ssl.entryPoints=port8086"
      - "traefik.http.routers.influx-ssl.rule=host(`influx.mydomain.com`)"
      - "traefik.http.routers.influx-ssl.tls=true"
      - "traefik.http.routers.influx-ssl.tls.certResolver=le-ssl"
      - "traefik.http.routers.influx-ssl.service=influx-ssl"
      - "traefik.http.services.influx-ssl.loadBalancer.server.port=8086"
    restart: always
    container_name: influxdb
    volumes:
      - influx-data:/var/lib/influxdb
    environment:
      # - INFLUXDB_DB=
      - INFLUXDB_HTTP_ENABLED=true
      - INFLUXDB_HTTP_AUTH_ENABLED=true

      # - INFLUXDB_ADMIN_USER=
      # - INFLUXDB_ADMIN_PASSWORD=

  grafana:
    labels:
      # SSL redirect requires a separate router (https://github.com/containous/traefik/issues/4688#issuecomment-477800500)
      - "traefik.http.routers.grafana.entryPoints=port80"
      - "traefik.http.routers.grafana.rule=host(`grafana.mydomain.com`)"
      - "traefik.http.middlewares.grafana-redirect.redirectScheme.scheme=https"
      - "traefik.http.middlewares.grafana-redirect.redirectScheme.permanent=true"
      - "traefik.http.routers.grafana.middlewares=grafana-redirect"
      # SSL endpoint
      - "traefik.http.routers.grafana-ssl.entryPoints=port443"
      - "traefik.http.routers.grafana-ssl.rule=host(`grafana.mydomain.com`)"
      - "traefik.http.routers.grafana-ssl.tls=true"
      - "traefik.http.routers.grafana-ssl.tls.certResolver=le-ssl"
      - "traefik.http.routers.grafana-ssl.service=grafana-ssl"
      - "traefik.http.services.grafana-ssl.loadBalancer.server.port=3000"
    image: grafana/grafana
    container_name: grafana
    restart: always
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.mydomain.com
      - GF_SERVER_DOMAIN=grafana.mydomain.com
      - GF_USERS_ALLOW_SIGN_UP=false
volumes:
  traefik-data:
  influx-data:
  grafana-data:

Notice the following:

We have added several labels to the usual configuration for grafana and influxDB.
Port 80 on grafana will redirect to HTTPS port 443.
traefik-data volume has been included as well

Running this will still not work because letsencrypt demands one last thing.

Generating certificates

For letsencrypt to generate the required certificates, it must make sure that the domains we claim here are ours. To do so we have to add one more DNS entry for each subdomain.

Each entry must be

Type: CAA
Name: Subdomain name, not the entire FQDN (in this case just grafana and influx)
Flags: 0
TTL: Auto
Tag: Only allow specific hostnames
CA domain name: letsencrypt.org

Done!

That's it! run all containers via

docker-compose up

(add -d for detacched mode)

and head over to https://grafana.mydomain.com to see your working grafana with custom domain and TLS!

Thanks for reading!

Have a lovely day!