Forem: Livio Ribeiro

Discuss: Return a response or write to the response?

Livio Ribeiro — Mon, 04 Aug 2025 22:36:55 +0000

In most web frameworks (Spring, Django, FastAPI, Axum) the handlers return an object to the framework that will be used to build the response that is sent to the client.

However, the Go lang net/http package uses a different approach: the handlers write data directly into the response and have no return value.

I believe that this behavior comes from a time that the most frameworks worked with blocking IO and employed optimizations using queues and threads dedicated to process the responses, so your code would return an object describing the response and the framework would put it in the queue to be processed by a thread.

With non-blocking IO, there is no need for these optimizations, since the async runtime already takes care of everything related to IO, allowing handlers to write data directly to the response.

So, which is better? Return a response or write to the response?

Announcing Selva web framework

Livio Ribeiro — Sat, 28 Oct 2023 10:12:59 +0000

I'd like to announce Selva web framework!

I've been working on it for quite a while, did a lot of back and forth, changed how some things work, then changed back... until I finally felt it was good enough to do a public announcement.

The framework uses under the hood another project of mine, asgikit, which is a toolkit that provides just the bare minimum to create an asgi application.

Quickstart
- What's going on?
- Why is that?
Controllers and handlers
- Path and query parameters
- Request body and Pydantic
Dependency injection
- Factory functions
Configuration
- Accessing the configuration
- Typed configuration
- Configuration profiles
Logging
Final words

Quickstart

First install the required dependencies:

pip install selva uvicorn[standard]

Create the application.py file:

from asgikit.requests import Request
from selva.web import controller, get


@controller
class Controller:
    @get
    async def hello(self, request: Request):
        response = request.response
        await response.start()
        await response.write("Hello, World!", end_response=True)

And then run uvicorn:

uvicorn selva.run:app --reload

What's going on?

First, you do not need to create an application, you just use selva.run:app. Selva will look for a module or package named application and scan it. Controllers will be registered in the routing system and services will be registered with the dependency injection system. Controllers are also services and take part in the dependency injection system as well.

But the most notable difference from other tools is that, in the request handler, instead of returning a response, you write to the response. This behavior comes from asgikit and is similar to how Go's net/http works.

Asgikit provides several helper functions to handle common use cases. The example above could be written as:

from asgikit.requests import Request
from asgikit.responses import respond_text
from selva.web import controller, get


@controller
class Controller:
    @get
    async def hello(self, request: Request):
        await respond_text(request.response, "Hello, World!")

Why is that?

In the common way of returning a response, extensibility is achieved by extending the usual Response class. Now you have to deal with inheritance and need to know, to some extent, the inner workings of the response class.

With the approach used by Asgikit/Selva (which was inspired by Go's net/http), extensibility is achieved through helper functions that only need to know how to write to the response.

Reading the request body uses a similar approach, you use helper function, such as asgikit.requests.read_json, to read the request body as json.

Controllers and handlers

Controllers are classes that have methods to handle request. They can be decorated with the path they respond to:

from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.web import controller, get


@controller("api")
class Controller:
    @get("hello")
    async def hello(self, request: Request):
        await respond_json(request.response, {"message": "Hello, World!"})

The method hello will respond in the path localhost:8000/api/hello:

$ curl http://localhost:8000/api/hello
{"message": "Hello, World!"}

Path and query parameters

Path parameters are defined in the handler decorator with the format :parameter to match path segments (between /), or *parameter to match paths with the / character.

Parameters can be extracted from the request by annotating the handler function arguments, such as selva.web.FromPath or selva.web.FromQuery:

from typing import Annotated
from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.web import controller, get, FromPath, FromQuery


@controller("api")
class Controller:
    @get("hello/:name")
    async def hello_path(self, request: Request, name: Annotated[str, FromPath]):
        await respond_json(request.response, {"message": f"Hello, {name}!"})

    @get("hello")
    async def hello_query(self, request: Request, name: Annotated[str, FromQuery]):
        await respond_json(request.response, {"message": f"Hello, {name}!"})

Type conversion is done based on the parameter type and can be extended.

Request body and Pydantic

If you annotate a handler parameter with a type that inherits from pydantic.BaseModel, or a list[pydantic.BaseModel], Selva will read the request body and parse using the pydantic.BaseModel class from the annotation:

from pydantic import BaseModel
from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.web import controller, post


class MyModel(BaseModel):
    property: int


@controller("api")
class Controller:
    @post("post")
    async def hello(self, request: Request, model: MyModel):
        await respond_json(request.response, {"model": model.model_dump_json()})

Dependency injection

Selva comes with a dependency injection system. You just need to decorate your services with @service and annotate the classes where the service will be injected:

from typing import Annotated
from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.di import service, Inject
from selva.web import controller, get, FromPath


@service
class GreeterService:
    def greet(self, name: str) -> str:
        return f"Hello, {name}!"


@controller("api")
class Controller:
    greeter: Annotated[GreeterService, Inject]

    @get("hello/:name")
    async def hello(self, request: Request, name: Annotated[str, FromPath]):
        message = self.greeter.greet(name)
        await respond_json(request.response, {"message": message})

Remember that classes decorated with @controller are services too and therefore can inject other services.

Factory functions

Services can also be defined by decorating a function with @service, making them factory functions. The same GreeterService example could be written could be written like this:

from typing import Annotated
from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.di import service, Inject
from selva.web import controller, get, FromPath


class GreeterService:
    def greet(self, name: str) -> str:
        return f"Hello, {name}!"


@service
def greeter_service_factory() -> GreeterService:
    return GreeterService()


@controller("api")
class Controller:
    greeter: Annotated[GreeterService, Inject]

    @get("hello/:name")
    async def hello(self, request: Request, name: Annotated[str, FromPath]):
        message = self.greeter.greet(name)
        await respond_json(request.response, {"message": message})

Configuration

Selva uses YAML files the provide configuration settings to the application. These files are located in the configuration directory:

project/
├── application/
│   ├── __init__.py
│   ├── controller.py
│   └── service.py
└── configuration/
    ├── settings.yaml
    ├── settings_dev.yaml
    └── settings_prod.yaml

The settings files are parsed using strictyaml, so only a subset of the yaml spec is used. This decisions is to make the settings files safer to parse and more consistent.

Selva provides a way to reference environment variables in the settings files:

property: ${PROPERTY}
with_default: ${PROPERTY:default}

Accessing the configuration

You can access the configuration by injection the class selva.configuration.Settings:


## settings.yaml
# message: Hello, World!

from typing import Annotated
from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.configuration import Settings
from selva.di import Inject
from selva.web import controller, get


@controller("api")
class Controller:
    settings: Annotated[Settings, Inject]

    @get("hello")
    async def hello(self, request: Request):
        await respond_json(request.response, {"message": self.settings.message})

Typed configuration

You can use Pydantic and the di system to provided typed settings to your services:


## settings.yaml
# application:
#   value: 42

from typing import Annotated
from pydantic import BaseModel
from asgikit.requests import Request
from asgikit.responses import respond_json
from selva.configuration import Settings
from selva.di import service, Inject
from selva.web import controller, get


class MySettings(BaseModel):
    value: int


@service
def my_settings_factory(settings: Settings) -> MySettings:
    return MySettings.model_validate(settings.application)


@controller("api")
class Controller:
    settings: Annotated[MySettings, Inject]

    @get("hello")
    async def hello(self, request: Request):
        await respond_json(request.response, {"value": self.settings.value})

Configuration profiles

We can create different configuration profiles that can be activated by settings the environment variable SELVA_PROFILE. When a profile is set, the framework will look for a file named configuration/settings_${PROFILE}.yaml and merge its settings into the main settings. For example, if we define SELVA_PROFILE=dev, the file configuration/settings_dev.yaml will be loaded.

# settings.yaml
property: value
mapping:
  property: nested value

# settings_dev.yaml
dev_property: dev value
mapping:
  property: dev nested value

# final settings
property: value
mapping:
  property: dev nested value
dev_property: dev value

Logging

Selva uses loguru for logging and provides some facilities on top of it: it configures by default an interceptor for the stardand logging module, so all logs go through loguru, and a custom filter that allows settings the log level for individual python packages in the yaml configuration.

# settings.yaml
logging:
  # log level to be used when not specified by package
  root: WARNING
  # specification of log level by package
  level:
    application: ERROR
    application.service: WARNING
    sqlalchemy: INFO
    sqlalchemy.orm: DEBUG

If a package has disabled loguru (logger.disable("package")) you can enable it again in the configuration file, and you can disable logging for a package as well:

logging:
  enable:
    - package
  disable:
    - other_package

Final words

Selva is still in its infancy, and there is work to do, more tests need to be written, documentation can be improved, but I believe it is a great tool and I hope it can have its place in the Python ecosystem.

Let me know in the comments what you think about Selva.

And by the way, "Selva" is portuguese for "Jungle" :)

Why I prefer Rust over Go

Livio Ribeiro — Sun, 06 Aug 2023 20:31:08 +0000

Recently I decided to learn Go and my first impressions were quite good. Go is a very nice language and easy to get started with. The go routines is probably the best feature of the language and I really wish other languages followed this model. The defer feature is also very nice.

However, as I continued learning, some things started feeling odd, some design decisions that looked strange. For example, the only loop construct is for, and there is no way to create a loop that iterates over a sequence of numbers, like for i in range(10) in Python or for i in 0..10 in Rust, you have to use a for loop as in C: for i := 0; i <= 10; i++.

After a while I realized: Go tries so hard to be like C that it does not even try make improvements that make it look too different than C.

Now let's compare the RWMutex from Go with RWLock from Rust:

lock := sync.RWMutex{}
lock.RLock()
// do someting
lock.Unlock()

This code will compile just fine, but will fail at runtime, because I called lock.Unlock() instead of lock.RUnlock(). You might say that is the developer's fault for no paying attention and calling the wrong method, and you are right, but let's take a look at the equivalent in Rust:

let lock = RwLock::new(1);

{
    let read = lock.read().unwrap();
    // do something
}

First thing we notice is that the RWLock operates with a value, since is very common the use a lock to synchronize access to something. Then we open curly braces to start a scope because lock.read() returns a guard that releases the lock when it goes out of scope, so there is no wrong method to be called. The error handling may look annoying but it is there for a reason.

For me this design decision for the Go RWMutex seems very strange, it could be much better if RLock returned a handle in which you would call Unlock and I cannot think of any reason for this other than to look more like C.

But Rust is not perfect, though, function signatures can become messy, specially with async, and dealing with lifetimes is not fun. And compilation time...

I will continue learning Go, it is really a nice language, but I still prefer Rust, I feel much safer with it (pun intended).

Angular dynamic settings

Livio Ribeiro — Fri, 27 Aug 2021 11:47:08 +0000

(You can find the code shown here in https://github.com/livioribeiro/angular-dynamic-settings-example)

When developing an Angular application, usually your application settings lives in src/environments/environment.ts for development and src/environments/environment.prod.ts for production, and Angular takes care of swapping those during a production build.

But if we have another environment, a staging environment, then we have a problem, since we expect to approve the application running in staging and promote the same code to production, but with Angular's approach to configuration we need to run another build to configure our app to production.

To overcome this problem, I came up a very simple, yet very effective, strategy:

Load a javascript file before the application starts that will define a settings object in window.$environment. This is essentially the same as environment.ts.
In environment.ts, export the object defined in window.$environment.
Tell Angular to add the configuration directory to the build output directory.

First we need to create a directory called src/config and put the javascript file environment.js there:

// src/config/environment.js
window.$environment = {
  production: false,
  api: "dev.my-backend.io",
  // and any other configuration that would go in "environment.ts"
};

And then load the script on index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>MyApp</title>
  <base href="/">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
  <!-- add the following line -->
  <script src="/config/environment.js"></script>
</head>
<body>
  <app-root></app-root>
</body>
</html>

Now, in environment.ts, export the configuration object:

// src/environments/environment.ts

// this interface is just to making things more typed
interface Environment {
  production: boolean;
  api: string;
}

export const environment = (window as any).$environment as Environment;

And finally, change angular.json build options, adding "src/config" the the assets, and remove the "fileReplacements" completely. I also changed "outputPath" to just "dist":

...
"build": {
  "builder": "@angular-devkit/build-angular:browser",
  "options": {
    // "outputPath" is just "dist"
    "outputPath": "dist",
    "index": "src/index.html",
    "main": "src/main.ts",
    "polyfills": "src/polyfills.ts",
    "tsConfig": "tsconfig.app.json",
    "assets": [
      "src/favicon.ico",
      "src/assets",
      // add the following
      "src/config"
    ],
    "styles": [
      "src/styles.css"
    ],
    "scripts": []
  },
    "configurations": {
      "production": {
        "budgets": [
          {
            "type": "initial",
            "maximumWarning": "500kb",
            "maximumError": "1mb"
          },
          {
            "type": "anyComponentStyle",
            "maximumWarning": "2kb",
            "maximumError": "4kb"
          }
        ],
        // "fileReplacements" is removed
        "outputHashing": "all"
      },
...

We can safely remove src/environments/environment.prod.ts, we don't need it anymore.

Now we can inject a configuration script instead of doing another build.

This approach works great with docker and kubernetes, and we can test it right now!

First, we need a Dockerfile:

FROM node:latest as builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build

FROM nginx:latest
COPY --from=builder /app/dist/ /usr/share/nginx/html/

And a configuration to inject (I called it "environment.js"):

// environment.js
window.$environment = {
  production: true,
  api: "prod.my-backend.io",
};

We now build the image and run the container with our new configuration:

docker build -t my-app .
docker run --name my-app \
  -it --rm \
  -p 8080:8080 \
  -v $PWD/environment.js:/usr/share/nginx/html/config/environment.js \
  my-app

With Kubernetes you can use a configmap to store the "environment.js" and mount it as a volume under "/usr/share/nginx/html/config".

And that's it! No more rebuilding angular for staging and production!

FastAPI with OIDC Discovery

Livio Ribeiro — Wed, 23 Jun 2021 11:34:53 +0000

FastAPI is a very nice tool for writing web applications in Python, it is very fast and is a joy to use. But saddly it still lacks support for openid-connect discovery, so I cannot simply point to my Keycloak instance and it configures itself.

Because of this I wrote FastAPI Resource Server, a helper that adds support to this scenario: setup issuer address, receive a JWT, validate and decode claims and return claims.

Usage is pretty straightforward, install with pip install fastapi-resource-server and configure your application:

from fastapi import Depends, FastAPI, Security
from pydantic import BaseModel

from fastapi_resource_server OidcResourceServer

app = FastAPI()

auth_scheme = OidcResourceServer(
    "https://issuer.url",
    scheme_name="My Issuer",
)


class User(BaseModel):
    sub: str
    username: str
    given_name: str
    family_name: str
    email: str


def get_current_user(claims: dict = Security(auth_scheme)):
    claims.update(username=claims["preferred_username"])
    user = User.parse_obj(claims)
    return user


@app.get("/users/me")
def read_current_user(current_user: User = Depends(get_current_user)):
    return current_user

Under the hood it will fetch the openid configuration from {issuer_url}/.well-known/openid-configuration, then fetch the jwks from the jwks_uri property and use python-jose to decode the JWT.

And it works with the swagger ui with no problems.

Manage frontend dependencies in Python web projects with FrontMan

Livio Ribeiro — Mon, 08 Feb 2021 12:09:27 +0000

Web applications always need some Javascript and CSS to look good and make user experience better (JQuery and Bootstrap, for instance), but managing all the libraries we need can be a source of trouble, specially when executing CI/CD pipelines.

Using npm to declare and retrieve frontend dependencies is a solution, but it is a bit too much if we are not building single page application with Angular or Vuejs, and besides that, it adds one significant dependency that in some cases could be avoided.

Because of that, and inspired by AspNet LibMan, I built FrontMan, a dependency manager written in Python for frontend dependencies that fetches files from CDNs (cdnjs, jsdelivr and unpkg).

You can install FrontMan by running pip install frontman (pipenv install --dev frontman for Pipenv and poetry add --dev frontman for Poetry)

Usage is simple, create a manifest (frontman.json) and execute frontman install.

The manifest is a json file like the following:

{
  "provider": "jsdelivr",
  "destination": "assets",
  "packages": [
    {
      "name": "jquery",
      "version": "3.5.1",
      "provider": "cdnjs",
      "destination": "jquery",
      "files": [
        "jquery.min.js"
      ]
    },
    {
      "name": "@popperjs/core",
      "version": "2.6.0",
      "path": "dist/umd",
      "files": [
        {
          "name": "popper.min.js",
          "rename": "popper.js",
          "destination":"popper"
        }
      ]
    },
    {
      "name": "bootstrap",
      "version": "4.6.0",
      "path": "dist",
      "destination": "bootstrap",
      "files": [
        "js/bootstrap.min.js",
        "css/bootstrap.min.css"
      ]
    }
  ]
}

The output will be like:

OK   https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js -> assets/jquery/jquery.min.js
OK   https://cdn.jsdelivr.net/npm/@popperjs/core@2.6.0/dist/umd/popper.min.js -> assets/popper/popper.js
OK   https://cdn.jsdelivr.net/npm/bootstrap@4.6.0/dist/js/bootstrap.min.js -> assets/bootstrap/js/bootstrap.min.js
OK   https://cdn.jsdelivr.net/npm/bootstrap@4.6.0/dist/css/bootstrap.min.css -> assets/bootstrap/css/bootstrap.min.css

After runnig frontman install there will be a file structure like this:

assets
├── bootstrap
│   ├── css
│   │   └── bootstrap.min.css
│   └── js
│       └── bootstrap.min.js
├── jquery
│   └── jquery.min.js
└── popper
    └── popper.js

Running the install command again will give the following output:

SKIP https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js -> assets/jquery/jquery.min.js
SKIP https://cdn.jsdelivr.net/npm/@popperjs/core@2.6.0/dist/umd/popper.min.js -> assets/popper/popper.js
SKIP https://cdn.jsdelivr.net/npm/bootstrap@4.6.0/dist/js/bootstrap.min.js -> assets/bootstrap/js/bootstrap.min.js
SKIP https://cdn.jsdelivr.net/npm/bootstrap@4.6.0/dist/css/bootstrap.min.css -> assets/bootstrap/css/bootstrap.min.css

If a file is already downloaded, FrontMan will not download it again, unless you pass the --force flag.

I hope you like FrontMan and find it useful.

Trying Hashicorp Nomad 1.0 with LXD and Terraform

Livio Ribeiro — Wed, 06 Jan 2021 19:05:50 +0000

A while ago I wrote an article about deploying Hashicorp Nomad using LXD and Ansible and since Nomad hit 1.0 I decided to revisit it. It is an interesting and I wanted to try Terraform and learn a bit about it.

You can find everything in this repository.

To deploy the cluster, install Terraform and Packer and then terraform apply.

I used Packer, another tool from Hashicorp, to create the container images and I also added a Prometheus instance to monitor the Nomad cluster and services. I will probably add Grafana in the future and if you know a good dashboard for Nomad, please share :)

Terraform is a great tool, but Ansible is still more flexible, although it can be a bit harder to deal with and sometimes I feel lost in its documentation.

Nomad is a good alternative to Kubernetes, specially if you are deploying your own cluster, with focus in simplicity and it is so easy to test it on your own machine.

Openshift behind a proxy with SSL termination

Livio Ribeiro — Tue, 06 Aug 2019 17:50:11 +0000

The Openshift Container Platform is a great tool to deploy your applications and services in a container environment: It hides the complexity of Kubernetes and offers many facilities to manage builds, configurations, volumes and environment variables.

Everything was fine until I had this weird problem where I could not get Keycloak to work properly. I was unable to authenticate because the service kept telling me that it required HTTPS... and I had HTTPS!

In order to track down the problem, I wrote a small service (in Rust :) that just returned the HTTP headers it received and the result was like this:

[other headers]
X-Forwarded-Port: 80
X-Forwarded-Proto: http
[some more headers]

That was strange! The proxy was sending the right X-Forwarded-* headers, but somewhere they were being overwriden!

I searched this behavior and found this anwswer (needs a redhat account to visualize) about changing the openshift router configuration. My solution was a bit different but worked the same.

We need to create a custom router configuration according to the documentation. First let's get a copy of the default configuration:

oc -n default rsh dc/router cat haproxy-config.template > haproxy-config.template

Now open the haproxy-config.template file and search for the following text:

http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto-Version h2 if { ssl_fc_alpn -i h2 }

These lines above override the X-Forwarded-* headers sent by the proxy. My solution was to add a conditional to only set the headers if they are not present:

http-request set-header X-Forwarded-Port %[dst_port]  if { req.hdr_cnt(X-Forwarded-Port) eq 0 }
http-request set-header X-Forwarded-Proto http        if { req.hdr_cnt(X-Forwarded-Proto) eq 0 } !{ ssl_fc }
http-request set-header X-Forwarded-Proto https       if { req.hdr_cnt(X-Forwarded-Proto) eq 0 } { ssl_fc }
http-request set-header X-Forwarded-Proto-Version h2  if { req.hdr_cnt(X-Forwarded-Proto-Version) eq 0 } { ssl_fc_alpn -i h2 }

With the file changed, now we can add it to the router:

# create a configmap to hold the file
oc -n default create configmap customrouter --from-file=haproxy-config.template
# add the configmap as a volume
oc -n default set volume dc/router --add --overwrite \
    --name=config-volume \
    --mount-path=/var/lib/haproxy/conf/custom \
    --source='{"configMap": { "name": "customrouter"}}'
# set an environment variable to tell the router were to find the configuration
oc -n default set env dc/router \
    TEMPLATE_FILE=/var/lib/haproxy/conf/custom/haproxy-config.template

After that, Openshift will redeploy the router and the configuration will take effect.

Solving minishift fresh instance problems

Livio Ribeiro — Sun, 04 Aug 2019 21:21:14 +0000

When I started working with minishift, I did not have much experience with Openshift. I was struggling with some problems and I had no idea what was going on. These were the problems:

Builds failed to push to registry
Deployments got stuck when pulling image
Could not access an application via a Route
Could not log in to the web console as administrator

The first three problems were caused by the "docker-registry" and "router" deployments on the "default" namespace that failed to start.

The solution to these problems is quite simple and involve just a few commands on the terminal:

# login first as admin user
oc login -u system:admin
# restart "docker-registry"
oc -n default rollout latest dc/docker-registry
# restart "router"
oc -n default rollout latest dc/router
# allow "admin" user to log in to the web console
oc adm policy add-cluster-role-to-user cluster-admin admin

Now you can work with minishift with no more problems!

Using LXD and Ansible to simulate infrastructure

Livio Ribeiro — Wed, 05 Dec 2018 01:25:33 +0000

When dealing with software like Kubernetes, Openshift and Rancher, it may be challenging to test an application locally. It is true that we have tools like minikube and minishift, but if we are working with the infrastructure, simulating a cluster with virtual machines can quickly consume all our available RAM.

LXD can help solve the memory problem by using containers as if they were full blown virtual machines! Unlike Docker, which uses containers to run a single process until it finishes, LXD uses containers to spawn an operating system image and use it as a complete server.

To install LXD you can follow the instructions on the official website, but after installing we have to initialize it:

# The default values are good enough
$ sudo lxd init

If you followed the getting started, you may have created some containers using the cli, but now comes the interesting part: We can use Ansible to automate the creation and provisioning of our servers!

To our LXD/Ansible project we are going to create a Nomad cluster with Consul and Traefik.

I chose Nomad because of its simplicity and versatility: it can run services as docker containers, like Kubernetes, but also java applications and any executable available in the host, isolated using the operating system resources (e.g. cgroups, namespaces and chroot on linux). You can see in the documentation what Nomad is able to run.

Consul is used by Nomad for service discovery, key-value storage and to bootstrap the cluster. Nomad can also run without Consul.

Traefik will proxy requests to the services deployed in the cluster. It will use Consul service catalog as the configuration backend, so the routes to services will be automatically configured.

The completed tutorial can be found in this repository.

Index

step 0: Planning
step 1: Consul
step 2: Nomad
step 3: Traefik
step 4: Deploying Services
Conclusion

step 0: Planning

Our cluster will need the following:

3 Consul nodes operating in server mode
- consul1: 10.99.0.101
- consul2: 10.99.0.102
- consul3: 10.99.0.103
3 Nomad nodes operating in server mode
- nomad-server1: 10.99.0.111
- nomad-server2: 10.99.0.112
- nomad-server3: 10.99.0.113
3 Nomad nodes operating in client mode
- nomad-client1: 10.99.0.121
- nomad-client2: 10.99.0.122
- nomad-client3: 10.99.0.123
- The nomad clients will docker and openjdk installed
1 Traefik node
- proxy: 10.99.0.100

All non Consul nodes will have Consul operating in client node.

And for ansible, we will create a project with the following structure:

~/projects/nomad-lxd-ansible
├── cache/
├── inventory/
│   └── hosts
├── roles/
├── ansible.cfg
└── playbook.yml

This structure can be created with the following commands:

mkdir -p ~/projects/nomad-lxd-ansible
cd ~/projects/nomad-lxd-ansible
mkdir cache inventory roles
touch ansible.cfg playboook.yml inventory/hosts

With the cache directory, we will download the Consul, Nomad and Traefik binaries only once in the host. This way, we avoid downloading again in every host.

In ansible.cfg we will tell ansible to use our inventory:

# ansible.cfg
[defaults]
inventory = inventory

Add our cluster nodes to the inventory:

# inventory/hosts
proxy           ip_address=10.99.0.100

[consul_servers]
consul1         ip_address=10.99.0.101
consul2         ip_address=10.99.0.102
consul3         ip_address=10.99.0.103

[nomad_servers]
nomad-server1   ip_address=10.99.0.111
nomad-server2   ip_address=10.99.0.112
nomad-server3   ip_address=10.99.0.113

[nomad_clients]
nomad-client1   ip_address=10.99.0.121
nomad-client2   ip_address=10.99.0.122
nomad-client3   ip_address=10.99.0.123

[all:vars]
ansible_connection=lxd
ansible_python_interpreter=/usr/bin/python3

The ip_address variable will be referenced in the playbook.

And to create the containers, add the following to playbook.yml:

# playbook.yml
---
- hosts: localhost
  # run this task in the host
  connection: local
  tasks:
    - name: create containers
      # get all host names from inventory
      loop: "{{ groups['all'] }}"
      # use lxd_container module from ansible to create containers
      lxd_container:
        # container name is the hostname
        name: "{{ item }}"
        state: started
        source:
          type: image
          mode: pull
          server: https://images.linuxcontainers.org
          alias: ubuntu/bionic/amd64
        config:
          # nomad clients need some privileges to be able to run docker containers
          security.nesting: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
          security.privileged: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
        devices:
          # configure network interface
          eth0:
            type: nic
            nictype: bridged
            parent: lxdbr0
            # get ip address from inventory
            ipv4.address: "{{ hostvars[item].ip_address }}"
        # # uncomment if you installed lxd using snap
        # url: unix:/var/snap/lxd/common/lxd/unix.socket

Now run ansible-playbook playbook.yml to create our nodes with lxd.

step 1: Consul

Let's tell ansible to download and setup Consul.

Edit the playbook to be as follows:

# playbook.yml
---
- hosts: localhost
  # run this task in the host
  connection: local
  # set urls as variables
  vars:
    consul_version: "1.4.0"
    consul_url: "https://releases.hashicorp.com/consul/{{ consul_version }}/consul_{{ consul_version }}_linux_amd64.zip"
  tasks:
    - name: create containers
      # get all host names from inventory
      loop: "{{ groups['all'] }}"
      # use lxd_container module from ansible to create containers
      lxd_container:
        # container name is the hostname
        name: "{{ item }}"
        state: started
        source:
          type: image
          mode: pull
          server: https://images.linuxcontainers.org
          alias: ubuntu/bionic/amd64
        config:
          # nomad clients need some privileges to be able to run docker containers
          security.nesting: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
          security.privileged: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
        devices:
          # configure network interface
          eth0:
            type: nic
            nictype: bridged
            parent: lxdbr0
            # get ip address from inventory
            ipv4.address: "{{ hostvars[item].ip_address }}"
        # # uncomment if you installed lxd using snap
        # url: unix:/var/snap/lxd/common/lxd/unix.socket

    # ensure cache directory exists
    - name: create cache directory
      file:
        path: cache
        state: directory

    - name: fetch applications
      unarchive:
        src: "{{ item.url }}"
        dest: cache
        creates: "cache/{{ item.file }}"
        remote_src: yes
      loop:
        - url: "{{ consul_url }}"
          file: consul

- hosts: consul_servers
  roles:
    - consul_server

The hosts belonging to the consul_servers group will have the role consul_server. We will also create another role called consul_service that will copy the consul binary to the host and setup the service. We split the role this way in order to have a role consul_client that also needs consul binary and service, but with different configuration.

Roles are located under the roles directory, and for the three roles for Consul we will have the following structure:

roles/
├── consul_client
│   ├── tasks
│   │   └── main.yml
│   └── templates
│       └── consul.hcl.j2
├── consul_server
│   ├── tasks
│   │   └── main.yml
│   └── templates
│       └── consul.hcl.j2
└── consul_service
    ├── files
    │   └── consul.service
    └── tasks
        └── main.yml

You can create the structure with the following:

mkdir -p \
  roles/consul_service/tasks \
  roles/consul_service/files \
  roles/consul_server/tasks \
  roles/consul_server/templates \
  roles/consul_client/tasks \
  roles/consul_client/templates \
&& touch \
  roles/consul_service/tasks/main.yml \
  roles/consul_service/files/consul.service \
  roles/consul_server/tasks/main.yml \
  roles/consul_server/templates/consul.hcl.j2 \
  roles/consul_client/tasks/main.yml \
  roles/consul_client/templates/consul.hcl.j2

Role: consul_service

Edit roles/consul_service/tasks/main.yml:

# roles/consul_service/tasks/main.yml
---
- name: install consul
  copy:
    src: cache/consul
    dest: /usr/local/bin/
    mode: 0755

- name: create consul service
  copy:
    src: consul.service
    dest: /etc/systemd/system/

- name: create consul directories
  file:
    path: "{{ item }}"
    state: directory
  loop:
    - /etc/consul.d
    - /var/consul

Edit roles/consul_service/files/consul.service:

# roles/consul_service/files/consul.service
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Restart=on-failure
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Role: consul_server

Edit roles/consul_server/tasks/main.yml:

# roles/consul_server/tasks/main.yml
---
- import_role:
    name: consul_service

- name: copy consul config
  template:
    src: consul.hcl.j2
    dest: /etc/consul.d/consul.hcl

- name: start consul
  service:
    name: consul
    state: restarted
    enabled: yes

Edit roles/consul_server/templates/consul.hcl.j2:

# roles/consul_server/templates/consul.hcl.j2
data_dir = "/var/consul"

server = true
advertise_addr = "{{ ansible_eth0.ipv4.address }}"

client_addr = "127.0.0.1 {{ ansible_eth0.ipv4.address }}"
enable_script_checks = true

{% if ansible_hostname == 'consul1' -%}
ui = true
bootstrap_expect = 3
{% else -%}
retry_join = [ "{{ hostvars.consul1.ansible_hostname }}" ]
{% endif %}

Role: consul_client

Edit roles/consul_client/tasks/main.yml:

# roles/consul_client/tasks/main.yml
---
- import_role:
    name: consul_service

- name: copy consul config
  template:
    src: consul.hcl.j2
    dest: /etc/consul.d/consul.hcl

- name: start consul
  service:
    name: consul
    state: restarted
    enabled: yes

Edit roles/consul_client/templates/consul.hcl.j2:

# roles/consul_client/templates/consul.hcl.j2
data_dir = "/var/consul"
server = false

advertise_addr = "{{ ansible_eth0.ipv4.address }}"
client_addr = "127.0.0.1 {{ ansible_eth0.ipv4.address }}"
enable_script_checks = true
retry_join = [ "{{ hostvars.consul1.ansible_hostname }}" ]

step 2: Nomad

Nomad setup will be very similar to Consul.

Edit the playbook to include Nomad:

# playbook.yml
---
- hosts: localhost
  # run this task in the host
  connection: local
  # set urls as variables
  vars:
    consul_version: "1.4.0"
    nomad_version: "0.8.6"
    consul_url: "https://releases.hashicorp.com/consul/{{ consul_version }}/consul_{{ consul_version }}_linux_amd64.zip"
    nomad_url: "https://releases.hashicorp.com/nomad/{{ nomad_version }}/nomad_{{ nomad_version }}_linux_amd64.zip"
  tasks:
    - name: create containers
      # get all host names from inventory
      loop: "{{ groups['all'] }}"
      # use lxd_container module from ansible to create containers
      lxd_container:
        # container name is the hostname
        name: "{{ item }}"
        state: started
        source:
          type: image
          mode: pull
          server: https://images.linuxcontainers.org
          alias: ubuntu/bionic/amd64
        config:
          # nomad clients need some privileges to be able to run docker containers
          security.nesting: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
          security.privileged: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
        devices:
          # configure network interface
          eth0:
            type: nic
            nictype: bridged
            parent: lxdbr0
            # get ip address from inventory
            ipv4.address: "{{ hostvars[item].ip_address }}"
        # uncomment if you installed lxd using snap
        url: unix:/var/snap/lxd/common/lxd/unix.socket

    # ensure cache directory exists
    - name: create cache directory
      file: { path: cache, state: directory }

    - name: fetch applications
      unarchive:
        src: "{{ item.url }}"
        dest: cache
        creates: "cache/{{ item.file }}"
        remote_src: yes
      loop:
        - url: "{{ consul_url }}"
          file: consul
        - url: "{{ nomad_url }}"
          file: nomad

- hosts: consul_servers
  roles:
    - consul_server

- hosts: nomad_servers
  roles:
    - consul_client
    - nomad_server

- hosts: nomad_clients
  roles:
    - consul_client
    - nomad_client

Similarly to Consul, we will have the roles nomad_service, nomad_server and nomad_client. But now we have two groups, nomad_servers and nomad_clients, each having its respective role, but both having the consul_client role.

We will also have a similar structure for the nomad roles:

roles/
├── nomad_client
│   ├── tasks
│   │   └── main.yml
│   └── templates
│       └── nomad.hcl.j2
├── nomad_server
│   ├── tasks
│   │   └── main.yml
│   └── templates
│       └── nomad.hcl.j2
└── nomad_service
    ├── files
    │   └── nomad.service
    └── tasks
        └── main.yml

We can create this structure with the following commands:

mkdir -p \
  roles/nomad_service/tasks \
  roles/nomad_service/files \
  roles/nomad_server/tasks \
  roles/nomad_server/templates \
  roles/nomad_client/tasks \
  roles/nomad_client/templates \
&& touch \
  roles/nomad_service/tasks/main.yml \
  roles/nomad_service/files/nomad.service \
  roles/nomad_server/tasks/main.yml \
  roles/nomad_server/templates/nomad.hcl.j2 \
  roles/nomad_client/tasks/main.yml \
  roles/nomad_client/templates/nomad.hcl.j2

Role: nomad_service

Edit roles/nomad_service/tasks/main.yml:

# roles/nomad_service/tasks/main.yml
---
- name: install nomad
  copy:
    src: cache/nomad
    dest: /usr/local/bin/
    mode: 0755

- name: create nomad service
  copy:
    src: nomad.service
    dest: /etc/systemd/system/

- name: create nomad directories
  file:
    path: "{{ item }}"
    state: directory
  loop:
    - /etc/nomad.d
    - /var/nomad

Edit roles/nomad_service/files/nomad.service:

# roles/nomad_service/files/nomad.service
[Unit]
Description="HashiCorp Nomad - Application scheduler"
Documentation=https://www.nomadproject.io/
Requires=network-online.target
After=network.target
ConditionFileNotEmpty=/etc/nomad.d/nomad.hcl

[Service]
Restart=on-failure
ExecStart=/usr/local/bin/nomad agent -config=/etc/nomad.d/nomad.hcl
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Role: nomad_server

Edit roles/nomad_server/tasks/main.yml:

# roles/nomad_server/tasks/main.yml
---
- import_role:
    name: nomad_service

- name: copy nomad config
  template:
    src: nomad.hcl.j2
    dest: /etc/nomad.d/nomad.hcl

- name: start nomad
  service:
    name: nomad
    state: restarted
    enabled: yes

Edit roles/nomad_server/templates/nomad.hcl.j2:

# roles/nomad_server/templates/nomad.hcl.j2
data_dir  = "/var/nomad"

advertise {
  http = "{{ ansible_eth0.ipv4.address }}"
  rpc  = "{{ ansible_eth0.ipv4.address }}"
  serf = "{{ ansible_eth0.ipv4.address }}"
}

server {
  enabled          = true
  bootstrap_expect = 3
  raft_protocol    = 3
}

Role: nomad_client

Edit roles/nomad_client/tasks/main.yml:

# roles/nomad_client/tasks/main.yml
---
- import_role:
    name: nomad_service

- name: update apt cache
  apt:
    update_cache: yes

- name: install docker and openjdk
  apt:
    name: "{{ packages }}"
    state: present
  vars:
    packages:
      - docker.io
      - openjdk-11-jdk-headless

- name: start docker service
  service:
    name: docker
    state: started

- name: copy nomad config
  template:
    src: nomad.hcl.j2
    dest: /etc/nomad.d/nomad.hcl

- name: start nomad
  service:
    name: nomad
    state: restarted
    enabled: yes

Edit roles/nomad_client/templates/nomad.hcl.j2:

data_dir  = "/var/nomad"

bind_addr = "{{ ansible_eth0.ipv4.address }}"

client {
  enabled = true
  network_interface = "eth0"
}

step 3: Traefik

Setting up Traefik will be similar to Consul and Nomad, but a bit simpler: there will be only one role named proxy.

Edit the playbook to include Traefik:

# playbook.yml
---
- hosts: localhost
  # run this task in the host
  connection: local
  # set urls as variables
  vars:
    consul_version: "1.4.0"
    nomad_version: "0.8.6"
    traefik_version: "1.7.5"
    consul_url: "https://releases.hashicorp.com/consul/{{ consul_version }}/consul_{{ consul_version }}_linux_amd64.zip"
    nomad_url: "https://releases.hashicorp.com/nomad/{{ nomad_version }}/nomad_{{ nomad_version }}_linux_amd64.zip"
    traefik_url: "https://github.com/containous/traefik/releases/download/v{{ traefik_version }}/traefik_linux-amd64"
  tasks:
    - name: create containers
      # get all host names from inventory
      loop: "{{ groups['all'] }}"
      # use lxd_container module from ansible to create containers
      lxd_container:
        # container name is the hostname
        name: "{{ item }}"
        state: started
        source:
          type: image
          mode: pull
          server: https://images.linuxcontainers.org
          alias: ubuntu/bionic/amd64
        config:
          # nomad clients need some privileges to be able to run docker containers
          security.nesting: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
          security.privileged: "{{ 'true' if item in ['nomad-client1', 'nomad-client2', 'nomad-client3'] else 'false' }}"
        devices:
          # configure network interface
          eth0:
            type: nic
            nictype: bridged
            parent: lxdbr0
            # get ip address from inventory
            ipv4.address: "{{ hostvars[item].ip_address }}"
        # # uncomment if you installed lxd using snap
        # url: unix:/var/snap/lxd/common/lxd/unix.socket

    # ensure cache directory exists
    - name: create cache directory
      file: { path: cache, state: directory }

    - name: fetch applications
      unarchive:
        src: "{{ item.url }}"
        dest: cache
        creates: "cache/{{ item.file }}"
        remote_src: yes
      loop:
        - url: "{{ consul_url }}"
          file: consul
        - url: "{{ nomad_url }}"
          file: nomad

    - name: fecth traefik
      get_url:
        url: "{{ traefik_url }}"
        dest: cache/traefik
        mode: 0755

- hosts: consul_servers
  roles:
    - consul_server

- hosts: nomad_servers
  roles:
    - consul_client
    - nomad_server

- hosts: nomad_clients
  roles:
    - consul_client
    - nomad_client

- hosts: proxy
  roles:
    - consul_client
    - proxy

The structure needed for the proxy role will be like this:

roles/
└── proxy
    ├── files
    │   └── traefik.service
    ├── tasks
    │   └── main.yml
    └── templates
        └── traefik.toml.j2

You can create the structure with the following:

mkdir -p \
  roles/proxy/tasks \
  roles/proxy/files \
  roles/proxy/templates \
&& touch \
  roles/proxy/tasks/main.yml \
  roles/proxy/files/traefik.service \
  roles/proxy/templates/traefik.toml.j2

Edit roles/proxy/tasks/main.yml:

# roles/proxy/tasks/main.yml
---
- name: install traefik
  copy:
    src: cache/traefik
    dest: /usr/local/bin/
    mode: 0755

- name: create traefik service
  copy:
    src: traefik.service
    dest: /etc/systemd/system/

- name: create traefik config directory
  file:
    path: /etc/traefik
    state: directory

- name: copy traefik config
  template:
    src: traefik.toml.j2
    dest: /etc/traefik/traefik.toml

- name: start traefik
  service:
    name: traefik
    state: restarted
    enabled: yes

Edit roles/proxy/files/traefik.service:

# roles/proxy/files/traefik.service
[Unit]
Description="Traefik Proxy"
Documentation=https://traefik.io
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/traefik/traefik.toml

[Service]
Restart=on-failure
ExecStart=/usr/local/bin/traefik --configfile=/etc/traefik/traefik.toml
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Edit roles/proxy/templates/traefik.toml.j2:

# roles/proxy/templates/traefik.toml.j2
[file]

# Backends
[backends]
  [backends.consul]
    [backends.consul.servers]
    {% for host in groups['consul_servers'] %}
      [backends.consul.servers.{{ host }}]
        url = "http://{{ hostvars[host].ansible_eth0.ipv4.address }}:8500"
    {% endfor %}

  [backends.nomad]
    [backends.nomad.servers]
    {% for host in groups['nomad_servers'] %}
      [backends.nomad.servers.{{ host }}]
        url = "http://{{ hostvars[host].ansible_eth0.ipv4.address }}:4646"
    {% endfor %}

# Frontends
[frontends]
  [frontends.consul]
  backend = "consul"
    [frontends.consul.routes.route1]
    rule = "Host:consul.{{ ansible_eth0.ipv4.address }}.nip.io"

  [frontends.nomad]
  backend = "nomad"
    [frontends.nomad.routes.route1]
    rule = "Host:nomad.{{ ansible_eth0.ipv4.address }}.nip.io"

[consulCatalog]
endpoint = "127.0.0.1:8500"
exposedByDefault = false
domain = "service.{{ ansible_eth0.ipv4.address }}.nip.io"

[api]
dashboard = true
debug = true

Traefik configuration grants access to Consul and Nomad dashboards and configures Consul Catalog backend. This way, services can be automatically discovered and exposed.

We set exposedByDefault = false so only the services marked with a specific tag will be exposed, therefore reducing the risk of accidentally making an internal service public.

step 4: Deploying Services

Now that we have everything in place, we can build our cluster and see it working:

ansible-playbook playbook.yml

If everything went OK, we can now access the Trafik dashboard at http://10.99.0.100:8080:

Not very impressive, since we do not have any service deployed yet, but if we go to the file tab:

There it is! It shows Consul and Nomad!... Still not very impressive, we configured them statically in Traefik configuration. Let's do something more interesting:

Enter any nomad server node:

lxc exec nomad-server1 -- bash

Create the nomad service definition

cat > hello.nomad <<EOF
job "hello-world" {
  datacenters = ["dc1"]

  group "example" {
    count = 3
    task "server" {
      # we will run a docker container
      driver = "docker"

      # resouces required by the task
      resources {
        network {
          # require a random port named "http"
          port "http" {}
        }
      }

      config {
        # docker image to run
        image = "hashicorp/http-echo"
        args = [
          "-listen", ":8080",
          "-text", "hello world",
        ]

        # map the random port to port 8080 on the task
        port_map = {
          http = 8080
        }
      }

      # exposed service
      service {
        # service name, compose the url like 'hello-world.service.myorg.com'
        name = "hello-world"
        # service will bind to this port
        port = "http"
        # tell traefik to expose this service
        tags = ["traefik.enable=true"]
      }
    }
  }
}
EOF

In the service section, tags = ["traefik.enable=true"] is what will tell Traefik to expose the service.

Deploy!

nomad job run hello.nomad

It will output something like this:

root@nomad-server1:~# nomad job run hello.nomad 
==> Monitoring evaluation "be583c44"
    Evaluation triggered by job "hello-world"
    Allocation "a19978c9" created: node "d9d3daa0", group "example"
    Allocation "0e0e0015" created: node "7fdbbd1f", group "example"
    Allocation "690efcc2" created: node "ab36a46e", group "example"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "be583c44" finished with status "complete"

Now, if we go to Traefik dashboard again:

It's there!

We can access the service by opening http://hello-world.service.10.99.0.100.nip.io.

But wait a minute, what is that 10.99.0.100.nip.io url? It is a service that to the ip address you put before it. Pretty handy for testing. Go to [nip.io] for more info.

This service, however, is quite boring, let's deploy something more interesting:

Create the nomad service definition

cat > gitea.nomad <<EOF
job "gitea" {
  datacenters = ["dc1"]

  group "gitea" {
    count = 1

    ephemeral_disk {
      # try to deploy this service on the same node every time
      sticky  = true
      # try to migrate the ephemeral disk if possible 
      migrate = true
      # set the ephemeral disk size to 2GB 
      size    = "2048"
    }

    task "server" {
      driver = "docker"

      config {
        image = "gitea/gitea:1.6"

        port_map = {
          http = 3000
        }

        # with docker driver, it is possible to mount volumes insinde the container from the ephemeral disk
        volumes = [
          "local/gitea-data:/data"
        ]
      }

      resources {
        network {
          port "http" {}
        }
      }

      service {
        name = "gitea"
        port = "http"
        tags = ["traefik.enable=true"]
      }
    }
  }
}
EOF

Plan the job deployment

nomad job plan gitea.nomad

It will output something like this:

+ Job: "gitea"
+ Task Group: "gitea" (1 create)
  + Task: "server" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 gitea.nomad

When running the job with the check-index flag, the job will only be run if the
server side version matches the job modify index returned. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

We went a bit different here. We planned the deployment or, in other words, we validated the deployment config (gitea.nomad) and generated an index number (0 in the case of a new deployment), so we do not risk updating a deployment after another operator did a deployment of his own.

To deploy the service, just follow the instructions nomad gave us:

nomad job run -check-index 0 gitea.nomad

In a few minutes it will appear in the Traefik dashboard and will be accessible at http://gitea.service.10.99.0.100.nip.io:

Conclusion

LXD is a very useful tool to test solutions that would be otherwise impossible or impractical with virtual machines. When combined with Ansible, you can quickly create test environments to evaluate these solutions in a way that is closer to a production environment than a scaled down tool like minikube or minishift (which are still completely valid tools if you are focusing only in the applications deployed in these solutions).

Nomad is a great software. Along with Consul, you have a simple yet very powerful solution to orchestrate your services. It can run docker, rkt and lxc containers, java applications packaged in a .jar file (like a Spring Boot application), and even binaries (like a Rocket application), which can be retrieved by nomad in the job definition and executed using the isolation primitives provided by the operating system. It is not as feature complete as Kubernetes but is a lot easier to operate.

Traefik integrates with a lot of services to provide auto configuration. Its Consul Catalog integration provides an incredible solution to a Nomad cluster.

Going further on Openshift

Livio Ribeiro — Sun, 19 Aug 2018 00:25:17 +0000

In my previous post I gave a brief introduction about what is Openshift (now OKD) and how we could easily deploy Gitea. Now we will focus on building and deploying java applications using the "source2image" (S2I) strategy.

But what is this S2I?

According to the S2I project site:

Source-to-Image (S2I) is a toolkit and workflow for building reproducible Docker images from source code. S2I produces ready-to-run images by injecting source code into a Docker container and letting the container prepare that source code for execution. By creating self-assembling builder images, you can version and control your build environments exactly like you use Docker images to version your runtime environments.

What will happen in practice is that Openshift will take your project's source, combine it with a builder docker image and generate the final runtime image used to run the application.

Preparing the development environment

To run our development environment, we will use minishift to create an openshift cluster. Use the commands below to download and install minishift (if you are using linux):

MINISHIFT_VERSION=1.22.0
MINISHIFT_URL=https://github.com/minishift/minishift/releases/download/v$MINISHIFT_VERSION/minishift-$MINISHIFT_VERSION-linux-amd64.tgz

curl -L -s -o /tmp/minishift.tgz $MINISHIFT_URL

tar -C /tmp --wildcards -x */minishift --strip 1 -zf /tmp/minishift.tgz
sudo install /tmp/minishift /usr/local/bin/minishift

With minishift installed, we can configure and start it (which may take several minutes):

# change values according to your workstation
MINISHIFT_VM_DRIVER=virtualbox
MINISHIFT_MEMORY=8G
MINISHIFT_CPUS=4
MINISHIFT_DISK_SIZE=100G

# configure minishift
minishift config set vm-driver $MINISHIFT_VM_DRIVER
minishift config set memory $MINISHIFT_MEMORY
minishift config set cpus $MINISHIFT_CPUS
minishift config set disk-size $MINISHIFT_DISK_SIZE

minishift start

To interact with Openshift, we need to download the oc command line tool, but with minishift we can use the bundled one, just execute the following:

eval $(minishift oc-env)

Then we login to openshift:

oc login 192.168.99.100:8443 -u admin
# use 'admin' as password

Deploying the application

Now that we have minishift up and running, we can start deploying some application. Let's start with a django example application.

But first, let's create the project:

oc new-project django-example --display-name 'Django Example'

It will print something like this

Now using project "django-example" on server "https://192.168.99.100:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.

(Do not worry with the message about a ruby application, we are actually deploying a python application ;)

A project is the openshift terminology to a namespace, a place to group related resources and define access rules to these resources. For example, we could allow user "snake" to view our "django-example" project, user "rattlesnake" to view and edit (trigger builds and deploys, etc.) and not give any access to user "badger".

With the project created, we can create the application:

oc new-app python:3.6~https://github.com/sclorg/django-ex.git

You might be asking "what is going on?". Well, we are telling openshift to take the python:3.6 image and combine it with the source code available at https://github.com/sclorg/django-ex.git. The syntax is "image"~"source".

But be careful! The python:3.6 is not the official python image from docker hub! It is instead from centos/python-36-centos7 and was imported to the openshift integrated registry! It is an S2I image to build and run python projects! You can view other images using the command oc get imagestream -n openshift

The output of the oc new-app will have some information about the application, with the following at the end:

--> Success
    Build scheduled, use 'oc logs -f bc/django-ex' to track its progress.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/django-ex' 
    Run 'oc status' to view your app.

The suggested command oc expose svc/django-ex will do something very important: it will expose a Service (a Kubernetes Service) to external traffic through a Route. The Route is very similar to a Service, but will handle external requests that matches a specific hostname and can also handle encryption (https).

Let's expose our application:

oc expose svc/django-ex

Unlike previous commands, exposing a service does not produce much output, so let's list the available routes to get the one we just created:

oc get routes

Which will output:

NAME        HOST/PORT                                        PATH      SERVICES    PORT       TERMINATION   WILDCARD
django-ex   django-ex-django-example.192.168.99.100.nip.io             django-ex   8080-tcp                 None

As we can see, the route listens for the hostname django-ex-django-example.192.168.99.100.nip.io. This nip.io is a special service service that will redirect requests to the ip address contained in the hostname itself, is this case, 192.168.99.100, the ip address of the minishift vm.

Accessing the route address will show the following screen:

Openshift web console

Openshift also has a very nice web console accessible at [https://192.168.99.100:8443/console]. Just login with the same credentials we used before ("admin" for both username and password).

Opening the link to our project on the right will take us to the following page:

It is the project dashboard where we have an overview of the project, the applications deployed, routes, container status, etc. We can even scale the container!

The web console has lots of featues, even though some actions still require using the cli, like creating a service by exposing the Deployment.

This is it for now. Next time we will get into the web console.

My journey through Openshift

Livio Ribeiro — Thu, 19 Jul 2018 02:20:41 +0000

Recently I started working with Openshift since the company I work for adopted it. It will be quite an interesting journey because I will have to adapt our services that currently run on JBoss and Tomcat to run on containers.

But first, what exactly is Openshift?

From my understanding, it is a distribution of Kubernetes focused on providing an easier way to migrate, deploy and manage your applications on a containerized environment. It hides some of the complexities of Kubernetes while providing some nice features like its dashboard and the source-to-image (S2I) strategy.

Openshift is maintained by RedHat.

The S2I is very interesting: it combines a specially built docker image with your application's source code and build another docker image ready to be deployed. Each S2I image is specialized in building and deploying a different technology stack. For example, the JBoss S2I image will look for a pom.xml in the sources, run mvn package, copy the generated artifact to the deployments directory and, when run, will configure the datasource using environment variables. The Tomcat S2I image works similarly. RedHat provides images for Java, Python, Ruby, Nodejs and others.

You can also deploy an image from docker hub.

Getting started

To get started with Openshift, you can download the minishift tool, which will start a virtual machine running an instance of Openshift.

But before we begin, we should tweak the virtual machine a little bit:

minishift config set vm-driver virtualbox
minishift config set memory 8G
minishift config set cpus 4
minishift config set disk-size 100G

These commands will tell minishift to:

use virtualbox instead of KVM/Hyper-V/xhyve (you can still use those)
set the memory to 8GB (but 4GB will do too)
set the number of cpus to 4, so we can run a few more containers
use a disk size of 100GB, so we do not run out of storage

You can use lower values for memory and cpus, but it will be very limited (and sometimes openshift may freeze).

To start minishift we can run:

minishift start

When the machine is ready, we can run:

eval $(minishift oc-env)

to have oc (the openshift command line tool) in our path.

Before we can interact with Openshift, we need to login (use admin/admin as credencials):

oc login

With everything running, it is time to deploy an application. For this example we will deploy Gitea, a git repository manager forked from Gogs.

But first, we need to create a project (which is effectively a kubernetes namespace):

oc new-project scm --display-name "Source Control Management"

Now we tell openshift to create an "app" from Gitea docker image:

oc new-app gitea/gitea:1.5

which will end up creating:

an Image Stream
a DeploymentConfig, similar to (and will eventually be replaced by) a kubernetes Deployment
a Service

To expose the service to the external world, we can create a Route

oc expose svc/gitea \
    --name gitea-http \
    --port 3000-tcp

The 3000-tcp is the name of the port on the service.

With the Route created, Gitea will now be available at http://gitea-scm.192.168.99.100.nip.io

You can also expose Gitea's ssh endpoint

oc expose svc/gitea \
    --name gitea-ssh \
    --port 22-tcp \
    --hostname gitea-scm-ssh.192.168.99.100.nip.io

Gitea is working, but its data will be lost on container restart, so we need a persistent volume to hold that data:

# remove automatically create volume
oc volume dc/gitea --remove --name gitea-volume-1

oc volume dc/gitea --add \
    --name 'gitea-volume-data' \
    --type 'pvc' \
    --mount-path '/data/' \
    --claim-name 'gitea-data' \
    --claim-size '1G'

One thing that is missing is configuration. Fortunately, Gitea can be configured using environment variables. So now we will create a ConfigMap to hold the environment:

oc create configmap gitea-config \
    --from-literal APP_NAME="My Gitea Instance" \
    --from-literal RUN_MODE=prod \
    --from-literal SSH_DOMAIN=gitea-scm-ssh.192.168.99.100.nip.io \
    --from-literal ROOT_URL=http://gitea-scm.192.168.99.100.nip.io \
    --from-literal DB_TYPE=sqlite3 \
    --from-literal INSTALL_LOCK=true

Now add the environment to the DeploymentConfig:

oc set env --from configmap/gitea-config dc/gitea

If you access our Gitea instance right now, you will notice that our configuration was not applied! This happened because Gitea already have a configuration file in the volume we created. To solve this, we need to add another volume, an EmptyDir volume, so Gitea can rebuild the configuration using the environment variables we set:

oc volume dc/gitea --add \
    --name 'gitea-volume-config' \
    --type 'emptyDir' \
    --mount-path '/data/gitea/conf'

It may take a bit longer for the application to be ready since it is creating the database tables, but then you will see that our settings were applied.

Now we just need to setup probes to monitor application health:

oc set probe dc/gitea --liveness \
    --failure-threshold 3 \
    --initial-delay-seconds 5 \
    -- echo ok

oc set probe dc/gitea --readiness \
    --failure-threshold 3 \
    --initial-delay-seconds 10 \
    --get-url=http://:3000/user/login

The readiness probe is particularly useful during rollouts so openshift can tell if it can safely remove the old container and redirect requests to the new one.

Update:

I made a Template to make easier to deploy Gitea. It is available here.

Gitea has more settings than the ones configurable through the environment variables, but right now setting the config file (app.ini) from a configmap does not work very well since Gitea tries to write to the file and fails because it is readonly. However, there is a pull request to fix that.