Forem: Justine Devasia

Building Production-Ready Nomad Clusters on AWS with Terraform

Justine Devasia — Fri, 11 Jul 2025 22:23:21 +0000

Setting up a proper production Nomad cluster on AWS involves significant infrastructure complexity. After implementing this setup across multiple projects, I've created a reusable Terraform infrastructure for teams with existing AWS and infrastructure automation experience.

Prerequisites: This requires solid experience with AWS, Terraform, and preferably some Nomad knowledge. The infrastructure is designed for teams who understand these tools but want to avoid rebuilding service discovery and cluster management from scratch.

What's Included

This infrastructure provides a complete AWS setup for running Nomad clusters:

Multi-AZ VPC with proper subnet design
Consul cluster for service discovery and configuration
Nomad servers with auto-scaling groups
Specialized client pools for different workload types
Application Load Balancers with SSL termination
Security groups following least-privilege principles
S3 + CloudFront for static asset delivery
CI/CD pipeline configurations

Why Choose Nomad?

Kubernetes excels with dedicated platform engineering teams, but Nomad offers a simpler alternative for smaller teams or when operational complexity needs to be minimized. Nomad provides straightforward container orchestration with a significantly reduced learning curve.

The job specification syntax is minimal and readable:

job "my-app" {
  datacenters = ["dc1"]

  group "web" {
    count = 3

    task "app" {
      driver = "docker"
      config {
        image = "my-app:latest"
        ports = ["http"]
      }
    }
  }
}

This approach eliminates the complexity of services, ingresses, config maps, and other Kubernetes abstractions while maintaining production-grade orchestration capabilities.

Deployment Process

Getting this infrastructure running involves several steps:

Build custom AMI using the included Packer configuration
Configure remote state with the provided Terraform backend setup
Deploy infrastructure after updating variables for your environment
Deploy applications using Nomad job specifications

The infrastructure implements security best practices:

No hardcoded secrets
Least-privilege IAM roles
Private subnets for workloads
Configurable CIDR blocks

Important: This isn't a one-click deployment. You'll need to understand the provisioning scripts, adjust networking configurations, and modify instance sizes for your specific requirements.

Specialized Node Pools

The infrastructure creates different node pools optimized for specific workloads:

Pool Type	Purpose	Constraints
Django	Python web applications	`node.class = "django"`
Elixir	Phoenix applications	`node.class = "elixir"`
Celery	Background job processing	`node.class = "celery"`
RabbitMQ	Message queue services	`node.class = "rabbit"`
Datastore	Database workloads	`node.class = "datastore"`
APM	Monitoring tools	`node.class = "apm"`

Nomad's constraint system automatically places jobs on appropriate nodes.

Multi-Environment Support

The configuration supports different environments with varying security profiles:

Development: More permissive settings for easier testing
Staging: Production-like with additional debugging capabilities
Production: Locked-down security with comprehensive monitoring

Implementation Considerations

Most infrastructure examples online fall into two categories: simplified demos that fail in production environments, or enterprise-grade solutions requiring dedicated platform teams. This infrastructure targets the middle ground - production-ready without excessive complexity.

The implementation has proven reliable across multiple projects, significantly reducing time spent on service discovery configuration and cluster bootstrapping.

However, this represents opinionated infrastructure decisions based on specific use cases. Production deployments will likely require modifications for different:

Instance types and sizing
Networking requirements
Compliance standards
Organizational policies

The codebase serves as a foundation for teams with the infrastructure expertise to adapt it appropriately.

Getting Started

The complete infrastructure is available as open source:

🔗 AWS Nomad Terraform Infrastructure

Quick Start

# 1. Build custom AMI
cd packer/
packer build ami.pkr.hcl

# 2. Setup remote state
cd tf-remote-state/dev/
terraform init && terraform apply

# 3. Deploy infrastructure  
cd tf-infra/
terraform init -backend-config="backend_develop.conf"
terraform apply -var-file="develop.tfvars"

The solution is suitable for teams experienced with Kubernetes complexity who want to evaluate Nomad, or those already familiar with HashiCorp tooling. The README provides deployment instructions, though understanding the underlying Terraform modules is recommended before production use.

Final Notes

This infrastructure is provided as-is for teams comfortable with the required technology stack. The code is meant to be read, understood, and modified for your specific environment rather than used as a black box.

Bug reports and improvements via pull requests are welcome.

What has been your experience with container orchestration platforms in production environments? How do you evaluate trade-offs between operational complexity and feature completeness?

How to Deploy Hashicorp Nomad Cluster on Vultr

Justine Devasia — Tue, 18 Jun 2024 07:43:30 +0000

Introduction

Hashicorp Nomad is a workload orchestrator to deploy and manage applications across large clusters of servers. Nomad is a single binary application which can run in both server and client modes. The server manages the cluster state and applications are deployed in the client machine.

Nomad support running various type of deployments using docker, binary files, java jar files, and Linux VMs using QEMU driver.

This article demonstrates step-by-step process to build a Nomad cluster on Vultr.

Prerequisites

Before you begin, you must have basic knowledge of Linux and Vultr services.

Example Nomad Cluster

The cluster consists of one Nomad server operating in a VM and three Nomad clients, each running on separate VMs, all forming a single compute plane. After deployment, the server will be accessible in the public IP on port 4646. To access client application, a load balancer will be attached to all clients on port 80. The entire cluster is located inside a VPC with a private IP range.

Set Up VPC 2.0

Open the Vultr Customer Portal
Navigate to Network and click VPC 2.0 on the main navigation menu.
Choose Add VPC 2.0 Network
Select a location.
Configure an IP range in the VPC, for example, 10.1.0.0/20.
Name the VPC 2.0 and click Add Network

This creates a vultr VPC 2.0 where the Nomad cluster will be deployed.

How to use Startup Script to Install Nomad and Docker

The bash script given below install Hashicorp Nomad and Docker in a Ubuntu VM.
A startup Script in Vultr allows the script to reuse by multiple VMs.

    #!/usr/bin/env bash
    set -e

    # Disable interactive apt prompts
    export DEBIAN_FRONTEND=noninteractive

    NOMAD_VERSION="${NOMAD_VERSION:-1.7.3}"

    # Update packages
    sudo apt-get -y update
    # Install software-properties-common
    sudo apt-get install -y software-properties-common

    # Add HashiCorp GPG key
    curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
    # Add HashiCorp repository
    sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
    # Update packages again
    sudo apt-get -y update

    # Install Nomad
    sudo apt-get install -y nomad="${NOMAD_VERSION}"-1


    # Disable the firewall
    sudo ufw disable || echo "ufw not installed"

    # Install Docker and associated dependencies
    sudo apt-get -y update
    sudo apt-get -y install \
        ca-certificates \
        curl \
        gnupg
    # Add Docker’s official GPG key
    sudo install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    sudo chmod a+r /etc/apt/keyrings/docker.gpg
    echo \
        "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
        "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |
        sudo tee /etc/apt/sources.list.d/docker.list >/dev/null  
    sudo apt-get -y update

    # Install Docker and required packages
    sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

    # Create daemon.json if it doesn't exist
    if [ ! -f /etc/docker/daemon.json ]; then
        sudo touch /etc/docker/daemon.json
    fi

    # Restart Docker
    sudo systemctl restart docker

    # Add the current user to the docker group
    sudo usermod -aG docker ubuntu

Open the Vultr Customer Portal
Navigate to Orchestration and click Scripts on the main navigation menu.
Choose Add Startup Script
Give the script a name and choose Type as Boot
Copy the above given bash script and paste it in the Script location
Click Add Script

This creates a Vultr startup script that can be used to deploy multiple VMs.

Nomad server User Data

Nomad uses a config file to run the server with its required configuration. The default file location is /etc/nomad.d/nomad.hcl. The bash script shown below will add the Nomad server configuration to the config file and run the server as a systemd service.

    #!/usr/bin/env bash
    set -Eeuo pipefail

    # Add Nomad client configuration to /etc/nomad.d/ folder

    cat <<EOF >/etc/nomad.d/nomad.hcl
    datacenter = "dc1"
    data_dir   = "/opt/nomad/data"
    bind_addr = "0.0.0.0"
    log_level = "INFO"

    advertise {
      http = "{{ GetInterfaceIP \"enp8s0\" }}"
      rpc  = "{{ GetInterfaceIP \"enp8s0\" }}"
      serf = "{{ GetInterfaceIP \"enp8s0\" }}"
    }

    server {
      enabled          = true
      bootstrap_expect = "1"
      encrypt          = "z8geXx7U+JPk6u/vlBRDhh81h5W12AXBN+7AUo5eXMI="
      server_join {
        retry_join = ["127.0.0.1"]
      }
    }

    acl {
      enabled = false
    }
    EOF

    sudo systemctl enable --now nomad
    sudo systemctl restart nomad

The Vultr Cloud-Init User-Data is a script that can be used to launch the server when it is booted for the first time.
Add the script to the Cloud-Init User-Data during the server launch.

Deploying Nomad server

Open the Vultr Customer Portal
Navigate to Compute on the main navigation menu.
Choose Deploy Server
Select the Type of Server
Select Location same as VPC
Select Ubuntu 22.04 LTS as the Operating System
Select a Plan
In the Additional Features section select Virtual Private Cloud 2.0 and Cloud-Init User-Data
Copy the Nomad server User Data from the previous section and add it in the user data input section
Select the VPC name created in the previous section
In the Server Settings section, choose the nomad-startup-script created.
Add Server Hostname & Label
Click Deploy Now to launch the Nomad server

The Nomad server is now deployed in the VPC.

Fetching the Private IP of Nomad server

To connect Nomad client with server, the private IP of Nomad server is required.

Select VPC 2.0 in the navigation menu and click the VPC ID
Note down the private IP of the Nomad server from the Attached Nodes

Nomad client User Data

The given bash script run a Nomad client as a systemd service along with its configuration during a server boot. Update the private IP of the Nomad server in the server_join block as shown below to make the script connect with the server.

Add the modified script to the Cloud-Init User-Data during the server launch.

    #!/usr/bin/env bash
    set -Eeuo pipefail

    # Add Nomad server configuration to /etc/nomad.d/ folder

    cat <<EOF >/etc/nomad.d/nomad.hcl
    datacenter = "dc1"
    data_dir  = "/opt/nomad/data"
    bind_addr = "0.0.0.0"
    log_level = "INFO"

    # add nomad advertise address
    advertise {
      http = "{{ GetInterfaceIP \"enp8s0\" }}"
      rpc  = "{{ GetInterfaceIP \"enp8s0\" }}"
      serf = "{{ GetInterfaceIP \"enp8s0\" }}"
    }


    # Enable the client
    client {
      enabled = true
      options {
        "driver.raw_exec.enable"    = "1"
        "docker.privileged.enabled" = "true"
      }
      # Configure the server_join option
      server_join {
        retry_join = [ "10.1.0.3" ]
      }

      # Configure the network interface
      network_interface = "enp8s0"

    }

    # Enable the docker plugin
    plugin "docker" {
      config {
        endpoint = "unix:///var/run/docker.sock"

        volumes {
          enabled      = true
          selinuxlabel = "z"
        }
        allow_privileged = true
      }
    }
    EOF

    # Enables nomad systemd service
    sudo systemctl enable --now nomad

    # Runs nomad service
    sudo systemctl restart nomad

Deploying Nomad clients

Open the Vultr Customer Portal
Navigate to Compute on the main navigation menu.
Choose Deploy Server
Select the Type of Server
Select Location same as VPC
Select Ubuntu 22.04 LTS as the Operating System
Select a Plan
In the Additional Features section select Virtual Private Cloud 2.0 and Cloud-Init User-Data
Copy the Nomad client User Data from the previous section and add it in the user data input section
Select the VPC name created in the previous section
In the Server Settings section, choose the nomad-startup-script created.
Increase the Server Qty to 3
Add Server Hostname & Label for each server
Click Deploy Now to launch all 3 Nomad clients

After both the server and clients are deployed in the VPC, a cluster is formed between the server and clients with the help of the server_join configuration in Nomad.

Accessing Nomad UI

Fetch the public IP of Nomad server from Vultr UI and browse http://public-ip:4646. The Nomad UI is accessible on port 4646, as shown in the screenshot below.

Navigate to the Clients and Server page in sidebar to view connected clients and servers.

Nomad server

Nomad clients

Attach Vultr Load Balancer to Nomad Clients

Open the Vultr Customer Portal
Select Load Balancers on the main navigation menu and select Add Load Balancer option.
Select the same location as the VPC 2.0
Use the default Load Balancer Configuration and Forwarding Rules
Select the VPC Network created in the previous section
Choose Add Load Balancer
After the Load Balancer is created, add the nomad clients as targets.

You have attached a Load Balancer to the Nomad clients.

Deploying Sample Web Application

You can deploy web application in a Nomad cluster using Nomad Job files. A Job file must include a .nomad extension written in the HCL (Hashicorp configuration language).

A sample nomad job file is shown below, which deploys a web api on the Nomad cluster.
The api is deployed on port 80 and uses a docker image called traefik/whoami, which returns the client IP address and port number when invoked.

    job "webapp" {
      datacenters = ["dc1"]

      type = "service"

      group "webapp" {
        count = 3

        network {
           mode = "host"
           port "http" {
             to = 80
             static = 80
           }
        }

        task "server" {
          env {
            WHOAMI_PORT_NUMBER = "${NOMAD_PORT_http}"
          }

          driver = "docker"

          config {
            image = "traefik/whoami"
            ports = ["http"]
          }
        }
      }
    }

Open Nomad UI
Navigate to Jobs and select Run Job
Add the above job file in text input and click Plan
Click Run
After the job is in running state, Open the browser and navigate to load balancer IP on port 80, and you can see the response from the application container. Refresh the page to see the load balancer in action.

Conclusion

You have created a Nomad cluster in the VPC and deployed a web application with load balancer. You can now access the Nomad UI and deploy various applications on it.

More Information

For more information, please see:

Docker Log Observability: Analyzing Container Logs in HashiCorp Nomad with Vector, Loki, and Grafana

Justine Devasia — Fri, 19 Apr 2024 17:45:44 +0000

Monitoring application logs is a crucial aspect of the software development and deployment lifecycle. In this post, we'll delve into the process of observing logs generated by Docker container applications operating within HashiCorp Nomad. With the aid of Grafana, Vector, and Loki, we'll explore effective strategies for log analysis and visualization, enhancing visibility and troubleshooting capabilities within your Nomad environment.

Steps:

Install Nomad in Linux
Install Docker
Run Nomad in both server and client mode
Run Loki in Nomad
Deploy Logging app
Deploy Vector in Nomad
Deploy Grafana
Observe logs in Grafana using Loki Datasource

Application structure

Install Nomad locally (Linux)

Nomad, much like Kubernetes, serves as a powerful container orchestration tool, facilitating seamless application deployment and management. In this guide, we'll walk through the installation process on a Linux machine, specifically Ubuntu 22.04 LTS. Let's dive in:

Install required packages

$ sudo apt-get update && sudo apt-get install wget gpg coreutils

Add the HashiCorp GPG key

$ wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

Add official Hasicorp Linux Repository

$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

Update and Install

$ sudo apt-get update && sudo apt-get install nomad

This command will install the latest Nomad binary. The installation can be confirmed by checking the version of nomad by running

$ nomad -v

Install Docker

To deploy containers, docker is essential and can be installed by the following steps. Nomad will detect the docker in a system using docker driver and use it to deploy containers.

Set up Docker's apt repository

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

Install Docker

$ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Restart docker and update permission

$ sudo systemctl restart docker
$ sudo usermod -aG docker ubuntu

Run Nomad in both server and client mode

Nomad binary can run in both server and client mode. The server manages the state of the cluster and clients are used to deploy applications. In production, multiple Nomad servers and clients running in separate machines are interconnected to form a cluster. Here, we will use a single machine to run both server and client.

Nomad require a config file to run the application in server and client mode. The config file is also required to add more capabilities to nomad such as adding docker driver, setting telemetry, autopilot and so on.

Here is the nomad configuration file which can be used to run nomad. It also contains the information required to run docker containers. Create a file nomad.hcl and copy the below content to the file.

datacenter = "dc1"
data_dir  = "/opt/nomad/data"
bind_addr = "0.0.0.0"
log_level = "INFO"

server {
  enabled          = true
  bootstrap_expect = 1
   search {
    fuzzy_enabled   = true
    limit_query     = 200
    limit_results   = 1000
    min_term_length = 5
  }
}

# Enable the client
client {
  enabled = true
  options {
    "driver.raw_exec.enable"    = "1"
    "docker.privileged.enabled" = "true"
  }

  server_join {
    retry_join = [ "127.0.0.1" ]
  }

}

plugin "docker" {
  config {
    endpoint = "unix:///var/run/docker.sock"

    extra_labels = ["job_name", "job_id", "task_group_name", "task_name", "namespace", "node_name", "node_id"]

    volumes {
      enabled      = true
      selinuxlabel = "z"
    }

    allow_privileged = true

  }
}

telemetry {
  collection_interval = "15s"
  disable_hostname = true
  prometheus_metrics = true
  publish_allocation_metrics = true
  publish_node_metrics = true
}

Run the following command in the terminal where the above file is located to start Nomad.

$ sudo nomad agent -config=nomad.hcl

This will start the application.

==> Loaded configuration from nomad.hcl
==> Starting Nomad agent...

==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.1.32:4646; RPC: 192.168.1.32:4647; Serf: 192.168.1.32:4648
            Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
                Client: true
             Log Level: INFO
               Node Id: 2921dae9-99dc-a65d-1a1f-25d9822c1500
                Region: global (DC: dc1)
                Server: true
               Version: 1.7.7

==> Nomad agent started! Log data will stream in below:

The Nomad UI will be available in the advertise address. Here it is, 192.168.1.32:4646, This will be based on the network interface you are connected and the value will be different for different machines. Going to this IP and port in web browser will open nomad UI as shown below.

You have installed both nomad and docker in the system and ready to deploy the observability applications!

Run Loki in Nomad

Loki is a scalable log aggregation system and is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. It is configured such that the logs are persisted to the file system. It can also be configured to store the data in various datastores like AWS S3, Minio and so on.

The nomad job file used to run loki is given below.

job "loki" {
  datacenters = ["dc1"]
  type        = "service"

  group "loki" {
    count = 1

    network {
      mode = "host"
      port "loki" {
        to = 3100
        static = 3100
      }
    }

    service {
      name = "loki"
      port = "loki"
      provider = "nomad"
    }

    task "loki" {
      driver = "docker"
      user = "root"
      config {
        image = "grafana/loki:2.9.7"
        args = [
          "-config.file",
          "local/config.yml",
        ]
        volumes = ["/loki_data:/loki"]
        ports = ["loki"]
      }
      template {
        data=<<EOH
auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  # Any chunk not receiving new logs in this time will be flushed
  chunk_idle_period: 1h
  # All chunks will be flushed when they hit this age, default is 1h
  max_chunk_age: 1h
  # Loki will attempt to build chunks up to 1.5MB, flushing if chunk_idle_period or max_chunk_age is reached first
  chunk_target_size: 1048576
  # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  chunk_retain_period: 30s
  max_transfer_retries: 0     # Chunk transfers disabled
  wal:
    enabled: true
    dir: "/loki/wal"
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
EOH
        destination = "local/config.yml"
        change_mode = "restart"
      }
      resources {
        cpu    = 1000 #Mhz
        memory = 1000 #MB
      }
    }
  }
}

Copy the above job configuration and add it in the nomad UI.

Deploy Logging app

The logging app is a simple docker container which emits logs randomly. It send 4 log levels, INFO, ERROR, WARNING and DEBUG to stdout. The job file is give below.

job "logger" {
  datacenters = ["dc1"]

  type = "service"

  group "logger" {
    count = 1

    task "logger" {

      driver = "docker"

      config {
        image = "chentex/random-logger:latest"
      }

      resources {
        cpu    = 100 # 100 MHz
        memory = 100 # 100MB
      }


    }
  }
}

Copy the above configuration and deploy another job from Nomad UI.

Deploy Vector in Nomad

Vector is log collecting agent which support many source and destination for log ingestion and export. Here, vector will collect the log data using docker_logs source and send to loki. The job file is given below.

job "vector" {
  datacenters = ["dc1"]
  # system job, runs on all nodes
  type = "system"

  group "vector" {
    count = 1

    network {
      port "api" {
        to = 8686
      }
    }

    ephemeral_disk {
      size    = 500
      sticky  = true
    }

    task "vector" {
      driver = "docker"
      config {
        image = "timberio/vector:0.30.0-debian"
        ports = ["api"]
        volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
      }
      env {
        VECTOR_CONFIG = "local/vector.toml"
        VECTOR_REQUIRE_HEALTHY = "false"
      }
      resources {
        cpu    = 100 # 100 MHz
        memory = 100 # 100MB
      }
      # template with Vector's configuration
      template {
        destination = "local/vector.toml"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }}
        left_delimiter = "[["
        right_delimiter = "]]"
        data=<<EOH
          data_dir = "alloc/data/vector/"
          [api]
            enabled = true
            address = "0.0.0.0:8686"
            playground = true
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
            target = "stdout"
          [sinks.loki]
            type = "loki"
            compression = "snappy"
            encoding.codec = "json"
            inputs = ["logs"] 
            endpoint = "http://[[ range nomadService "loki" ]][[.Address]]:[[.Port]][[ end ]]"
            healthcheck.enabled = true
            out_of_order_action = "drop"
            # remove fields that have been converted to labels to avoid having the field twice
            remove_label_fields = true
              [sinks.loki.labels]
              # See https://vector.dev/docs/reference/vrl/expressions/#path-example-nested-path
              job = "{{label.\"com.hashicorp.nomad.job_name\" }}"
              task = "{{label.\"com.hashicorp.nomad.task_name\" }}"
              group = "{{label.\"com.hashicorp.nomad.task_group_name\" }}"
              namespace = "{{label.\"com.hashicorp.nomad.namespace\" }}"
              node = "{{label.\"com.hashicorp.nomad.node_name\" }}"
        EOH
      }
      kill_timeout = "30s"
    }
  }
}

Copy the above configuration and deploy the job from Nomad UI.Once vector is deployed, it will start collecting the logs from docker logs and send it to Loki.

Deploy Grafana

Grafana is a popular tool to visualize logs, metrics and traces. Here we will use grafana and its Loki Data source connector to view and explore log data sent by docker applications running in Nomad.
The grafana nomad job file is given below.

job "grafana" {
  datacenters = ["dc1"]
  type        = "service"


  group "grafana" {
    count = 1

    network {
      mode = "host"

      port "grafana" {
        to = 3000
        static = 3000
      }
    }

    task "grafana" {
      driver = "docker"

      env {
        GF_LOG_LEVEL          = "ERROR"
        GF_LOG_MODE           = "console"
        GF_PATHS_DATA         = "/var/lib/grafana"
      }

      user = "root"

      config {
        image = "grafana/grafana:10.4.2"
        ports = ["grafana"]
        volumes = ["/grafana_volume:/var/lib/grafana"]
      }

      resources {
        cpu    = 2000
        memory = 2000
      }

    }
  }

}

Deploy this job using the Nomad UI as described in previous section.

After deploying all the jobs, the Nomad UI looks like this.

Connecting to Loki Data source in Grafana

After confirming all jobs are running as expected, open the grafana UI on :3000. For me the address is my host machine's private IP which is http://192.168.1.32:3000. Login to the Dashboard, using default username admin and password admin.

Select the Data sources option, press Add Data sources

Search for loki and select it

Update the connection URL of loki. Based on the loki nomad job file, it is listening on static port 3100. The connection url will be like this, http://(machine-ip):3100.

keep other options default and click on Save & Test
If the connection is successful, it shows Data source successfully connected.

Visualising Loki Logs in Grafana

Select the Explore option in Grafana

Select Loki
Select the Label jobs and choose logger Nomad job.

Click Run Query to view the logs

You have successfully installed Nomad, Docker, and run applications on top of it, and queried the logs emitted by that application in Grafana.

References

Running Docker based web applications in Hashicorp Nomad with Traefik Load balancing

Justine Devasia — Fri, 15 Mar 2024 09:19:37 +0000

In previous post, we discussed creating a basic Nomad cluster in the Vultr cloud. Here, we will use the cluster created to deploy a load-balanced sample web app using the service discovery capability of Nomad and its native integration with the Traefik load balancer. The source code is available here for the reference.

Traefik acts as an API gateway for your online services, ensuring that incoming requests are properly routed to the appropriate parts of your system. It can perform host-based or path-based routing, allowing the server to run a wide variety of services with varied domain names.

What makes traefik stand out is its ability to automatically detect and configure services without any extra effort on your part. It's like having a helpful assistant that knows exactly where everything belongs. This information is obtained through connection to Nomad services, a process we will undertake in this article.

To run the service, all we need is a Nomad job file and a nomad cluster. A nomad job file is configuration file which contains information about the application to run, its networking and the service discovery information.

This is how the job file looks like:

job "traefik" {
  datacenters = ["dc1"]
  type        = "system"

  group "traefik" {

    network {
      mode = "host"
      port  "http"{
         static = 80
      }
      port  "admin"{
         static = 8080
      }
    }

    task "server" {
      driver = "docker"
      config {
        image = "traefik:2.11"
        ports = ["admin", "http"]
        args = [
          "--api.dashboard=true",
          "--api.insecure=true", # not for production
          "--entrypoints.web.address=:${NOMAD_PORT_http}",
          "--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
          "--providers.nomad=true",
          "--providers.nomad.endpoint.address=http://<nomad server ip>:4646" 
        ]
      }

      resources {
        cpu    = 100 # Mhz
        memory = 100 # MB
      }
    }

    service {
      name = "traefik-http"
      provider = "nomad"
      port = "http"
    }
  }
}

The main components of a Job file are the job, group and task stanzas. A job file can have multiple groups, and each group can have multiple tasks. The group contains networking information like the type of network, ports exposed and the service info. If you are interested in nomad networking, this article is a great source of information. The task can run docker containers using a driver and configs

To successfully route traffic, the traefik proxy needs to have information about the IP address and port of applications running in the cluster. The arguments in the configuration contain information about how traefik will obtain details about applications running in Nomad. Nomad offers a native service discovery option, and in this case, traefik takes advantage of this service discovery information to retrieve application details. While running the job, it is important to modify the endpoint address to the Nomad server address, here: providers.nomad.endpoint.address=http://<nomad server ip>:4646. More configuration options are available in the traefik documentation.

Running Traefik Job file in the cluster

Let us try to run the job file in a nomad cluster. In my previous blog, I have created a nomad cluster in vultr cloud and will use the same cluster to deploy this job file. The cluster consists of 1 server and 3 clients, both traefik and sample web app will be deployed in the cluster and we will observe traefik distributing traffic effectively with minimal configuration.

There are two ways to run a job, manually deploying the job file from Nomad UI or running it with nomad cli. In this case manually triggering the job from UI is suitable. Before triggering the job, I will update the job file with nomad server endpoint address, here the endpoint is the address of nomad server load balancer we created earlier.

Traefik job file in nomad UI:

Planning:

Job run Result:

Once traefik started running, it will be accessible in port 80 of the vultr load balancer created. The configuration of attaching loadbalancer to the vm is done with the help of terraform.

The 404 error is expected as there is no running services present. Now we are confident that the traefik is running and waiting for redirecting the requests.

Running sample webapp in the cluster

A sample web app is also deployed along with traefik and its function is to return the IP of the machine it is running. This is ideal for testing to verify the proxy is working as expected.The job file for the web app is present here.

job "webapp" {
  datacenters = ["dc1"]

  type = "service"

  group "webapp" {
    count = 1

    network {
       mode = "host"
       port "http" {
         to = 80
       }
    }

    service {
      name = "webapp"
      port = "http"
      provider = "nomad"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.webapp.rule=Path(`/`)",
      ]
    }

    task "server" {
      env {
        WHOAMI_PORT_NUMBER = "${NOMAD_PORT_http}"
      }

      driver = "docker"

      config {
        image = "traefik/whoami"
        ports = ["http"]
      }
    }
  }
}

This is a simple job configuration for the sample web app and the important thing to note is the tags.

Traefik proxy find the service to load balance using this tags and based on the tag information, it can do host based or path based routing. With this approach, the proxy get information about the web app without any modification in its configuration and restarts unlike proxies like nginx. This is a great advantage of using traefik for load balancing.

Deploying the job:

Jobs successfully ran:

Now we have both webapp and traefik running in nomad cluster. Doing a curl on the client load balancer give us back the response from any of these clients with the help of traefik.

When we do curl on the client load balancer, we get back a different remote address each time, indicating the proxy is doing load balancing between the 3 instances of the web app deployed.

In this blog post, we explored the seamless integration of HashiCorp Nomad and Traefik for deploying a load-balanced web application infrastructure in the Vultr cloud. By utilizing Nomad's service discovery and Traefik's dynamic routing capabilities, we demonstrated the straightforward setup and management of a scalable cluster environment. In the upcoming posts, we will deploy more services and dig deeper into the nomad ecosystem.

Building HashiCorp Nomad Cluster in Vultr Cloud using Terraform

Justine Devasia — Mon, 11 Mar 2024 13:31:45 +0000

Nomad is really awesome!

In this blog post, let us see how to build and automate a Nomad cluster using Terraform and the Vultr cloud computing platform. The previous blog post discusses how to create a machine image in Packer. Here we will use the machine image created to deploy a Nomad cluster with both servers and clients. They are attached to Vultr load balancers and protected by firewalls.
The source code is available here

HashiCorp Nomad is a workload orchestrator to deploy applications in the cloud or on-premise. It enables deployment and management of various workloads including containers, jar files, exec jobs etc.. It is highly scalable and can run millions of containers in a single cluster.

Terraform is an infrastructure as code application where cloud infrastructure deployment can be automated and codified.

This Nomad cluster will have a single server and 3 clients. The server manages the state of the cluster and job deployments. The clients are the machines on which actual applications run.

To deploy the infrastructure, we will require Terraform, Vultr cloud, and some shell scripts. All the scripts required to spawn up infra are available here.

Firstly, let us install Terraform using HashiCorp's documentation. Once Terraform is set up, we need to create a Vultr account. Vultr cloud is providing free $250 credits for first-time signup. Generate an API key from the dashboard and keep it safe, as it is required for creating the infrastructure.

Steps to build the cluster

Clone the git repo https://github.com/justinepdevasia/nomad-terraform-vultr-cloud
Open terminal inside the repo and cd terraform
Replace the terraform.tfvars variables with the values you require. The VM will be created from the snapshot created, and the proper snapshot ID needs to be provided in the file. If you haven't done the snapshot creation, please read this article.

The terraform.tfvars file looks like this:

region = "bom"
plan = "vc2-1c-2gb"
snapshot_id = "c31a3a09-8b8b-4b96-a56f-a020606d4cd4"
private_network_label = "nomad-network"
nomad_server_hostname_prefix = "nomad-server"
nomad_client_hostname_prefix = "nomad-client"
lb_server_name = "nomad-servers-lb"
lb_client_name = "nomad-clients-lb"

Run command export VULTR_API_KEY="your-vultr-api-key" - this is to store the API key value in an environment variable. We do not want to expose it in the code.
Run command terraform init
Run command terraform plan -var="vultr_api_key=${VULTR_API_KEY}" to get information about the changes happening to infra when the application is deployed. Here the variable Vultr API key is fetched from the environment variable.

Run command terraform apply -var="vultr_api_key=${VULT R_API_KEY}" to build the infra.

In this process, the Terraform will connect to Vultr cloud and execute the creation of various services like Virtual machines, load balancers, and firewall. The output of the Terraform shows the load balancer IPs.

The Vultr cloud shows both Nomad servers and clients:

We can verify the cluster is active by pasting the IP of the Nomad server in the browser. It will be like this:

Clients and servers:

Using Terraform, we can easily add more clients to the clusters by increasing the count of Nomad clients in the client configuration. On updating the number of clients to 3 and running Terraform apply, I was able to add 2 more clients to the system. More clients, more compute!

Destroying the Cluster

The entire infrastructure can be dismantled by running the command terraform destroy -var="vultr_api_key=${VULT R_API_KEY}"

Terraform provides us a quick and convenient way to build and tear down infrastructures on demand. This enables the creation of multiple identical environments for test, prod, and QA. In the current setup, the Terraform state file is stored in the local machine, but it can also be stored in a remote backend like S3 or Terraform Cloud as well.

source github repo: https://github.com/justinepdevasia/nomad-terraform-vultr-cloud/tree/main

In the upcoming post, we will discuss how to run various workloads in Nomad including stateless and stateful applications.

Accelerating Deployment: Creating HashiCorp Nomad Machine Images on Vultr Cloud via Packer

Justine Devasia — Sun, 03 Mar 2024 06:16:12 +0000

Immutable infrastructure creation in the cloud requires on-demand machine images. Whenever we need to make changes to the system, the old image is torn down and a new one is deployed. Packer is a tool provided by HashiCorp to build machine images easily, which can then be used to deploy cloud VMs. In this post, we will see how we can create a machine image in Vultr Cloud using Packer.

Vultr is a cloud computing platform which enables us to deploy virtual machines and various cloud services like loadbalancers, CDNs etc. . In Vultr, the machine image is called snapshots. We will install Nomad and Docker within the snapshot. Later, when a cluster is created, this snapshot ID can be utilized to swiftly spawn the cluster with all the packages pre-installed.

Firstly, let us install packer using hashicorp documentation. Once packer is setup, we need to create a vultr account. Vultr cloud is providing free $250 credits for first time signup. Generate api key from dashboard and keep it safe, as it is required for creating the snapshot.

Once both packer and vultr is ready, let us use the script and config file. The source code is available here.

Two files are used in creating the image,

setup.sh - contains some basic shell script to install docker and nomad in an ubuntu machine.
ubuntu.pkr.hcl - packer template to build the image

Let us go through the contents of ubuntu.pkr.hcl file.

 variable "vultr_api_key" {
   type      = string
   default   = "${env("VULTR_API_KEY")}"
   sensitive = true
 }

 packer {
   required_plugins {
     vultr = {
       version = ">=v2.3.2"
       source = "github.com/vultr/vultr"
     }
   }
 }

 source "vultr" "ubuntu-nomad" {
   api_key              = "${var.vultr_api_key}"
   os_id                = "2179"
   plan_id              = "vc2-1c-2gb"
   region_id            = "bom"
   snapshot_description = "Ubuntu 23.04 Nomad ${formatdate("YYYY-MM-DD hh:mm", timestamp())}"
   ssh_username         = "root"
   state_timeout        = "25m"
 }

 build {
   sources = ["source.vultr.ubuntu-nomad"]

   provisioner "shell" {
     script = "setup.sh"
   }
 }

The variable vultr_api_key is used to connect with vultr cloud securely. It acquires the value from VULTR_API_KEY env variable.
The above packer stanza download the correct vultr plugin to use in the machine image creation.
The source stanza contains information about the cloud provider, the region in which image required, base os used etc..
The build stanza connects to a source and use a provisioner to build the image. Here it execute the build process by running the setup.sh file provided by us.

Now let us see how to run this setup to build our customized os image.

Clone the git repo https://github.com/justinepdevasia/nomad-terraform-vultr-cloud
open terminal inside the repo and cd packer
run command export VULTR_API_KEY="your-vultr-api-key"
run command packer init .
run command packer build .

This will trigger a packer build session, the application will connect to vultr cloud, spawn a temporary vm and install the packages. Once setup.sh file is fully executed, snapshot creation is triggered and we get back the snapshot id.

This snapshot id can be used to spawn new vms with all the packages, in this case nomad and docker in built.
Finally we can see the created snapshot in Vultr cloud dashboard.

Packer has really enabled the creation of machine images a smooth and enjoyable process. Combining it with the simplicity and ease of use of vultr, we get a powerful system for automation of cloud VM image creation. With this approach we can pre-install any package in a snapshot and spawn the VMs without any additional installation after launch.

In the upcoming article, we will discuss how to use this snapshot to deploy a Nomad cluster using Terraform.

Embracing Simplicity: The Advantages of Nomad over Kubernetes

Justine Devasia — Sat, 16 Dec 2023 13:07:29 +0000

In the rapidly evolving landscape of container orchestration and management, two prominent players have emerged: Kubernetes and HashiCorp's Nomad. While Kubernetes has gained widespread adoption and popularity, Nomad provides a compelling alternative that stands out for its simplicity and efficiency. In this blog post, we'll explore the advantages of using Nomad over Kubernetes and why it might be the right choice for certain use cases.

HashiCorp Nomad stands out as an easy-to-use and flexible cluster scheduler, capable of efficiently running a diverse range of workloads, including micro-services, batch processes, and both containerized and non-containerized applications. This powerful tool seamlessly integrates with HashiCorp Consul for service discovery and HashiCorp Vault for secrets management, enhancing its capabilities. Nomad's architecture is notably simpler than that of Kubernetes, as it operates as a single binary for both clients and servers, eliminating the need for external coordination or storage services. Its distributed, highly available, and operationally simple design has made it a go-to choice for companies looking to deploy containerized workloads efficiently and manage clusters at any scale, all while minimizing costs.

Simplicity

In terms of simplicity, Nomad offers a streamlined architecture compared to Kubernetes. While Kubernetes relies on over half a dozen interoperating services, including etcd for coordination and storage, Nomad stands out as a single binary for both clients and servers, requiring no external services. Nomad combines a lightweight resource manager and scheduler into one system, defaulting to a distributed, highly available, and operationally simple setup. This architectural simplicity makes Nomad an efficient choice for users seeking straightforward container orchestration.

Wide variety of workload support

Unlike Kubernetes, which is tailored for Docker containers, Nomad boasts a broader purpose. Nomad accommodates virtualized, containerized, and standalone applications, encompassing technologies such as Docker, Java, IIS on Windows, Qemu, and more. Nomad's design emphasizes extensible drivers, and ongoing efforts aim to expand support for all prevalent drivers.

Consistency in Deployment

Challenges in achieving consistent deployment often arise with full Kubernetes installations in production due to their time-consuming, operationally complex, and resource-intensive nature. Although the Kubernetes community has introduced streamlined versions like minikube, kubeadm, and k3s to ease development and testing, this approach results in inconsistencies in capabilities, configuration, and management when transitioning to production.

Nomad, in contrast, offers a solution to this issue. As a single lightweight binary, Nomad can be deployed consistently across various environments, including local development, production, on-premises, at the edge, and in the cloud. This uniform deployment approach ensures the same level of operational ease-of-use throughout all these environments, setting Nomad apart from the fragmented distributions of Kubernetes.

Scalability

Kubernetes, according to documentation, supports clusters up to 5,000 nodes and 300,000 total containers. However, as the environment expands, operational complexity increases due to the compounded constraints of interoperating components.

In contrast, Nomad has demonstrated real-world scalability, surpassing 10,000 nodes in production environments. It excels in handling multi-cluster deployments seamlessly across availability zones, regions, and data centers. Nomad's native support for multi-cluster deployments simplifies scaling applications across various data centers, regions, and clouds without introducing additional complexity. Rigorous scalability benchmarks, including the 1 million container challenge in 2016 and the 2 million container challenge in 2020, validate Nomad's architectural design and its ability to perform under the most demanding requirements.

Conclusion

In conclusion, Nomad proves to be a compelling alternative to Kubernetes, especially when prioritizing operational efficiency, cost-effectiveness, and streamlined maintenance. Its versatility extends to non-containerized workloads, making it a flexible choice for diverse application requirements. Notably, Nomad stands out for its user-friendly maintenance, well-suited for smaller teams of infrastructure engineers. With its simplicity and robust capabilities, Nomad offers an attractive solution that aligns with the evolving needs of efficient and cost-conscious deployment environments.