Forem: Tarek N. Elsamni

Introducing 'terraform-state-split': A New Tool for Reorganizing Terraform States

Tarek N. Elsamni — Thu, 06 Jul 2023 09:07:14 +0000

As many Terraform users know, managing and organizing resources across different states can be a complex task. The conventional methods often require substantial manual effort and technical expertise. Today, I'm thrilled to share a tool that I've developed to simplify this process - the 'terraform-state-split' CLI.

The 'terraform-state-split' CLI is an intuitive, interactive tool designed to facilitate the transition of Terraform resources between different states. This tool not only helps to reduce the size of your Terraform states, but it also enables you to divide your resources into multiple state files or reorganize your Terraform states as per your requirements.

A Real-life Success Story: From 1400 Resources to Manageable Chunks

The real power of 'terraform-state-split' is highlighted through its practical application. I had been grappling with a giant Terraform state that comprised over 1400 resources. Such a massive state was cumbersome to manage, created complexities during updates or modifications and consumed so much time during running plans to do minor changes.

With 'terraform-state-split', I was able to break down this monolithic Terraform state into multiple smaller, more manageable states. This didn't just improve the organization of the resources, but it also made modifications and updates considerably more straightforward and less error-prone. The tool has truly transformed the way I work with Terraform.

DEMO

Here’s How You Can Get Started

Ready to give 'terraform-state-split' a try? Here's how you can get it installed:

First, tap into the shebang-labs tap via Homebrew with the following command:

brew tap shebang-labs/tap

Next, install the 'terraform-state-split' CLI:

brew install terraform-state-split

And that's it! You are all set to start leveraging the power of 'terraform-state-split' CLI to better manage and reorganize your Terraform states.

The source code for this CLI is publicly accessible.

You can find it at: https://github.com/shebang-labs/terraform-state-split.

I encourage you to explore it, play around with it, and provide your valuable feedback.

In conclusion, 'terraform-state-split' is a tool developed out of necessity, and it embodies the essence of "making things easier." So, whether you're looking to reduce your Terraform states, divide them into multiple state files, or simply reorganize them, give 'terraform-state-split' a try and see the difference for yourself.

Horizontal scaling WebSockets on Kubernetes and Node.js

Tarek N. Elsamni — Tue, 09 Feb 2021 00:04:09 +0000

The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can't be scaled, for example, DaemonSets.

How does the Horizontal Pod Autoscaler work?

The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behavior of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user.

To learn more about how Kubernetes HPA works you can read this detailed article from the official kubernetes.io.

The most common example of the HPA configurations are based on CPU/Memory utilisation metrics provided by metrics-server. In this article I'll give an example of scaling up/down a Kubernetes deployment based on application-specific custom metrics. The application will be a Node.js (Express) server with WebSockets support and the goal will be to scale up/down the deployment based on number of connected clients (connections count).

To achieve this goal, this post will focus on:

Creating a demo app with WebSocket support.
Integrating prometheus-client to expose WebSocket stats as a prometheus metric.
Configuring Prometheus to harvest the exposed metrics.
Setting up prometheus-adapter to convert the prometheus metric to HPA complaint metric.
Configuring HPA to utilise and consume the complaint metric.

Creating a demo app with WebSocket support

The following code will create a demo Express app and integrate WebSocket on /ws/ path.

https://github.com/shebang-labs/websocket-prometheus-hpa-example/blob/main/app.js

Integrating prometheus-client to expose WebSocket stats as a prometheus metric

The following code will integrate a prometheus client and expose a prometheus standard/complaint websockets_connections_total metric on port 9095. Next step is to guide prometheus to start harvesting and collecting this metric and persist the stats over time.

https://github.com/shebang-labs/websocket-prometheus-hpa-example/blob/main/app.js

Configuring Prometheus to harvest the exposed metrics

In this stage, I will use Helm to deploy prometheus on the kubernetes cluster. First, we need to add the helm repo for prometheus using this command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Then we can install prometheus with a persistent volume to store and persist the metrics data over time with the following command:

helm upgrade --install prometheus prometheus-community/prometheus --namespace prometheus --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

At this point we should have the prometheus components up and running perfectly on the kubernetes clsuter on the prometheus namespace as shown in the following:

Prometheus Namespace (Kubernetes)

To guide prometheus to start scraping/collecting the application exposed metric websockets_connections_total over time, we need to annotate the pod which runs the Express app with the following annotations:

prometheus.io/scrape: 'true'
prometheus.io/port: '9095'

So the application deployment would look something like:

https://github.com/shebang-labs/websocket-prometheus-hpa-example/blob/main/deployment.yaml

Setting up prometheus-adapter to convert the prometheus metric to HPA complaint metric

At this stage Prometheus is scraping the metrics every 1 second from port 9095 from all pods in this deployment. To verify this, you can port-forward the prometheus server to localhost and access its query/dashboard UI using the following command:

kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090

which will make the dashboard accessible on localhost:9090. Then you can search for websockets_connections_total to see the scraped metrics over time as shown here:

In this example the query returned 2 graphs as there are 2 pods in this deployment generating different websockets_connections_total values. One of the pods has 1-2 websocket connections overtime and the other has 0 connections.

In the next step we will start using averages (sum of reported connections counts from different pods / pods count) to decide on how scale up and down. But first we need to transform this Prometheus metrics into HPA complaint metric. We can achieve this using prometheus-adapter.

You can install prometheus-adapter as a helm chart. You need to point the adapter to the prometheus instance to query the data from there. Also you will need to tell the adapter how to query the metrics, transform and format it.

This can be done using the following custom helm configurations:

prometheus:
  url: http://prometheus-server.prometheus.svc
  port: 80

rules:
  custom:
    - seriesQuery: '{__name__=~"^myapp_websockets_connections_total$"}'
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: "^(.*)_total"
        as: "${1}_avg"
      metricsQuery: (avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>))

prometheus-adapter-values.yaml

Now, you can use this file to install a custom prometheus-adapter as follows:

helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter --values=./prometheus-adapter-values.yaml --namespace prometheus

To verify that the adapter did work as expected, you should be able to query the HPA custom metrics using the following command:

# I'm using jq for better formatting. You can omit it if needed.
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/myapp-namespace/pods/*/myapp_websockets_connections_avg" | jq .

This should show a result like:

Configuring HPA to utilise and consume the complaint metric

Using the following HPA definition we can control the deployment scaling up and down configs based on the avg websockets connections per pod:

https://github.com/shebang-labs/websocket-prometheus-hpa-example/blob/main/hpa.yaml

In this example, I've configured the min replicas to be 2 and the max to be 10 and then Kubernetes will use the myapp_websockets_connections_avg value over time to align with the target 5 connections per pod and it will scale up and down dynamically to match this target 🎉🎉

Rails Antivirus validator as a service on K8s

Tarek N. Elsamni — Wed, 05 Feb 2020 10:44:06 +0000

Originally posted on https://www.shebanglabs.io/rails-antivirus-clamby-clamav/

Carrierwave, ClamAV and Clamby

If you are building a web application, you definitely will want to enable file uploading. File uploading is an important feature in modern-day applications. Carrierwave is a famous ruby gem that works perfectly with Rack based web applications, such as Ruby on Rails to provide file uploading out of the box with a long list of other features around this speciality.

If you have a file upload on your web application and you do not scan the files for viruses then you not only compromise your software, but also the users of the application and their files.

To avoid such scenarios we tend often to whitelist allowed file extensions and content types. This approach might not be enough if you decided to allow/whitelist executable uploads or if the attacker is uploading a malicious image or any file of an allowed file extension or content-type.

In this tutorial, I will show you how to utilize Rails ActiveModel::Validator class to build a modular validator to scan each file upload in real-time using ClamAV and Clamby gem.

ClamAV® is an open source antivirus engine for detecting trojans, viruses, malware & other malicious threats.

Clamby gem depends on the clamscan daemons to be installed already. If you installed clamscan and tried to run Clamby, you will notice that it takes few seconds (around ~10 depending on available computing resources). This is because every time you run a scan, a new process of clamscan gets initiated to run the scan which takes some time to load the antivirus database, check viruses signatures, run other boating routines and finally start the actual scan.

To overcome this issue. Clamby creator is highly recommending to use the daemonize set to true option. This will allow for clamscan to remain in memory and will not have to load for each virus scan. It will save several seconds per request.

The bad news is a single process of ClamAV is consuming an average of 600-800MB.

For every rails server/pod running you will consume such expensive memory for nothing but preloading the viruses database in memory to deliver real-time antivirus scans!

Fortunately, ClamAV has a TCP/IP socket based interface. Which means we could run a single shared process and access it remotely using TCP/IP sockets. Or even better to run a cluster of distributed processes and loadbalance the virus scans across them. This sounds like a good plan 👌.

Assumptions And Prerequisites

The following part of this post will show you how to deploy ClamAV as a service on K8s, access it from other pods (Rails) over a TCP/IP socket and how to configure Rails to utilize this service in a modular and DRY implementation.

This post makes the following assumptions:

You have basic knowledge of how to build Docker images.
You have a Docker environment running.
You have a Kubernetes cluster running.
Your Ruby on Rails application is containerized and running on - Kubernetes.
You have the kubectl command line (kubectl CLI) installed.

Step 1: Deploy ClamAV as a service on Kubernetes

To deploy ClamAV on Kubernetes, you need to configure a kubernetes deployment and make it accessible through a kubernetes service. The service will expose the deployment using a FDQN DNS that loadbalances the traffic to the deployment replicas without any unfamiliar service discovery mechanisms (which makes the antivirus horizontally scalable).

The kubernetes deployment will look like:

# k8s/clamav-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: clamav
  namespace: shared
spec:
  replicas: 1
  minReadySeconds: 30
  template:
    metadata:
      labels:
        app: clamav
    spec:
      containers:
      - name: clamav
        image: quay.io/ukhomeofficedigital/clamav:v1.7.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3310
          name: api
          protocol: TCP
        livenessProbe:
          exec:
            command:
            - /readyness.sh
          initialDelaySeconds: 20
          timeoutSeconds: 2

The exposing service will look like:

# k8s/clamav-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: antivirus-svc
  namespace: shared
spec:
  selector:
    app: clamav
  clusterIP: None
  ports:
  - name: zombie-port # Actually, we do not use this port but it is still needed to allow the service to receive TCP traffic.
    port: 1234
    targetPort: 1234

Now, you can create the deployment and its exposing service using kubectl as follows:

kubectl apply -f k8s/clamav-deployment.yaml -f k8s/clamav-service.yaml
kubectl -n shared get svc
NAME             TYPE        CLUSTER-IP  EXTERNAL-IP  PORT(S)   AGE
antivirus-svc    ClusterIP   None        <none>       1234/TCP  20s

Step 2: Configure Clamby to use ClamAV service

As shown in the previous step, ClamAV is now up and running as a kubernetes deployment with 1 replica (you could add more replicas to make it horizontal scalable) and listening to port 3310 with protocol TCP. Also, the kubernetes service will make sure that the traffic going to antivirus-svc.shared.svc.cluster.local is being load balanced across the replicas automagically.

To configure Clamby ruby gem to connect to the ClamAV daemon at antivirus-svc.shared.svc.cluster.local using port 3310 and over TCP sockets we need to use the following Rails initializer:

# config/initializers/clamby.rb

clamby_configs = {
  daemonize: true
}

clamby_configs[:config_file] = '/etc/clamav/clamd.conf'

Clamby.configure(clamby_configs)

This initializer is instructing the Clamby gem to use a clamav config file located at /etc/clamav/clamd.conf. This file is not created yet but we will now create it as a part of building the RoR docker image used to run the application.

So, your RoR Dockerfile should look something like:

FROM bitnami/rails:latest

# Install OS dependencies
# COPY Gemfile $APP_PATH/Gemfile
# COPY Gemfile.lock $APP_PATH/Gemfile.lock

# Install bundler
# bundle install

# COPY . $APP_PATH

# Precompile assets

RUN echo "TCPSocket 3310" > /etc/clamav/clamd.conf
RUN echo "TCPAddr antivirus-svc.shared.svc.cluster.local" >> /etc/clamav/clamd.conf

# Entrypoint and CMD

Now, if you run rails c from a container running on the kubernetes cluster and using this Dockerfile image. Then you should be able to run the following command to do ClamAV scans using the remote service over TCP:

# rails c
Loading development environment (Rails 5.2.3)
[1] pry(main)> Clamby.virus?('SOME_LOCAL_FILE_PATH')
ClamAV 0.101.1/25431/Fri Apr 26 08:57:33 2019
/app/SOME_LOCAL_FILE_PATH: OK
false # no virus 🎉

Step 3: An activemodel validator to utilize Clamby

After getting all of the infrastructure in place for running ClamAV as a remote service over TCP and configuring the RoR app to connect to it. It is time to write a modular, DRY and reusable ActiveModel validator that could be used to scan every file the user uploads in real-time.

An ActiveModel validator could look like:

# app/validators/antivirus_validator.rb

class AntivirusValidator < ActiveModel::Validator
  def validate(record)
    if file(record).path && File.exist?(file(record).path) && Clamby.virus?(file(record).path)
      record.errors.add(options[:attribute_name].to_sym, I18n.t('infected_file'))
    end
  end

  private

  def file(record)
    record.public_send(options[:attribute_name].to_sym)
  end
end

Then you could use the validator with the following one line inside any ActiveRecord model:

# app/models/some_model.rb

class SomeModel < ActiveRecord::Base
  mount_uploader :image, PictureUploader
  validates_with AntivirusValidator, attribute_name: 'image'
end

Whenever you need to scan a file uploaded by a mounted uploader in an ActiveModel object, all you need to do is to add the following validation to the model:

validates_with AntivirusValidator, attribute_name: 'image'

Because the ClamAV process is preloaded, up and running already on the remote deployment. and because the deployment is running on the same kubernetes cluster so all traffic goes local. A file scan process takes ~20ms for small files < 1MB and little bit more for bigger files. Do not hesitate to scan every single file uploaded by the end users as the process is not expensive and everything is now in-place to do scans with an extra one line of code.

Happy virus 🦠 scanning 👋