Forem: Alexey Melezhik

Dead Simple CI Introduction

Alexey Melezhik — Sat, 14 Feb 2026 13:11:55 +0000

Dsci is a brand new kid on the cicd area. It allows use of general programming languages to create ci scenarios.

——-

Consider some imaginary case of building and deploying an application as docker container. So, logically we have two main stages:

build docker image from source code
deploy docker image using docker pull and docker run commands

Let’s see how dsci could tackle such a scenario …

——-

Pipeline file

The main concept we start with is so called ‘pipeline file’.

It’s just yaml file containing a list of jobs to be performed. Every job is executed on isolated docker container, execution logic is always sequential - one job, by another.

This pattern yet simple however covers the majority of use cases:

jobs.yaml

jobs:
  -
     id: build
     path: build/
  -
     id: deploy
     path: deploy/

Thus, every element of job list is a job, where job has some unique identifier and path to location of job file.

Conditional job logic

Usually there are some conditions when and what ci scripts are triggered. Those conditions may be based on different criteria, but usually they depend on branch names.

Dsci utilizes a special syntax to define those rules. Let’s say we only want to deploy if source branch is main, and perform build for all branches:

jobs:
  -
     id: build
     path: build/
  -
     id: deploy
     path: deploy/ 
     only: .<ref> eq "refs/heads/main"

Read more on job conditions on dsci documentation - https://github.com/melezhik/DSCI/blob/main/job-cond.md

Passing parameters to job

To pass parameters to job, just use params: key:

jobs:
  -
     id: build
     path: build/
     params:
       foo: bar
       version: 0.1.0
  -
     id: deploy
     path: deploy/

Those parameters could be thought as overrides for default ones, however dsci allows more then that - pass results ( states ) from one job to another, see later.

Job file

The next important building block of the hierarchy is so called “job file” defining main job logic.

Job file needs to be named depending on language of choice, in our example we use Bash as it usually is enough for many use cases. But using other languages is also possible. If one needs more flexibility they may chose Python or Golang

build/task.bash

version=$(config version)
tag_version=$(date +%s).$version
docker build . -t app:$tag_version
docker push repo/app:$tag_version

In this simple example our job file is just a single task job. However if there is a need to split complex job into several tasks they may do so by using job.$ext file approach. This file needs to be named according language of choice.

Let’s say we have three tasks - configure, build and test - incorporated into build job, we may run them one by one like this:

build/job.py

#!/bin/python
run_task(“configure”);
run_task(“build”);
run_task(“test”);

And if we put those tasks under build/tasks/ directory like this:

build/tasks/configure/task.bash
build/tasks/build/tasks.bash
build/tasks/tests/task.bash

In this case we have modular setup of our ci job.

The neat thing about this DSL, dsci provides the same SDK for all supported programming languages.

Read more about jobs and tasks on dsci documentation web site:

Passing results between jobs

Let’s make our last job example more realistic and return docker image tag dynamically created by build job back to deploy job. We may want to do so deploy jobs know what tag to pull before deploy. All we need is to add extra line to save the job state into internal dsci cache:

build/task.bash

version=$(config version)
tag_version=$(date +%s).$version
docker build . -t app:$tag_version
docker push repo/app:$tag_version
update_state tag $tag_version

update_state() is very handy function allowing to pass states between different tasks and jobs. It’s implemented for all supported languages.

To pick up tag name in deploy job we can use already mentioned config() function:

deploy/task.bash

set -e
tag=$(config tag)
docker pull repo/app@$tag
docker stop -t 1 container || :
docker run -rm -name container -td repo/app@$tag

Read more about job and tasks states on dsci documentation ( the links above ^^ )

Secrets

Let’s modify build job example with pushing image to docker registry. To do a push we need to authenticate against a docker registry first. Dsci enables simple way to pass secrets to pipelines.

This time let’s rewrite our job file on Python for convenience:

build/job.py

#!/bin/python
import os
password  = os.environ['password'])

run_task(“build”, { “password”: password })

And then modify build task to handle password parameter:

version=$(config version)
tag_version=$(date +%s).$version
docker build . -t app:$tag_version
update_state tag $tag_version

echo $password | docker login --username your-username --password-stdin

docker push repo/app:$tag_version

As we can see secrets passed into pipelines as environment variables.

And unlike other task and jobs parameters they never saved in job reports or cache files.

—-

Deployment on localhost

By default dsci pipelines run inside some docker container, this fits situation when one needs run purely ci code - for example build and run some unit tests. CD part comes to play when build artifacts are ready for deploy.

Dsci allows switch to deployment environment by using localhost switcher:

jobs:
  -
     id: build
     path: build/
     params:
       foo: bar
       version: 0.1.0
  -
     id: deploy
     path: deploy/
     localhost: true

In that case deployment occurs on VM running dsci orchestrator and which allows to restart docker container with new image version right on VM

——-

That is it.

This simple but real life example shows how easy and one can write cicd pipelines using dsci framework and how flexible it is.

Hopefully you like it.

—-

For comprehensive documentation and more information please visit dsci web site - http://deadsimpleci.sparrowhub.io/doc/README

DTAP - super simple testing protocol for infrastructure audit

Alexey Melezhik — Thu, 08 Jan 2026 12:54:15 +0000

DTAP is a new testing protocol allowing to test infrastructure with just a Bash scripts. Here is a quick example, let's test that /etc/dhcp/ directory and /etc/dhcpcd.conf file exists:

#!/bin/bash

session=$(date +%s)

ls /etc/dhcp/ 2>&1 | dtap --box - \
--session $session \
--params path=/etc/dhcp/ \
--check path-ok \
--desc "dhcp/ dir"

ls /etc/dhcpcd.conf  2>&1 | dtap --box - \
--session $session \
--params path=/etc/dhcpcd.conf \
--check path-ok \
--desc "dhcpcd.conf file"

dtap  --report  --session $session

The result will be:

DTAP report
session: 1767875428
===
dhcp/ dir ...... OK
dhcpcd.conf file ...... OK

Plain and simple

As one can see test scripts are just plain Bash commands, no fancy YAML or even high level programming languages.

Also DTAP follows WYSIWYG principle when we get exactly what we see, in a sense this is something we would do trying to check existence of the mentioned directory and file:

ls /etc/dhcp/
ls /etc/dhcpcd.conf

And if any errors occurr we will get exactly what we are asking for - output of ls command which most of the Linux users probably are familiar with:

!/bin/bash
ls  /etc/does-not-exist 2>&1 | dtap --box - \
--session $session \
--params path=/etc/does-not-exist \
--check path-ok \
--desc "/etc/does-not-exist dir"

Output:

DTAP report
session: 1767875841
===
/etc/does-not-exist dir ...... FAIL
[report]
15:37:22 :: [sparrowtask] - run sparrow task .@path=/etc/does-not-exist
15:37:22 :: [sparrowtask] - run [.], thing: .@path=/etc/does-not-exist
[task run: task.bash - .]
[task stdout]
15:37:23 :: ls: cannot access '/etc/does-not-exist': No such file or directory
[task check]
stdout match <^^ "/etc/does-not-exist"  $$> False
=================
TASK CHECK FAIL
2

How this works?

There are two essentials primitives in DTAP:

boxes
and - checks

Box is an abstraction for everything we want to test - from web server to messages in syslog files. Boxes produces some output to be checked against check rules ( aka checks ). In the previous examples - boxe is just ls command which output redirected to a certain check rule via --box - notation ( meaning read box output from STDIN ). DTAP comes with some predefined boxes user can use them without writing any piece of code, but most of the time boxes - are something a user would write as a chain of Bash commands something likes that:

#!/bin/bash
( 
  2>&1
  cmd1;
  cmd2;
  cmd3;
  # ...
) | tap --box -

Or with a single script:

#!/bin/bash
. /some/box/script.sh | tap --box -

Checks are rules written on formal DSL and executed remotely on DTAP server, so users don't need to install anything, only small tap binary written on golang that interacts with a server send output from boxes to a server and get results back. Checks DSL is based on regular expressions and is super flexible, allowing many things to do including extension by using many general programming languages.

As an example if you look inside path-ok check that verifies file/directory existence you'll see something like this:

generator: <<OK
!bash
echo "regexp: ^^ \"$(config path)\"  \$\$"
OK

To list available checks just run:

tap --check_list

Follow double tap web site documentation to get details on using available checks.

It's easy to create new checks and add them to DTAP distribution, if you are instead please let me know. There is quick start introduction into check DSL could be found here

Conclusion.

DTAP is a new kid on the block, that allows to test infrastructure with using just a Bash yet very flexible and powerful. I encourage you to play with it, you can start with installation guide

Thanks for reading

raku-sparrow6 - swiss army knife for alpine linux

Alexey Melezhik — Wed, 23 Jul 2025 14:30:07 +0000

Sparrow is a Rakulang based framework that allows to automate routine tasks, the beauty of it - one can use prepacked plugins available from public Sparrow repository - https://sparrowhub.io

Install Sparrow framework

Sparrow is installed as native Alpine package:

sudo apk add \
--no-cache --wait 120 -u \
--repository=http://dl-cdn.alpinelinux.org/alpine/edge/community \
raku-sparrow6

Set up plugins repository

export SP6_REPO=http://sparrowhub.io/repo
s6 --index-update

Search alpine plugins

Sparrowhub has a lot of plugins, to limit search to Alpine related plugins:

s6 --search alpine

There are now not too many of them for Alpine, eventually I am going to add more, put in comments what sort of plugins one would like to see

Run plugin

To run plugin, use Sparrow cli, you can also pass parameters to a plugin. For example, to validate APKBUILD file one can use apkbuild-strict plugin:

s6 --plg-run apkbuild-strict@path=/path/to/apkbuild/file

That way Sparrow plugins could be run from terminal or get inlined into shell scripts

Conclusion

It'd be great to hear feedback from (Alpine) users, what sort of tasks could be implemented by Sparrow

Credits

Kudos to Celeste - Alpine maintainer for raku-sparrow6 Alpine package creation

Validating configuration files with Raku and Sparrow Task::Check DSL

Alexey Melezhik — Thu, 30 Jan 2025 08:53:50 +0000

In real world we have a lot of pure structured or structured in hard to parse formats configuration files:

/etc/default/nginx, /etc/default/grub, logrotate.conf, sysctl.conf - to name a few.

In configuration management it's not only important to maintain specific configurations but also validate those changes to make it sure we don't break things.

Sparrow - is an automation tool written on Raku that allow to make such a validation by leveraging Task::Check DSL rules which underneath are just Raku regular expressions.

In the rest of this post we take a look at real life example.

Grub

Consider linux GRUB boot loader file /etc/default/grub with following configuration:

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="rootfstype=ext4 quiet splash acpi_osi="
GRUB_CMDLINE_LINUX=""
GRUB_ENABLE_CRYPTODISK=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

GRUB_DISABLE_OS_PROBER="true"

Though configuration format is quite simple, based on VAR=value semantics, it still allows some freedom in specification.

For example to enable variables we could use one of many forms - yes/"yes", true/"true" and just 1. Another challenge is to check that some variable are disabled - they might be just commented or “switched off” via negation form, e.g. VAR=0,false,""

As rigorous operations we want to validate a couple of things:

Boot on crypted disks is supported by setting GRUB_ENABLE_CRYPTODISK to one of the values: 1,yes,true and optional quoted form - "yes","true"
OS prober capability is disabled or in other words GRUB_DISABLE_OS_PROBER is not set by using any of forms - 1,yes,true,"yes","true"

Before go parsing let's create simple Raku program (Sparrow task), that dumps out configuration removing empty and commented lines, the whole output will be placed inside >>>, <<< markers for visibility:

task.raku

#!raku

my $path = config()<path>;

say "validate config {$path} ...";

say ">>>";

my @out;

for $path.IO.lines -> $i {
  $i.chomp;
  next unless $i ~~ /\S+/;
  @out.push($i) unless $i ~~ /^ \s* '#'/;
};

say sort(@out).join("\n");
say "<<<";

Now let's write verification scenario by creating rules in Sparrow::Task::Check DSL format:

task.check

note: validate config

between: {'>>>'}  {'<<<'}

  note: allow to boot on crypted disks
  regexp: ^^ \s* "GRUB_ENABLE_CRYPTODISK=" (<[y 1 true]>  | '"true"' | '"yes"' )

  note: never allow to enable os prober
  !regexp: ^^ \s* "GRUB_DISABLE_OS_PROBER=" (<[y 1 true]> | '"true"' | '"yes"' )

end:

The DSL code has there major parts:

between: {'>>>'} {'<<<'}

Setting a context for a search in between >>> ... <<< delimiter, this is an example of so called search context modifiers, in this case - Range modifier.

In other words this is precaution that we only search within specific boundaries of text output

regexp: ^^ \s* "GRUB_ENABLE_CRYPTODISK=" (<[y 1 true]> | '"true"' | '"yes"' )

Verifying that we have GRUB_ENABLE_CRYPTODISK set to one of values (y,1,true,"true","yes").

regexp: marker means Raku regular expression is used for validation.

note: never allow to enable os prober !regexp: ^^ \s* "GRUB_DISABLE_OS_PROBER=" (<[y 1 true]> | '"true"' | '"yes"' )

Verifying we DON'T(*) have GRUB_ENABLE_CRYPTODISK set to one of values (y,1,true,"true","yes"), in other words that GRUB_ENABLE_CRYPTODISK is disabled.

(*) ! sign before regexp: marker means negation logic.

Note: expressions are human readable comments that are showed up in a report to help us correlate every check to business logic

Test report

Now let's run our script and get a result of test. As this is Sparrow scenario, we use s6 - Sparrow CLI runner:

#!bash
s6 --task-run .@path=/etc/default/grub

14:09:20 :: [sparrowtask] - run sparrow task .@path=/etc/default/grub
14:09:20 :: [sparrowtask] - run thing .
[task run: task.raku - .]
[task stdout]
14:09:21 :: validate config examples/grub ...
14:09:21 :: >>>
14:09:21 :: GRUB_CMDLINE_LINUX=""
14:09:21 :: GRUB_CMDLINE_LINUX_DEFAULT="rootfstype=ext4 quiet splash acpi_osi="
14:09:21 :: GRUB_DEFAULT=0
14:09:21 :: GRUB_DISABLE_OS_PROBER="true"
14:09:21 :: GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
14:09:21 :: GRUB_ENABLE_CRYPTODISK=true
14:09:21 :: GRUB_HIDDEN_TIMEOUT_QUIET=true
14:09:21 :: GRUB_TIMEOUT=2
14:09:21 :: <<<
[task check]
# validate config
# allow to boot on crypted disks
stdout match (r) <^^ \s* "GRUB_ENABLE_CRYPTODISK=" (<[y 1 true]>  | '"true"' | '"yes"' )> True
# never allow to enable os prober
stdout match (r) <!^^ \s* "GRUB_DISABLE_OS_PROBER=" (<[y 1 true]> | '"true"' | '"yes"' )> False
=================
TASK CHECK FAIL

PS for colorful report please click here

Expectedly we have following results:

GRUB_ENABLE_CRYPTODISK check passed
GRUB_DISABLE_OS_PROBER fails

Pretty impressive results considering such a little amount of code has been written!

Sparrow Task::Checks is comprehensive and super flexible tool allowing one to write validation scenarios for arbitrary data formats and complexity, check out documentation pages to know more.

That is it. Thanks for reading

Useful terminal plugins to build golang code

Alexey Melezhik — Fri, 24 Jan 2025 09:16:01 +0000

Recently, I've released 2 new Sparrow plugins to help build Golang code:

go-build

Simple golang code builder

go-format

Golang code formatter

They are simple yet useful, providing two essentials tools to build Golang code in terminal:

Build code
Format code

The rest of the post shows how to use the plugins

Installation

To install plugins we need to use Tomtit task runner with profiles - preset of Raku scenarios that grouped by topics:

#!bash

cd /to/you/source/code
tom --init # initialize Tomtit for the first time
tom --profile go # install go profile scenarios

Now once we've installed go profile, we have an access to go-build and go-format scenarios which are just a Raku wrappers to run mentioned plugins. Code is simple enough and editable to adjust your project specific needs

Go-build scenario:

#!bash

tom --cat go-build

#!raku

my $path = [
  "cmd/app/main.go",
  "cmd/cli/cli.go"
];


task-run "build", "go-build", %(
  :$path,
);

Go-format scenario:

#!bash

tom --cat go-format

#!bash

task-run "go format", "go-format";

Customization

In case of go-build scenario - adjust path array with files to build for your project specific:

#!bash

export EDITOR=nano
tom --edit go-build

#!raku

my $path = [
  "cmd/app/app.go",
  "cmd/app2/app2.go",
];

# the rest of the code stays the same

Run

To run plugins - use tom cli runner:

#!bash
tom go-build
11:56:26 :: go build cmd/main.go
11:56:27 :: cmd/main.go

#!bash
tom go-format
11:57:18 :: cmd/main.go

That is it.

Follow plugins documentation to get familiar with plugins parameters.

Thanks for reading

Simple search in source code with Tomtit and Sparrow

Alexey Melezhik — Thu, 23 Jan 2025 20:21:22 +0000

Recently, I've released a simple Sparrow plugin called find to search text in source code, which is based on find and grep core linux utilities.

The rest of the document shows how to perform search in terminal by using Tomtit task runner.

Installation

To install the plugin, we need to use Tomtit task runner with profiles - preset of Raku scenarios that grouped by topics:

#!bash

cd /to/you/source/code
tom --init # initialize Tomtit for the first time
tom --profile code # install code profile scenarios

Now once we've installed code profile scenarios (for now it's just a single one), we have an access to search scenario which is just a Raku wrapper to do search with find plugin:

#!bash

tom --cat search

#!raku

my $ext = prompt("ext (go): ");

$ext = "go" unless $ext;

my $search1 = prompt("search1: ");

my $search2 = prompt("search2: ");

my $exclude = prompt("exclude: ");

say "find [$search1] [$search2] !$exclude in $ext";

task-run "find $search1 $search2 in $ext", "find", %(
  :$ext,
  :$search1,
  search2 => $search2 || "",
  exclude => $exclude || "",
);

The logic is following:

‘search1’ filters results out by first string and then ‘search2’ filter is applied for final results, ‘ext’ defines files extension to search.

Customization

Stub code shipped with Tomtit profile is good enough, we only need to change it a bit, to make default choice for file extenstion suitable for Raku project:

#!bash

export EDITOR=nano
tom --edit search

#!raku

my $ext = prompt("ext (rakumod): ");

$ext = "rakumod" unless $ext;

# ... the rest of the code is the same

Search

Now let's search files in my Tomtit repository. Say, I want to find all exported functions.

#!bash

tom search

load configuration from /home/melezhik/projects/Tomtit/.tom/env/config.raku
ext (rakumod): 
search1: sub
search2: is export
exclude: 
find [sub] [is export] ! in rakumod
23:09:06 :: find [sub] [is export] [!] in *.rakumod
23:09:06 :: ===
23:09:06 :: find . -name *.rakumod -exec grep --color -H 'sub' {} \; | grep 'is export'
23:09:06 :: case2
23:09:06 :: ./lib/Tomtit/Completion.rakumod:sub complete () is export {
23:09:06 :: ./lib/Tomtit.rakumod:our sub check-if-init ( $dir ) is export {
23:09:06 :: ./lib/Tomtit.rakumod:our sub init ($dir) is export {
23:09:06 :: ./lib/Tomtit.rakumod:our sub load-conf () is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub tomtit-usage () is export  {
23:09:06 :: ./lib/Tomtit.rakumod:sub tomtit-help () is export  {
23:09:06 :: ./lib/Tomtit.rakumod:sub tomtit-clean ($dir) is export { 
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-last ($dir) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-run ($dir,$scenario,%args?) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-remove ($dir,$scenario) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-cat ($dir,$scenario,%args?) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-edit ($dir,$scenario) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub environment-edit ($dir,$env) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub environment-list ($dir) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub environment-set ($dir,$env) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub environment-show ($dir) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub environment-cat ($dir,$env,%args?) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-doc ($dir,$scenario) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-list ($dir) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub scenario-list-print ($dir) is export {
23:09:06 :: ./lib/Tomtit.rakumod:multi sub profile-list  is export {
23:09:06 :: ./lib/Tomtit.rakumod:multi sub profile-list($dir,$profile is copy)  is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub profile-install ($dir, $profile is copy, %args?) is export {
23:09:06 :: ./lib/Tomtit.rakumod:sub completion-install () is export {

That is it. Follow plugin documentation to get familiar with plugin parameters.

Thanks for reading

Developing a simple Sparrow plugin with Raku and Bash

Alexey Melezhik — Mon, 20 Jan 2025 14:27:08 +0000

Sparrow is very efficient in writing scripts to automate system management using Raku and Bash, in today's post I am going to show an example ...

Deploy configuration file and restart service upon changes

Let's create a simple plugin that enable / disable some configuration flags in service configuration file and if any changes occur reload a service. Format of flags in configuration file is following:

var=true|false

Here is an example of configuration file:

config.txt

debug=true
sentry=false
tls=false

Usually services are managed as systemd units, so service reload could be used to reload service configuration.

——

Let’s get started and create root folder for the plugin:

#!bash

mkdir -p service-config-reload
cd service-config-reload

Now inside plugin root directory:

hook.raku:

#!raku

my $cfg-orig = config()<path>.IO.slurp();

my $cfg = $cfg-orig;

for config()<enable><> -> $f {
   $cfg ~~ s/<?wb> "$f=" \S+ /$f=true/;
}

for config()<disable><> -> $f {
   $cfg ~~ s/<?wb> "$f=" \S+ /$f=false/;
}

set_stdout($cfg);

if $cfg ne $cfg-orig {
   set_stdout("config changed");
   config()<path>.IO.spurt($cfg);
   set_stdout("config updated");
   run_task "service_restart", %(
       service => config()<service>,
       path => config()<path>,
   );
}

mkdir -p tasks/service_restart/

tasks/service_restart/task.bash

#!bash

sudo service $service reload

In real life applications services might have a linter capability where before configuration is applied it's checked and if any error occur application of changes is terminated:

#!bash

set -e
sudo /usr/bin/$service --check $path
sudo service $service reload

Publish plugin

Packaging scenario as a plugin will allow users to reuse it as a library.

sparrow.json

{
  "name" : "service-config-reload",
  "description" : "Deploy configuration file and restart service upon changes",
  "version" : "0.0.1",
  "category" : "linux",
  "url" : "https://github.com/melezhik/sparrow-plugins/tree/master/service-config-reload"
}

This last command will upload plugin to local Sparrow repository (see this doc for more details)

#!bash

s6 --upload

Use of plugins

Anywhere in Raku scenario, just do this:

#!raku

use Sparrow6::DSL;

task-run "apply and reload", "service-config-reload", %(
   :path<config.txt>,
   :service<app>,
   enable => <tls sentry>, # enable TLS and use of Sentry for production
   disable => <debug>, # disable debug mode for production 
)

Thanks for reading!

PS

Source code of the plugin could be found here - https://github.com/melezhik/sparrow-plugins/tree/master/service-config-reload

Go pipelines with Raku interfaces

Alexey Melezhik — Tue, 05 Nov 2024 20:47:34 +0000

So, let's say you're golang developer and want pure Go to write some CICD task:

cat task.go

package main

import "fmt"

func main() {
    fmt.Println("Hello, pipeline")
}

Go is cool, but there is one thing that makes it difficult to use in high level scenarios - its verbosity. Passing parameters to go tasks and return them back to main scenario takes some efforts and a lot of boilerplate code. It'd be good to keep main code concise and easy to read.

Rakulang in other hand is perfect language when it comes to munge data in and out, due it's extreme flexibility and expressiveness.

In this post I am going to show how to embed golang tasks into CICD pipelines, with a little help of Sparrow framework.

Let's, first of all, modify our golang task code, new version will be:

cat task.go

package main

import (
  "fmt"
  "github.com/melezhik/sparrowgo"
)

func main() {

  type Params struct {
    Message string
  }

  type Result struct {
    Message string
  }

  var params Params

  // sparrowgo takes care about passing 
  // input parameters from Raku to Go
  sparrowgo.Config(&params)

  // read params from pipeline
  fmt.Printf("Task params: %s\n", params.Message)

  // return results back to pipeline 
  sparrowgo.UpdateState(&Results{Message : "Hello from Go"})

}

All we've done here is utilized Sparrowgo package that "convert" golang task into Sparrow task with a benefits of passing and returning data from and to Rakulang.

Finally this how our pipeline will look like, now it's Raku part:

#!raku

my $s = task-run ".", %(
  :message<Hello from Raku>
);

say "Result: ", $s<Message>;

High level design.

Now, once we have some prove of concept code in place, we can get a high level picture of what our pipeline system could look like:

      [ Raku scenario to pass and handle data in and out ]
        \                    \                     \          
      task.go -> result -> task.go -> result -> task.go -> ...

So, we have the best of two worlds - Raku to write scenarios with less of code and Golang to do all heaving lifting where performance and strict type checking is required.

Sparky - composable user interfaces for internal services

Alexey Melezhik — Mon, 01 Jul 2024 15:09:30 +0000

Sparky recent releases have introduced a lot of features allowing to build user interfaces for internal web applications.

In a nutshell Sparky is a web platform to run automation tasks, and Sparky equips you with a rich feature set to build a frontend to launch tasks with parameters.

HTML controls

Let's brew some coffee, even though I've been trying to slow down on caffeine recently.

allow_manual_run: true
vars:
  -
    name: Flavor
    default: "latte"
    type: select 
    values: [ espresso, amerikano, latte ]

  -
    name: Topic  
    default: "milk"
    type: select
    values: [ milk, cream, cinnamon ]
    multiple: true

  -
    name: Step3 
    default: "boiled water"
    type: input

This simple Sparky job definition will spin up a simple UI with 3 controls:

In Raku scenario those input parameters are handled like that:

my $flavor = tags()<Flavor>;
my $water = tags()<Step3>;
my @topics = [];

# tags() needs :array modifier
# as multiple choices are passed

for tags(:array)<Topic><> -> $t {
   @topics.push: $t;
};

Simple, huh?

One more coffee, please!

Or maybe you like a tea?

Another cool feature of Sparky is a multi scenario flows or group variables. Let's say you would like to give a choice of coffer or tea, so instead of having 2 job definitions with different sets of variables, let's create just one:

vars:

  # tea vars

  -
    name: Flavor
    default: "black" 
    type: select 
    values: [ black, green ]
    group: [ tea ]

  -
    name: Topic
    default: "milk"
    type: select
    values: [ milk, cream ]
    group: [ tea ]

  # coffee vars

  -
    name: Flavor
    default: "latte"
    type: select 
    values: [ espresso, amerikano, latte ]
    group: [ coffee ]

  -
    name: Topic  
    default: "milk"
    type: select
    values: [ milk, cream, cinnamon ]
    group: [ coffee ]
    multiple: true

  # common vars

  -

    name: Step3
    default: "boiled water"
    type: input
    group: [ tea, coffee ]

# every var
# in group vars
# is a separate
# scenario 

group_vars:
  - tea
  - coffee

Now, when the job gets run, we have a choice:

And then (when click on proper link), let's enjoy some tea:

Conclusion

Sparky is a nice web console to build internal automation services with HTML and write automation scenarios with Raku. Thanks for reading.

Sparky - hacking minikube with mini tool

Alexey Melezhik — Fri, 14 Jun 2024 13:56:10 +0000

TL; DR: How to deploy docker to minikube when one does not need anything fancy, but pure Raku.

So, you have your own pet K8s cluster deployed as minikube and you want to play with it. You have few microservices to build and you don't want to bother with kubernetes low level commands at all.

On the other hand, you setup is complex enough to express it in a bunch of yaml files or kubectl commands. Here is an elegant way to handle this in pure Raku, and it's called Sparky ...

Show me the design

      |-------------------------------------|
      | Sparky -> kubectl -> MiniKube       | 
      |                     /\  /\  /\      |
      |                     pod pod pod     |
      |-------------------------------------|

So the infrastructure part is simple - on the same host we install minikube and Sparky that underlying uses kubectl to deploy containers into k8s cluster.

Show me the code

As usually Sparky job is a pure Raku code, but this time some plugins will be of use as well ...

Sparky is integrated with - Sparrowhub - https://sparrowhub.io - repository of Sparrow plugins - useful building blocks for any sort of automation.

Let's use a couple of them - k8s-deployment and k8s-pod-check to deploy and check Kubernetes pods. From Sparky point of view those are just Raku functions, with some input parameters.

task-run "dpl create", "k8s-deployment", %(
  :deployment_name<nginx>,
  :app_name<nginx>,
  :image<nginx:1.14.2>,
  :3replicas,
);


# give it some time to allow all pods to start ...
sleep(5);

task-run "nginx pod check", "k8s-pod-check", %(
  :name<nginx>,
  :namespace<default>,
  :die-on-check-fail,
  :3num,
);

For the tutorial purpose we are going to deploy nginx server with 3 replicas, by using k8s-deployment plugin.

Let's give it a try.

First run

And the very first deploy ... fails:

... some output ...

11:03:34 :: deployment.apps/nginx created
11:03:34 :: [repository] - installing k8s-pod-check, version 0.000012
11:03:34 :: [repository] - install Data::Dump to /home/astra/.sparrowdo/minikube/sparrow6/plugins/k8s-pod-check/raku-lib
All candidates are currently installed
No reason to proceed. Use --force-install to continue anyway
[task run: task.pl6 - nginx pod check]
[task stdout]
11:03:41 :: ${:die-on-check-fail(Bool::True), :name("nginx"), :namespace("default"), :num(3)}
11:03:41 :: ===========================
11:03:41 :: NAME                           READY   STATUS             RESTARTS         AGE
11:03:41 :: nginx-77d8468669-5gxbf         0/1     ErrImagePull       0                5s
11:03:41 :: nginx-77d8468669-c5vbl         0/1     ErrImagePull       0                5s
11:03:41 :: nginx-77d8468669-lhc54         0/1     ErrImagePull       0                5s
11:03:41 :: ===========================
11:03:41 :: nginx-77d8468669-5gxbf POD_NOT_OK
11:03:41 :: nginx-77d8468669-c5vbl POD_NOT_OK
11:03:41 :: nginx-77d8468669-lhc54 POD_NOT_OK
[task check]
stdout match <^^ 'nginx' \S+ \s+ POD_OK $$> False
---

Although Kubernetes deployment has been successfully created, further k8s-pod-check failed to verify that all pods are running.

Use of die-on-check-fail option made the job stops strait away after this point.

The reason is in ErrImagePull - nginx docker image is not accessible from within a minikube, which a known minkube DNS issue, which is easy to fix.

It's fixed!

All we need to do is to upload nginx docker image manually, so that minukube will pick it up from a file cache:

minikube image load nginx:1.14.2

Now, when have restarted the failed job we get this:

... some output ...

11:05:42 :: deployment.apps/nginx unchanged
[task run: task.pl6 - nginx pod check]
[task stdout]
11:05:47 :: ${:die-on-check-fail(Bool::False), :name("nginx"), :namespace("default"), :num(3)}
11:05:47 :: ===========================
11:05:47 :: NAME                           READY   STATUS             RESTARTS         AGE
11:05:47 :: nginx-77d8468669-5gxbf         1/1     Running            0                2m10s
11:05:47 :: nginx-77d8468669-c5vbl         1/1     Running            0                2m10s
11:05:47 :: nginx-77d8468669-lhc54         1/1     Running            0                2m10s
11:05:47 :: ===========================
11:05:47 :: nginx-77d8468669-5gxbf POD_OK
11:05:47 :: nginx-77d8468669-c5vbl POD_OK
11:05:47 :: nginx-77d8468669-lhc54 POD_OK
[task check]
stdout match <^^ 'nginx' \S+ \s+ POD_OK $$> True
<3 pods are running> True
---

The last deployment has not changed (with is denoted by "deployment.apps/nginx unchanged" line), as we did not change anything, however minikube now is able to pick the recently uploaded docker image and all pods now are running.

Congratulation with the very first successfully deployment to Kubernetes via Sparky!

Clean up

In the end let's remove our test pods, by using k8s-deployment plugin:

task-run "dpl delete", "k8s-deployment", %(
  :deployment_name<nginx>,
  :action<delete>,
);

Further thoughts

This simple scenario is going to give us some ideas on how to deploy to Kubernetes in imperative way using pure Raku, I, personally like this approach better, as having a bunch of helm charts and yaml files seems overkill when one need just to deploy some none production code, however, as always YMMV, thanks for reading ...

Sparky - simple and efficient alternative to Ansible

Alexey Melezhik — Fri, 31 May 2024 13:22:49 +0000

So, you have a hundred VMs you need to manage, and you have ... Ansible ? I should stop here, as this is a tool that is standard in configuration management nowadays, but I'd dare to continue and say there is a better alternative to it.

But before we get into it, why am I so frustrated with Ansible? Here are my points:

YAML based declarative DSL really stinks on complex tasks as it lacks the flexibility that imperative languages have.
YAML is not even a programming language, and you gonna pay the price very soon.
To keep ansible code clean and simple, extra efforts are required, one need to refactor out all the complexity from YAML to python modules and this feels like "why I even start using YAML DSL"?
Ansible reports are frustrating as I always need to add these debug tasks to show real STDOUT/STDERR emitted from commands, where it should just work out of the box.
Ansible ties me to the idea of "running on a host," where sometimes I need to run tasks not tied to hosts, yes, you can still use "ansible_connection=local" but this feels awkward.

Meet Sparky

So, meet Sparky - elegant, efficient and all-battery included automation tool. It's written on powerful and modern Raku language, with bulma css frontend and web sockets.

To install Sparky - install Rakudo first and then install Sparky itself as a Raku module:

curl https://rakubrew.org/install-on-perl.sh | sh 
eval "$(~/.rakubrew/bin/rakubrew init Bash)" 
rakubrew download moar-2024.05
git clone https://github.com/melezhik/sparky.git
cd sparky/
# install Sparky and it's dependencies
zef install --/test .
# init sparky sqlite database
raku db-init.raku
# run sparky job runner
nohup sparkyd >~/.sparkyd.log < /dev/null &
# run sparky web console
cro run

This simple scenario gets it up and running; if you go to http://127.0.0.1:4000 you'll see a nice Sparky web console. We use the console to run sparky jobs.

Show me the design

So we have a control plane that would manage many hosts over ssh, using push mode:

         ---------------
         | CP , Sparky | 
         ---------------
              [ssh] 
       /   /    |    \    \
     host host host host host

This is pretty much what ansible does ...

Show me the code

Now say, we have 5 NGINX servers we need to restart, let's drop a simple Sparky job to do this in pure Raku language:

use Sparky::JobApi;
class Pipeline does Sparky::JobApi::Role { 
  method stage-main {
    for 1..5 -> $i {
      my $j = self.new-job :workers<5>; 
      $j.queue: %(
        sparrowdo => %(
          bootstrap => true,
          host => "nginx_{$i}.local.domain"
        ),
       tags => %(
         stage => "child",
         i => $i
       )
      ); 
   }
  }
  method stage-child {
    service-restart "nginx" 
  }
}

In this scenario, Sparky will run five parallel jobs that restart nginx on five hosts. Simple and elegant.

Moreover those five jobs will appear as five separate reports in Sparky UI …

Got interested?

Of course, this is only a quick glance at Sparky architecture and features, things to cover further:

Job orchestration (DAGs)
Core DSL (pure Raku)
Custom UIs
Authentication (oauth2) and security access list
Writing more sophisticated scenarios
Extending Sparky with plugins by using many programming languages
Installing Sparky with MySQL/Postgresql database storage (instead of sqlite)
Using Sparky as CI server (SCM triggering and cron jobs)

Links

Sparky project - https://github.com/melezhik/sparky

Sparky - Self-hosted Task Runner with Easily Customizable UI

Alexey Melezhik — Tue, 02 Jan 2024 14:25:42 +0000

Sparky is a task-runner allow teams to automate their daily tasks by creating #Rakulang scenarios and customizable UI

mkdir -p ~/.sparky/projects/build-rakudo/

nano ~/.sparky/projects/build-rakudo/sparky.yaml

sparrowdo:
  no_sudo: true
  no_index_update: true
  bootstrap: false
  format: default
allow_manual_run: true
vars:
  -
      name: version
      default: "2023.12"
      type: input
  -
      name: arch
      values: [ alpine, debian, ubuntu ]
      type: select
      default: alpine

nano ~/.sparky/projects/build-rakudo/sparrowfile

#!raku
task-run "files/build-rakudo", %(
  rakudo_version => tags()<version>,
  arch => tags()<arch>,
);

In this imaginary scenario we want to build a Rakudo docker image for a specific Rakudo version and Linux distribution:

mkdir -p ~/.sparky/projects/build-rakudo/files/build/

nano ~/.sparky/projects/build-rakudo/files/build/task.bash


cat << HERE > $cache_root_dir/install-rakudo.sh
mkdir ~/rakudo && cd $_
curl -LJO https://rakudo.org/dl/rakudo/rakudo-$1.tar.gz
tar -xvzf rakudo-*.tar.gz
cd rakudo-*
perl Configure.pl --backend=moar --gen-moar
make
make install
HERE

if test $(config arch) == "alpine"; then
cat << HERE > $cache_root_dir/Dockerfile
FROM alpine:latest
ARG rakudo_version=2023.12
RUN apk update && apk add git make gcc musl-dev
RUN adduser -D -h /home/worker -s /bin/bash -G wheel worker
USER worker
ENV PATH="/home/worker/rakudo-$rakudo_version/install/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/vendor/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/core/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/site/bin:/home/worker/.raku/bin:${PATH}"
COPY install-rakudo.sh . 
RUN sh ./install-rakudo.sh $rakudo_version
HERE
elif test $(config arch) == "debian"; then
cat << HERE > $cache_root_dir/Dockerfile
FROM debian:latest
ARG rakudo_version=2023.12
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -q -o Dpkg::Use-Pty=0
RUN apt-get install -q -y -o Dpkg::Use-Pty=0 build-essential curl git
RUN useradd -m  -d /home/worker --shell /bin/bash worker
USER worker
ENV PATH="/home/worker/rakudo-$rakudo_version/install/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/vendor/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/core/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/site/bin:/home/worker/.raku/bin:${PATH}"
COPY install-rakudo.sh . 
RUN sh ./install-rakudo.sh $rakudo_version
HERE
elif test $(config arch) == "ubuntu"; then
cat << HERE > $cache_root_dir/Dockerfile
FROM ubuntu:latest
ARG rakudo_version=2023.12
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -q -o Dpkg::Use-Pty=0
RUN apt-get install -q -y -o Dpkg::Use-Pty=0 build-essential curl git
RUN useradd -m  -d /home/worker --shell /bin/bash worker
USER worker
ENV PATH="/home/worker/rakudo-$rakudo_version/install/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/vendor/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/core/bin:/home/worker/rakudo-$rakudo_version/install/share/perl6/site/bin:/home/worker/.raku/bin:${PATH}"
COPY install-rakudo.sh . 
RUN sh ./install-rakudo.sh $rakudo_version
HERE
else
echo "$(config arch) is not supported"
exit 1
fi

docker build $cache_root_dir/ \
-f $cache_root_dir/Dockerfile \
--build-arg rakudo_version=$(config rakudo_version) \
-t team/rakudo:$(config arch)-$(config version)

docker push team/rakudo:$(config arch)-$(config version)

Once we've created all necessary files we can navigate to a "build-rakudo" project in Sparky UI and hit "build now" button:

By choosing a Rakudo version and Linux distribution and launching a new build, within a few minutes we get a new Rakudo docker image published to an internal docker registry.

This is just a simple example of how one can use Sparky for automation, there is more then that, but the main idea is to spin up jobs quickly with simple web interfaces generated from YAML specifications, resulting in various kinds of centralized tools available for needs of your team.

For the scaling it's even possible to convert scenarios into plain Raku modules or Sparrow plugins and distribute them across many teams.

Conclusion

Sparky is versatile task runner enable small teams of developers of self-hosted platform to automate all boring and manual stuff they might have during development cycle, check it out for more details - https://raku.land/zef:melezhik/Sparky