Forem: dyrector.io

Goodbye, 2023! dyrector.io’s Annual Recap

Geri Máté — Wed, 20 Dec 2023 11:04:33 +0000

2023 is coming to an end, which means it's time to revisit what happened with the team and the project of dyrector.io in the past 12 months.

January – Full Stack Highlighted dyrector.io

After the lengthy Christmas break with a full stomach and a couple extra kilograms the real surprise caught us blind-sided. The Full Stack platform featured dyrector.io in its highlights.

Team-wise the most notable event was our Minus 30 hike in the pleasant January weather, which was a great occasion to have a chat about both technology related and unrelated things, and also to taste some pálinka.

February – dyrector.io Alpha Dropped

The first weeks of February were all about attending FOSDEM and the upcoming launch of dyrector.io on Product Hunt. On the day of the launch we made alpha access available.

Our Product Hunt launch turned out to be a shot at the buzzer, but we still did nice. With a launch 6 hours into the voting, we reached the #11 spot. The same day we made a new release and a demo video. Busier than planned, but we did good.

At the conference in Belgium, we were able to catch up with a lot of likeminded people eager to learn about open-source software.

At the same time, our teammate, Levi showed up in the local cloud meetup scene as organizer and a presenter, too. Another teammate of ours, Nándi was interviewed in the podcast series of Uptime Community about DevOps, ChatGPT, and open-source.

March – Three (Hundred) Is the Magic Number

We doubled down on catering to a self-hosting audience in the first months of 2023, which helped us reach 300 stars on GitHub on the 3rd of March. We published a bunch of blog posts about self-hosting certain types of applications, which you can find here.

In March, we published our Awesome repository containing infrastructure related questions. We consider it useful when someone is onboarded to a new project maintaining infrastructure.

Another important event of the month was when Docker announced the end of Free Teams on Docker Hub. Backlash was inevitable and so was the organization backing out of their plans of monetizing Free Teams.

April – Adventures in the UK and Hungary

A portion of our team took a business trip in the UK to visit Hanover Displays at their HQ in Brighton. While Levi and Gopher was there, they paid a visit to the LEGO HQ for a meetup, as well.

After the trip in the UK, we went out of the office for a few days of team building when we could unwind with the whole team.

Levi attended KubeCon in Amsterdam, too, which turned out to be the funniest way to reach 420 stars on GitHub on April 20th. Trust me, we didn’t plan this whatsoever.

May – 0.4.0 & Roadmap Published

After a Q1 busy with refactoring and making dyrector.io’s code more efficient, we started to make new releases faster. The first step was making 0.4.0, which didn’t deliver any significant changes to functionality, but it was important to accelerate our release cycle in the long run.

At the same time, we published our roadmap on GitHub and added new issues to the repository.

We also made some new friends: ConfigCat reviewed the platform on their blog.

June – Team Building in Croatia & 1000000000 Stars

Release 0.5.0 was a special moment for our team. It was the first version in months that included new features. To celebrate this special moment, we went to Croatia to finish working on the new version and chill at the sunny beach.

This was the perfect way to kick off our summer. After the trip to Croatia, we were able to consistently release on a bi-weekly basis, shipping new features again and again.

After 0.5.0 dropped, we passed 512 stars on GitHub, or 1000000000 in binary.

July – Automated Deployments With dyrector.io

One of the most significant features we added this year was the auto-deployment capability. The GitHub Actions compatible feature came out on July 14 in release 0.6.0.

A very pleasant surprise was when Nevo David mentioned dyrector.io in his blog post, which resulted in increased exposure and interest in the platform. In a few days we gained hundreds of stars on GitHub.

Even though it was the middle of the summer, we took no breaks. Between publishing new releases full of new features, we went to Lake Balaton to sail and Nándi and Geri even completed the Lake Balaton Cross Swimming.

At the end of July Levi attended WeAreDevelopers 2023 in Berlin.

August – dyrector.io Turns International

The most significant change was an internal change: our teammate, Nándi moved to the Netherlands with his girlfriend. We officially became a remote-first company, while the rest of the team still showed up at the office every day. We had a goodbye party for him where we said farewell with a few cans of his favorite beverages for the road.

We launched dyrector.io on a new platform called Dev Hunt, which is an open-source Product Hunt alternative. With the help of our community, we were able to reach the #1 spot and the Developer Tool of the Week title that comes with it.

In other cloud-related news HashiCorp announced they're changing their products' license, including Terraform’s, to Business Source License, which sparked the foundation of OpenTF, which later was named OpenTofu.

September – Product Hunt Launch #2

The majority of August was spent on preparations for our Product Hunt launch in September. The date was set – September 8th. We knew a product like ours only has a chance of a significant result on a Friday.

The result: #6 in the daily rankings, top 50 in the weekly with around 260 votes. Definitely an impressive result with a heavily developer-focused tool.

In the meantime, Levi took care of networking: he appeared in the Follow The Pattern podcast, attended InfoBip’s Shift conference in Croatia, and went to Kubernetes Community Days in Vienna.

October – darklens Enters the Scene

The biggest achievement of October in our household was a one-week sprint when more than half of the team was on vacation. Three teammates of ours joined forces, two developers and one marketer, to develop a complimentary product to dyrector.io.

We named this tool darklens, which makes Docker logs and container settings available in your browser. A week after the sprint we launched darklens on Product Hunt for an impressive #14 spot with 140 upvotes.

November – Team Building in Portugal

Over the summer, the whole team was able to snag developer tickets to Web Summit in Lisbon. Soon as we got the confirmation, we started planning our travel to Portugal. With a little sightseeing and networking at the conference, the week we spent in Lisbon turned out to be a blast. We made a lot of new connections.

One of the coolest things of the year was when people found the invitation card for our CTF puzzle and came to our Discord channel or stopped by to say hi at Web Summit.

December – 0.10.0. Dropped

The latest release of dyrector.io, 0.10.0 dropped in early December. You can find out more about it on GitHub.

That’s it for 2023. So long, and thanks for all the fish!

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.

Support us with a star on GitHub.

5 Use Cases When Containerization Is Absolutely Useless for You

Geri Máté — Thu, 30 Nov 2023 14:19:59 +0000

#1 Static, Unchanging Environments

If your application has minimal dependencies and operates consistently across different environments without the need for isolation, containerization may offer little benefit.

Example:

If your application will be the only process executed on the machine.

#2 Limited Scalability Needs

For applications with predictable and steady workloads that do not require rapid scaling or dynamic resource allocation, the overhead of containerization might outweigh the advantages.

Example:

Small scale IoT apps.

#3 Simple, Standalone Applications

In cases where your application is straightforward, lacks dependencies, and isn't part of a larger ecosystem with varied technologies, containerization may introduce unnecessary complexity.

Example:

Zero dependency binaries, and also debugging a host process is more straightforward than doing the same with a container.
Offline applications installed from external medium, running without internet connection.

#4 Resource-Constrained Environments

On systems with extremely limited resources, such as embedded devices or constrained hardware, the overhead of running containerization platforms might not be justified.

Example:

Microelectronics.

#5 Desktop Applications

Sounds exotic, huh? For a good reason. It would be very unusual to use containers for desktop applications. Though similar isolation techniques exist, it is not widespread.

Example:

cs_16_nosteam_portable.exe😅

If You Really Need to Containerize...

You can use dyrector.io to deploy and manage containerized services.

⭐ Star dyrector.io on GitHub:

https://github.com/dyrector-io/dyrectorio

Dagger 101: How to Get Started with Containerized CI Workflows

Geri Máté — Thu, 23 Nov 2023 11:04:26 +0000

Continuous Integration and Continuous Delivery are the secret sauces of shipping new features consistently and reliably to your software. However, the effectiveness of this process is closely tied to the tooling that orchestrates it. Some of the pain points of CI/CD systems are slow feedback loops, vendor lock-in, lack of abstraction, limited composability, or YAML itself. This is where Dagger comes into the spotlight, promising a more unified and accelerated path.

Introduction

The development and deployment process at dyrector.io has already become much faster each year as we adopt and integrate better tools and methods. However, we aim to further unify and accelerate this. Dagger philosophy aligns with what we consider crucial for a truly rapid and seamless process:

Local testing: Enable developers to test their code instantly, locally
Programmable CI: Replace messy YAML-based, complex CI with code
Compatibility: If it runs in a container, you can add it to your pipeline
Portability: The same pipeline can run on your local machine, a CI runner, a dedicated server, or any container hosting service
Universal caching: Every operation is cached by default, and caching works the same everywhere

Currently, we have the option to use our own dyrector.io (we’ll refer to it as dyo many times in this blog post) go CLI with our commands or Docker Compose with its YAML to spin up our stack for local testing, while we also maintain a GitHub Actions workflow for running end-to-end tests on GitHub. This setup lacks coherence, as we cannot employ the specialized GitHub Actions workflow YAML in a local setting or with a different CI/CD environment.

We want to get closer to being able to ship every single day, or even multiple times a day, as quickly as we possibly can, using the same tool running locally and in CI. Dagger feels like an actual innovation in CI/CD, and it seems it will enable us to do that. There is also a strong focus on getting feedback from the community and utilizing it when we’re designing and building something that people really need.

Setting up Dagger CI/CD

We would like to use Dagger locally with the dyo Go CLI, and for this we need the Dagger Go SDK for integration (there are many Dagger SDKs) and the Dagger Engine, which will run our pipelines. We developed a small proof of concept (POC) to evaluate if we could use our entire stack locally with Dagger. If this POC will be successful, we plan to use the same setup in our GitHub workflow, essentially using GitHub Actions just to trigger the Dagger pipeline.

Steps to set up Dagger for our project:

Install the Dagger Go SDK (again, you can use any other Dagger SDK for your project, but we use Go) Go to your existing project – in our case it is dyrectorio.

$ go get dagger.io/dagger
$ go mod tidy

Add local Dagger test to our Makefile It is for simple and fast “make test” (similarly to our other commands).

# Shortcut for local testing
.PHONY: test
test:
    go run golang/cmd/dagger/main.go

Create Dagger main.go
We already have dyo, dagent and crane in our golang/cmd, so put dagger here too.
Import Dagger SDK

Create a Dagger client using the SDK
This will allow you to interact with the Dagger Engine and create pipelines.

Create Dagger pipelines

Additional note:
We can also install the Dagger CLI if we want to, but this is an optional tool to interact with the Dagger Engine from the command-line – it has a nice terminal UI though, with parallel progress bars that are visually impressive if you are into that sort of thing.

Install the Dagger CLI

$ cd /usr/local
$ curl -L https://dl.dagger.io/dagger/install.sh | sh

Workflow Integration

As you will see, the “Dagger way” is a very “Docker-ish” way - no surprise, one of the co-founders of Dagger is Solomon Hykes, earlier founder and technical director of Docker.

To show you concrete code examples from our POC:

Import Dagger SDK
In our main.go:

import (
    "context"
    "dagger.io/dagger"
    …)

Create a Dagger client using the SDK

func initDaggerClient(ctx context.Context) *dagger.Client {
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        panic(err)
    }
    return client
}

And we can call this initDaggerClient() function in our main() like this:

    ctx := context.Background()
    client := initDaggerClient(ctx)
    defer client.Close()

Run unit tests on our NestJS-based Crux backend:

func runCruxUnitTestPipeline(ctx context.Context, client *dagger.Client) {
    log.Info().Msg("Run crux unit test pipeline...")

    _, err := client.Container().From("node:20-alpine").
        WithDirectory("/src", client.Host().Directory("web/crux/"), dagger.ContainerWithDirectoryOpts{
            Exclude: []string{"node_modules"},
        }).
        WithWorkdir("/src").
        WithExec([]string{"npm", "ci"}).
        WithExec([]string{"npm", "run", "test"}).
        Stdout(ctx)
    if err != nil {
        panic(err)
    }

    log.Info().Msg("Crux unit test pipeline done.")
}

We can call this runCruxUnitTestPipeline() function in our main():
runCruxUnitTestPipeline(ctx, client)

Run unit tests on our Next.js-based Crux UI frontend is very similar to the above code, we only need to change the host directory to “web/crux-ui/” and an additional “.next” exclusion, everything else remains the same:

    WithDirectory("/src", client.Host().Directory("web/crux-ui/"), dagger.ContainerWithDirectoryOpts{
        Exclude: []string{"node_modules", ".next"},
    }).

A slightly more advanced example when we run our Crux backend in production mode (as we do for e2e test) with a connected PostgreSQL DB service container:

func getEnv(envPath string) map[string]string {
    cruxEnv, err := godotenv.Read(envPath)
    if err != nil {
        panic(err)
    }
    return cruxEnv
}

func getCruxPostgres(client *dagger.Client, cruxEnv map[string]string) *dagger.Container {
    databaseURL := cruxEnv["DATABASE_URL"]
    parsedURL, err := url.Parse(databaseURL)
    if err != nil {
        panic(err)
    }
    postgresUsername := parsedURL.User.Username()
    postgresPassword, _ := parsedURL.User.Password()
    postgresDB := strings.TrimPrefix(parsedURL.Path, "/")

    dataCache := client.CacheVolume("data")

    cruxPostgres := client.Pipeline("crux-postgres").Container().From("postgres:14.2-alpine").
        WithMountedCache("/data", dataCache).
        WithEnvVariable("POSTGRES_USER", postgresUsername).
        WithEnvVariable("POSTGRES_PASSWORD", postgresPassword).
        WithEnvVariable("POSTGRES_DB", postgresDB).
        WithEnvVariable("PGDATA", "/data/postgres").
        WithExposedPort(5432)

    return cruxPostgres
}

func runCruxProd(ctx context.Context, client *dagger.Client, cruxPostgres *dagger.Container) *dagger.Container {
    crux := client.Pipeline("crux").Container().From("node:20-alpine")
    crux = crux.
        WithDirectory("/src", client.Host().Directory("web/crux/"), dagger.ContainerWithDirectoryOpts{
            Exclude: []string{"node_modules"},
        }).
        WithWorkdir("/src").
        WithServiceBinding("localhost", cruxPostgres).
        // WithEnvVariable("NOCACHE", time.Now().String()).
        WithExec([]string{"npm", "ci"}).
        WithExec([]string{"npm", "run", "build"}).
        WithExec([]string{"npm", "run", "prisma:migrate"}).
        WithExec([]string{"npm", "run", "start:prod"})

    _, err := crux.Stdout(ctx)
    if err != nil {
        panic(err)
    }

    return crux
}

We can run the above code in our main() like this:

    cruxEnv := getEnv("web/crux/.env") 
    cruxPostgres := getCruxPostgres(client, cruxEnv) 
    runCruxProd(ctx, client, cruxPostgres)

We would like to note that we made our POC with Dagger 0.8.x during September, so the code snippets above will show that. But even then the new API development of Dagger Services v2 (which we will need for our complex e2e pipeline) was in progress at Dagger in a separate feature branch and they promised on their Discord forum back then that this new API with some breaking changes will be included in Dagger 0.9. It wasn’t just us showing demand for parallel long running service containers - and they kept their word and it is indeed included in Dagger 0.9.0 released at the end of October. Shouts to Team Dagger!

We put our POC on hold in October, but we have been keeping an eye on Service v2 developments and news. We will try out Service v2 in the near future and dedicate another blog post to whether we managed to solve our entire e2e pipeline with Dagger.

Dagger efficiently caches each step of the pipelines, automatically handling the caching of source code copies, containers and builds, and when developers configure it programmatically, it also caches mounted volumes such as database data, node_modules, and Go build-cache. Our logs provide clear examples of this on reruns without code modifications.

    copy web/crux/ CACHED
    > in host.directory web/crux/
    …
    pull docker.io/library/postgres:14.2-alpine CACHED
    > in crux-postgres > from postgres:14.2-alpine
    > in crux > service bvqf991cmob5i.97ul8ph8qf1qc.dagger.local
    …
    exec docker-entrypoint.sh postgres
    > in crux > service bvqf991cmob5i.97ul8ph8qf1qc.dagger.local
    [0.15s] PostgreSQL Database directory appears to contain a database; Skipping initialization
    …
    [0.30s] 2023-11-08 10::11.131 UTC [15] LOG:  database system is ready to accept connections
    ...
    exec docker-entrypoint.sh npm run build CACHED
    > in crux
    exec docker-entrypoint.sh npm run prisma:migrate CACHED
    > in crux
    exec docker-entrypoint.sh npm ci CACHED
    > in crux
    copy / /src CACHED
    > in crux
    exec docker-entrypoint.sh npm run start:prod
    > in crux
    [0.57s] > crux@0.7.0 start:prod
    [0.57s] > node dist/main
    [2.31s] [Nest] 33  - 11/07/2023, 14:24:13.142 AM     LOG [NestFactory] Starting Nest application...
    ...

Challenges and Lessons Learned

We were able to run most of our stack with Dagger 0.8.x, the Crux backend and the Crux-UI frontend separately, but our entire e2e test will require Dagger 0.9.x with the Services v2 API that we can run Crux, Crux-ui, Traefik and Kratos as long running service containers for the Playwright e2e container.

If you want to know more about the Services v2, Dagger wrote a blog post about it here:

Dagger 0.9: Host-to-container, container-to-host, and other networking improvements: https://dagger.io/blog/dagger-0-9

Best Practices for Dagger CI/CD

The fact that we can write the CI/CD code in Go and in a docker-like style had a refreshing effect on us. Here are some general tips:

Iterate small: Start with a small POC to understand how Dagger fits into your workflow before scaling up
Community engagement: Stay active in Dagger's community forums or Discord channels for support and to keep up with the latest developments
Documentation: Keep your Dagger configurations well-documented to ease onboarding and maintenance
Monitor and optimize: Regularly review the performance of your pipelines and optimize caching strategies for better efficiency

Conclusion

We have seen firsthand the transformative nature of Dagger and the flexibility of its programmable pipelines. It stands out as a forward-thinking solution, addressing typical CI/CD bottlenecks with a developer-centric approach. Since Dagger is relatively new and evolving, keeping an eye on updates and community feedback can help in adopting best practices as they emerge.

Dagger Resources

There's still lot to learn about Dagger, so it might be worth the time to check out the following resources to learn about this tool:

You can explore further on Dagger's official website: https://dagger.io
For those eager to dive deeper into Dagger's capabilities, the Dagger documentation is an excellent resource: https://docs.dagger.io
For absolute hackers: https://github.com/dagger/dagger
Dagger Discord community: https://discord.gg/dagger-io

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.

Support us with a star on GitHub.

Why You Should Self-Host GitHub Runners – Or Stay Away from It

Geri Máté — Wed, 08 Nov 2023 12:23:24 +0000

GitHub Actions is the Alfred to your Batman. When you don’t feel like doing something or simply don’t have the capacity to handle various tasks, you can rely on GitHub Actions to automate workflows. You can take GitHub Actions to the next level by self-hosting runners, though. But should you? Let’s find out!

Why Self-Hosted GitHub Runners Are Beneficial

We’ve been managing dyrector.io’s code on GitHub for more than a year now. One thing we’ve always struggled with was slow GitHub Actions workflows. Here’s why we’ve been contemplating switching to self-hosting our GitHub Runners.

Speed

The default GitHub runner takes longer to execute as it initializes an ephemeral runner for each job in a workflow from scratch. This method, chosen by GitHub for its simplicity and security, has its merits. Compared to this, self-hosted runners remain active, bypassing the initialization phase for every job, thus providing quicker execution. This continuous availability demands proper management to ensure subsequent runs are not interfered with by remnants from previous executions.

Control

GitHub-hosted runners run on Ubuntu with a 2-core CPU, limiting parallel job executions to four. In a self-hosted scenario, we have the liberty to choose other OSes. We opted for Rocky Linux over Ubuntu for its open-source, enterprise-grade, and 100% Red Hat compatibility. This choice also allowed us to define the VM's hardware parameters like CPU, memory and disk type/size. However, this freedom comes at the cost of increased maintenance overhead.

Debugging / Monitoring

Debugging is more challenging on GitHub-hosted runners as only error messages and logs are retained. In the meantime, self-hosted runners keep everything in the “_work” and “_diag” directories, allowing real-time monitoring to understand precisely what is happening and the resources being consumed, as the running VM is under our control.

As we look into the future and explore opportunities for further improvement:

Writing in YAML, especially for CI/CD purposes, often necessitates additional scripts to handle various build and runtime conditions in a workflow. This can result in a fragmented view of the process.
Alternatively, or in addition, leveraging the power of Dagger CI/CD could offer a more streamlined approach to creating workflows. Dagger CI/CD allows you to use real programming languages through the Dagger SDK.
For example, we have chosen to use the Dagger Go SDK, which enables the creation of unified workflows. These workflows can run seamlessly, whether it's locally, on GitHub-hosted runners, self-hosted runners, or other CI/CD frameworks, with minimal or no need for significant modifications. This approach entirely avoids the need for extensive YAML configurations, providing a more efficient and flexible way to manage your CI/CD pipelines.

Few Reasons Why You Shouldn’t Self-Host Runners

Convenience

The default GitHub hosted runner functionality is free and comes with autoscaling if we look at the submitted parallel pull requests, so you don't have to do anything for them, they are simply there and doing their job. We obviously lose this default behavior if we go on the self-hosted route.

Setup/Maintenance

The initial setup requires a learning curve, and maintaining the runners can demand a fair share of time. It is not so much the setting of the runners themselves, but rather the maintenance, updating, securing the VM(s) and the correct initial setting of the workflow to manage the clean up and teardown side steps for every job and job step, if necessary.

Security Concerns

Self-hosted runners may expose your environment to potential security risks if not configured and managed properly. Something even GitHub recommends in its official docs is to use self-hosted runners with private repositories. Here's a more detailed description about security measures for GitHub runners.

Setting Up Self-Hosted Runners

Ensure your system meets GitHub's minimum requirements, which include:

2-core CPU
7 GB RAM
14 GB SSD storage

We used a larger machine with the following specifications:

16 vCPUs
32 GiB memory
Initially, 16 GB SSD (later upgraded to 64 GB)

The upgrade was necessary due to the combined temporary space needs of our code, node.js, and about 10 docker containers, including a playwright container for testing. Our runners resided on an additional data disk, leaving about 8 GB free on the system disk.

Instead of using multiple small VMs with one runner each, we chose to use one large VM hosting several parallel runners. This approach minimizes VM maintenance overhead and is designed to efficiently handle multiple parallel GitHub pull requests.

Future scaling is straightforward as setting up additional runners and/or VMs is not complicated; runners distribute workflow jobs based on common labels regardless of their VM location.

We set up our self-hosted runners with these steps, here we will show actions-runner-001, but it was done in a similar way for our runners 002, 003 and so on.

Step 1: Create a new runner

At your GitHub repository’s Settings, in the left sidebar click Actions, then click Runners and finally click New self-hosted runner. Select the OS image and architecture of your self-hosted runner machine. In our case it is Linux and x64.

Step 2: Download the runner installer

# Create a folder
$ mkdir actions-runner-001 && cd actions-runner-001
# Download the latest runner package
$ curl -o actions-runner-linux-x64-2.311.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
# Optional: Validate the hash
# On Rocky Linux you may need to install shasum once for this validation
$ sudo dnf update
$ sudo dnf install -y perl-Digest-SHA
$ echo "29fc8cf2dab4c195bb147384e7e2c94cfd4d4022c793b346a6175435265aa278  actions-runner-linux-x64-2.311.0.tar.gz" | shasum -a 256 –c
# Extract the installer
$ tar xzf ./actions-runner-linux-x64-2.311.0.tar.gz

Step 3: Install runner dependencies (if needed)

We only need to do this step once per VM, not per runner. You can skip this step if your OS already contains these dependencies, but for Rocky Linux 9.2 it was necessary.

# Install dependencies (on Rocky Linux dotnet core 6 was missing by default) 
$ sudo ./bin/installdependencies.sh

We also installed node.js, go and docker on our VM for our workflow, but these are not runner dependencies, so we will not go into detail about that here.

Step 4: Configure the runner

# Create the runner and configure it
$ ./config.sh --url https://github.com/dyrector-io/dyrectorio --token <RUNNER_TOKEN>

During the configuration process, you can keep most settings at their default values, but we chose to make our runners easily identifiable by giving them unique names and adding extra labels. Initially, the configuration script provides a common name, but our objective was to test multiple runners on a single VM.

By default, a runner is tagged with three labels for Linux x64: self-hosted, Linux, and X64. However, you have the flexibility to specify additional labels during the initial configuration or later on the GitHub repository website. Unlike the default labels, you can add or remove these custom labels at any time. These labels come in handy for targeting specific groups of runners or individual runners within your workflow.

Step 5: Set pre-job script

Pre-job script is not mandatory if you do not want to use it, but we need it.
In the runner directory just create a .env file with this content:

ACTIONS_RUNNER_HOOK_JOB_STARTED=pre-job-script.sh

And in the pre-job bash script file you can use your additional VM specific logic which will run before every job. Important to write “exit 0” at the end of the script file, because this means the script run without errors – otherwise or if you return any other value the runner will skip this job. You can also use this to your advantage for pre checks.

Step 6: Start the runner

You can start the runner with its run script ($ ./run.sh), but we want to run it as a service so first need to install the service and on Rocky Linux we also need to set the SELinux security context for the runsvc.sh file to ensure it operates correctly within the SELinux security policy (otherwise it will be blocked). We only need to set SELinux context and service install once.

# Set SELinux context for the runsvc script to s0 (standard security level)
$ sudo chcon system_u:object_r:usr_t:s0 runsvc.sh
# Install the runner as a service
$ sudo ./svc.sh install

Now we can use the service with its start, stop, status commands.

# Start the runner service
$ sudo ./svc.sh start

After completing these steps, the runner and its status are now listed under "Runners" of the GitHub repository.

Step 7: Execute a workflow on self-hosted runners

In your workflow file, use the following YAML for each job, adjusting the label(s) as per your runner configuration:

runs-on: self-hosted

Security Tips

Additional security measures for our public open-source repository:

We use CODEOWNERS file for our repository.
In the repository settings, we have the "Require approval for all outside collaborators" option enabled instead of the default "Require approval for first-time contributors".
Before allowing any external pull requests to run, we check if any workflow files have been modified! (It is easy to spot if anything appears in .github/workflows, without much approval overhead)
We use our self-hosted GitHub runner with an isolated Azure VM in its own resource group.
We take care of updating the runner VM's OS to ensure it is always up to date from a security perspective.
We run external pull requests on a GitHub runner, while we run our own pull requests on our self-hosted runner. This is determined by a necessary pre-job in our workflows, based on the submitter's identity, assigning the appropriate "runs-on" label to the subsequent jobs.
In the runner's “_diag“ and “_work“ directories, we can review diagnostic logs for both the workflow runs and the runner itself, as well as the checked-out code in the "workflows private" directory."

Conclusion

Self-hosted GitHub runners offer more freedom and level of control that can significantly boost the efficiency of your development workflow. However, they come with the overhead of setup, maintenance, and potential security concerns. Assessing your project’s needs and your team’s capacity to manage self-hosted runners is crucial before diving in. With proper setup and management, self-hosted runners can indeed be a valuable asset to your development process.

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.

Support us with a star on GitHub.

Easily accessible docker logs & inspect with darklens

Geri Máté — Fri, 20 Oct 2023 11:46:22 +0000

darklens is an open-source lightweight container viewer. In other words, darklens is a tool you can use to check out your containers across multiple environments without opening dozens of terminals and digging up SSH keys.

The never-ending pain of endless terminal windows

Have you felt the confusion when you have at least 4 terminal windows in front of you and after 5 minutes you have to double check what’s going on in either of them? Well, the guy who accidentally deleted GitLab’s database might know a thing or two about that.

Hold on, we have no personal feelings against terminals. They’re cool until the point when you have terminal window blindness. We kind of like how you can just type npm i and things work out. To be able to use darklens, you’ll actually need a shell. Thought we’d mention that part to put our love and hate relationship into perspective about terminal windows. But when an environment – or multiple environments! – hogs a terminal window, that’s the opposite of cool.

To prevent this phenomenon, and possibly fatal database deletions, we came up with the idea to deliver a centralized UI for your containerized stack. Whether you have one home server, or have ten or twenty VMs, you can add them all to darklens and check out what’s going on. Is your self-hosted Minecraft server having problems? Cool, let's just give darklens a quick look to see what’s happening.

Why use JSON when a GUI will do?

Be honest: when was the last time you looked at a JSON and had complete understanding of the situation? Maybe it’s just us, but when it comes to container configs, we prefer to take a glance at settings in an organized way. Standards are fine, but that doesn’t mean configs need to come exclusively in JSON format.

That’s why we made a table view for container configs. Essentially, what you’d see in a terminal window after sending a docker inspect command, you’d see the same data in darklens.

Agent installation is like a breeze

darklens might remind you of a couple of already existing tools. We tried some of them, and we faced problems getting on-board time to time. These tools are certainly popular for a reason. We had two thoughts about this, however:

When on-boarding becomes complex for an abstraction, such as darklens, the whole thing might fail at the beginning.
It doesn’t take too much to improve the user experience to spare users the trouble of setting up ports and other things.

That's why we came up with the idea to offer an easy to install agent. You can install the agent with 3 easy steps:

name the environment where you’ll set the agent up,
generate a one-liner script,
run it in a shell,

and you’re good to go.

Based on the feedback of a couple of folks who we asked, our agent is good to use because it just works. Therefore, the whole process facilitates towards a smooth user experience, so mission accomplished.

Get started with darklens

Follow these two simple steps to set up darklens:

Enter docker run -p 8000:8000 -d ghcr.io/dyrector-io/darklens:latest in terminal
Open localhost:8000 in browser

Check out darklens on GitHub

Give the tool a closer look on GitHub:

https://github.com/dyrector-io/darklens/

If you bump into anomalies, feel free to create an issue, or reach out to us on Discord.

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.

Support us with a star on GitHub.

What have you accomplished during this Hacktoberfest so far?

Geri Máté — Fri, 06 Oct 2023 12:35:50 +0000

Show off your accomplishments in the comments!

If you're a maintainer, give credit to your contributors below!

What have you accomplished during this Hacktoberfest so far?

Geri Máté — Fri, 06 Oct 2023 12:35:50 +0000

Show off your accomplishments in the comments!

If you're a maintainer, give credit to your contributors below!

Hacktoberfest Is the Ultimate Developer Experience Lesson

Geri Máté — Mon, 02 Oct 2023 07:30:00 +0000

Helping developers to simply deploy any applications they develop to their desired environment is our core mission. It was a no-brainer for us to participate in last year's Hacktoberfest as a project to the key open-source event of the year when we were able to engage with other developers. It'll be helpful for us to provide the best developer experience to our users in the long run.

Open-source software changes lives. It was the same to our community this Hacktoberfest, too. We hardly imagined contributors would come to us but to our biggest surprise, we doubled the stars on our GitHub repository and the feedback is clear: the community loves dyrector.io.

Support is Key for High Developer-Experience

We talked to some of our 11 contributors who made 19 commits in total. This might not sound like a lot, but as a project at the starting line, we weren’t expecting any of it. We made issues mainly related to Kubernetes, Golang and Typescript. Our goal was to help first time contributors and be as supportive as we could be with the people who chose to devote their free time to work on improving the platform.

One of our contributors, Ritvik highlighted our helpful approach to contributors. “I really liked how helpful you guys were during the entire time. I honestly had no idea of Docker or Golang before contributing but I somehow managed to do the required task with your help”, he said. “I was really worried that you guys would doubt my competency judging by the kind of work I did but it all worked out just fine, I guess", he added explaining he participated in Hacktoberfest while his university exams were going on.

“I'm glad to have contributed in this ambitious project", said Kit, another community member of ours.

“With modern tech stacks and continuously improving documentations, the developer experience is enjoyable.

You guys created a very friendly environment where help is prompt and with patience. I believe more and more contributors will be attracted here after this Hacktoberfest. I'm also looking forward to making more contributions in the future", he explained.

Based on this feedback, our advice for other open-source maintainers is to be supportive and appreciative of the efforts of others. You never know what difficulties others face in relation to your project or outside of it. Going out of your way to help people to help you will leave a positive impression and increase the chance of them coming back to the project.

Ways to Support Open-Source Contributors

Our main channel to promote the project was the official Hacktoberfest server on Discord. It was the easiest way to point contributors to our server, where we could directly engage with them. We also made a channel on our server for them where they could ask questions or notify us about issues. At the same time, we promoted the issues of our GitHub repo on Twitter and other channels.

There’s Always Room for Improvement

While the feedback and the experience of our first Hacktoberfest was positive, we still found things we can improve.

Guide contributors who need it: One of the main problems we experienced with less experienced contributors is that they didn’t sign their commits. We shouldn’t assume that everyone understands the importance of commit signatures.
Issues with more context: We made sure the issue descriptions are as detailed as possible, but similar to commit signatures, we shouldn’t expect users to have the same context as we, who work on the project every day have. More context is required in the future. It might be useful to break down larger issues into smaller pieces to help contributors.

Why Do People Participate in Hacktoberfest?

One major talking point of our Hacktoberfest retrospective was when we discussed how we should treat contributors. For more context, it’s a more nuanced topic because the main incentive of Hacktoberfest is to get 4 pull requests merged to win the swag or the opportunity to plant a tree.

There was one issue which was stalled for more than 10 days. We reached out to the contributor to see what’s up, they replied and unassigned the issue from themselves without completing it. We assume one of the two following scenarios might have taken place:

We came across as a-holes: While our intention was to ask the person assigned to the issue if we can help, we might have come across as pushy to them unintentionally. We understand most contributors hack things in their free time, so we can’t expect prompt or on-demand activity from them.
They no longer needed the issue: The incentives of Hacktoberfest make some contributors take on more issues than they'll need to complete the challenge, which they’ll abandon once they reach the required number of PRs merged. They forgot about this particular issue and didn’t come back to unassign it from themselves.

While the ticket itself wasn’t urgent or that important, stalling it took the opportunity from other contributors to complete it. This might have been the difference for someone from completing Hacktoberfest. We can’t be sure but there’s a chance. Our lesson to learn from this is to fine-tune our communication with contributors based on the importance of the ticket they get assigned to so we can make sure it’s not our mistake if we lose a contributor.

A very interesting situation was when one contributor went missing and another one stepped forward asking if they could work on the issue instead. So, we can see how the community itself can resolve this type of situation, but we’d rather avoid this.

Another thing we were aware of, but it was also fascinating to see happen is how some contributors take the absolute least effort to commit something. An example of that was when someone opened a PR for adding a period to a readme file. We did have a good chuckle when we witnessed it.

Starcount After Hacktoberfest

We went open source in Summer 2022. Halfway through Hacktoberfest we passed 100 stars on our main repository on GitHub. It's fair to say that Hacktoberfest is a great way to bring attention to your open-source product. If you'd like to get a boost for stars on your repository, check out How To Get GitHub Stars, a weekly newsletter for tricks and tips.

Summary

Hacktoberfest was an absolute banger for our community. We doubled the fans of the platform on GitHub and made some new friends. In fact, we enjoyed Hacktoberfest so much we’re considering ways to make long-term open-source incentives around the project in the future.

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.

Find the project on GitHub.

Go Hacktoberfest Issues

Geri Máté — Thu, 28 Sep 2023 13:05:54 +0000

We rolled out our first batch of issues on GitHub. And we offer swag to the best contributors.

Our issues are related to the following topics:

Golang
Node.js
NestJS
Typescript
Prisma

Check out the repository at the link below:

https://github.com/dyrector-io/dyrectorio

More issues are in the works.

Last year we sent one of our contributors a swag pack with a hat and a bunch of stickers, as seen below. This year we plan to ship to certain parts of Europe (we're located in Hungary) but we're happy to meet up with our contributors in case they visit KubeCon Europe, Web Summit and other conferences we usually attend and give you your swag pack in person.

Product Hunt Is for the Cool Kids

Geri Máté — Mon, 25 Sep 2023 12:11:18 +0000

We launched again on Product Hunt. While we did fine, we had to come to the realization that we’re not fit for Product Hunt but it’s totally fine. But if you feel like you shouldn’t launch on Product Hunt, what are your options?

We launched for the 2nd time on Product Hunt this month. The results were fine: with a relatively niche product we finished 6th with around 260 upvotes. Amongst the open-source projects that week, our launch was one of the most successful.

As a result, we gained a few sign-ups, the traffic on our website was one of the highest of all time, some users gave our GitHub repository a look and starred it, too. As you can see below, our website traffic spiked for a 10x number compared to the day before, and GitHub traffic went 5x.

It’s worth noting that because of Product Hunt’s mechanics, we accidentally had our product’s log in site as the Visit link for 6 hours into launch day, so we missed out on some traffic. My hypothesis is that we experienced this increase because it was probably our most active day on social media, but we definitely weren’t posting this much since the Web Summit in 2021 (which we’ll attend again this year).

Anyway, we consider the launch a success as it made some publicity for our platform, however, we could’ve done better. But the bittersweet reality is that we didn’t really have a chance.

Product Hunt Became a Cheap Thrill

Back in February, when I connected folks for our first launch on LinkedIn, I thought these connections would stay somewhat relevant in the future. Obviously, this couldn’t be any further from the truth.

Similar to the majority of products that are on Product Hunt, the connections that come from the platform are so superficial that after 5 minutes it won’t matter at all. On the flipside of this, there are a couple of people who I’ve met because of Product Hunt, and they seem genuine.

The result of these superficial connections was that the majority of folks who supported us in February didn’t bother responding, let alone upvoting us. I didn’t count but I’d estimate that their support could’ve landed us in the top 4 on Friday.

Product Hunt Lost Its Charm

The prestige of Product Hunt is quickly fading. Once upon a time Product Hunt was the place where you could meet innovators trying to make their mark on how you’re getting things done and deliver value.

The “community” Product Hunt has to offer picks up and puts down products like a kid does with toys in recess. I saw a crypto project that earned a Web3 project of the year badge in 2021, launched again this year for a daily #6 and weekly #82 spot. The new cool kid is generative AI and Notion until the next cool kid shows up, even though, it’s hard to see the value most of these projects offer.

Now you can barely find anything on the platform that’s not a GPT wrapper or a Notion related project that even the maker doesn’t care about after launch day. I’ve seen makers who essentially don’t work on anything else than GPT wrappers, launch every other week and move on to the next wrapper. Not surprisingly, these makers post discussions that are filled with so many buzzwords and copywriting trigger words that you might need ChatGPT to decipher to a human language.

Your best bet on Product Hunt is with a product aimed at a general audience or a few niches that tend to utilize automation before anyone else – such as sales. You’ll have to spend a few weeks before launch connecting with as many people as you can because your competition will be hungry in this area. Ideally these people fit into your ICP. Plus, your messaging needs to be on point.

Where Else Can You Showcase Your Product?

For general B2C tools, I’m not sure. Maybe Show HN is a good place, but your stuff needs to slap to gain any attention from the Hacker News community and you pretty much have to nail everything. Most importantly: when you make it on Show HN, you know you’re on to something with your stuff.

For developer tools, such as dyrector.io, DevHunt seems like a promising platform. It’s still early on the journey to make a global impact. But they recently launched on Product Hunt and finished #1 on the daily vote, meaning there's demand for actual products. By the way, we were developer tool of the week on DevHunt last month, and if you’re curious about up-and-coming developer tools, you should check them out. In the meantime, I launched the weekly series of DevHunt Digest where I share my thoughts about the products launching on the platform. Check it out on Dev.to for the 1st part of the series.

Support Us With a Star on Github

Launch day is over for a couple days now. We appreciate the support of everyone who showed up. If you haven’t, you can still support us by giving a star on GitHub at the link below:

https://github.com/dyrector-io/dyrectorio

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.

Support us with a star on GitHub.

How to Automate Deployments with GitHub Actions

Geri Máté — Wed, 26 Jul 2023 10:21:58 +0000

GitHub Actions is one of the most popular ways to cut waste from your processes when you maintain software. It’ll take care of mundane, repetitive tasks of integration and delivery while you take care of some other business. Here’s how you can extend your GitHub Actions pipeline with automated deployments to any environment you use.

How GitHub Actions Works?

GitHub Actions is a platform that automates developer workflows. It’s popular as it’s modular, community focused, and reusable, however some might not be a fan of it being too verbose, atomic, and untestable locally. One of the most common use cases of GitHub Actions is to create CI/CD pipelines, as it’s able to automate all the tasks of building a container or pushing an image to a registry.

Some CI/CD pipelines stop there but others prefer to deploy the new image for their users right away in a continuous manner. GitHub has preset deployment workflows available in the Actions marketplace, but those are cloud provider specific ones.

Create Your Continuous Deployment Workflow

For the purpose of this blog post, we’re going to use dyrector.io, which is an open-source continuous delivery platform that you can use to deploy applications to all cloud providers and on-premises infrastructures.

Generate Your Continuous Deployment Token

Step 1: Install agent

After creating an account on dyrector.io, install the platform's agent where you’d like to continuously deploy your application. You can do so by generating a Shell or a PowerShell install script depending on your OS. The script will take care of making a connection between the platform and your infrastructure. The agent – written in Go – will run as a Docker container. You can set up the agent by following these steps.

Step 2: Add your registry

When your infrastructure is ready for deployments, which is signaled by node status turning green, navigate to registries and add your registry. Any V2 compatible registry can be added, even locally stored images as an unchecked registry. Check the docs for more info.

Step 3: Create a project out of your images

The next step will be to create the deployable stack, a project on the platform. You can do so by navigating to the project page. The new project needs to be a versioned one. When it’s done, add a new rolling version to the project, and click the Add images button to browse added registries for your images. The whole process is documented here.

Step 4: Create the CD token

When you have all the images added, move forward by adding a deployment, where you can create a continuous deployment token.

Token creation goes as below:

Click on the Create token button.
Enter a name for your token and click the Create button.
The platform will generate the token and the curl command. Make sure you copy and paste it somewhere secure because you won’t be able to retrieve it otherwise.

Add the curl Command to GitHub Actions

The last step will be pretty simple. Take the curl and create a job for it in GitHub Actions. The result should look like the example below:

name: "github actions example" 

on: 
  push: 
    tags: 
      - 'latest' 

jobs: 
  run-updater: 
    runs-on: ubuntu-latest 
    steps: 
    - name: continuous deployment 
      run: | 
        curl -X POST -H 'Authorization: Bearer ${YOUR_JWT_TOKEN}' https://app.dyrectorio.com/api/deployments/${YOUR_DEPLOYMENT_ID}/start

The tags in the example refers to when the pipeline pushes an image with the latest tag to the container registry. This event will run the curl command, which we named continuous deployment.

YOUR_JWT_TOKEN refers to the token generated on the platform, and it should be used as a secret in your pipeline to keep it secure. YOUR_DEPLOYMENT_ID is the deployment ID in the platform, and you can use it as is because it can't be utilized without the JWT token.

How Does Continuous Deployment Work?

As you can see in the flowchart below, dyrector.io’s continuous deployment functionality fits into a schematic CI/CD workflow on GitHub Actions. The pipeline triggers continuous deployment on the platform when the image is pushed to the registry.

The purple area in the flowchart represents a GitHub Actions pipeline which builds the container and pushes the image to a repository. Then it triggers the platform to execute the deployment by signaling the node to pull and start the container with the tag which already exists on the node.

Why Is It Worth It?

Short answer: you’ll save a lot of time.

As already mentioned, it’s a lot more flexible solution compared to GitHub Actions marketplace alternatives as it’s a technology and cloud provider agnostic way for deployments.

One of the key factors why it’ll save you time is that you won’t need to work with SSH keys. Other solutions, like GitLab, will be a slower process because you’re going to need SSH keys as shown in this tutorial.

When you work with multiple environments, as in you’d like to deploy the application to different servers, you won’t need multiple GitHub Actions pipelines. Instead, one will trigger the deployment to all the environments you want to deploy your software.

In case you need to adjust some configuration variables, like exposing a port or adding a new environment variable, you can do so without touching any of the JSON or YAML files and make the changes via the platform.

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source container management platform.

Find the project on GitHub.

Portainer vs. dyrector.io: Which one is for you?

Geri Máté — Thu, 13 Jul 2023 13:31:54 +0000

Portainer and dyrector.io are both container management platforms with some overlapping scopes and capabilities. But there are some important differences.

Both platforms make life easier for Docker and Kubernetes users but they’re for different use cases.

Portainer identifies itself as a universal container management platform that doesn’t lock users into a single technology or vendor. dyrector.io establishes itself as an open-source continuous deployment platform.

For the purpose of this blog post, we compared the free edition of both open-source platforms. Portainer offers a wide range of functionalities without limitations in their platform’s Community Edition (CE), while packaging up some premium features in their Business Edition not available in CE. At the same time, self-managed dyrector.io comes with no restrictions, as all features are 100% open-source.

What’s Common Between the Two

The cool thing about both is that they aim to be technology and vendor agnostic. This means that both platforms can be useful when you work with Docker or K8s, and you can deploy containerized apps to all kinds of providers, even on-premises environments. This is a useful trait for users who already have their infrastructure set, or teams who still have the infrastructure design part ahead of them and would prefer to pick the best provider for their solutions.

Another common trait between the two is that they offer CD capabilities. Portainer offers both polling and webhook for this purpose, while dyrector.io operates with webhooks. One more similarity they share is that both provide API to their users, so they’re able to customize Portainer and dyrector.io’s functionalities to their purposes.

Besides, both of them offer preset applications as templates which you can deploy with little to no configuration.

How They Differ

Pros to Portainer

At first glance, Portainer seems like a Docker GUI. Portainer lists all kinds of information about not only the containers, but the images and the volumes stored on the environment where Portainer is hosted. Extended Docker capabilities include the management of Docker Swarm, as well, something that dyrector.io doesn’t deal with.

Besides the GUI, Portainer has API capabilities for Docker management and provisioning. Essentially, Portainer, as a more mature product, has a wide range of integrations. Teams in Portainer enable users to collaborate when they manage their containers.

Portainer Business Edition provides S3 compatible configuration backups and host management, which might be useful to users who don’t feel like working with SSH keys.

Pros to dyrector.io

Compared to Portainer, dyrector.io acts as a GUI where you can configure and manage versions of your containerized applications. For example, when you develop an application that uses a PostgreSQL database, you can configure both images as a deployable stack. The platform is designed to support de facto release management practices, as it offers rolling and incremental version types, rollbacks, and tracks the deployment statuses of said versions.

The configuration management capabilities provide deep level customization with Docker and Kubernetes specific options, such as ingress variables. Configuration backups are built-in, as the platform saves them automatically. Contrary to Portainer, dyrector.io doesn’t support Docker Swarm.

As an emerging platform, a long list of integrations includes chat notifications in Slack, Teams, and Discord, where users can stay up to date on the recent activities of their teammates. The platform provides file injection from S3 buckets.

Similar to Portainer, you’re able to gain information about the runtimes on your infrastructure, inspect and log them through the platform. Therefore, the platform offers a shortcut to users who need a quick look at their containers without SSH access.

Multi-instance deployment is another interesting use case of dyrector.io. It’s useful when you have dozens or hundreds of deployment targets where you’d like to set up your containers. Instead of manually going through the deployments, you can select them and trigger the deployment to all the targets at the same time.

Agent Comparison

Both platform’s agent setup workflows are similar at the beginning. You’ll have to run a script that’s going to install the agent on the eventual deployment target. Portainer works with a general script that’s only responsible for the agent’s runtime, you’ll have to manually enter your environment’s IP address and port. On the contrary, authorization is baked into dyrector.io agent’s install script, as the platform generates a unique script for a one-time setup.

Due to this difference, we were unable to connect Portainer’s agent to our instance of the platform. It was even more problematic as we didn’t receive a meaningful error message to solve the problem, no matter what we did to firewall rules on said environments. In the meantime, we were able to spark up dyrector.io’s agent to multiple VPS providers with ease.

Conclusion

Portainer acts like an in-depth Docker GUI that leaves room for lots of Docker specific capabilities, including GitOps integrations to facilitate to CD pipelines. The biggest difference between Portainer and dyrector.io is the latter’s ability to manage and deploy versions with several capabilities designed to serve collaboration. Portainer offers detailed Docker and infrastructure management functionality, however, dyrector.io focuses on release management instead.

This blogpost was written by the team of dyrector.io. dyrector.io is an open-source container management platform.

Find the project on GitHub.