Forem: Ivan Filippov

Rufler: One YAML File to Rule Your Autonomous AI Swarm

Ivan Filippov — Wed, 15 Apr 2026 16:55:11 +0000

The Problem: Too Many Commands, Not Enough Orchestration

If you've worked with ruflo (the TypeScript AI agent orchestration CLI from the claude-flow family), you know it's powerful. It gives you swarm topologies, hive-mind coordination, shared memory, autopilot loops, and the ability to spawn Claude Code as an autonomous agent.

But actually using it on a project looks something like this:

ruflo init --force
ruflo daemon start
ruflo memory init --backend=hybrid
ruflo swarm init --topology=hierarchical --max-agents=8 --strategy=specialized
ruflo hive-mind init --topology=hierarchical --consensus=raft --max-agents=8
ruflo autopilot config --max-iterations=100 --timeout=180
ruflo autopilot enable
ruflo hive-mind spawn --count=4 --role=specialist --claude \
  --non-interactive --dangerously-skip-permissions=true \
  --objective="$(cat my_assembled_prompt.txt)"

That's 8 commands just to get started. And the objective prompt? You have to assemble it by hand -- project context, task description, every agent's role and prompt, autonomy instructions, memory checkpointing rules.

Do this across multiple projects, add multi-task workflows, background runs, log management, and graceful teardown -- and you're writing a lot of shell scripts.

Enter Rufler

Rufler is a Python wrapper that replaces all of the above with:

rufler run

One command. One YAML file. The entire ruflo pipeline runs automatically.

Here's what a rufler_flow.yml looks like:

project:
  name: my-api
  description: Go microservice with REST API and PostgreSQL.

task:
  main: |
    Build a Go HTTP server on :8080 with CRUD endpoints,
    PostgreSQL storage, and integration tests.
  autonomous: true

agents:
  - name: architect
    type: system-architect
    role: specialist
    seniority: lead
    prompt: Design the API schema and package layout.

  - name: coder
    type: coder
    role: worker
    seniority: senior
    prompt: Implement the server following the architect's design.
    depends_on: [architect]

  - name: reviewer
    type: reviewer
    role: specialist
    seniority: senior
    prompt: Review all code for quality and security.
    depends_on: [coder]

Rufler reads this, composes a structured objective prompt (agents sorted by seniority, with GATE/HANDOFF blocks for depends_on), and runs the full ruflo pipeline.

What Rufler Adds Over Raw Ruflo

Raw ruflo	With rufler
8+ shell commands to boot	`rufler run` (one command)
Assemble objective prompts by hand	Auto-composed from YAML
No multi-task support	Sequential or parallel task groups
No task decomposition	AI-powered decomposition (`task.decompose: true`)
Mixed ANSI/JSON/text logs	Clean NDJSON -- every line is parseable JSON
No way to see live progress	`rufler follow` -- 4-panel TUI dashboard
No run history	Docker-style registry: `rufler ps`, `rufler tasks`, `rufler tokens`
No resume after interruption	Automatic resume from last completed task
Manual MCP server setup	Declare MCP servers in YAML

In short: ruflo is the engine; rufler is the ignition, dashboard, and cruise control.

Features Worth Highlighting

Docker-Style Run Lifecycle

Every rufler run gets an 8-char hex id. You can list runs with rufler ps, inspect tasks with rufler tasks, check tokens with rufler tokens, and stop gracefully with rufler stop. Works from any directory.

rufler run -d          # detach (like docker run -d)
rufler ps              # list running
rufler follow a1b2     # live TUI for run a1b2
rufler stop a1b2       # graceful shutdown

Task Tracking and Resume

Each task gets a sub-id (a1b2c3d4.01) with a tracked lifecycle: queued -> running -> exited / failed / stopped. Token usage is counted per task.

When you stop a run mid-way and restart, rufler automatically resumes from the first incomplete task:

rufler run -d        # tasks 1-2 complete, stopped on task 3
rufler stop
rufler run -d        # resumes from task 3
# => "resuming: skipping 2 completed tasks, starting from task_3"

AI-decomposed task files are reused too -- no wasted claude calls on re-decomposition. Use --new to force a clean start.

Live TUI Dashboard

rufler follow renders a 4-panel Rich dashboard that tails all per-task logs:

+-----------------------------------------------------------------+
| rufler follow  [running]  claude-opus-4  tokens: 94.6K  00:04:23|
+-----------------------------------------------------------------+
| Tasks  2/4            | Session                                 |
| > task_1     done     | model: claude-opus-4-6                  |
| > task_2     running  | tokens: in=98  out=829                  |
|   task_3     queued   | turns: 14                               |
|   task_4     queued   | last tool: Write                        |
+-----------------------+-----------------------------------------+
| Conversation (task_2)                                           |
| 14:23  think  Planning the API routes...                        |
| 14:24  tool   Write(src/api/routes.go)                          |
| 14:24  result File created successfully                         |
+-----------------------------------------------------------------+
| Log                                                             |
| 14:22  OK    task_end task_1                                    |
| 14:23  INFO  session init (claude-opus-4-6)                     |
+-----------------------------------------------------------------+

You see the AI's thinking, text output, tool calls, and results in real time.

MCP Server Management

Declare MCP servers directly in the YAML -- rufler registers them with Claude Code via claude mcp add:

mcp:
  servers:
    - name: my-db
      command: npx
      args: ["-y", "@anthropic/mcp-postgres"]
      env:
        DATABASE_URL: "postgresql://localhost/mydb"

    - name: sentry
      transport: http
      url: "https://mcp.sentry.dev/mcp"

Supports stdio, HTTP, and SSE transports with env vars and headers.

Agent Dependencies (Soft DAG)

Declare depends_on and rufler injects coordination prompts:

agents:
  - name: architect
    ...
  - name: coder
    depends_on: [architect]  # waits for architect's approval

The coder agent won't start until the architect writes an approval to shared memory. Cycles and self-dependencies are rejected at config load time.

Getting Started

# Install
git clone https://github.com/lib4u/rufler.git
cd rufler && pip install -e .

# Verify dependencies
rufler check

# Create a sample flow
rufler init

# Edit and run
$EDITOR rufler_flow.yml
rufler run --dry-run    # inspect the plan
rufler run -d           # launch detached
rufler follow           # watch it work

Prerequisites: Python 3.9+, Node.js 20+, Claude Code CLI (claude), and ruflo (resolved automatically via npx).

When to Use Rufler vs Raw Ruflo

Use rufler when:

You want a single config file checked into your repo
You're running multi-task workflows (sequential or parallel)
You need task-level tracking, resume, and token accounting
You want a live dashboard instead of raw log tailing
You're managing MCP servers across projects

Use ruflo directly when:

You need fine-grained control over individual ruflo commands
You're building custom orchestration scripts
You need features rufler doesn't wrap yet

Rufler doesn't replace ruflo -- it's a layer on top. Every ruflo command still works alongside rufler.

Repository: github.com/lib4u/rufler
Built on: ruflo by @ruvnet + Claude Code CLI by Anthropic
License: MIT

Golang with Colly: Use Random Fake User-Agents When Scraping

Ivan Filippov — Fri, 10 Jan 2025 16:13:28 +0000

One of the primary reasons for getting blocked while performing web scraping is the use of improper or default user-agents.

Fortunately, adding random fake user-agents to your Go Colly scrapers is straightforward and simple.

What Are Fake User-Agents?

User-agents are strings used by websites to identify the client making the request, providing information about the application, operating system (e.g., Windows, macOS, Linux), and browser (e.g., Chrome, Firefox, Safari) being used. These strings are sent to servers as part of the HTTP request headers.

For instance, here’s an example of a user-agent when accessing a website using Chrome on an Android device:

'User-Agent': 'Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36'

When scraping a website, it’s crucial to set a user-agent for each request. If you don’t, the website can detect non-human traffic and block your scraper.

By default, Go Colly uses this user-agent for its requests:

"User-Agent": "colly - https://github.com/gocolly/colly",

This default user-agent reveals that Colly is being used, making your scraper easy to detect and block. To avoid this, customizing the user-agent is essential.

That is why we need to manage the user-agents Go Colly sends with our requests.

How To Set A Fake User-Agent In Go Colly

Implementing a fake user-agent with Go Colly is a breeze. You can modify the request headers by setting a custom user-agent in the OnRequest() callback, ensuring each outgoing request uses a different or randomized string.


package main

import (
    "bytes"
    "log"
    "github.com/gocolly/colly"
)

func main() {
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Set Fake User Agent
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("User-Agent", "1 Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148")
    })

    // Print the Response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/headers five times
    for i := 0; i < 5; i++ {
        c.Visit("http://httpbin.org/headers")
    }
}

From here our scraper will use this user-agent for every request.

However, if you are scraping at scale then using the same user-agent for every request isn't best practice as it makes it easier for the website to detect you as a scraper.

To solve this problem we will need to configure our Go Colly scraper to use a random user-agent with every request.

How To Rotate Through Random User-Agents

For using the generated user agent, it is enough to use special packages, in the following examples we will use github.com/lib4u/fake-useragent

package main

import (
    "bytes"
    "log"
    "github.com/gocolly/colly"
    uaFake "github.com/lib4u/fake-useragent"
)

func main() {

    // Init user-agent faker
    ua, err := uaFake.New()
    if err != nil {
        fmt.Println(err)
    }
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Set Fake User Agent
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("User-Agent", ua.Filter().GetRandom())
    })

    // Print the Response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/headers five times
    for i := 0; i < 5; i++ {
        c.Visit("http://httpbin.org/headers")
    }
}

Now thanks to the fact that we added just a couple of lines of code, we can get the random user-agent.

However, using a simple random user-agent is not always enough; below we will look at options where we will use specific fake user agents.

The library github.com/lib4u/fake-useragent offers thousands of fake user agents, from the real world database.

// Get random user-agent in string
fmt.Println(ua.GetRandom())  // Mozilla/5.0 (iPhone; CPU iPhone OS 18_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Mobile/15E148 Safari/604.1

// Get user-agent string from a specific browser
fmt.Println(ua.Filter().Chrome().Get())
// Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36

fmt.Println(ua.Filter().Firefox().Get())
//Mozilla/5.0 (Android 14; Mobile; rv:133.0) Gecko/133.0 Firefox/133.0

fmt.Println(ua.Filter().Safari().Get())
//Mozilla/5.0 (iPhone; CPU iPhone OS 18_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Mobile/15E148 Safari/604.1

Now I’ll show you using the Go Colly library as an example.

package main

import (
    "bytes"
    "log"
    "github.com/gocolly/colly"
    uaFake "github.com/lib4u/fake-useragent"
)

func main() {

    // Init user-agent faker
    ua, err := uaFake.New()
    if err != nil {
        fmt.Println(err)
    }
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Set Fake User Agent
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("User-Agent", ua.Filter().Chrome().Platform(uaFake.Desktop).Get())
    })

    // Print the Response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/headers five times
    for i := 0; i < 5; i++ {
        c.Visit("http://httpbin.org/headers")
    }
}

This way we set up the generation of random user-agents for the desktop version of the browser Google Chrome.

For now every time we visit, the website will identify us as a random desktop user, thus a simple substitution of the user agent can help you when parsing websites, but do not forget about the use of proxies and other system headers.

https://github.com/lib4u/fake-useragent
https://github.com/gocolly/colly

Grequest is inspired by the Request library for Python for GO

Ivan Filippov — Mon, 06 Jan 2025 20:49:48 +0000

Simple and lightweight golang package for http requests. based on powerful net/http

Grequest is inspired by the Request library for Python and Guzzle in PHP, the goal is to make a simple and convenient library for making http requests in go

The library has a flexible API with methods that return a pointer to the library structure, which allows you to declaratively describe a request using a chain of methods.

Installation

go get github.com/lib4u/grequest

Usage

Simple get request anf get string response

req := app.Get("https://jsonplaceholder.typicode.com/todos/1").Do()
    response := req.Body().GetStrings()
    fmt.Println(response)

Get a response and writing it to a json structure

type AutoGenerated struct {
    UserID    int    `json:"userId"`
    ID        int    `json:"id"`
    Title     string `json:"title"`
    Completed bool   `json:"completed"`
}
...
   var myJsonResponse AutoGenerated
    req := app.Get("https://jsonplaceholder.typicode.com/todos/1").Do()
    err := req.Body().GetWithJsonStruct(&myJsonResponse)
    if err != nil {
        fmt.Println("do ..")
    }
    fmt.Println(myJsonResponse.Title)

Simple post request with json payload body and get status of response

data := LoginRequest{
        Username: "example",
        Password: "12345",
    }
    req := app.Post("https://example.site/login").Body().SetJson(data).Do()
    fmt.Println(req.Status().GetCode())

Simple save file from response

// file will saved as ../files/image.png
app.Get("https://example.com/image.png").Do().Body().SaveFile()
//OR
app.Get("https://example.com/image.png").Do().Body().Path("/user/files").SaveFile()
//OR 
app.Get("https://example.com/image.png").Do().Body().ToFile("path/savedimage.png")

Sending form with files and text fields and set header

req := app.Post("https://example.site/form/")
    req.Header().Set("Client", "number_1")
    form := req.FormData().WithMultipart()
    form.AddField("first_name", "John")
    form.AddField("last_name", "Doe")
    form.AddFile("photo", "my_photo.png")
    form.AddFile("documents", "example.txt")
    form.AddFile("documents", "example2.txt")
    form.Push()
    req.Do()
    .....

Request with authentication

//With basic auth
req := app.Post("https://example.site/secret)
    req.Header().Set("Client", "number_1")
    req.Auth().SetBasic("user", "password")
    req.Do()
    ...
    //Sets bearer token
    req := app.Post("https://example.site/secret)
    req.Header().Set("Client", "number_1")
    req.Auth().SetBearer("myToken")
    req.Do()

Work with cookie

//Save cookie to file 
//By default this saved in  cookies/example.site/cookies.json
req := app.Post("https://example.site/cookies")
    req.Cookie().Save()
    ...
    // Load saved cookies form cookies/example.site/cookies.json
    reqWithCookie := app.Post("https://example.site/cookies")
    reqWithCookie.Cookie().Load()
    reqWithCookie.Do()
    ...