Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

Hello everyone, I'm glad to see you. Today, I want to share my new experience with building Kafka HTTP Scanner, an HTTP vulnerability scanner that is fully developed in Go.

Initially, a security testing weekend challenge turned out to be a complete Command Line Interface Tool (lmao), something I am (half) proud to say I created.

🔍 Why Another Security Scanner?

With the rapid development of IT technology, we are surrounded by multiple security tools, so I ask myself and you: "Why?"

I have two answers:

• An opportunity to learn – I wanted to strengthen my understanding of Web security vulnerabilities as well as develop my competency in Go.
• Flexibility – I needed a tool that could be tailored for specific needs without going through over-complicated enterprise-grade tools.

Moreover, let us refuse to lie to one another, building security systems is quite an enjoyable task to partake.

⚡ Core Features That Make It Stand Out

The tool is capable of identifying a wide array of threats to security:

• HTTP Request Smuggling (this was fascinatingly difficult to integrate)
• SQL Injections
• Cross-Site Scripting (XSS)
• CORS Misconfigurations
• Security Headers Neglection
• SSL/TLS Vulnerable Areas
• And plenty more...

In addition to detection of vulnerabilities, I wanted it to be:

✅ Fast – Able to scan concurrently (Go’s goroutines served well for this function)

✅ Modular – Having an easily deployable architecture to introduce new vulnerability scan checks

✅ Friendly – With detailed instructions to fix the problems instead of just the pointers containing the problems

The Technical Architecture

Building Blocks

The project follows a clean, modular structure:

this diagram has been made with help of dirscanner

📂 cmd/              # CLI commands using Cobra
📂 internal/
  📂 core/           # Core scanner logic
  📂 http/           # Custom HTTP client
  📂 model/          # Data structures
  📂 modules/        # Vulnerability detection modules
  📂 output/         # Report formatting
📂 pkg/utils/        # Utility functions

The Module System

The heart of the scanner is its module system. Each vulnerability check is implemented as a separate module that implements a simple interface:

type ScanModule interface {
    Name() string
    Description() string
    Run(scanner Scanner) ([]model.Vulnerability, error)
}

This made it incredibly easy to add new vulnerability checks. Here's a simplified example of the XSS module:

func (m *XSSVulnerabilityModule) Run(s Scanner) ([]model.Vulnerability, error) {
    // Send requests with XSS payloads
    resp, err := s.SendHTTPRequest("GET", "/?q=<script>alert(1)</script>", nil, nil)

    // Check if the payload was reflected without encoding
    if strings.Contains(bodyStr, "<script>alert(1)</script>") {
        return []model.Vulnerability{{
            ID:      "XSS-01",
            Name:    "Reflected Cross-Site Scripting",
            Severity: model.SeverityHigh,
            // ...details, remediation info, etc.
        }}, nil
    }

    return nil, nil
}

Technical Challenges I Tackled

1. HTTP Request Smuggling Detection

Identifying HTTP request smuggling was surprisingly intricate. I first had to design synthetic HTTP requests that had mismatched and contradictory ‘Content-Length’ and ‘Transfer Encoding’ headers, then examine the responses from the server. This was an involved process that entailed me creating a custom raw HTTP client that had full control over the precise construction of the request:

func (c *Client) SendRawRequest(target model.Target, payload string) (string, error) {
    // Establish raw TCP or TLS connection
    if target.SSL {
        conn, err = tls.DialWithDialer(dialer, "tcp", address, tlsConfig)
    } else {
        conn, err = net.DialTimeout("tcp", address, timeout)
    }

    // Send the exact bytes we've crafted
    fmt.Fprintf(conn, payload)

    // Read the raw response
    // ...
}

2. Concurrency Control

Security scanning is perhaps the single most useful application for concurrency – the more security checks we can do in parallel, the better. Conversely, waiting for one task to finish before starting another is inefficient in this case. Unfortunately, unregulated parallelism can lead to a denial of service for the target server or invoke rate limits.

scanner.semaphore = make(chan struct{}, config.Concurrency)

// In the scan routine
s.semaphore <- struct{}{} // Acquire token
defer func() { <-s.semaphore }() // Release token

3. Robust Error Handling

When you're sending malformed requests and probing security boundaries, things break. A lot. Graceful error handling was critical to prevent the scanner from crashing mid-scan:

go func() {
    defer func() {
        if r := recover(); r != nil {
            s.logger.Errorf("Module %s panicked: %v", m.Name(), r)
            errorChan <- fmt.Errorf("module %s panicked: %v", m.Name(), r)
        }
    }()

    vulnerabilities, err := m.Run(s)
    // ...
}()

By wrapping module execution in a panic-recovering goroutine, a single misbehaving module can't bring down the entire scan.

Reflections

I learned much more than I initially planned from this project:

• Security vulnerabilities are nuanced – Successfully detecting them requires an understanding of both the techniques employed by the attacker and the server-side behaviors.

• Go's concurrency model shines for network tools – The parallel implementation of the scanning procedure was surprisingly easy, thanks to goroutine and channels.

• CLI UX matters – A well-designed interface at the command-line terminal along with properly formatted output improves the quality of the developer tools.

What's Next?

I have a lot of ideas to work on for Kafka HTTP Scanner:

• Active Exploitation Modules (with appropriate protections in place)
• CI/CD integration
• Custom reporting designs
• API fuzzing

Try It Out!

Feel free to test out Kafka HTTP Scanner if you want to dabble in the world of security testing, or if you are looking for a Go-based command-line tool.

git clone https://github.com/aymaneallaoui/kafka-http-scanner.git
cd kafka-http-scanner
go build -o httpscan
Copy
To execute a basic scan:

./httpscan scan --url https://your-target-site.com

Since the tool is open-sourced, feel free to make any changes that fit your requirements. I would gladly welcome any feedback and contributions as well.

In case you created any security tools using Go, what were some of the obstacles that you had to overcome? Share your thoughts in the comments!