Forem: aymane aallaoui

[Boost]

aymane aallaoui — Wed, 18 Jun 2025 11:09:10 +0000

aymane aallaoui

Jun 11 '25

Embracing TypeScript Principles in Go: The Creation of a Zod-Inspired Validation Library

#programming #webdev #go #javascript

Comments 3

10 min read

Embracing TypeScript Principles in Go: The Creation of a Zod-Inspired Validation Library

aymane aallaoui — Wed, 11 Jun 2025 12:54:03 +0000

Introduction to TypeScript and Go

TypeScript and Go now occupy a prominent place in the minds of contemporary engineers. The former grafts optional, static typing onto JavaScript's freewheeling syntax, thereby easing the perennial pain of runtime errors. The latter pares programming down to its essentials, affording near-aphasic developers performance and effortless concurrency.

Lately I've been flitting between both languages on separate projects, and the split in their validation cultures jumps out immediately. TypeScript contributors lean on expressive, DSL-like utilities such as Zod that reduce schema verification to a handful of fluent method calls. Go teams, by contrast, often revert to reflection-centred code that is both bulky and, in some edge cases, a noticeable drain on execution speed.

That asymmetry gnawed at me until it posed an almost gluttonous challenge: Why couldn't Gopher's ecosystem borrow at least a slice of TypeScript's validation magic? Would it be possible to import its clean syntax and ergonomic chainability without smuggling in the heaviness that usually tags along with transpiled JavaScript?

Understanding the Need for a Validation Library

Every production application, regardless of stack, must grapple with the drudgery of data validation. User forms, API payloads, even the sporadic upstream feeds from third-party services demand a guarantee that incoming structures match the formats we expect. When that guarantee is absent, bugs slip through, lose the light of day, and often show up at the worst imaginable moment.

In short, prolonged confidence in data hygiene is not a luxury; it is the unobtrusive backbone that allows software to change confidently. Each time an assumption goes unverified, the cost compounds until fixing it becomes harder than building the feature in the first place. Thus a tidy, readable validation layer is less an optional ornament than an embedded organ of any well-mannered production codebase.

Validation in Go often feels sluggish because the standard routines seem stuck in quicksand.

In a typical project, a developer might write 50 lines of ad-hoc code just to check a user struct.

// The old way - verbose, error-prone, no reusability
if user.Name == "" {
    return errors.New("name is required")
}
if len(user.Name) < 3 {
    return errors.New("name must be at least 3 characters")
}
if len(user.Email) == 0 {
    return errors.New("email is required")
}
// ... 50 more lines of repetitive validation

None of those blocks—require non-empty string, enforce minimum length, repeat—errors reusable or neat.

Reflection-Based Tools

Many existing libraries lean on reflection, which makes the checks slower than hand-rolled loops.

When speed matters, that penalty is more than a micro-benchmark—frequent tests drag the entire CI.

Error messages from these libraries tend to land as something like field_name: invalid without context.

Fluent chaining, another popular request, is absent or half-baked in most canon packages.

A New Benchmark

Programmers have been asking for a validator that pairs raw speed with decent developer love.

Some even hint at TypeScript's Zod when they sketch wish-lists on Slack.

Inspiration from Zod: What Makes it a Popular Choice in TypeScript?

What Zod Gets Right

A fluent, chainable API lets you read a schema almost like a sentence.

const userSchema = z.object({
  name: z.string().min(3).max(50),
  email: z.string().email(),
  age: z.number().min(18).max(120)
});

Errors arrive in structured JSON, so front ends can parse them but so can log analyzers.

TypeScript's type system quietly inhales every Zod shape, giving compile-time safety all the way to the network.

Small, self-contained code means zero hidden dependencies that bloated the bundle.

Zod schemas stack neatly, letting a password rule reuse the email rule without copy-paste hell.

Those guiding ideas eventually coalesced into a validation library the team called zod-go.

The Challenges of Translating TypeScript Principles to Go

Moving Zod's distinctive design across languages exposed several friction points.

First, the two type systems do not speak the same grammatical rule book. Go is statically typed yet has only minimal inference compared to TypeScript. Crafting an expressive API while locking down the type surface took painful care.

Performance quickly surfaced as another non-negotiable. Most Go users expect libraries to hover near native speed; anything slower griefed their error log.

Flexibility in Go arrives via interface{} but eats compile-time checks for breakfast. Balancing runtime liberty with compile-time confidence became a constant tug-of-war.

Method chaining, an idiomatic TypeScript flourish, sits awkwardly in Go. Each call must return the precise interface if fluent syntax is to survive.

Concurrency sits at the heart of the Go ethos. To audit large slices without blocking the universe, the design had to embrace goroutines without losing clarity.

Three months of headaches later a prototype landed that neither crawled nor wrote spaghetti. zod-go passed the benchmarks and still felt light in the hands.

Pleasingly, syntax remained close to the original Zod.

schema := validators.String().
    Min(8).
    Pattern(`[A-Z]`).
    WithMessage("pattern", "Must contain uppercase").
    Required()

The Creation of zod-go: A Performance-driven Validation Library

Overview of Key Features and Functionalities

Rich Data Type Support:

A single library handles multiple formats: strings accept regular-expression matching plus length and format guards, numbers come with precise range and type limits, boolean fields can be marked mandatory, and array entries undergo element checks while enforcing uniqueness. Nested objects and maps are each governed by their own schema rules.

Detailed Error Reporting:

The core validation error type packages useful metadata:

type ValidationError struct {
    Field   string        `json:"field"`
    Value   interface{}   `json:"value"`
    Message string        `json:"message"`
    Rule    string        `json:"rule"`
    Details []ErrorDetail `json:"details,omitempty"`
}

Concurrent Processing:

Heavy workloads benefit from parallelism:

results := zod.ValidateConcurrently(schemas, dataList, 4) // launches four goroutines and aggregates outcomes

Custom Validator Support:

Special cases travel via plug-in functions. An email-domain check looks like this:

if !strings.HasSuffix(email, "@company.com") {
    return zod.NewValidationError("domain", email, "must be company email")
}

Performance Benchmarks and Improvements Compared to Existing Solutions

Results from my test suite arrived almost shockingly fast. A simple string test hit 50 million calls in 32.1 nanoseconds per op with no allocations.

Comparison Numbers:

BenchmarkZodGo_SimpleString-8           50000000    32.1 ns/op    0 B/op
BenchmarkGoPlayground_SimpleString-8     5000000   385.2 ns/op   64 B/op
BenchmarkGoValidator_SimpleString-8      3000000   441.8 ns/op   96 B/op

BenchmarkZodGo_ComplexObject-8           1000000  1247.3 ns/op  128 B/op
BenchmarkGoPlayground_ComplexObject-8     100000 12847.1 ns/op 2048 B/op

Key Performance Achievements:

Zod now outpaces well-known reflection validators by at least 10 to 12 times. Routine checks incur zero heap churn, memory pressure eases via pooling, and concurrent jobs scale almost linearly with CPU cores.

Several optimizations drove the latest performance boosts:

Direct type assertions supplanted the reflective calls once used.
Error structs now borrow from an object pool, reducing heap churn.
A worker pool pattern handles concurrent validation in tidy batches.
Memory allocation paths have been pruned and trimmed for lower latency.

How TypeScript Principles were Incorporated into zod-go

Zod-go channels TypeScript ideals even within Go's statically typed world.

Type Safety and Schema Validation

Go lacks the deep compile-time inference that TypeScript fans celebrate, yet zod-go secures type safety by confining methods to narrow interfaces. For instance, the StringValidator interface exposes only those functions meaningful for string checks.

type StringValidator interface {
    Schema
    Min(int) StringValidator
    Max(int) StringValidator
    Pattern(string) StringValidator
    Email() StringValidator
    Required() StringValidator
    WithMessage(rule, message string) StringValidator
}

Individual validators guard themselves, preventing accidental calls to length methods on an email-checking context.

Schema Composition

Road address formats nest cleanly inside larger structures:

addressSchema := validators.Object(map[string]zod.Schema{
    "street": validators.String().Required(),
    "city": validators.String().Required(),
    "zipCode": validators.String().Pattern(`^\d{5}$`),
})

User records build atop that blueprint:

userSchema := validators.Object(map[string]zod.Schema{
    "name": validators.String().Min(2).Required(),
    "email": validators.String().Email().Required(),
    "address": addressSchema,
})

Error Handling and Custom Error Messages

When validation stalls, zod-go packages feedback in a hierarchical tree rather than a flat list.

Custom Error Messages

Password rules show the structure at work:

schema := validators.String().
    Min(8).WithMessage("minLength", "Password must be at least 8 characters").
    Pattern(`[A-Z]`).WithMessage("pattern", "an uppercase letter is mandatory in the password field")

Developers receive both the policy name and its user-friendly string.

Structured Error Output

{
  "field": "user.address.zipCode",
  "value": "invalid-zip",
  "message": "must match pattern ^\\d{5}$",
  "rule": "pattern",
  "details": [
    {
      "field": "zipCode",
      "message": "the provided ZIP code does not conform to the expected format"
    }
  ]
}

Error Aggregation

In contrast to some libraries that halt validation upon the first failure, zod-go accumulates every encountered error. This all-at-once reporting is invaluable for diagnosing input problems.

Extensibility and Modularity

Custom Validators

type CustomValidator func(data interface{}) error

func isValidProductCode(data interface{}) error {
    code, ok := data.(string)
    if !ok {
        return zod.NewValidationError("type", data, "input must be of type string")
    }

    if !regexp.MustCompile(`^[A-Z]{2}\d{6}$`).MatchString(code) {
        return zod.NewValidationError("format", code, "product code must follow the TWOLETTERSYYYYYY pattern")
    }

    return nil
}

schema := validators.String().Custom(isValidProductCode)

Middleware Support

// Prepend a logging layer to track which fields are checked
loggingValidator := WithLogging(userSchema)

// Use an in-memory cache to avoid re-validating expensive computations for five minutes
cachedValidator := WithCache(complexSchema, 5*time.Minute)

Real-life Use Cases: Examples of zod-go in Action

Validating User Input in Web Applications

User Registration Endpoint

type RegistrationRequest struct {
    Username string `json:"username"`
    Email    string `json:"email"`
    Password string `json:"password"`
    Profile  struct {
        FirstName string `json:"firstName"`
        LastName  string `json:"lastName"`
        Bio       string `json:"bio"`
    } `json:"profile"`
}

var registrationSchema = validators.Object(map[string]zod.Schema{
    "username": validators.String().
        Min(3).Max(20).
        Pattern(`^[a-zA-Z0-9_]+$`).
        WithMessage("pattern", "username may only consist of alphanumeric characters and underscores").
        Required(),

    "email": validators.String().
        Email().
        WithMessage("email", "the supplied email address appears to be invalid").
        Required(),

    "password": validators.String().
        Min(8).
        Pattern(`[A-Z]`).WithMessage("uppercase", "An uppercase letter is required").
        Pattern(`[a-z]`).WithMessage("lowercase", "At least one lowercase letter is needed").
        Pattern(`\d`).WithMessage("digit", "Include a digit for security").
        Required(),

    "profile": validators.Object(map[string]zod.Schema{
        "firstName": validators.String().Min(1).Max(50).Required(),
        "lastName":  validators.String().Min(1).Max(50).Required(),
        "bio":       validators.String().Max(500).Optional(),
    }).Required(),
})

func handleRegistration(w http.ResponseWriter, r *http.Request) {
    var req RegistrationRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }

    if err := registrationSchema.Validate(req); err != nil {
        if validationErr, ok := err.(*zod.ValidationError); ok {
            w.Header().Set("Content-Type", "application/json")
            w.WriteHeader(http.StatusBadRequest)
            w.Write([]byte(validationErr.ErrorJSON()))
            return
        }
        http.Error(w, "Validation failed", http.StatusBadRequest)
        return
    }

    // Process valid registration...
}

Data Validation in API Requests

Governance of Product Data

var productSchema = validators.Object(map[string]zod.Schema{
    "name":        validators.String().Min(1).Max(200).Required(),
    "description": validators.String().Min(10).Max(2000).Required(),
    "price":       validators.Number().Min(0.01).Required(),
    "category":    validators.String().OneOf([]string{"electronics", "clothing", "books", "home", "sports"}).Required(),
    "tags":        validators.Array(validators.String().Min(1).Max(50)).Min(1).Max(10).Unique(),
    "dimensions": validators.Object(map[string]zod.Schema{
        "length": validators.Number().Min(0),
        "width":  validators.Number().Min(0),
        "height": validators.Number().Min(0),
        "weight": validators.Number().Min(0),
    }),
    "inStock": validators.Bool().Required(),
})

func createProduct(w http.ResponseWriter, r *http.Request) {
    var product map[string]interface{}
    if err := json.NewDecoder(r.Body).Decode(&product); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }

    if err := productSchema.Validate(product); err != nil {
        if validationErr, ok := err.(*zod.ValidationError); ok {
            response := map[string]interface{}{
                "error":   "Validation failed",
                "details": validationErr,
            }
            w.Header().Set("Content-Type", "application/json")
            w.WriteHeader(http.StatusBadRequest)
            json.NewEncoder(w).Encode(response)
            return
        }
        http.Error(w, "Validation failed", http.StatusBadRequest)
        return
    }

    // Create product in database...
}

Bulk Data Processing

The following function accepts a slice of arbitrary user records and assembles a schema-per-record blueprint:

func processBulkUsers(users []interface{}) []Result {
    schemas := make([]zod.Schema, len(users))
    for i := range schemas {
        schemas[i] = userSchema
    }

    // Processing thousands of records simultaneously is accomplished via 
    // zod.ValidateConcurrently and a concurrency limit of eight workers.
    results := zod.ValidateConcurrently(schemas, users, 8)

    validUsers := make([]interface{}, 0)
    errors := make([]zod.ValidationError, 0)

    for i, result := range results {
        if result.IsValid {
            validUsers = append(validUsers, users[i])
        } else {
            errors = append(errors, *result.Error.(*zod.ValidationError))
        }
    }

    return results
}

A compact slice validUsers captures entries that pass all checks, while another slice errors collects the specific validation failures encountered along the way.

The function finally surfaces a set of Result objects that indicate the overall validation outcome for each initial user entry.

Future Development and Roadmap

The collaboration with zod-go is in its infancy, and several significant enhancements are on the horizon.

Schema Inference from Go Structs

Future releases will support automatic schema creation directly from struct tags, allowing a definition such as:

type User struct {
    Name  string `validate:"min=3,max=50,required"`
    Email string `validate:"email,required"`
    Age   int    `validate:"min=18,max=120"`
}

to yield a complete zod schema with only the invocation:

schema := zod.InferSchema[User]()

JSON Schema Export

Each zod schema can later be serialized into standard JSON Schema format, making it straightforward to feed it into OpenAPI documentation pipelines or share with frontend developers bound by TypeScript types.

jsonSchema := schema.ToJSONSchema()

Transformation and Coercion

The library will gain built-in data manipulation operators, so a declaration such as:

validators.Object(map[string]zod.Schema{
    "name": validators.String().Transform(strings.ToTitle),
    "age":  validators.String().Coerce().Number().Min(18),
})

will automatically title-case names and cast age inputs before numeric validation.

Plugin Ecosystem

Community authors will, over time, release add-on validators for niche domains; an example usage could be:

import "github.com/zod-go-plugins/financial"

schema := validators.Object(map[string]zod.Schema{
    "creditCard": financial.CreditCard(),
    "ssn":        financial.SSN(),
})

IDE Integration and Code Generation

Plans are underway for a VS Code extension that highlights schema errors inline, completes struct tags, and even spins up a stub HTTP middleware layer based on the active schema design.

Command-line utilities now let developers extract schemas directly from running APIs. Such automation spares programmers the tedium of hand-coding type mirrors every time an endpoint shifts.

The new module plugs seamlessly into popular Go frameworks like Gin, Echo, and Fiber. One command wires existing handlers to a schema printer, turning live routes into machine-readable contracts on the fly.

Advanced Performance Optimizations

Researchers chasing speed have several fresh levers embedded in the release. Zero-allocation validation paths generated at build-time eliminate even transient memory waste.

String patterns now track processor SIMD lanes, cutting runtime char-by-char scans to single CPU cycles. Larger slices of log data validate against on-disk maps, removing the need to load everything into RAM.

Conclusion: Bridging Communities Through Innovation

This project began as a simple ports exercise but quickly turned into something bigger. It proved TypeScript's Zod ideas adapt smoothly, without sacrifice, to Go's lower-level idioms.
Long-term, the frontier of software won't be confined by syntax or runtime; it'll be defined by reusable concepts. zod-go is merely the first line of proof that good ideas jump fences without hesitation.

Get Started Today

Installation Command

In the terminal, run:

go get github.com/aymaneallaoui/zod-go

Minimal Program

package main

import (
    "fmt"
    "github.com/aymaneallaoui/zod-go/zod/validators"
)

func main() {
    schema := validators.String().
        Min(3).
        Max(50).
        Required()

    if err := schema.Validate("Hello, zod-go!"); err != nil {
        fmt.Printf("Validation failed: %v\n", err)
    } else {
        fmt.Println("Validation passed! 🎉")
    }
}

[Boost]

aymane aallaoui — Mon, 07 Apr 2025 19:05:03 +0000

aymane aallaoui

Mar 26 '25

Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

#webdev #programming #go #security

Comments 3

4 min read

new post

aymane aallaoui — Wed, 26 Mar 2025 20:57:23 +0000

Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

aymane aallaoui ・ Mar 26

#webdev #programming #go #security

Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

aymane aallaoui — Wed, 26 Mar 2025 20:57:06 +0000

Hello everyone, I'm glad to see you. Today, I want to share my new experience with building Kafka HTTP Scanner, an HTTP vulnerability scanner that is fully developed in Go.

Initially, a security testing weekend challenge turned out to be a complete Command Line Interface Tool (lmao), something I am (half) proud to say I created.

🔍 Why Another Security Scanner?

With the rapid development of IT technology, we are surrounded by multiple security tools, so I ask myself and you: "Why?"

I have two answers:

An opportunity to learn – I wanted to strengthen my understanding of Web security vulnerabilities as well as develop my competency in Go.
Flexibility – I needed a tool that could be tailored for specific needs without going through over-complicated enterprise-grade tools.

Moreover, let us refuse to lie to one another, building security systems is quite an enjoyable task to partake.

⚡ Core Features That Make It Stand Out

The tool is capable of identifying a wide array of threats to security:

HTTP Request Smuggling (this was fascinatingly difficult to integrate)
SQL Injections
Cross-Site Scripting (XSS)
CORS Misconfigurations
Security Headers Neglection
SSL/TLS Vulnerable Areas
And plenty more...

In addition to detection of vulnerabilities, I wanted it to be:

✅ Fast – Able to scan concurrently (Go’s goroutines served well for this function)

✅ Modular – Having an easily deployable architecture to introduce new vulnerability scan checks

✅ Friendly – With detailed instructions to fix the problems instead of just the pointers containing the problems

The Technical Architecture

Building Blocks

The project follows a clean, modular structure:

this diagram has been made with help of dirscanner

📂 cmd/              # CLI commands using Cobra
📂 internal/
  📂 core/           # Core scanner logic
  📂 http/           # Custom HTTP client
  📂 model/          # Data structures
  📂 modules/        # Vulnerability detection modules
  📂 output/         # Report formatting
📂 pkg/utils/        # Utility functions

The Module System

The heart of the scanner is its module system. Each vulnerability check is implemented as a separate module that implements a simple interface:

type ScanModule interface {
    Name() string
    Description() string
    Run(scanner Scanner) ([]model.Vulnerability, error)
}

This made it incredibly easy to add new vulnerability checks. Here's a simplified example of the XSS module:

func (m *XSSVulnerabilityModule) Run(s Scanner) ([]model.Vulnerability, error) {
    // Send requests with XSS payloads
    resp, err := s.SendHTTPRequest("GET", "/?q=<script>alert(1)</script>", nil, nil)

    // Check if the payload was reflected without encoding
    if strings.Contains(bodyStr, "<script>alert(1)</script>") {
        return []model.Vulnerability{{
            ID:      "XSS-01",
            Name:    "Reflected Cross-Site Scripting",
            Severity: model.SeverityHigh,
            // ...details, remediation info, etc.
        }}, nil
    }

    return nil, nil
}

Technical Challenges I Tackled

1. HTTP Request Smuggling Detection

Identifying HTTP request smuggling was surprisingly intricate. I first had to design synthetic HTTP requests that had mismatched and contradictory ‘Content-Length’ and ‘Transfer Encoding’ headers, then examine the responses from the server. This was an involved process that entailed me creating a custom raw HTTP client that had full control over the precise construction of the request:

func (c *Client) SendRawRequest(target model.Target, payload string) (string, error) {
    // Establish raw TCP or TLS connection
    if target.SSL {
        conn, err = tls.DialWithDialer(dialer, "tcp", address, tlsConfig)
    } else {
        conn, err = net.DialTimeout("tcp", address, timeout)
    }

    // Send the exact bytes we've crafted
    fmt.Fprintf(conn, payload)

    // Read the raw response
    // ...
}

2. Concurrency Control

Security scanning is perhaps the single most useful application for concurrency – the more security checks we can do in parallel, the better. Conversely, waiting for one task to finish before starting another is inefficient in this case. Unfortunately, unregulated parallelism can lead to a denial of service for the target server or invoke rate limits.

scanner.semaphore = make(chan struct{}, config.Concurrency)

// In the scan routine
s.semaphore <- struct{}{} // Acquire token
defer func() { <-s.semaphore }() // Release token

3. Robust Error Handling

When you're sending malformed requests and probing security boundaries, things break. A lot. Graceful error handling was critical to prevent the scanner from crashing mid-scan:

go func() {
    defer func() {
        if r := recover(); r != nil {
            s.logger.Errorf("Module %s panicked: %v", m.Name(), r)
            errorChan <- fmt.Errorf("module %s panicked: %v", m.Name(), r)
        }
    }()

    vulnerabilities, err := m.Run(s)
    // ...
}()

By wrapping module execution in a panic-recovering goroutine, a single misbehaving module can't bring down the entire scan.

Reflections

I learned much more than I initially planned from this project:

Security vulnerabilities are nuanced – Successfully detecting them requires an understanding of both the techniques employed by the attacker and the server-side behaviors.
Go's concurrency model shines for network tools – The parallel implementation of the scanning procedure was surprisingly easy, thanks to goroutine and channels.
CLI UX matters – A well-designed interface at the command-line terminal along with properly formatted output improves the quality of the developer tools.

What's Next?

I have a lot of ideas to work on for Kafka HTTP Scanner:

Active Exploitation Modules (with appropriate protections in place)
CI/CD integration
Custom reporting designs
API fuzzing

Try It Out!

Feel free to test out Kafka HTTP Scanner if you want to dabble in the world of security testing, or if you are looking for a Go-based command-line tool.

git clone https://github.com/aymaneallaoui/kafka-http-scanner.git
cd kafka-http-scanner
go build -o httpscan
Copy
To execute a basic scan:

./httpscan scan --url https://your-target-site.com

Since the tool is open-sourced, feel free to make any changes that fit your requirements. I would gladly welcome any feedback and contributions as well.

In case you created any security tools using Go, what were some of the obstacles that you had to overcome? Share your thoughts in the comments!

[Boost]

aymane aallaoui — Thu, 02 Jan 2025 23:17:13 +0000

Golang Field ordering matters?

aymane aallaoui ・ Sep 7 '24

#webdev #go #performance #beginners

Turbocharge Your Go App: Mastering Blazing-Fast Static File Serving Over TCP

aymane aallaoui — Sun, 08 Sep 2024 17:20:17 +0000

Hey gophers 🐹!

Have you ever thought of the best ways to ensure faster serving of static files using TCP in Go? Although there are built-in functions such as http.ServeFile that get the job done in simple file serving tasks, these functions become a hindrance when it comes to extremely large files or when carried out under a substantial load. In this article, we want to tackle advanced problem areas of this process so that people who want to go beyond the typical level of Go development will be pleased.

The Problem

Special attention has to be paid to the file serving speed as it is particularly important in case of heavy traffic. When serving static content through solutions such as http.ServeFile, there are following problems to address:

Buffering in One Layer: Data is loaded into memory first and only then sent over the network, creating unneeded memory footprint and delays.
Blocking I/O: Performing blocking operations on files can negatively affect the speed, especially if the files are several megabytes.
Poor Load Balance: There is no provision for performing file transfers in a more concurrent manner which means that speed is lost.

New Solution: More Optimizations

This is how you can get around these constraints and improve performance:

Zero-Copy File Transfer

Reduce on memory consumption and increase the speed of transfers by using the sendfile system call from the syscall package to accomplish a zero-copy file transfer. Memory in the user space is not involved and the data is ‘sent’ directly from the file descriptor to the socket.



import (

    "syscall"

    "net"

    "os"

)

func serveFile(conn net.Conn, filePath string) error {

    file, err := os.Open(filePath)

    if err != nil {

        return err

    }

    defer file.Close()

<span class="n">fileStat</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">file</span><span class="o">.</span><span class="n">Stat</span><span class="p">()</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">err</span>
<span class="p">}</span>

<span class="c">// Directly transfer file content to the connection socket</span>
<span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">syscall</span><span class="o">.</span><span class="n">Sendfile</span><span class="p">(</span><span class="kt">int</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">net</span><span class="o">.</span><span class="n">TCPConn</span><span class="p">)</span><span class="o">.</span><span class="n">File</span><span class="p">()</span><span class="o">.</span><span class="n">Fd</span><span class="p">()),</span> <span class="kt">int</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">Fd</span><span class="p">()),</span> <span class="no">nil</span><span class="p">,</span> <span class="kt">int</span><span class="p">(</span><span class="n">fileStat</span><span class="o">.</span><span class="n">Size</span><span class="p">()))</span>
<span class="k">return</span> <span class="n">err</span>



}

Gouroutines As An External Async I/O Mechanism

Utilise the concurrency framework in Go by dividing a file transfer into asynchronous pieces. Offload these pieces in parallel employing goroutines to shorten the time wasted in waiting for I/O call to finish.



func asyncServeFile(conn net.Conn, filePath string) error {

    file, err := os.Open(filePath)

    if err != nil {

        return err

    }

    defer file.Close()

<span class="n">buf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="m">32</span><span class="o">*</span><span class="m">1024</span><span class="p">)</span> <span class="c">// 32KB buffer</span>
<span class="k">var</span> <span class="n">wg</span> <span class="n">sync</span><span class="o">.</span><span class="n">WaitGroup</span>

<span class="k">for</span> <span class="p">{</span>
    <span class="n">n</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">file</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">n</span> <span class="o">&gt;</span> <span class="m">0</span> <span class="p">{</span>
        <span class="n">wg</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="m">1</span><span class="p">)</span>
        <span class="k">go</span> <span class="k">func</span><span class="p">(</span><span class="n">data</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">)</span> <span class="p">{</span>
            <span class="k">defer</span> <span class="n">wg</span><span class="o">.</span><span class="n">Done</span><span class="p">()</span>
            <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
        <span class="p">}(</span><span class="n">buf</span><span class="p">[</span><span class="o">:</span><span class="n">n</span><span class="p">])</span>
    <span class="p">}</span>
    <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
        <span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="n">io</span><span class="o">.</span><span class="n">EOF</span> <span class="p">{</span>
            <span class="k">break</span>
        <span class="p">}</span>
        <span class="k">return</span> <span class="n">err</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="n">wg</span><span class="o">.</span><span class="n">Wait</span><span class="p">()</span>
<span class="k">return</span> <span class="no">nil</span>



}

Focus On The Critical Sections

All the sections of the file may not be of equal merit. For illustration, video files which can start playing may require video metadata. Focus on such sections in order to enhance the perceived speed within the user interface.



func serveCriticalSections(conn net.Conn, filePath string, criticalSections []fileRange) error {

    file, err := os.Open(filePath)

    if err != nil {

        return err

    }

    defer file.Close()

<span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">section</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">criticalSections</span> <span class="p">{</span>
    <span class="n">buf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="n">section</span><span class="o">.</span><span class="n">length</span><span class="p">)</span>
    <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">file</span><span class="o">.</span><span class="n">ReadAt</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="n">section</span><span class="o">.</span><span class="n">offset</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">err</span>
    <span class="p">}</span>
    <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
<span class="p">}</span>

<span class="k">return</span> <span class="no">nil</span>



}

Conclusion

There is more to optimizing the handling of static file transfers over TCP in Go than just making use of the built-in facilities. Enhanced performance of the application can be achieved through the utilization of zero-copy transfer of files, asynchronous file I/O and management of critical segments of files. These methods enable high traffic and handling of huge files without losing user satisfaction.

That said happy coding and hope that you will not have any problem transferring your files next time. and always remember to just beat it

Golang Field ordering matters?

aymane aallaoui — Sat, 07 Sep 2024 11:13:19 +0000

Greetings, Gophers! During the initial year of working with Golang, I kept thinking that there must be ordering of fields and I thought why should I be bothered about it? Well, it is only fields, only something wrong could it be, I right? As most of the other novices, I thought it was not worth bothering with. In other words, how vital could it be that certain fields in a struct, would be put in a specific order? Well, a great deal!

Field ordering is one of the aspects ignored at the beginning but as one proceeds with the tutorial this understanding particularly of how Go works with pointers is understood as being very critical. Actually, it is precisely this order that is critical when it comes to improving application performance especially when working with large data sets or operations that are too heavy on memory. That unfortunate deficiency will be mended in a better understanding why it is so critical with Go field ordering.

How Does Go Store Structs In Memory?

When placed in memory, structs are represented as a block of consecutive memory, in which all the fields are located one after another according to their definition in a structure. This might seem rather simple, but this kind of linear organization plays some quite significant effects too, particularly in areas such as memory alignment and padding.

Memory Alignment and Padding

Memory alignment is about how data is placed and accessed from memory. Normally, CPUS can have a bias in terms of where data is fetched in memory, which is referred to as alignment boundaries. For instance, a 32 bit integer should be placed or fetched from the 4th byte address. In cases where there are fields in your struct which are not aligned properly, flicking through the pages, a Go compiler may add padding bytes, and so on. This becomes quite wasteful. For example, look at this struct.



struct example{

a bool   // 1 byte

b int32 // 4bytes;

c bool   // 1byte 

d int64 //8 bytes
}

In this improper struct because of alignment rules, the Go compiler might add one or more padding bytes in middle of these fields:

a is 1 byte but b wants 4 bytes align hence padding is inserted 3bytes
b length is of 4bytes
c length is of 1 byte but to align d which requires 8 bytes there is 7asing hence padding is introduced.
d length is of 8 bytes

How wood be structural wood, still is 24 in size because of the legs though the content only takes 14 however look at the volume of the actual content plus that of the padding.

Reordering Fields for Minimum Padding

Field order and structure finding can help to avoid wasting space in the form of negative margin. In another words:



type Example struct { 

d int64 // 8 bytes

b int32 // 4 bytes

a bool // 1 byte

c bool // 1 byte

}

In the above optimized struct:

d occupies 8 bytes.
b occupies 4 bytes.
a and c occupies 1 byte each without the need of the padding.

This structure is now only 16 bytes in size and this is better than the former 24 byte sized structure.

Why This Matters

As one considers common small-scale applications, he or she will most probably find the amount of memory in use by the application to be no different from the latter. This however is not the case in construction where performance and even memory space are critical consider a system embedded, super fast high frequency trading applications, or applications meant to process tremendous bulk of data these faithful restraint can add up fast. This becomes even more evident when one constructs or operates using many large arrays or concatenated slices of structs. It is not as easy to notice bias or load union when a structure has only a” s few bytes higher capacity. When low memory architecture is mass produced, with the amount of millions of instances to be handled, little by little, this addicted overdose of waste is no longer unheard of.

Conclusion

Ordering fields is not only nice from the viewpoint of Golang struct design but also plays an important role in memory optimization. Understanding this aspect of how Go does memory layout for your structs and their pixels enables more effective struct design in practice. Such an insignificant adjustment can cause a considerable performance boost when applications that make heavy use of memory are concerned. When the next opportunity comes for you to be defining a struct in Go, do not just spill those fields around. Instead, spend one minute considering the sequence- you will be thankful to yourself and your application for it in days to come!

Mastering JWT Security

aymane aallaoui — Fri, 05 Apr 2024 14:48:13 +0000

Ensuring user authentication and protecting an on-going session are vital parts of modern web development. Among the many options for managing authentication and authorization in web applications, JSON Web Tokens (JWTs) have become popular due to their simplicity, efficiency and flexibility. However, just like any other technology, JWTs have their own security considerations. In this article we will explore how to effectively secure JWTs by discussing the best practices.

Understanding JWTs:

Let’s quickly recap what JWTs are and how they work before delving into security measures. They are concise self-contained tokens that consist of three components –header, payload and signature. Generally these tokens are employed for the purposes of authentication or information exchange between parties. After a user has successfully logged in, a new JWT is issued; it then gets sent back to the client who will include it in subsequent requests thus authenticating them.

Best Practices for Securing JWTs:

use HTTPS: this will always guarantee that JWTs which are transmitted are encrypted on the move and hence there is no eavesdropping or man-in-the-middle attack. When HTTP is employed, JWTs become an easy target to intercept and therefore your system becomes insecure.

Keep JWTs Stateless: as opposed to keeping server’s database/session storage with sensitive data/state of sessions, you need to know these tokens are created for being stateless. This way the application can get better scalability and reduce cases of data breaches.

Implement Proper Token Expiry: JWTs should have a reasonable expiration time so that they won’t last long thus giving hacker a little time window. Shorter token expiration times make it hard for hackers who may steal them leading to unauthorized access.

Use Strong and Unique Keys: CRT algorithms like RSA with adequate key length or HMAC SHA-256 are recommended when signing JWTs. Besides, each JTW must always be signed using a unique key to eliminate substitution attacks on tokens.

Validate JWT Signatures: validation must be done in order to confirm whether incoming JTWs were signed by its author or not. Such omissions will lead you into accepting forged or tampered token while threatening your security.

Token revocation: A blacklist or revocation list should be maintained in order to invalidate JWTs where token revocation is necessary i.e. when a user logs out or an account is suspended. This makes it impossible for users to use tokens that may have been compromised or are no longer current to access protected resources.

Do not Store Sensitive Data: Do not put sensitive information which include passwords or personally identifiable data (PII) in the JWT payload; instead, securely store such data on the server and just include a reference or identifier in the token.

Rate Limiting and Throttling: To protect your authentication endpoints from brute force attacks and denial-of-service (DoS) attacks, rate limiting and throttling mechanisms must be implemented. This will prevent malicious actors from bombarding the server with too many authentication requests.

Code Examples

To bring out these best practices, we will walk through some basic code example using jsonwebtoken and nodejs.

const jwt = require('jsonwebtoken');
require('dotenv').config(); 

/* use a secure secret key at least 32 characters long eg '9wJMN71@Dx5#p%bTqY!6Rs*eK$A&zP2H' */

const secretKey = process.env.JWT_SECRET;


const createToken = (payload) => {
  try {
    const token = jwt.sign(payload, secretKey, { expiresIn: '1h' });
    return token;
  } catch (error) {
    console.error('Error creating JWT:', error.message);
    return null;
  }
};


const verifyToken = (token) => {
  try {
    const decoded = jwt.verify(token, secretKey);
    return decoded;
  } catch (error) {
    console.error('Error verifying JWT:', error.message);
    return null;
  }
};


const payload = { user_id: 123456 };
const token = createToken(payload);
if (token) {
  console.log('JWT created successfully:', token);

  const decoded = verifyToken(token);
  if (decoded) {
    console.log('JWT verified successfully. Decoded payload:', decoded);
  } else {
    console.log('JWT verification failed.');
  }
}

Conclusion

To effectively secure JWTs, one must have a full understanding of how they function and the various weaknesses that could be attacked. Comply with recommendations such as utilizing HTTPS, enforcing token expiration times and using strong cryptographic implementations; these steps will improve the security of your web apps thus ensuring protection for confidential user information. Always remember that it is an evolving process so you should constantly audit and change your JWT based security strategies according to current threats or best practice. With this kind of behavior that protects against JWT dangers, you can minimize potential hazards and develop confidence within your client base.

MVC Sucks, and We Should Learn from It

aymane aallaoui — Tue, 02 Apr 2024 11:22:09 +0000

Introduction:

Countless framework designers and programmers have found that MVC (Model-View-Controller) is the most successful way of organizing code in web development. It has been used in various frameworks and projects guiding the arrangement of applications by developers. However, let’s be honest, MVC is not perfect. This means that it contains multiple weaknesses that make coding complex; maintainability can turn into a nightmare and may lead to disappointments among the developers. When we are building a web application, it will be important for us to learn from such weaknesses without completely dismissing MVC. In this blog post, we shall discuss some of these shortcomings associated with MVC.

The Problem with MVC:

According to its proponents, Model should manage data while View should render UI and Controller should act as an intermediary between both of them. At first sight, this separation seems reasonable enough but at practice it results in bulky controllers, thin models and tight coupling components.

Let's take a look at a typical MVC setup in PHP:

class PostController {
    public function index() {
        $posts = Post::getAll();
        require('views/posts/index.php');
    }
}


class Post {
    public static function getAll() {
        // Fetch posts from the database
    }
}

// View 
foreach ($posts as $post) {
    echo "<h2>{$post->title}</h2>";
    echo "<p>{$post->content}</p>";
}

It is the controller’s responsibility in this example to retrieve information from the model and pass it on to the view for rendering. Nonetheless, with the growth of an application, a controller tends to become overloaded with logic for handling various requests thereby adding onto its difficulty of maintainability and testing.

The Proposal: Going beyond MVC

Instead of blindly following MVC, we could adopt a more elastic architecture that addresses the inadequacies of MVC. One of them involves blending Domain-Driven Design (DDD) with CQRS (Command Query Responsibility Segregation).

DDD puts greater emphasis on what are referred to as domain-driven models rather than the technical frameworks used when building applications. By defining richly structured domains that include distinct entities, value objects, and aggregates, we can confine most business rules within applications.

CQRS helps split our application into read or write operations thus optimizing read and write models independently such that they improve performance and scalability.

Let's refactor our previous example using DDD and CQRS principles:

// Controller
class PostController {
    private $postService;

    public function __construct(PostService $postService) {
        $this->postService = $postService;
    }

    public function index() {
        $posts = $this->postService->getAllPosts();
        require('views/posts/index.php');
    }
}

// Service
class PostService {
    private $postRepository;

    public function __construct(PostRepository $postRepository) {
        $this->postRepository = $postRepository;
    }

    public function getAllPosts() {
        return $this->postRepository->getAll();
    }
}

// Repository
class PostRepository {
    public function getAll() {
        // Fetch posts from the database
    }
}

// View 
foreach ($posts as $post) {
    echo "<h2>{$post->getTitle()}</h2>";
    echo "<p>{$post->getContent()}</p>";
}

So in this in example, we have introduced a service layer responsible for orchestrating business logic and a repository layer responsible for interacting with the data store. This separation of concerns makes our code base more maintainable and testable.

Conclusion:

So, the end point of this discussion will be that while MVC has been used as a model for web based applications for years, it is important to realise its inadequacies. By taking lessons from MVC’s flaws as well as embracing other flexible architectures like DDD and CQRS we can come up with web applications that are easier to maintain, scale up and follow clean code and good design principles. Thus let us go beyond MVC and embrace the changing structure of web applications.

Forem: aymane aallaoui

[Boost]

Embracing TypeScript Principles in Go: The Creation of a Zod-Inspired Validation Library

Embracing TypeScript Principles in Go: The Creation of a Zod-Inspired Validation Library

Introduction to TypeScript and Go

Understanding the Need for a Validation Library

Reflection-Based Tools

A New Benchmark

Inspiration from Zod: What Makes it a Popular Choice in TypeScript?

What Zod Gets Right

The Challenges of Translating TypeScript Principles to Go

The Creation of zod-go: A Performance-driven Validation Library

Overview of Key Features and Functionalities

Performance Benchmarks and Improvements Compared to Existing Solutions

How TypeScript Principles were Incorporated into zod-go

Type Safety and Schema Validation

Schema Composition

Error Handling and Custom Error Messages

Extensibility and Modularity

Real-life Use Cases: Examples of zod-go in Action

Validating User Input in Web Applications

Data Validation in API Requests

Future Development and Roadmap

Schema Inference from Go Structs

JSON Schema Export

Transformation and Coercion

Plugin Ecosystem

IDE Integration and Code Generation

Advanced Performance Optimizations

Conclusion: Bridging Communities Through Innovation

Get Started Today

Links

[Boost]

Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

new post

Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

aymane aallaoui ・ Mar 26

Building Kafka HTTP Scanner: An Open-Source Web Vulnerability Scanner in Go

🔍 Why Another Security Scanner?

⚡ Core Features That Make It Stand Out

The Technical Architecture

Building Blocks

The Module System

Technical Challenges I Tackled

1. HTTP Request Smuggling Detection

2. Concurrency Control

3. Robust Error Handling

Reflections

What's Next?

Try It Out!

[Boost]

Golang Field ordering matters?

aymane aallaoui ・ Sep 7 '24

Turbocharge Your Go App: Mastering Blazing-Fast Static File Serving Over TCP

The Problem

New Solution: More Optimizations

Zero-Copy File Transfer

Gouroutines As An External Async I/O Mechanism

Focus On The Critical Sections

Conclusion

Golang Field ordering matters?

How Does Go Store Structs In Memory?

Memory Alignment and Padding

Reordering Fields for Minimum Padding

Why This Matters

Conclusion

Mastering JWT Security

Understanding JWTs:

Best Practices for Securing JWTs:

Code Examples

Conclusion

MVC Sucks, and We Should Learn from It

Introduction:

The Problem with MVC:

Let's take a look at a typical MVC setup in PHP:

The Proposal: Going beyond MVC

Let's refactor our previous example using DDD and CQRS principles:

Conclusion: