Forem: Joao Oliveira

Building an API in Go to Manage Files on Amazon S3

Joao Oliveira — Tue, 20 Jan 2026 21:03:58 +0000

When I decided to create my blog, I chose Amazon S3 to host the images for my posts. The decision was practical (S3 is reliable, inexpensive, and scales well), but I also saw it as an opportunity: using a real, everyday problem to deepen my fluency in the AWS ecosystem, especially around security, automation, and integrations.

After creating a Go program that allows me to optimize and convert images to WebP, which I detailed in this article, I decided to build an API—also in Go—that allows me to manage my AWS account. With it, I can perform single or multiple file uploads, manage buckets, perform intelligent listings, and handle efficient streaming downloads.

That’s how this API came to life: a Go application that allows me to manage files and buckets in S3 with a productivity-focused workflow, supporting single and multiple uploads, paginated listings, streaming downloads, and the generation of temporary URLs (presigned URLs) for secure access.

To ensure agility in day-to-day usage, the application is fully containerized with Docker. This way, I spin up the container and use the endpoints (which I keep saved in Apidog). With each upload, I instantly receive the final file link, ready to paste into a blog post. This is exactly the “fast and easy” workflow I was aiming for.

What This API Does

In short, this API covers:

Single and multiple uploads (with concurrency).
Paginated listing of objects with useful metadata.
Streaming downloads, avoiding loading entire files into the API’s memory.
Presigned URLs for temporary and secure access to private buckets.
Bucket management (create, list, view statistics, empty, and delete).

Architecture and Project Structure

I structured the project following the recommendations of the Go Standards Project Layout. This organization, inspired by Clean Architecture, allows the application to start robustly while making it easier to evolve features and maintain the system over time.

How Each Part Connects

A simple way to understand the architecture is to imagine a flow:

Handler (HTTP): translates the protocol, performs minimal validation, and returns the response.
Service: applies business rules, validations, security, concurrency, and timeouts.
Repository (interface): defines the storage contract.
S3 Repository (implementation): handles integration details with the AWS SDK.

Directory Structure

s3-api/
├── .github/
│   └── workflows/            # CI: pipelines (tests, lint, build) with GitHub Actions
├── cmd/
│   └── api/
│       └── main.go           # Bootstrap: loads config, initializes dependencies, starts the HTTP server
├── internal/
│   ├── config/               # Centralized configuration: typed env vars + defaults
│   ├── middleware/           # HTTP middlewares: logging, timeout, recovery, etc.
│   └── upload/               # Domain module (files and buckets in S3)
│       ├── entity.go         # Domain entities and response DTOs
│       ├── errors.go         # Domain errors (no HTTP coupling)
│       ├── repository.go     # Storage contract (interface)
│       ├── s3_repository.go  # Contract implementation using AWS SDK v2
│       ├── service.go        # Business rules: validations, concurrency, timeouts
│       ├── handler.go        # HTTP handlers: translate request/response and call the Service
│       └── service_test.go   # Unit tests for the Service (with repository mocks)
├── .env                      # Local config (NEVER commit; use .gitignore)
├── Dockerfile                # Multi-stage build (slim binary)
└── docker-compose.yml        # Local environment: runs API with env vars and ports

Important Decisions in This Structure

cmd/api/main.go: This is where Dependency Injection is applied. The main function creates the AWS client, the repository, and the service, wiring them together. If tomorrow we decide to replace S3 with a local database, we only need to change a single line here.
internal/middleware/: This is where we implement structured logging middleware (using slog) and a timeout middleware, ensuring that no request hangs indefinitely and protecting the overall health of the application.
internal/upload/service.go: This is where we use advanced concurrency with errgroup. When performing multiple uploads, the service launches several goroutines to communicate with S3 in parallel, drastically reducing response time.
internal/upload/errors.go: Instead of returning generic strings, we use typed errors. This allows the Handler to elegantly determine whether it should return a 400 Bad Request or a 404 Not Found.

Endpoints

The API is divided into two main groups: File management and Bucket management.

Files

Method	Endpoint	Description
POST	/api/v1/upload	Upload a single file
POST	/api/v1/upload-multiple	Multiple upload
GET	/api/v1/list	Paginated listing
GET	/api/v1/download	Streaming download
GET	/api/v1/presign	Generate temporary (presigned) URL
DELETE	/api/v1/delete	Remove an object from the bucket

Buckets

Method	Endpoint	Description
GET	/api/v1/buckets/list	List account buckets
POST	/api/v1/buckets/create	Create bucket
GET	/api/v1/buckets/stats	Statistics
DELETE	/api/v1/buckets/delete	Delete bucket
DELETE	/api/v1/buckets/empty	Empty bucket

Key Highlights and Technical Decisions

Decisions that have the greatest impact on real-world API usage.

UUID v7 for file names

For uploads, I use UUID v7 to rename files. In addition to avoiding collisions and preventing exposure of original names, v7 preserves temporal ordering. This helps with organization in S3, facilitates auditing, and leaves the door open for future evolution (for example, database indexing without losing order).

Streaming to avoid memory spikes

For downloads, the goal is to keep RAM usage stable. Instead of “downloading everything and then returning it,” I treat it as a stream: data flows from S3 directly to the client without becoming a large in-memory buffer inside the API.

Real file type (MIME) validation

File extensions are easy to spoof. For this reason, instead of trusting .jpg/.png, I validate the file content (header) to identify the actual type. This reduces the risk of uploading malicious content disguised as images.

Timeouts as resource protection

Since the API depends on external services (S3), I do not want requests to hang indefinitely. Timeouts prevent excessive consumption of goroutines and connections when the network is unstable or the provider is slow.

Initializing the Project

The first step is to prepare the development environment. Go uses the module system (go mod) to manage dependencies, ensuring that the project is reproducible on any machine.

# Create the root folder and enter the directory
mkdir s3-api && cd s3-api

# Initialize the module (replace with your repository if needed)
go mod init github.com/JoaoOliveira889/s3-api

# Create the directory tree following the Go Standard Project Layout
mkdir -p cmd/api internal/upload internal/config internal/middleware

Dependency Management

I use the AWS SDK for Go v2 and a few libraries to make the project more robust (HTTP routing, env loading, UUIDs, testing). The goal is to keep the dependency set lean and well-justified, avoiding anything unnecessary.

# SDK core and configuration/credentials management
go get github.com/aws/aws-sdk-go-v2
go get github.com/aws/aws-sdk-go-v2/config

# S3 service client
go get github.com/aws/aws-sdk-go-v2/service/s3

# Gin Gonic: high-performance HTTP framework
go get github.com/gin-gonic/gin

# GoDotEnv: load environment variables from a .env file
go get github.com/joho/godotenv

# UUID: generate unique identifiers (v7)
go get github.com/google/uuid

# Testify: assertions and mocks for tests
go get github.com/stretchr/testify

Domain Layer

This is where the entities and contracts live. The core idea is: the domain describes what the system does, without being coupled to S3 itself.

Entities

Entities represent the main data structures of the application: files, listing metadata, bucket statistics, and pagination.

package upload

import (
    "io"
    "time"
)

type File struct {
    Name        string            `json:"name"`         // final object name in storage
    URL         string            `json:"url"`          // resulting URL after upload
    Content     io.ReadSeekCloser `json:"-"`            // Content is not serialized to JSON because it represents the file stream
    Size        int64             `json:"size"`
    ContentType string            `json:"content_type"`
}

type FileSummary struct {
    Key               string    `json:"key"`            // full object key in S3
    URL               string    `json:"url"`
    Size              int64     `json:"size_bytes"`
    HumanReadableSize string    `json:"size_formatted"`
    Extension         string    `json:"extension"`
    StorageClass      string    `json:"storage_class"`
    LastModified      time.Time `json:"last_modified"`
}

type BucketStats struct {
    BucketName         string `json:"bucket_name"`
    TotalFiles         int    `json:"total_files"`
    TotalSizeBytes     int64  `json:"total_size_bytes"`
    TotalSizeFormatted string `json:"total_size_formatted"`
}

type BucketSummary struct {
    Name         string    `json:"name"`
    CreationDate time.Time `json:"creation_date"`
}

type PaginatedFiles struct {
    Files     []FileSummary `json:"files"`
    NextToken string        `json:"next_token,omitempty"` // continuation token (pagination token)
}

Repository Interface

The repository defines a storage contract. This way, the Service layer does not “know” it is S3. It only knows that there is an implementation capable of storing, listing, downloading, and deleting.

Note: by using streaming for uploads/downloads, the API avoids loading entire files into memory, keeping RAM usage much more predictable.

package upload

import (
    "context"
    "io"
    "time"
)

// Repository defines the storage contract.
type Repository interface {
    // Upload stores the file in the bucket and returns the final object URL.
    Upload(ctx context.Context, bucket string, file *File) (string, error)

    // GetPresignURL generates a temporary URL for secure downloads.
    GetPresignURL(ctx context.Context, bucket, key string, expiration time.Duration) (string, error)

    // Download returns a stream (io.ReadCloser) to enable streaming without loading everything into memory.
    Download(ctx context.Context, bucket, key string) (io.ReadCloser, error)

    // List returns a page of files using a continuation token.
    List(ctx context.Context, bucket, prefix, token string, limit int32) (*PaginatedFiles, error)

    // Delete removes a specific object from the bucket.
    Delete(ctx context.Context, bucket string, key string) error

    // CheckBucketExists verifies whether the bucket exists and is accessible.
    CheckBucketExists(ctx context.Context, bucket string) (bool, error)

    // CreateBucket creates a bucket (usually with prior validations in the Service layer).
    CreateBucket(ctx context.Context, bucket string) error

    // ListBuckets lists all buckets available in the account.
    ListBuckets(ctx context.Context) ([]BucketSummary, error)

    // GetStats aggregates bucket statistics.
    GetStats(ctx context.Context, bucket string) (*BucketStats, error)

    // DeleteAll removes all objects from the bucket.
    DeleteAll(ctx context.Context, bucket string) error

    // DeleteBucket deletes the bucket.
    DeleteBucket(ctx context.Context, bucket string) error
}

Infrastructure Layer

With the contract defined, I can create the concrete implementation that talks to AWS. The goal here is to translate the domain’s needs into calls to the AWS SDK v2.

package upload

import (
    "context"
    "fmt"
    "io"
    "path/filepath"
    "strings"
    "time"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/aws-sdk-go-v2/service/s3/types"
)

type S3Repository struct {
    client *s3.Client
    region string
}

func NewS3Repository(client *s3.Client, region string) Repository {
    return &S3Repository{
        client: client,
        region: region,
    }
}

func (r *S3Repository) Upload(ctx context.Context, bucket string, file *File) (string, error) {
    input := &s3.PutObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(file.Name),
        Body:   file.Content, // stream: avoids loading the whole file into memory
    }

    _, err := r.client.PutObject(ctx, input)
    if err != nil {
        return "", fmt.Errorf("failed to upload: %w", err)
    }

    // Direct URL
    return fmt.Sprintf("https://%s.s3.%s.amazonaws.com/%s", bucket, r.region, file.Name), nil
}

func (r *S3Repository) List(ctx context.Context, bucket, prefix, token string, limit int32) (*PaginatedFiles, error) {
    input := &s3.ListObjectsV2Input{
        Bucket:            aws.String(bucket),
        Prefix:            aws.String(prefix),
        ContinuationToken: aws.String(token),
        MaxKeys:           aws.Int32(limit),
    }

    // In the SDK, an empty token must be nil to avoid sending an invalid token.
    if token == "" {
        input.ContinuationToken = nil
    }

    output, err := r.client.ListObjectsV2(ctx, input)
    if err != nil {
        return nil, fmt.Errorf("failed to list objects: %w", err)
    }

    var files []FileSummary
    for _, obj := range output.Contents {
        key := aws.ToString(obj.Key)
        size := aws.ToInt64(obj.Size)
        files = append(files, FileSummary{
            Key:               key,
            Size:              size,
            HumanReadableSize: formatBytes(size),
            StorageClass:      string(obj.StorageClass),
            LastModified:      aws.ToTime(obj.LastModified),
            Extension:         strings.ToLower(filepath.Ext(key)),
            URL:               fmt.Sprintf("https://%s.s3.%s.amazonaws.com/%s", bucket, r.region, key),
        })
    }

    next := ""
    if output.NextContinuationToken != nil {
        next = *output.NextContinuationToken
    }

    return &PaginatedFiles{Files: files, NextToken: next}, nil
}

func (r *S3Repository) Delete(ctx context.Context, bucket, key string) error {
    _, err := r.client.DeleteObject(ctx, &s3.DeleteObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    })
    return err
}

func (r *S3Repository) Download(ctx context.Context, bucket, key string) (io.ReadCloser, error) {
    output, err := r.client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    })
    if err != nil {
        return nil, err
    }

    // Return Body as a stream so the handler can io.Copy directly to the response.
    return output.Body, nil
}

func (r *S3Repository) GetPresignURL(ctx context.Context, bucket, key string, exp time.Duration) (string, error) {
    // The presign client generates temporary URLs without making the object public.
    pc := s3.NewPresignClient(r.client)
    req, err := pc.PresignGetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }, s3.WithPresignExpires(exp))
    if err != nil {
        return "", err
    }
    return req.URL, nil
}

func (r *S3Repository) CheckBucketExists(ctx context.Context, bucket string) (bool, error) {
    _, err := r.client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: aws.String(bucket)})
    if err != nil {
        return false, nil
    }
    return true, nil
}

func (r *S3Repository) CreateBucket(ctx context.Context, bucket string) error {
    _, err := r.client.CreateBucket(ctx, &s3.CreateBucketInput{Bucket: aws.String(bucket)})
    return err
}

func (r *S3Repository) ListBuckets(ctx context.Context) ([]BucketSummary, error) {
    out, err := r.client.ListBuckets(ctx, &s3.ListBucketsInput{})
    if err != nil {
        return nil, err
    }
    var res []BucketSummary
    for _, b := range out.Buckets {
        res = append(res, BucketSummary{
            Name:         aws.ToString(b.Name),
            CreationDate: aws.ToTime(b.CreationDate),
        })
    }
    return res, nil
}

func (r *S3Repository) DeleteBucket(ctx context.Context, bucket string) error {
    _, err := r.client.DeleteBucket(ctx, &s3.DeleteBucketInput{Bucket: aws.String(bucket)})
    return err
}

func (r *S3Repository) DeleteAll(ctx context.Context, bucket string) error {
    out, err := r.client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{Bucket: aws.String(bucket)})
    if err != nil || len(out.Contents) == 0 {
        return err
    }

    var objects []types.ObjectIdentifier
    for _, obj := range out.Contents {
        objects = append(objects, types.ObjectIdentifier{Key: obj.Key})
    }
    _, err = r.client.DeleteObjects(ctx, &s3.DeleteObjectsInput{
        Bucket: aws.String(bucket),
        Delete: &types.Delete{Objects: objects},
    })
    return err
}

func (r *S3Repository) GetStats(ctx context.Context, bucket string) (*BucketStats, error) {
    out, err := r.client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{Bucket: aws.String(bucket)})
    if err != nil {
        return nil, err
    }
    var totalSize int64
    for _, obj := range out.Contents {
        totalSize += aws.ToInt64(obj.Size)
    }
    return &BucketStats{
        BucketName:         bucket,
        TotalFiles:         int(len(out.Contents)),
        TotalSizeBytes:     totalSize,
        TotalSizeFormatted: formatBytes(totalSize),
    }, nil
}

func formatBytes(b int64) string {
    const unit = 1024
    if b < unit {
        return fmt.Sprintf("%d B", b)
    }
    div, exp := int64(unit), 0
    for n := b / unit; n >= unit; n /= unit {
        div *= unit
        exp++
    }
    return fmt.Sprintf("%.1f %cB", float64(b)/float64(div), "KMGTPE"[exp])
}

Technical Highlights in the Infrastructure Layer

Context (context.Context): network operations respect cancellation and timeouts; if the request is dropped, unnecessary work and resource consumption are avoided.
Efficient pagination: essential for large buckets; clients can navigate results without overloading the API.
Presigned URLs: temporary and secure access to private buckets without exposing them publicly.
Data presentation: formatting sizes and metadata significantly improves the experience for API consumers.

Service Layer

This layer is responsible for orchestration: validations, business rules, timeouts, and concurrency.

One detail I particularly like here: the Service interface exposes what the application does, while the concrete implementation remains private, enforcing the use of a constructor (factory). This helps keep the design consistent and controlled.

package upload

import (
    "context"
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "path/filepath"
    "regexp"
    "strings"
    "time"

    "github.com/google/uuid"
    "golang.org/x/sync/errgroup"
)

type Service interface {
    UploadFile(ctx context.Context, bucket string, file *File) (string, error)
    UploadMultipleFiles(ctx context.Context, bucket string, files []*File) ([]string, error)
    GetDownloadURL(ctx context.Context, bucket, key string) (string, error)
    DownloadFile(ctx context.Context, bucket, key string) (io.ReadCloser, error)
    ListFiles(ctx context.Context, bucket, ext, token string, limit int) (*PaginatedFiles, error)
    DeleteFile(ctx context.Context, bucket string, key string) error
    GetBucketStats(ctx context.Context, bucket string) (*BucketStats, error)
    CreateBucket(ctx context.Context, bucket string) error
    ListAllBuckets(ctx context.Context) ([]BucketSummary, error)
    DeleteBucket(ctx context.Context, bucket string) error
    EmptyBucket(ctx context.Context, bucket string) error
}

const (
    uploadTimeout       = 60 * time.Second
    deleteTimeout       = 5 * time.Second
    maxBucketNameLength = 63
    minBucketNameLength = 3
)

var (
    // S3 bucket names follow DNS rules (min/max length and charset). The regex covers the general pattern.
    bucketDNSNameRegex = regexp.MustCompile(`^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$`)

    // Allowlist: validates the actual file type (detected MIME) instead of trusting the extension.
    allowedTypes = map[string]bool{
        "image/jpeg":      true,
        "image/png":       true,
        "image/webp":      true,
        "application/pdf": true,
    }
)

type uploadService struct {
    repo Repository
}

func NewService(repo Repository) Service {
    return &uploadService{repo: repo}
}

func (s *uploadService) UploadFile(ctx context.Context, bucket string, file *File) (string, error) {
    ctx, cancel := context.WithTimeout(ctx, uploadTimeout) // prevents "hanging" requests in S3 calls
    defer cancel()

    if err := s.validateBucketName(bucket); err != nil {
        return "", err
    }

    if err := s.validateFile(file); err != nil {
        slog.Error("security validation failed", "error", err, "filename", file.Name)
        return "", err
    }

    // UUID v7 preserves temporal ordering and avoids collisions/exposure of original names.
    id, err := uuid.NewV7()
    if err != nil {
        slog.Error("uuid generation failed", "error", err)
        return "", fmt.Errorf("failed to generate unique id: %w", err)
    }

    file.Name = id.String() + filepath.Ext(file.Name)

    url, err := s.repo.Upload(ctx, bucket, file)
    if err != nil {
        slog.Error("repository upload failed", "error", err, "bucket", bucket)
        return "", err
    }

    file.URL = url
    slog.Info("file uploaded successfully", "url", url)
    return url, nil
}

func (s *uploadService) UploadMultipleFiles(ctx context.Context, bucket string, files []*File) ([]string, error) {
    // errgroup cancels the context if any goroutine fails.
    g, ctx := errgroup.WithContext(ctx)
    results := make([]string, len(files))

    for i, f := range files {
        i, f := i, f // avoids incorrect capture of loop variables
        g.Go(func() error {
            url, err := s.UploadFile(ctx, bucket, f)
            if err != nil {
                return err
            }
            results[i] = url
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }

    return results, nil
}

func (s *uploadService) GetDownloadURL(ctx context.Context, bucket, key string) (string, error) {
    if err := s.validateBucketName(bucket); err != nil {
        return "", err
    }

    // Presign is the recommended way to provide temporary access to private buckets.
    return s.repo.GetPresignURL(ctx, bucket, key, 15*time.Minute)
}

func (s *uploadService) DownloadFile(ctx context.Context, bucket, key string) (io.ReadCloser, error) {
    if err := s.validateBucketName(bucket); err != nil {
        return nil, err
    }
    return s.repo.Download(ctx, bucket, key)
}

func (s *uploadService) ListFiles(ctx context.Context, bucket, ext, token string, limit int) (*PaginatedFiles, error) {
    if err := s.validateBucketName(bucket); err != nil {
        return nil, err
    }

    if limit <= 0 {
        limit = 10
    }

    res, err := s.repo.List(ctx, bucket, "", token, int32(limit))
    if err != nil {
        return nil, err
    }

    if ext == "" {
        return res, nil
    }

    var filtered []FileSummary
    target := strings.ToLower(ext)

    if !strings.HasPrefix(target, ".") {
        target = "." + target
    }

    for _, f := range res.Files {
        if strings.ToLower(f.Extension) == target {
            filtered = append(filtered, f)
        }
    }

    res.Files = filtered
    return res, nil
}

func (s *uploadService) DeleteFile(ctx context.Context, bucket string, key string) error {
    ctx, cancel := context.WithTimeout(ctx, deleteTimeout)
    defer cancel()

    if key == "" {
        return fmt.Errorf("file key is required")
    }

    if err := s.validateBucketName(bucket); err != nil {
        return err
    }

    return s.repo.Delete(ctx, bucket, key)
}

func (s *uploadService) GetBucketStats(ctx context.Context, bucket string) (*BucketStats, error) {
    if err := s.validateBucketName(bucket); err != nil {
        return nil, err
    }
    return s.repo.GetStats(ctx, bucket)
}

func (s *uploadService) CreateBucket(ctx context.Context, bucket string) error {
    if err := s.validateBucketName(bucket); err != nil {
        return err
    }

    exists, err := s.repo.CheckBucketExists(ctx, bucket)
    if err != nil {
        return err
    }
    if exists {
        return ErrBucketAlreadyExists
    }

    return s.repo.CreateBucket(ctx, bucket)
}

func (s *uploadService) DeleteBucket(ctx context.Context, bucket string) error {
    if err := s.validateBucketName(bucket); err != nil {
        return err
    }
    return s.repo.DeleteBucket(ctx, bucket)
}

func (s *uploadService) EmptyBucket(ctx context.Context, bucket string) error {
    if err := s.validateBucketName(bucket); err != nil {
        return err
    }
    return s.repo.DeleteAll(ctx, bucket)
}

func (s *uploadService) ListAllBuckets(ctx context.Context) ([]BucketSummary, error) {
    return s.repo.ListBuckets(ctx)
}

func (s *uploadService) validateBucketName(bucket string) error {
    bucket = strings.TrimSpace(strings.ToLower(bucket))
    if bucket == "" {
        return ErrBucketNameRequired
    }

    if len(bucket) < minBucketNameLength || len(bucket) > maxBucketNameLength {
        return fmt.Errorf("bucket name length must be between %d and %d", minBucketNameLength, maxBucketNameLength)
    }

    if !bucketDNSNameRegex.MatchString(bucket) {
        return fmt.Errorf("invalid bucket name pattern")
    }

    if strings.Contains(bucket, "..") {
        return fmt.Errorf("bucket name cannot contain consecutive dots")
    }

    return nil
}

func (s *uploadService) validateFile(f *File) error {
    seeker, ok := f.Content.(io.Seeker)
    if !ok {
        return fmt.Errorf("file content must support seeking")
    }

    buffer := make([]byte, 512)
    n, err := f.Content.Read(buffer)
    if err != nil && err != io.EOF {
        return fmt.Errorf("failed to read file header: %w", err)
    }

    // Reset the stream back to the beginning before upload.
    if _, err := seeker.Seek(0, io.SeekStart); err != nil {
        return fmt.Errorf("failed to reset file pointer: %w", err)
    }

    detectedType := http.DetectContentType(buffer[:n])
    if !allowedTypes[detectedType] {
        slog.Warn("rejected file type", "type", detectedType)
        return ErrInvalidFileType
    }

    return nil
}

Technical Highlights in the Service Layer

Concurrency with errgroup

In multiple uploads, I don’t upload files one by one. I trigger uploads in parallel and let errgroup manage cancellation: if one upload fails, the others are signaled to stop. In practice, the total time tends to be close to the slowest upload, rather than the sum of all uploads.

Security: file type validation

The validation reads a portion of the content to identify the actual file type, reducing the risk of malicious files with a “spoofed” extension.

Resilience: per-operation timeouts

Operations such as upload and delete have different characteristics; therefore, it makes sense to define distinct time limits to keep the API responsive and avoid stuck resources.

Encapsulation and dependency injection

The Service depends on a Repository. This makes the code highly testable: in tests, I can mock the repository without needing AWS.

Handler Layer

The Handler translates HTTP into the application domain: it extracts parameters, performs minimal validation, calls the Service, and returns JSON (or streaming in the case of downloads).

I chose Gin for its combination of performance, simplicity, and a rich middleware ecosystem.

package upload

import (
    "errors"
    "io"
    "net/http"
    "strconv"

    "github.com/gin-gonic/gin"
)

type Handler struct {
    service Service
}

func NewHandler(s Service) *Handler {
    return &Handler{service: s}
}

func (h *Handler) UploadFile(c *gin.Context) {
    bucket := c.PostForm("bucket")
    fileHeader, err := c.FormFile("file")
    if err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": "file field is required"})
        return
    }

    openedFile, err := fileHeader.Open()
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to open file"})
        return
    }
    defer openedFile.Close()

    file := &File{
        Name:        fileHeader.Filename,
        Content:     openedFile,
        Size:        fileHeader.Size,
        ContentType: fileHeader.Header.Get("Content-Type"),
    }

    url, err := h.service.UploadFile(c.Request.Context(), bucket, file)
    if err != nil {
        h.handleError(c, err)
        return
    }

    c.JSON(http.StatusCreated, gin.H{"url": url})
}

func (h *Handler) UploadMultiple(c *gin.Context) {
    bucket := c.PostForm("bucket")
    form, err := c.MultipartForm()
    if err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": "invalid multipart form"})
        return
    }

    filesHeaders := form.File["files"]
    if len(filesHeaders) == 0 {
        c.JSON(http.StatusBadRequest, gin.H{"error": "no files provided"})
        return
    }

    var filesToUpload []*File
    for _, header := range filesHeaders {
        openedFile, err := header.Open()
        if err != nil {
            continue
        }

        filesToUpload = append(filesToUpload, &File{
            Name:        header.Filename,
            Content:     openedFile,
            Size:        header.Size,
            ContentType: header.Header.Get("Content-Type"),
        })
    }

    defer func() {
        for _, f := range filesToUpload {
            f.Content.Close()
        }
    }()

    urls, err := h.service.UploadMultipleFiles(c.Request.Context(), bucket, filesToUpload)
    if err != nil {
        h.handleError(c, err)
        return
    }

    c.JSON(http.StatusCreated, gin.H{"urls": urls})
}

func (h *Handler) GetPresignedURL(c *gin.Context) {
    bucket := c.Query("bucket")
    key := c.Query("key")

    url, err := h.service.GetDownloadURL(c.Request.Context(), bucket, key)
    if err != nil {
        h.handleError(c, err)
        return
    }

    c.JSON(http.StatusOK, gin.H{"presigned_url": url})
}

func (h *Handler) DownloadFile(c *gin.Context) {
    bucket := c.Query("bucket")
    key := c.Query("key")

    stream, err := h.service.DownloadFile(c.Request.Context(), bucket, key)
    if err != nil {
        h.handleError(c, err)
        return
    }
    defer stream.Close()

    c.Header("Content-Disposition", "attachment; filename="+key)
    c.Header("Content-Type", "application/octet-stream")

    _, _ = io.Copy(c.Writer, stream)
}

func (h *Handler) ListFiles(c *gin.Context) {
    limit, _ := strconv.Atoi(c.DefaultQuery("limit", "10"))

    result, err := h.service.ListFiles(
        c.Request.Context(),
        c.Query("bucket"),
        c.Query("extension"),
        c.Query("token"),
        limit,
    )

    if err != nil {
        h.handleError(c, err)
        return
    }

    c.JSON(http.StatusOK, result)
}

func (h *Handler) DeleteFile(c *gin.Context) {
    err := h.service.DeleteFile(c.Request.Context(), c.Query("bucket"), c.Query("key"))
    if err != nil {
        h.handleError(c, err)
        return
    }

    c.Status(http.StatusNoContent)
}

func (h *Handler) GetBucketStats(c *gin.Context) {
    bucket := c.Query("bucket")
    if bucket == "" {
        c.JSON(http.StatusBadRequest, gin.H{"error": "bucket parameter is required"})
        return
    }

    stats, err := h.service.GetBucketStats(c.Request.Context(), bucket)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, stats)
}

func (h *Handler) CreateBucket(c *gin.Context) {
    var body struct {
        Name string `json:"bucket_name" binding:"required"`
    }

    if err := c.ShouldBindJSON(&body); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": "valid bucket_name is required"})
        return
    }

    if err := h.service.CreateBucket(c.Request.Context(), body.Name); err != nil {
        h.handleError(c, err)
        return
    }

    c.Status(http.StatusCreated)
}

func (h *Handler) ListBuckets(c *gin.Context) {
    buckets, err := h.service.ListAllBuckets(c.Request.Context())
    if err != nil {
        h.handleError(c, err)
        return
    }
    c.JSON(http.StatusOK, buckets)
}

func (h *Handler) DeleteBucket(c *gin.Context) {
    if err := h.service.DeleteBucket(c.Request.Context(), c.Query("name")); err != nil {
        h.handleError(c, err)
        return
    }
    c.Status(http.StatusNoContent)
}

func (h *Handler) EmptyBucket(c *gin.Context) {
    bucket := c.Query("bucket")
    if bucket == "" {
        c.JSON(http.StatusBadRequest, gin.H{"error": "bucket parameter is required"})
        return
    }

    err := h.service.EmptyBucket(c.Request.Context(), bucket)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.Status(http.StatusNoContent)
}

func (h *Handler) handleError(c *gin.Context, err error) {
    switch {
    case errors.Is(err, ErrInvalidFileType),
        errors.Is(err, ErrBucketNameRequired):
        c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})

    case errors.Is(err, ErrBucketAlreadyExists):
        c.JSON(http.StatusConflict, gin.H{"error": err.Error()})

    case errors.Is(err, ErrFileNotFound):
        c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})

    case errors.Is(err, ErrOperationTimeout):
        c.JSON(http.StatusGatewayTimeout, gin.H{"error": "request timed out"})

    default:
        c.JSON(http.StatusInternalServerError, gin.H{"error": "an unexpected error occurred"})
    }
}

Technical Decisions in the Handler

Multipart/form-data without “blowing up RAM”: I open the file as a stream, avoiding loading the entire content into memory.
Download streaming: I create a “pipe” between S3 and the client. This makes it possible to support larger files with modest hardware.
Centralized error handling: instead of scattering if err != nil across all routes, the Handler maps semantic domain errors to consistent HTTP status codes (400, 404, 409, 504, etc.).

The Entry Point (main)

The main function loads configuration, instantiates dependencies, and starts the server. This is where one of the most useful concepts for keeping projects healthy comes into play: dependency injection.

In practical terms: main creates the AWS client → passes it to the repository → passes it to the service → passes it to the handler. This keeps the system modular and easy to test.

package main

import (
    "context"
    "log/slog"
    "os"
    "time"

    // Internal packages
    appConfig "github.com/JoaoOliveira889/s3-api/internal/config"
    "github.com/JoaoOliveira889/s3-api/internal/middleware"
    "github.com/JoaoOliveira889/s3-api/internal/upload"
    "github.com/gin-gonic/gin"

    // External packages
    configAWS "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/joho/godotenv"
)

func main() {
    _ = godotenv.Load()

    cfg := appConfig.Load()

    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    slog.SetDefault(logger)

    r := gin.New()

    r.Use(middleware.RequestTimeoutMiddleware(cfg.UploadTimeout))
    r.Use(middleware.LoggingMiddleware())
    r.Use(gin.Recovery())

    ctx := context.Background()
    awsCfg, err := configAWS.LoadDefaultConfig(ctx, configAWS.WithRegion(cfg.AWSRegion))
    if err != nil {
        slog.Error("failed to load AWS SDK config", "error", err)
        os.Exit(1)
    }

    s3Client := s3.NewFromConfig(awsCfg)
    repo := upload.NewS3Repository(s3Client, cfg.AWSRegion)
    service := upload.NewService(repo)
    handler := upload.NewHandler(service)

    api := r.Group("/api/v1")
    {
        api.GET("/health", func(c *gin.Context) {
            c.JSON(200, gin.H{
                "status":    "healthy",
                "env":       cfg.Env,
                "timestamp": time.Now().Format(time.RFC3339),
            })
        })

        api.GET("/list", handler.ListFiles)
        api.POST("/upload", handler.UploadFile)
        api.POST("/upload-multiple", handler.UploadMultiple)
        api.GET("/download", handler.DownloadFile)
        api.GET("/presign", handler.GetPresignedURL)
        api.DELETE("/delete", handler.DeleteFile)

        buckets := api.Group("/buckets")
        {
            buckets.POST("/create", handler.CreateBucket)
            buckets.DELETE("/delete", handler.DeleteBucket)
            buckets.GET("/stats", handler.GetBucketStats)
            buckets.GET("/list", handler.ListBuckets)
            buckets.DELETE("/empty", handler.EmptyBucket)
        }
    }

    slog.Info("server successfully started",
        "port", cfg.Port,
        "env", cfg.Env,
        "region", cfg.AWSRegion,
    )

    if err := r.Run(":" + cfg.Port); err != nil {
        slog.Error("server failed to shut down properly", "error", err)
        os.Exit(1)
    }
}

Technical Highlights in `main`

Structured logging: JSON logs are more useful for indexing and searching in tools such as CloudWatch, ELK, or Datadog.
Chained middlewares: logging and global timeouts are applied consistently across all routes.
Recovery: prevents a panic from bringing down the entire API.
Grouping and versioning: /api/v1 makes it easier to evolve the API without breaking existing clients.

Security and configuration: managing credentials

For the API to communicate with AWS, it needs credentials. However, exposing keys in the code is a serious risk. For this reason, in local development I use environment variables (via .env), and in production the natural evolution is to use IAM Roles.

`.env` file (local development)

At the root of the project, we create the .env file. It is used to store server configuration and AWS secret keys.

Warning: This file must be included in your .gitignore. Never commit your keys to GitHub or any other version control system.

# Server Settings
PORT=8080
APP_ENV=development

# AWS Configuration
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=SUA_ACCESS_KEY_AQUI
AWS_SECRET_ACCESS_KEY=SUA_SECRET_KEY_AQUI

# Application Settings
UPLOAD_TIMEOUT_SECONDS=60

Obtaining Credentials in the AWS Console (IAM)

Access keys are generated in IAM. For study purposes, you can use broad permissions, but the recommended approach is to follow the principle of least privilege and restrict access to only the required bucket and actions.

Step by step:

Access IAM: In the AWS Console, search for “IAM”.
Create a User: Go to Users > Create user.
Configuration: Define a name (e.g., s3-api-manager). It is not necessary to enable access to the AWS Management Console for this user, as it will be used only via code.
Permissions (Principle of Least Privilege): Choose Attach policies directly.
- Note: For learning purposes, you may select AmazonS3FullAccess. In a real production scenario, the ideal approach is to create a custom policy that grants access only to the specific bucket you intend to use.
Generate the Keys: After creating the user, click on the user name, go to the Security credentials tab, and look for the Access keys section.
Creation: Click Create access key, select the “Local code” option, and proceed.

IMPORTANT: You will see the Access Key ID and the Secret Access Key. Copy and paste them into your .env file immediately, as the Secret Key will never be shown again.

Middlewares: observability and resilience

Middlewares are the right place for global behaviors: logging, timeouts, correlation, and protection. Instead of replicating these concerns per endpoint, I centralize them.

Structured logging

The log is written after the request finishes. This makes it possible to accurately record the status code and latency, making performance and error analysis much easier.

package middleware

import (
    "log/slog"
    "time"

    "github.com/gin-gonic/gin"
)

func LoggingMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        path := c.Request.URL.Path
        raw := c.Request.URL.RawQuery

        c.Next()

        if raw != "" {
            path = path + "?" + raw
        }

        slog.Info("incoming request",
            "method", c.Request.Method,
            "path", path,
            "status", c.Writer.Status(),
            "latency", time.Since(start).String(),
            "ip", c.ClientIP(),
            "user_agent", c.Request.UserAgent(),
        )
    }
}

Timeout control

In systems that depend on external services, leaving connections open indefinitely is risky. The timeout middleware cancels the request when it exceeds a defined limit, freeing up resources.

package middleware

import (
    "context"
    "net/http"
    "time"

    "github.com/gin-gonic/gin"
)

func RequestTimeoutMiddleware(timeout time.Duration) gin.HandlerFunc {
    return func(c *gin.Context) {
        ctx, cancel := context.WithTimeout(c.Request.Context(), timeout)
        defer cancel()

        c.Request = c.Request.WithContext(ctx)

        finished := make(chan struct{}, 1)

        go func() {
            c.Next()
            finished <- struct{}{}
        }()

        select {
        case <-finished:
            return
        case <-ctx.Done():
            if ctx.Err() == context.DeadlineExceeded {
                c.AbortWithStatusJSON(http.StatusGatewayTimeout, gin.H{
                    "error": "request timed out",
                })
            }
        }
    }
}

Technical Highlights in Middlewares

c.Next() at the right moment: ensures that logs capture the actual status code and real latency.
select and channels: a clear pattern to observe “completed vs. timed out,” maintaining control over the request lifecycle.
Structured logs: make it easier to build dashboards and alerts.

Unit Tests

One of the greatest benefits of this architecture is the ability to test business logic in isolation, without AWS and without an internet connection.

I use mocks to simulate the repository and focus on validating: bucket rules, file type validations, identifier generation, and error handling.

package upload

import (
    "context"
    "io"
    "time"

    "github.com/stretchr/testify/mock"
)

type RepositoryMock struct {
    mock.Mock
}

func (m *RepositoryMock) Delete(ctx context.Context, bucket string, key string) error {
    panic("unimplemented")
}

func (m *RepositoryMock) DeleteAll(ctx context.Context, bucket string) error {
    panic("unimplemented")
}

func (m *RepositoryMock) DeleteBucket(ctx context.Context, bucket string) error {
    panic("unimplemented")
}

func (m *RepositoryMock) GetStats(ctx context.Context, bucket string) (*BucketStats, error) {
    panic("unimplemented")
}

func (m *RepositoryMock) ListBuckets(ctx context.Context) ([]BucketSummary, error) {
    panic("unimplemented")
}

func (m *RepositoryMock) Upload(ctx context.Context, bucket string, file *File) (string, error) {
    args := m.Called(ctx, bucket, file)
    return args.String(0), args.Error(1)
}

func (m *RepositoryMock) Download(ctx context.Context, bucket, key string) (io.ReadCloser, error) {
    args := m.Called(ctx, bucket, key)
    if args.Get(0) == nil {
        return nil, args.Error(1)
    }
    return args.Get(0).(io.ReadCloser), args.Error(1)
}

func (m *RepositoryMock) GetPresignURL(ctx context.Context, bucket, key string, expiration time.Duration) (string, error) {
    args := m.Called(ctx, bucket, key, expiration)
    return args.String(0), args.Error(1)
}

func (m *RepositoryMock) List(ctx context.Context, bucket, prefix, token string, limit int32) (*PaginatedFiles, error) {
    args := m.Called(ctx, bucket, prefix, token, limit)
    return args.Get(0).(*PaginatedFiles), args.Error(1)
}

func (m *RepositoryMock) CheckBucketExists(ctx context.Context, bucket string) (bool, error) {
    args := m.Called(ctx, bucket)
    return args.Bool(0), args.Error(1)
}

func (m *RepositoryMock) CreateBucket(ctx context.Context, bucket string) error {
    args := m.Called(ctx, bucket)
    return args.Error(0)
}

Implementing Service Tests

In the service_test.go file, we focus on testing business rules: bucket name validation, UUID generation, and error handling.

package upload

import (
    "context"
    "strings"
    "testing"
    "time"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/mock"
)

type readSeekCloser struct {
    *strings.Reader
}

func (rsc readSeekCloser) Close() error { return nil }

func TestUploadFile_InvalidBucket(t *testing.T) {
    mockRepo := new(RepositoryMock)
    service := NewService(mockRepo)

    result, err := service.UploadFile(context.Background(), "", &File{})

    assert.Error(t, err)

    assert.Empty(t, result)

    assert.ErrorIs(t, err, ErrBucketNameRequired)
}

func TestUploadFile_Success(t *testing.T) {
    mockRepo := new(RepositoryMock)
    service := NewService(mockRepo)
    ctx := context.Background()

    content := strings.NewReader("\x89PNG\r\n\x1a\n" + strings.Repeat("0", 512))
    file := &File{
        Name:    "test-image.png",
        Content: readSeekCloser{content},
    }

    bucket := "my-test-bucket"
    expectedURL := "https://s3.amazonaws.com/my-test-bucket/unique-id.png"

    mockRepo.On("Upload", mock.Anything, bucket, mock.AnythingOfType("*upload.File")).Return(expectedURL, nil)

    resultURL, err := service.UploadFile(ctx, bucket, file)

    assert.NoError(t, err)
    assert.NotEmpty(t, resultURL)
    assert.Equal(t, expectedURL, resultURL)

    mockRepo.AssertExpectations(t)
}

func TestGetDownloadURL_Success(t *testing.T) {
    mockRepo := new(RepositoryMock)
    service := NewService(mockRepo)

    bucket := "my-bucket"
    key := "image.png"
    expectedPresignedURL := "https://s3.amazonaws.com/my-bucket/image.png?signed=true"

    mockRepo.On("GetPresignURL", mock.Anything, bucket, key, 15*time.Minute).
        Return(expectedPresignedURL, nil)

    url, err := service.GetDownloadURL(context.Background(), bucket, key)

    assert.NoError(t, err)
    assert.Equal(t, expectedPresignedURL, url)
    mockRepo.AssertExpectations(t)
}

Technical Highlights in Tests

Test the rule, not the integration: the Service must be predictable; integrations are left for dedicated tests.
Security validation reflected in tests: if the API validates MIME types, the tests must respect this to remain realistic.
errors.Is and semantic errors: this strengthens consistency and makes assertions easier and clearer.

Automation with GitHub Actions

On every push or pull request, the CI installs dependencies and runs tests. This keeps the main branch stable and reduces the risk of regressions.

Configuring the Workflow (`go.yml`)

The configuration file should be created at .github/workflows/go.yml. It defines the steps required to validate the health of the project.

# .github/workflows/go.yml
name: Go CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    name: Run Tests 
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.25.5'
          cache: true

      - name: Install dependencies
        run: go mod download

      - name: Verify dependencies
        run: go mod verify

      - name: Run tests
        run: go test -v -race ./...

Technical Highlights in Automation

Race detector (-race): important because there is concurrency (parallel uploads).
Dependency caching: speeds up builds and reduces execution time.
Secure pipeline: since the Service is tested with mocks, the CI does not require AWS credentials.

Centralized Configuration

In small projects, it is common to scatter os.Getenv calls. As projects grow, this becomes technical debt: dependencies become implicit and hard to track.

Centralizing configuration creates a “single source of truth”: validation and typing at startup, and simple consumption throughout the rest of the codebase.

package config

import (
    "os"
    "strconv"
    "time"
)

type Config struct {
    Port          string
    AWSRegion     string
    UploadTimeout time.Duration
    Env           string
}

func Load() *Config {
    return &Config{
        Port:          getEnv("PORT", "8080"),
        AWSRegion:     getEnv("AWS_REGION", "us-east-1"),

        UploadTimeout: time.Duration(getEnvAsInt("UPLOAD_TIMEOUT_SECONDS", 30)) * time.Second,
        Env:           getEnv("APP_ENV", "development"),
    }
}

func getEnv(key, defaultValue string) string {
    if value, exists := os.LookupEnv(key); exists {
        return value
    }
    return defaultValue
}

func getEnvAsInt(key string, defaultValue int) int {
    valueStr := getEnv(key, "")
    if value, err := strconv.Atoi(valueStr); err == nil {
        return value
    }
    return defaultValue
}

Semantic Errors

The Service should not return HTTP status codes. It should return errors that make sense within the domain, and the Handler decides the appropriate status. This keeps the domain reusable (HTTP today, gRPC tomorrow, CLI later).

package upload

import "errors"

var (
    ErrBucketNameRequired  = errors.New("bucket name is required")
    ErrFileNotFound        = errors.New("file not found in storage")
    ErrInvalidFileType     = errors.New("file type not allowed or malicious content detected")
    ErrBucketAlreadyExists = errors.New("bucket already exists")
    ErrOperationTimeout    = errors.New("the operation timed out")
)

Technical Highlights in Error Handling

Domain-agnostic design: errors remain meaningful outside of HTTP.
Robust comparison with errors.Is: safer than comparing strings.
Consistency: stable and predictable messages for API consumers.

Containerization with Docker

Containerization ensures that the API runs the same way in any environment. Here I use a multi-stage build: compilation happens in an image with the toolchain, and the final runtime image is smaller and more secure.

Optimized Dockerfile

services:
  s3-api:
    build: .
    ports:
      - "8080:8080"
    env_file:
      - .env
    restart: always

Orchestration with Docker Compose

The docker-compose.yml file simplifies application startup by managing ports and automatically loading our secrets file (.env).

# Stage 1: Build the Go binary
FROM golang:1.25.5-alpine AS builder

WORKDIR /app

# Copy dependency files
COPY go.mod go.sum ./
RUN go mod download

# Copy the source code
COPY . .

# Build the application with optimizations
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o main ./cmd/api/main.go

# Stage 2: Create the final lightweight image
FROM alpine:latest

RUN apk --no-cache add ca-certificates

WORKDIR /root/

# Copy the binary from the builder stage
COPY --from=builder /app/main .
# Copy the .env file (optional, better to use environment variables in prod)
COPY --from=builder /app/.env . 

EXPOSE 8080

CMD ["./main"]

Technical Highlights in Docker

SSL certificates: minimal images do not always include CA certificates; without them, HTTPS calls may fail.
Slim binary: build flags reduce size and speed up deployment.
Smart caching: copying go.mod/go.sum before the source code helps Docker reuse layers efficiently.

Useful Commands

Tests

go test ./...

Running the Project

go run cmd/api/main.go

Build the image and start the container in the background

# Build the image and start the container in the background
docker-compose up --build -d

# View application logs in real time
docker logs -f go-s3-api

# Stop and remove the container
docker-compose down

Conclusion

Building this management API for Amazon S3 was a practical way to combine a real need (automating my blog workflow) with solid learning (Go, AWS, security, observability, and architecture).

Project Links

Main GitHub Repository: This is the “living” repository, which may evolve and differ from what is described in this article.
Article Version Code: To see the project exactly as it was built and explained here, in a static state.

References

Go Standard Project Layout: https://github.com/golang-standards/project-layout
Clean Architecture (The Clean Code Blog): https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html
AWS SDK for Go v2 – S3 Developer Guide: https://aws.github.io/aws-sdk-go-v2/docs/getting-started/
Go Concurrency Patterns (Context Package): https://go.dev/blog/context
Gin Web Framework: https://gin-gonic.com/docs/
Testify – Testing Toolkit: https://github.com/stretchr/testify
Google UUID Package (v7 support): https://github.com/google/uuid
Errgroup Package (Concurrency): https://pkg.go.dev/golang.org/x/sync/errgroup
Dockerizing a Go App: https://docs.docker.com/language/golang/build-images/
GitHub Actions for Go: https://github.com/actions/setup-go

Redis Implementing Pub Sub and Streams in NET 10

Joao Oliveira — Thu, 15 Jan 2026 14:02:02 +0000

Redis (Remote Dictionary Server) can do much more than just act as a cache. I know that this is the most common use case because it protects your database from excessive requests and improves response times by keeping data in memory, but Redis offers much more.

Redis can also working as a powerful message broker. With native support for Pub/Sub and Streams, if you already have an instance of Redis running, you may want to take a look at other features that can do your job better.

In this article, I'll demonstrate how to implement both Pub/Sub and Streams as message brokers using Redis and .NET. By the end, you should be able to evaluate which approach is the best option for your specific scenario.

Concepts: Pub/Sub vs Stream

Think of Pub/Sub like a live Radio Broadcast. If your radio is off, you miss the song. Streams, on the other hand, are more like YouTube. You can watch the video whenever you want, and it remains available until it is deleted. Below is a comparison between the

Feature	Pub/Sub	Streams
Persistence	No. Messages are lost if no one is listening	Yes. Messages are stored in Redis until deleted
Delivery	One-to-Many	Many-to-Many
History	No	Yes. You can read "past" messages
Use Case	Real-time chats, notifications	Order processing, Audit logs, Event Sourcing

Environment Configuration

We’ll build this project with .NET 10. First, create a Minimal API project from the terminal:

# 1. Create project folder
mkdir RedisMessageLab && cd RedisMessageLab

# 2. Create Minimal API project
dotnet new webapi -n MessagingApi -minimal

# 3. Add the Redis Driver
cd MessagingApi
dotnet add package StackExchange.Redis

# 4. Create the Docker Compose file in the root
cd ..
touch docker-compose.yml

Docker Setup

Create a docker-compose.yml file and add the following configuration:

services:
  redis:
    image: redis:7.4-alpine
    container_name: redis-lab
    ports:
      - "6379:6379"
    command: ["redis-server", "--appendonly", "yes"] # Enable persistence for Streams

Start the container with:

docker-compose up -d

Project Implementation

Model

Create an AppMessage record to pass data through our system.

public record AppMessage(string Id, string Content, string Sender);

Interface

Create an IRedisService interface.

public interface IRedisService 
{
    Task<long> PublishAsync(AppMessage message);
    Task AppendAsync(AppMessage message);
    Task<AppMessage?> ConsumeAndStackDeleteAsync();
    Task<List<AppMessage>> PeekHistoryAsync();
}

Service

Create a RedisService class.

public class RedisService(IConnectionMultiplexer redis) : IRedisService
{
    private const string Channel = "live_updates";
    private const string Stream = "audit_stream";

    public async Task<long> PublishAsync(AppMessage message)
    {
        var sub = redis.GetSubscriber();
        string json = JsonSerializer.Serialize(message);

        // Publish returns the count of active subscribers
        long subscribersCount = await sub.PublishAsync(RedisChannel.Literal(Channel), json);

        if (subscribersCount != 0) return subscribersCount;

        // This will show up in your .NET Terminal/Console
        Console.ForegroundColor = ConsoleColor.Yellow;
        Console.WriteLine($"[PUB/SUB WARNING] Message {message.Id} was sent, but 0 subscribers were listening. Data is lost!");
        Console.ResetColor();

        return subscribersCount;
    }

    public async Task AppendAsync(AppMessage message)
    {
        var db = redis.GetDatabase();
        await db.StreamAddAsync(Stream, [
            new NameValueEntry("id", message.Id), 
            new NameValueEntry("content", message.Content)]);
    }

    public async Task<AppMessage?> ConsumeAndStackDeleteAsync()
    {
        var db = redis.GetDatabase();
        // Read the oldest message
        var messages = await db.StreamReadAsync(Stream, "0-0", count: 1);
        if (messages.Length == 0) return null;

        var msg = messages.First();
        var note = new AppMessage(msg.Values[0].Value!, msg.Values[1].Value!, "Stream");

        // Delete after consuming
        await db.StreamDeleteAsync(Stream, [msg.Id]);
        return note;
    }

    public async Task<List<AppMessage>> PeekHistoryAsync()
    {
        var db = redis.GetDatabase();

        // XRANGE audit_stream - + (Read everything from start to finish)
        var entries = await db.StreamRangeAsync(Stream, "-", "+");

        return entries.Select(e => new AppMessage(
            e.Values.FirstOrDefault(v => v.Name == "id").Value!,
            e.Values.FirstOrDefault(v => v.Name == "content").Value!,
            "Stream History"
        )).ToList();
    }
}

Workers

Create a LiveNotificationWorker worker to handle live notifications.

public class LiveNotificationWorker : BackgroundService 
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
        var redis = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
        var sub = redis.GetSubscriber();

        await sub.SubscribeAsync(RedisChannel.Literal("live_updates"), (channel, message) => {
            // This proves Pub/Sub success in the logs
            Console.WriteLine($"[PUB/SUB SUCCESS] Received Live: {message}");
        });
    }
}

Program.cs

Open Program.cs and add the code below.

var builder = WebApplication.CreateBuilder(args);
var multiplexer = ConnectionMultiplexer.Connect("localhost:6379");
builder.Services.AddSingleton<IConnectionMultiplexer>(multiplexer);
builder.Services.AddScoped<IRedisService, RedisService>();
builder.Services.AddHostedService<LiveNotificationWorker>();

var app = builder.Build();

// Endpoint 1: Test Pub/Sub
app.MapPost("/broadcast", async (AppMessage message, IRedisService service) => {
    long listeners = await service.PublishAsync(message);
    return Results.Ok(new { 
        Mechanism = "Pub/Sub", 
        ActiveListeners = listeners,
        Status = listeners > 0 ? "Delivered" : "Lost (No Listeners)"
    });
});

// Endpoint 2: Test Stream Storage
app.MapPost("/log", async (AppMessage message, IRedisService service) => {
    await service.AppendAsync(message);
    return Results.Ok("Message logged in Stream.");
});

// Endpoint 3: Test Stream Consume & Delete
app.MapPost("/log/process", async (IRedisService service) => {
    var note = await service.ConsumeAndStackDeleteAsync();
    return note is null ? Results.NotFound("No logs left.") : Results.Ok(note);
});

// Endpoint 4: Peek at History (Without deleting)
app.MapGet("/log/history", async (IRedisService service) =>
{
    var history = await service.PeekHistoryAsync();
    return history.Count != 0 ? Results.Ok(history) : Results.NotFound("The Stream is empty.");
});

app.Run();

Comparison

Feature	Pub/Sub Success	Stream Success
Data Delivery	Instant, only to active users	Stored, can be claimed later
Cleanup	Automatic (nothing is stored)	Manual (you must `XDEL` or `XTRIM`)
Guarantee	None	At-least-once

Testing the endpoints

I'll use the Apidog to test the endpoints, but you can use your preferred tool.

Run the project from the terminal with:

dotnet run

The project should then be running at http://localhost:5040

Pub/Sub Broadcast

This endpoint sends a message that the background worker should receive instantly in the terminal.

Method: POST
Endpoint: /broadcast

Body (JSON)

{
    "id": "PS-100",
    "content": "Live alert: System update at 10 PM",
    "sender": "AdminPanel"
}

In Apidog, you will see the message"Message broadcasted. Check terminal for success." and in the terminal you should see the message as shown in the image below.

Simulate a Pub/Sub without Worker

If you want to simulate a scenario where no worker is running, follow these steps:

Comment out this line in Program.cs: builder.Services.AddHostedService<LiveNotificationWorker>();
Send a request to /broadcast.
The API returns 200 OK, but the message is gone forever. You will not see the success message in the terminal, as you did in the previous test.

Stream Log

This endpoint sends a message to a Redis Stream, where it will remain until you manually process it.

Method: POST
Endpoint: /log

Body (JSON)

{
    "id": "STR-500",
    "content": "CRITICAL: Database connection retry",
    "sender": "DB-Monitor"
}

If the message is published successfully, you will see in the response "Message logged in Stream."

Note: You can send multiple messages; just change something in the body to better illustrate the test.

Peak History on Stream

With this endpoint, you can retrieve all messages published to the Stream that have not been consumed.

Method: GET
Endpoint: /log/history
Body: None

You will see all published messages in the response:

Stream Process

This request fetches the oldest message from the stream and then deletes it from Redis.

Method: POST
Endpoint: /log/process
Body: None

The first time you click Send, you will receive the JSON for STR-500.
The second time you click Send, if you have not sent any additional messages, you will receive a 404 Not Found response (because the message was deleted after the first consumption). If you send more messages to the /log endpoint, you must consume all of them before receiving a 404, which indicates that the stream is empty.

To view the stored messages in the terminal, you can use the following command:

docker exec -it redis-lab redis-cli XRANGE audit_stream - +

Process Flow

The diagram illustrates the fundamental difference between Redis Pub/Sub and Redis Streams in terms of message delivery and persistence.

Conclusion

As shown throughout this article, Redis is a solid option for use as message broker, especially when response time is critical for your application, or when you already have a Redis instance running and do not want to introduce an additional messaging tool.

You can choose Pub/Sub when you need to build features such as a live chat, a gaming lobby, or a proximity-based matching system. If your application requires monitoring active users or real-time updates, and occasional message loss is acceptable, Pub/Sub is a good fit. It is fast, lightweight, and does not consume memory by storing messages.

On the other hand, when message retention is critical and you cannot afford to lose data, such as in payment flows or order processing. Streams are the better choice. Messages are persisted, ordered, and can be processed later, even if consumers are temporarily unavailable.

In short, Pub/Sub is about speed and immediacy, while Streams focus on durability and control.

Project Source Code

You can find the complete implementation of this project on my GitHub: [Your GitHub Repository Link Here]

References & Further Reading

Redis Documentation: Pub/Sub – Official guide on the publish/subscribe messaging paradigm.

Redis Documentation: Streams – Deep dive into the stream data type and consumer groups.

StackExchange.Redis GitHub – Documentation for the leading .NET Redis client used in this project.

Microsoft Docs: Minimal APIs Overview – Reference for building high-performance APIs with .NET 10.

Docker Docs: Compose File Reference – Guide for setting up multi-container applications.

Enrich Logs with CorrelationID in .NET

Joao Oliveira — Tue, 09 Dec 2025 19:22:15 +0000

A Correlation ID is a good technique when you need to track a log journey in your system. This is especially true when a single user request spans multiple services and layers in your architecture. For example, when your log is added across these various components, and you need to track the entire log lifecycle, it is very difficult and painful if you don't have a way to track it by a common value.

This is where the Correlation ID comes in to make your life better. This technique will automatically add a unique identifier to all your logs that you can use in our preferred analytic tool to view the entire sequence of events, even when the request is propagated to other internal services. This provides an easy way to filter and display only what you need to know, providing end to end traceability.

In .NET, we have an easy way to implement this, allowing us to create a unique identifier in all logs that we can use for a single track. In this article, I'll show you how to do it using the native ILogger in .NET, and how to ensure this ID travels with your outbound HTTP requests.

So, let's build it together.

Create a Middleware

The first step is to create a middleware with a default configuration to include the Correlation ID in each logger. To make it unique, we will use a GUID (Globally Unique Identifier) to generate a unique identifier. This middleware is responsible for receiving and establishing the trace ID for the current service.

public sealed class CorrelationIdMiddleware(RequestDelegate next, ILogger<CorrelationIdMiddleware> logger)
{
    public async Task InvokeAsync(HttpContext context)
    {
        string correlationId =
            context.Request.Headers.TryGetValue(CorrelationIdConstants.HeaderName, out var headerValue)
            && !string.IsNullOrWhiteSpace(headerValue)
                ? headerValue.ToString()
                : Guid.NewGuid().ToString();

        context.Items[CorrelationIdConstants.LogPropertyName] = correlationId;
        context.Response.Headers[CorrelationIdConstants.HeaderName] = correlationId;

        using (logger.BeginScope(new { CorrelationId = correlationId }))
        {
            await next(context);
        }
    }
}

This middleware does several important things:

Retrieval and Generation: It checks the incoming HTTP headers for an existing Correlation ID. If found (meaning the request came from another service), it reuses it. If not found, it generates a new unique GUID.
Response Header: It adds the Correlation ID to the outgoing response headers so the client or next service can use it for tracing.
Context Storage: It stores the ID in context.Items. This is the key piece that will allow us to access the ID for outbound propagation later.
Logging Scope: It uses logger.BeginScope() to link the Correlation ID to the current request's entire logging lifecycle. This ensures every subsequent log entry (ILogger) automatically includes the ID.
Pipeline Execution: It calls await next(context) to process the request, and thanks to the using block, the log scope is automatically cleaned up when the request is finished.

Defining Correlation ID Constants

To keep your project clean and follow best practices, let's create a file to add our constant values. By defining these values in a static class, we avoid "magic strings" and prevent potential runtime errors caused by typos, ensuring consistency across all components.

Create a static class named CorrelationIdConstants:

public static class CorrelationIdConstants
{
    public const string HeaderName = "X-Correlation-ID";
    public const string LogPropertyName = "CorrelationId";
}

Purpose of the Constants

HeaderName: Defines the specific HTTP header key that our middleware will look for in incoming requests and use to stamp outgoing responses.
LogPropertyName: Defines the property name that will appear in your log output.

Registering the Middleware

To use this middleware, you need to register it in your Program.cs file. This is the application's entry point where the request pipeline is configured.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();
builder.Services.AddOpenApi(); 

var app = builder.Build();

app.UseMiddleware<CorrelationIdMiddleware>();
app.UseHttpsRedirection();
app.MapControllers();

app.Run();

NOTE: It's critical to register this middleware early in the pipeline, to ensure the Correlation ID is established and available for all subsequent Middleware, Filters, and Controllers that handle the request.

Configure appsettings.json for Console Visibility

To ensure the Correlation ID is visible in your console logs, you must explicitly instruct the default console logging provider to include log scopes.

 {
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.EntityFrameworkCore": "Warning"
    },
    "Console": {
      "IncludeScopes": true
    }
  }
}

NOTE: The line IncludeScopes: true is needed to show in the .NET Console Logger, this is crucial for developers relying on the console during local development and testings.

However, if you were using a structured logging provider like Serilog or NLog, or sending logs to a specialized tool like Splunk, Kibana or Elastic, this setting is often not required. Those systems usually capture data automatically or use custom configurations to capture scope data or use custom configurations to include the properties directly.

Propagating the Correlation ID to External Services

When making HTTP requests to external services, we need to create a custom DelegatingHandler paired with the HttpClientFactory. This pattern allows us to intercept every outbound request made by a specific HttpClient and inject the CorrelationIdheader, ensuring our log trace is continuous across service boundaries.

The CorrelationIdDelegatingHandler Implementation This handler uses the injected IHttpContextAccessor to retrieve the active Correlation ID set by our inbound middleware and adds it to the outbound request headers.

public class CorrelationIdDelegatingHandler(IHttpContextAccessor httpContextAccessor) : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var httpContext = httpContextAccessor.HttpContext;

        if (httpContext == null ||
            !httpContext.Items.TryGetValue(CorrelationIdConstants.LogPropertyName, out object? correlationIdObject))
            return await base.SendAsync(request, cancellationToken);

        string? correlationId = correlationIdObject.ToString();
        request.Headers.TryAddWithoutValidation(CorrelationIdConstants.HeaderName, correlationId);

        return await base.SendAsync(request, cancellationToken);
    }
}

Creating the Generic HTTP Client Service The HttpClientService is designed to be your reusable client. It takes the array of DelegatingHandler services resolved by DI and manually chains them together with a base transport handler (SocketsHttpHandler) to create a correctly configured HttpClient.

public class HttpClientService(IHttpClientFactory httpClientFactory, DelegatingHandler[] handlers)
    : IHttpClientService
{
    public Task<HttpResponseMessage> CallServiceAsync(
        string baseUrl, string authKey, HttpMethod method, string endpoint)
    {
        HttpMessageHandler currentHandler = new SocketsHttpHandler(); 

        foreach (var handler in handlers.Reverse()) 
        {
            handler.InnerHandler = currentHandler;
            currentHandler = handler;
        }

        var client = new HttpClient(currentHandler);

        client.BaseAddress = new Uri(baseUrl);
        client.DefaultRequestHeaders.Add("Authorization", $"Bearer {authKey}");

        var request = new HttpRequestMessage(method, endpoint);

        return client.SendAsync(request);
    }
}

The Centralized Registration in Program.cs This is the foundation of the advanced propagation pattern. We use a Factory Method to correctly resolve the collection of DelegatingHandler services into the array required by the HttpClientService.

builder.Services.AddHttpContextAccessor();
builder.Services.AddHttpClient();
builder.Services.AddTransient<DelegatingHandler, CorrelationIdDelegatingHandler>();
builder.Services.AddSingleton<IHttpClientService>(provider => 
{
    var factory = provider.GetRequiredService<IHttpClientFactory>();
    var handlers = provider.GetServices<DelegatingHandler>().ToArray(); 
    return new HttpClientService(factory, handlers);
});

Tracing the Log Journey (Error Scenario)

To illustrate the power of this implementation, observe the console log below. This example captures a single user request for an endpoint that retrieves a user by ID.

Crucially, the request failed because the provided ID does not exist in the database, demonstrating how the Correlation ID tracks the operation across multiple application layers and during an error condition.

Notice how the unique CorrelationId links logs from the API down to the Domain and Infrastructure layers.

Conclusion

Using the Correlation ID is a very powerful and essential way to improve the observability in distributed applications.

By implementing this native .NET pattern, we established a single, reliable identifier that automatically tracks requests from the initial inbound API call, through internal application logic, and finally across all outbound service calls via the custom DelegatingHandler. This robust, end to end solution transforms debugging and monitoring, allowing you to trace complex operations with ease and speed.

Stack, Queue and PriorityQueue in C#

Joao Oliveira — Tue, 04 Nov 2025 17:31:59 +0000

Stack, Queue and PriorityQueue are fundamental data structures used to store, organize and manage data efficiently in C#.

They are not only useful for solving algorithm challenges, but also extremely valuable in real-world applications such as searching, ordering tasks, inserting, removing and updating data in different workflows.
In this article, I will show the main characteristics of each one, explain how they work, and demonstrate real-world usage scenarios to help reinforce memorization and understanding.
So, let's start talking about these important data structures in C#.

Stack

A Stack<T> represents a LIFO (Last In, First Out) collection.
This means the last item added will be the first one removed.

Constructor

Constructor	Description
`Stack()`	Creates an empty stack.
`Stack(int capacity)`	Creates a stack with an initial internal capacity. Useful when the expected size is known.
`Stack(IEnumerable<T> collection)`	Creates a stack with items copied from another collection (the last element becomes the top).

Core Operations

Member	Description	Example
`Push(T item)`	Adds an item to the top	`stack.Push("A");`
`Pop()`	Removes and returns the top item. Throws if empty	`var item = stack.Pop();`
`Peek()`	Returns the top item without removing it. Throws if empty	`var item = stack.Peek();`
`TryPop(out T result)`	Safe Pop (returns false if empty)	`if(stack.TryPop(out var value)){...}`
`TryPeek(out T result)`	Safe Peek (returns false if empty)	`if(stack.TryPeek(out var value)){...}`

Utility Methods

Member	Description
`Count`	Number of elements in the stack
`Clear()`	Removes all items
`Contains(T item)`	Checks if the stack contains the given item
`ToArray()`	Returns a new array in LIFO order
`GetEnumerator()`	Iterates through the stack (top → bottom)
`TrimExcess()`	Reduces memory usage after many removals

Example

    Stack<string> stack = new();

    stack.Push("First");
    stack.Push("Second");
    stack.Push("Third");

    Console.WriteLine("Top of the stack: " + stack.Peek());

    Console.WriteLine("Removing items:");
    while (stack.Count > 0)
        Console.WriteLine(stack.Pop());

    Console.WriteLine("Stack empty? " + (stack.Count == 0));

Output

Top of the stack: Third
Removing items:
Third
Second
First
Stack empty? True

Queue

A Queue<T> represents a FIFO (First In, First Out) collection.
The first element inserted is the first one removed.

Constructor

Constructor	Description
`Queue()`	Creates an empty queue.
`Queue(int capacity)`	Creates a queue with a predefined capacity.
`Queue(IEnumerable<T> collection)`	Initializes a queue with elements, where the first item becomes the front.

Core Operations

Member	Description	Example
`Enqueue(T item)`	Adds an item to the back (tail).	`queue.Enqueue("A");`
`Dequeue()`	Removes and returns the front item. Throws if empty.	`var item = queue.Dequeue();`
`Peek()`	Returns the front item without removing it.	`var item = queue.Peek();`
`TryDequeue(out T result)`	Safe dequeue	`if(queue.TryDequeue(out var x)) { ... }`
`TryPeek(out T result)`	Safe peek	`if(queue.TryPeek(out var x)) { ... }`

Utility Methods

Member	Description
`Count`	Number of elements
`Contains(T item)`	Checks if the item exists
`ToArray()`	Returns items in FIFO order
`Clear()`	Removes all elements
`GetEnumerator()`	Iterates through the queue front → back
`EnsureCapacity(int capacity)`	Ensures enough internal storage to avoid resizing
`TrimExcess()`	Reduces unused memory

Code:

Queue<string> queue = new();

queue.Enqueue("First");
queue.Enqueue("Second");
queue.Enqueue("Third");

Console.WriteLine("Front of the queue: " + queue.Peek());

Console.WriteLine("Processing queue:");
while (queue.Count > 0)
    Console.WriteLine(queue.Dequeue());

Console.WriteLine("Queue empty? " + (queue.Count == 0));

Output

Front of the queue: First
Processing queue:
First
Second
Third
Queue empty? True

PriorityQueue

A PriorityQueue<TElement, TPriority> is similar to a Queue, except element removal is based on priority, not insertion order.
By default, it behaves as a Min-Heap (lower priority value = higher priority).

Type:

PriorityQueue<TElement, TPriority>

TElement → the value you are storing
TPriority → determines the processing order (lower numbers = higher priority by default)

Constructors:

Consctructor	Description
`PriorityQueue()`	Creates an empty priority queue
`PriorityQueue(int initialCapacity)`	Pre-allocates capacity to reduce resizing
`PriorityQueue(IEnumerable<(TElement,TPriority)> items)`	Creates a queue with the specified items
`PriorityQueue(IEnumerable<(TElement,TPriority)> items, IComparer<TPriority>? priorityComparer)`	Same, but allows custom priority logic

Core Methods

Member	Description	Example
`Enqueue(TElement element, TPriority priority)`	Adds an element with its priority	`queue.Enqueue("Task", 2);`
`Dequeue()`	Removes and returns the element with highest priority	`var item = queue.Dequeue();`
`Peek()`	Returns the highest priority element without removing	`var item = queue.Peek();`
`TryDequeue(out TElement element, out TPriority priority)`	Safe dequeue	`if(queue.TryDequeue(out var e, out var p)) {...}`
`TryPeek(out TElement element, out TPriority priority)`	Safe peek	`if(queue.TryPeek(out var e, out var p)) {...}`

Utility Methods

Member	Description
`Count`	Number of items
`Clear`	Removes all elements
`EnsureCapacity(int capacity)`	Pre-allocates memory
`TrimExcess()`	Reduces memory usage
`EnqueueDequeue(element, priority)`	Inserts item then removes the highest priority item
`UnorderedItems`	Returns internal items in no particular order

Code:

PriorityQueue<string, int> queue = new();

queue.Enqueue("Fix critical bug", 1);
queue.Enqueue("Team meeting", 2);
queue.Enqueue("Review PR", 3);

Console.WriteLine($"Count: {queue.Count}");
Console.WriteLine($"Peek: {queue.Peek()}");

if (queue.TryDequeue(out string? task, out int priority))
    Console.WriteLine($"Dequeued: {task} (Priority {priority})");

queue.EnqueueRange([
    ("Refactor service", 4),
    ("Write documentation", 5)
]);

string removed = queue.EnqueueDequeue("Urgent hotfix", 0);
Console.WriteLine($"Removed during EnqueueDequeue: {removed}");

Console.WriteLine("\nRemaining items by priority:");
while (queue.TryDequeue(out string? t, out int p))
    Console.WriteLine($"→ {t} (p={p})");

queue.TrimExcess();

Output

Count: 3
Peek: Fix critical bug
Dequeued: Fix critical bug (Priority 1)
Removed during EnqueueDequeue: Urgent hotfix

Remaining items by priority:
→ Team meeting (p=2)
→ Review PR (p=3)
→ Refactor service (p=4)
→ Write documentation (p=5)

Turning PriorityQueue into Max-Heap

To return the highest number instead of the lower in a PriorityQueue you can create a custom comparer:

var queue = new PriorityQueue<string, int>( comparer: Comparer<int>.Create((x, y) => y.CompareTo(x)));

Example:

var queue = new PriorityQueue<string, int>( comparer: Comparer<int>.Create((x, y) => y.CompareTo(x)));

queue.Enqueue("Low importance", 1);
queue.Enqueue("Medium importance", 5);
queue.Enqueue("High importance", 10);

while (queue.TryDequeue(out string? task, out int priority))
    Console.WriteLine($"{task} (priority {priority})");

High importance (priority 10)
Medium importance (priority 5)
Low importance (priority 1)

Stack vs Queue vs PriorityQueue (Quick Comparison)

Structure	Order Rule	Core Operations	Example Use Case
Stack	LIFO (Last In, First Out)	Push / Pop / Pee	Undo systems, Call stack, DFS
Queue	FIFO (First In, First Out)	Enqueue / Dequeue / Peek	Scheduling, Task processing, BFS
PriorityQueue	Priority-based ordering	Enqueue(element, priority) / Dequeue	Job scheduling, AI pathfinding, Hospital triage

Real Word Examples

Stack - Undo History

When typing in a text editor, each action gets pushed into a history stack.
If the user presses Undo, we pop the last action.

var undoStack = new Stack<string>();

Type("Hello");
Type("Hello World");
Type("Hello World!");

Undo(); // removes "Hello World!"
Undo(); // removes "Hello World"
return;

void Undo()
{
    if (undoStack.TryPop(out string? lastAction))
        Console.WriteLine($"Undo → removed: {lastAction}");
}

void Type(string text)
{
    undoStack.Push(text);
    Console.WriteLine($"Typed: {text}");
}

Queue - Print Queue

Print Queue (Printers process jobs in arrival order)

var printQueue = new Queue<string>();

AddPrintJob("Report.pdf");
AddPrintJob("Invoice.docx");
AddPrintJob("Presentation.pptx");

ProcessNextJob(); // Report
ProcessNextJob(); // Invoice
ProcessNextJob(); // Presentation
return;

void ProcessNextJob()
{
    if (printQueue.TryDequeue(out string? job))
        Console.WriteLine($"Printing: {job}");
}

void AddPrintJob(string document)
{
    printQueue.Enqueue(document);
    Console.WriteLine($"Added: {document}");
}

PriorityQueue - Hospital ER Triage

Hospital Emergency Room Triage
Patients are not seen in order of arrival — they are seen by severity.

var erQueue = new PriorityQueue<string, int>(
    comparer: Comparer<int>.Create((a, b) => a.CompareTo(b)) // lower value = higher priority
);

erQueue.Enqueue("Head trauma", 1);
erQueue.Enqueue("Broken arm", 4);
erQueue.Enqueue("High fever", 3);
erQueue.Enqueue("Cardiac arrest", 0);

Console.WriteLine("Patients being treated:");
while (erQueue.TryDequeue(out string? patient, out int severity))
    Console.WriteLine($"{patient} (severity {severity})");

Scalable Distributed Web Crawler

Joao Oliveira — Fri, 24 Oct 2025 16:34:31 +0000

Goal: To design a distributed web crawler capable of processing 100 million pages per day, demanding high throughput (2,500+ RPS peak) while strictly adhering to web politeness protocols.

System Requirements

Functional Requirements (FR)

FR-01 Seed URL Ingestion: The system must be able to accept initial URL seeds to begin the crawl.
FR-02 Text Extraction and Storage: Text content must be extracted from pages and stored in a durable persistence layer for later analysis.
FR-03 Freshness / Re-crawling: URLs must be revisited periodically to ensure data freshness, respecting specified Crawl-delay or pre-set recrawl intervals.

Non-Functional Requirements (NFR)

NFR-01 Performance (Scale): The system must sustain 100 million pages per day (2,500 + RPS peak).
NFR-02 Politeness (Rate Limiting): A distributed Rate Limiting mechanism is mandatory to enforce a minimum delay (1-2 seconds or Crawl-delay) per host.
NFR-03 Resilience & Fault Tolerance: The system must be resilient to failures, implementing Retry logic with Exponential Backoff and using a Dead-Letter Queue (DLQ).
NFR-04 Optimization: The system must utilize HTTP headers (If-Modified-Since / ETag) to avoid downloading unchanged content (bandwidth optimization).

Back-of-the-Envelope Estimation

The capacity estimates justify our choice of distributed technologies:

Throughput: The average requirement of ≈ 1,157 RPS escalates to a peak design capacity of 2,500 to 3,000 RPS to handle overhead and retries.
Content Storage: At 10 KB of text per page, we face 1 TB per day (365 TB annually). This mandates a low-cost, high-capacity Object Storage solution (like AWS S3).
Metadata Storage: Storing critical URL state (timestamps, ETag) for 5 billion unique URLs requires a highly performant NoSQL database optimized for high read/write volume (like Cassandra).

High-Level Design (HLD) Components and Data Flow

The architecture is separated into specialized services, orchestrated by asynchronous queues to decouple network-bound and CPU-bound tasks.

Core Components and Politeness Enforcement

URL Frontier Queue (Apache Kafka): Serves as the central persistent queue for all URLs waiting to be crawled.
Scheduler Service: Pulls URLs from the Frontier, enforces politeness checks, and manages the scheduling of the next crawl.
Rate Limiter (Redis): An in-memory key-value store used to quickly check the last crawl time for any given host.
Downloader Workers: A massive, scalable pool responsible for making the actual HTTP requests.
Persistence Layer (Cassandra): The NoSQL database storing URL metadata (state, ETag, last crawl time) for Deduplication (FR-04) and Freshness checks (FR-03).

Architectural Deep Dive and Optimization

The Scalable URL Frontier and Host-Based Partitioning

To maximize throughput while strictly adhering to politeness (NFR-02), the Kafka Frontier is optimized using Host-Based Partitioning.

Partitioning Strategy: URLs are partitioned based on their hostname hash (e.g., hash(google.com)).
Benefit: All URLs belonging to the same host are guaranteed to land on the same Kafka partition. This allows a single, dedicated instance of the Scheduler Service to process all requests for that host sequentially, making the distributed rate-limiting check much simpler and more reliable. This effectively transforms the distributed problem into a per-partition serial processing task.

Politeness Enforcement (Scheduler and Redis)

The Scheduler Service and Redis work together to strictly enforce the per-host delay (NFR-02).

Scheduler Action: The Scheduler pulls the next URL from its assigned Kafka partition.
Redis Check: It queries Redis using the key structure HOST:{hostname}.
Atomic Politeness Check (Optimization): To prevent race conditions among multiple Schedulers, the check-and-update operation must be atomic. This is best achieved using a Redis Lua Script that performs both steps:
1. Check: Read the timestamp of the last access for the host.
2. Update: If the required delay has passed, update the timestamp with the current time and return ALLOW. Otherwise, return DELAY.
URL Requeueing: If the check returns DELAY the URL is immediately sent back to the Kafka Frontier (or an internal Delay Queue) with a time-to-wait parameter.

Deduplication, Freshness, and the Cassandra Data Model

The NoSQL Storage (Cassandra) is crucial for managing the state of billions of URLs. Its data model must support two extremely fast lookups:

Politeness: Quick retrieval of the last crawl time.
Deduplication (FR-04): Quick check if a newly discovered URL has already been processed.

Cassandra Schema Design:

Field	Description	Purpose
url_hash (Primary Key)	A unique hash of the full URL (e.g., SHA-256).	Fast Deduplication (FR-04) and uniqueness guarantee.
hostname	The host portion of the URL.	Used for politeness lookups and partitioning data.
last_crawl_time	Timestamp of the last successful fetch.	Required by the Scheduler for Politeness (NFR-02).
etag_header	The ETag value from the last response.	Required for Optimization (NFR-04).
status	Current state (e.g., TO_CRAWL, CRAWLED, PERMANENT_FAIL).

Optimization and Resilience Flow
The design incorporates advanced features for efficiency and reliability:

Bandwidth Optimization (NFR-04): Downloader Workers check the Cassandra store for the etag_header and last_crawl_time. They then use the HTTP headers If-Modified-Since and If-None-Match(using the ETag) in the request. If the content is unchanged, the server returns a 304 (Not Modified), saving significant bandwidth and processing load.
Resilience (NFR-03): The retry mechanism is implemented in the Downloader Worker:
- Retry Path: Transient errors (e.g., timeouts, 5xx errors) trigger a message sent back to the Download Queue with a scheduled delay tag, implementing Exponential Backoff (2^n delay, max 3 retries).
- Permanent Fail Path: After the retry limit is exhausted, the URL is routed to the Dead-Letter Queue (DLQ) (NFR-03) for out-of-band analysis, preventing toxic messages from clogging the system.
DNS Caching: The Downloader Workers utilize a local DNS Cache (Redis) to minimize latency and reduce reliance on external DNS lookups for frequently accessed hosts.

Native Mapping in .NET: Take Control with Extension Methods

Joao Oliveira — Thu, 16 Oct 2025 03:00:00 +0000

In the .NET universe, transforming objects from one type to another — like from Entities to DTOs — is part of the routine.
Most of us use libraries like AutoMapper or Mapster, which do this job elegantly. But... what if you want to have total control, ditch the magic, and even improve performance in some scenarios?
If this idea makes sense to you, come with me, because today we're going to talk about how to do native mapping in .NET using extension methods.

Why manual mapping?

Zero external dependency: One less library in your project.
Absolute control over conversion rules: No surprises, no implicit behavior.
Better performance in critical scenarios: Manual mapping is often faster than solutions based on reflection or expression.

Point of attention: You'll have to write (and maintain) the mapping by hand. The question is: is it worth it for your project?

Practical scenario: DTOs + User Entity.

DTOs

public class UserRequestDto
{
    public string Name { get; set; } = null!;
    public string Email { get; set; } = null!;
    public string Password { get; set; } = null!;
}

public class UserResponseDto
{
    public Guid Id { get; set; }
    public string Name { get; set; } = string.Empty;
    public string Email { get; set; } = string.Empty;
    public DateTime CreatedAt { get; set; }
    public DateTime? UpdatedAt { get; set; }
}

public class UserUpdateDto : UserRequestDto
{
    public Guid Id { get; set; }
}

Entity

public class User
{
    public Guid Id { get; set; }
    public string Name { get; set; } = string.Empty;
    public string Email { get; set; } = string.Empty;
    public string Password { get; set; } = string.Empty;
    public DateTime CreatedAt { get; set; }
    public DateTime? UpdatedAt { get; set; }
}

Mapping class using extension methods

public static class UserMapping
{
    // Creation
    public static User ToCreateUser(this UserRequestDto dto) => new()
    {
        Name = dto.Name,
        Email = dto.Email,
        Password = dto.Password,
        CreatedAt = DateTime.UtcNow
    };

    // Update
   public static void MapToUpdate(this User existingUser, UserUpdateDto userUpdateDto)
    {
        existingUser.Name = userUpdateDto.Name;
        existingUser.Email = userUpdateDto.Email;
        existingUser.Password = userUpdateDto.Password;
        existingUser.UpdatedAt = DateTime.UtcNow;
    }

    // Return
    public static UserResponseDto ToReturnUser(this User user) => new()
    {
        Id = user.Id,
        Name = user.Name,
        Email = user.Email,
        CreatedAt = user.CreatedAt,
        UpdatedAt = user.UpdatedAt
    };
}

Attention: The password is being mapped directly here only for didactic example purposes. In practice, always implement secure hashing (and never store passwords in plain text).

Create a user:

public async Task<string> CreateUser(UserRequestDto dto)
{
    var user = dto.ToCreateUser();
    var result = await _userRepository.CreateUser(user);

    return result ? "User created successfully." : "Creation failed.";
}

Update an existing user

public async Task<string> UpdateUser(UserUpdateDto dto)
{
    var user = await _userRepository.GetUserById(dto.Id);
    if (user is null)
        return "User not found.";

    user.MapToUpdate(dto);

    var result = await _userRepository.UpdateUser(user);

    return result ? "User updated successfully." : "Error updating.";
}

Get by Id

public async Task<UserResponseDto?> GetUserById(Guid id)
{
    var user = await _userRepository.GetUserById(id);
    return user?.ToReturnUser();
}

List all

public async Task<IEnumerable<UserResponseDto>> GetAllUsers()
{
    var users = await _userRepository.GetAllUsers();
    return users.Select(user => user.ToReturnUser());
}

Advantages and Disadvantages

Pros:

Simple, explicit, and direct code.
Zero magic: you know exactly what is happening.
Better performance in many cases, especially when compared to libraries that use reflection or expression.

Cons:

Manual code requires maintenance. If the model changes, you need to remember to update the methods.
In very large systems, the volume of mappings can grow considerably.
Without care, it can generate repetitive code.

Example project
I uploaded a complete project with this example on GitHub:

Shall we exchange ideas?

If this content helped you, leave a comment! Tell me:
Do you prefer manual mapping, AutoMapper, Mapster, or another solution?
Have you ever had performance problems because of mapping?

Distributed Cache in .NET: How to Configure and Use Redis

Joao Oliveira — Mon, 13 Oct 2025 19:45:57 +0000

In this article, I will show how to integrate Redis into a .NET API, using it as a Distributed Cache layer for the database. In high-read scenarios, this not only ensures an incredibly faster response time but also reduces the load and operational cost of your primary database.

For our example, we will use Redis in a local Docker container, configuring the application to communicate directly with it via the mapped port.

We will create a Generic Repository Class (ICacheRepository<T>) that will encapsulate the methods for key creation, retrieval, and removal. This is the most recommended pattern for maintaining a clean and reusable architecture.

Configuring the Environment with Docker and Redis

Installing and Starting Redis

Execute the commands below in your terminal. The first downloads the official Redis image, and the second starts the service.

docker pull redis:latest
docker run --name my-redis -d -p 6379:6379 redis

Technical Note: The docker run command uses the -p 6379:6379 mapping to expose the container's default Redis port to your local machine. my-redis is the name of the container we will use for subsequent interactions.

Verifying the Connection

To ensure the service is active and accessible, use the command-line client (redis-cli) through your container:

docker exec -it my-redis redis-cli PING

If the response is PONG, Redis is working perfectly and is ready to be integrated into your .NET API.

Structuring the .NET 9 Solution via CLI

Below are the commands to set up the project structure, separating the Web API layer from the Services and Cache layer.

Creating the Solution and Directories

Start by creating the root project folder and the solution file (.sln).

Create the Root Directory and Navigate to the Root

mkdir RedisCacheApi
cd RedisCacheApi

Create the Source Directory and Navigate to src

mkdir src
cd src

Create the Solution File

dotnet new sln -n RedisCacheApi

Create the API Project (Controllers) and Create the Service Project (Class Library)

dotnet new webapi -n RedisCache.Api -f net9.0
dotnet new classlib -n RedisCache.Service -f net9.0
cd ..

Adding the Projects to the Solution

dotnet sln add ./src/RedisCache.Api/RedisCache.Api.csproj
dotnet sln add ./src/RedisCache.Service/RedisCache.Service.csproj

Add the Reference

dotnet add ./src/RedisCache.Api/RedisCache.Api.csproj reference ./src/RedisCache.Service/RedisCache.Service.csproj

Add the Redis and Logger Packages

dotnet add src/RedisCache.Service/RedisCache.Service.csproj package Microsoft.Extensions.Caching.Abstractions

dotnet add src/RedisCache.Service/RedisCache.Service.csproj package Microsoft.Extensions.Logging.Abstractions

Cache Configuration and Abstraction

To integrate Redis into your API, we start by defining the connection string in the API project's configuration file.

Defining the Connection in appsettings.json

In the RedisCache.Api project, this file defines the port your application will use to connect to the Docker container (localhost:6379).

{
  "ConnectionStrings": {
    "RedisConnection": "localhost:6379"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

Note: .NET Core will use the "RedisConnection" key to configure the Distributed Cache service in Program.cs

Configuring the Pipeline and Dependency Injection (Program.cs)

This is the point where we connect the Redis client with the generic abstraction and the business service. The key is the registration of generic types in the AddScoped line.

var builder = WebApplication.CreateBuilder(args);

// 1. Pipeline Configuration
builder.Services.AddControllers(); 
builder.Services.AddLogging(); 

// 2. Distributed Redis Configuration
builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = builder.Configuration.GetConnectionString("RedisConnection");
    options.InstanceName = "RedisCacheApi_"; // Optional prefix for keys in Redis
});

// 3. Registration of Services and Repositories (Dependency Injection)
builder.Services.AddScoped(typeof(ICacheRepository<>), typeof(CacheRepository<>));

// Registers the Business Service.
builder.Services.AddScoped<IUserService, UserService>(); 

var app = builder.Build();

app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers(); 

app.Run();

Technical Configuration Notes

AddStackExchangeRedisCache: This method, provided by the package you added, injects the Redis implementation for .NET into the system as IDistributedCache.
options.InstanceName: The prefix is a good practice to prevent key collisions if you use the same Redis instance for multiple applications (e.g., RedisCacheApi_user:1).
AddScoped(typeof(ICacheRepository<>), typeof(CacheRepository<>)): This is the crucial line. It registers the open generic repository. The .NET Dependency Injection system is smart enough that when UserService requests ICacheRepository<User> (the closed generic type), it correctly instantiates the CacheRepository<User>.

The Controller (UsersController.cs)

This class acts as the entry point of your API, dealing only with translating HTTP requests into calls to the Business Service (IUserService).

namespace RedisCache.Api.Controllers;

[ApiController]
[Route("api/[controller]")]
public class UsersController(IUserService userService) : ControllerBase
{
        [HttpGet("GetAllUsers")]
        public async Task<ActionResult<List<User>>> GetAllUsers()
        {
            try
            {
                var users = await userService.GetAllUsersAsync();
                return Ok(users);
            }
            catch (Exception ex)
            {
                return StatusCode(500, "Internal error during list retrieval: " + ex.Message);
            }
        }

        [HttpGet("GetUser/{id:int}")]
        public async Task<ActionResult<User>> GetUser(int id)
        {
            try
            {
                var user = await userService.GetUserByIdAsync(id);
                return Ok(user);
            }
            catch (KeyNotFoundException ex)
            {
                return NotFound(ex.Message);
            }
            catch (Exception ex)
            {
                return StatusCode(500, "Internal error: " + ex.Message);
            }
        }

        [HttpPost("CreateUser")]
        public async Task<ActionResult<User>> CreateUser([FromBody] CreateUserDto userDto)
        {
            try
            {
                var newUser = await userService.CreateUserAsync(userDto);
                return CreatedAtAction(nameof(GetUser), new { id = newUser.Id }, newUser); 
            }
            catch (Exception ex)
            {
                return StatusCode(500, "Internal error: " + ex.Message); 
            }
        }

        [HttpPut("UpdateUser/{id:int}")]
        public async Task<ActionResult<User>> UpdateUser(int id, [FromBody] CreateUserDto userDto)
        {
            try
            {
                var updatedUser = await userService.UpdateUserAsync(id, userDto);
                return Ok(updatedUser);
            }
            catch (KeyNotFoundException ex)
            {
                return NotFound(ex.Message);
            }
            catch (Exception ex)
            {
                return StatusCode(500, "Internal error during update: " + ex.Message);
            }
        }

        [HttpDelete("DeleteUser/{id:int}")]
        public async Task<IActionResult> DeleteUser(int id)
        {
            try
            {
                await userService.DeleteUserAsync(id);
                return NoContent();
            }
            catch (KeyNotFoundException ex)
            {
                return NotFound(ex.Message);
            }
            catch (Exception ex)
            {
                return StatusCode(500, "Internal error during deletion: " + ex.Message);
            }
        }
}

The Generic Cache Repository

This class implements the generic contract, translating high-level calls from your services (e.g., userCache.GetAsync("1")) to low-level Redis operations (e.g., GetStringAsync("User:1"))

namespace RedisCache.Service.Services;

public class CacheRepository<T>(IDistributedCache cache) : ICacheRepository<T>
    where T : class
{
    private readonly string _keyPrefix = $"{typeof(T).Name}:";
    private readonly TimeSpan _defaultExpiration = TimeSpan.FromMinutes(10);

    private string GetCacheKey(string id) => $"{_keyPrefix}{id}";

    public async Task<T?> GetAsync(string id)
    {
        string key = GetCacheKey(id);
        string? cachedJson = await cache.GetStringAsync(key);

        // Returns the deserialized T object or null
        return cachedJson is null ? null : JsonSerializer.Deserialize<T>(cachedJson);
    }

    public async Task SetAsync(string id, T item, TimeSpan? expiration = null)
    {
        string key = GetCacheKey(id);
        string json = JsonSerializer.Serialize(item);

        // Sets the expiration, using the default if none is provided
        var options = new DistributedCacheEntryOptions()
            .SetAbsoluteExpiration(expiration ?? _defaultExpiration);

        await cache.SetStringAsync(key, json, options);
    }

    public Task RemoveAsync(string id)
    {
        string key = GetCacheKey(id);
        // Removes the key directly from Redis
        return cache.RemoveAsync(key);
    }
}

The Business Service with Caching

This class implements the Cache-Aside Pattern, utilizing the abstraction of the two injected repositories (userCache and allUsersCache) to manage keys optimally.

namespace RedisCache.Service.Services;

public class UserService(
    ICacheRepository<User> userCache,
    ICacheRepository<List<User>> allUsersCache, 
    ILogger<UserService> logger)
    : IUserService
{
    private const string AllUsersCacheKey = "all_users_list";

    // --- CREATE: Cache-Through and List Invalidation ---
    public async Task<User> CreateUserAsync(CreateUserDto userDto)
    {
        User newUser = new(_nextId++, userDto.Username, userDto.Email);
        Database.Add(newUser.Id, newUser);

        logger.LogInformation("User created in DB with ID {Id}", newUser.Id);

        string idString = newUser.Id.ToString();
        await userCache.SetAsync(idString, newUser); // 1. Cache-Through (saves individual)

        await allUsersCache.RemoveAsync(AllUsersCacheKey); // 2. Invalidates the list

        logger.LogInformation("All Users cache list invalidated.");

        return newUser;
    }

    // --- READ: Get By ID (Cache-Aside) ---
    public async Task<User> GetUserByIdAsync(int id)
    {
        string idString = id.ToString(); 
        var user = await userCache.GetAsync(idString); 

        if (user is not null)
        {
            logger.LogInformation("Cache HIT for user {Id}", id);
            return user;
        }

        logger.LogWarning("Cache MISS for user {Id}. Accessing DB.", id);
        await Task.Delay(200); 

        if (!Database.TryGetValue(id, out var userFromDb))
            throw new KeyNotFoundException($"User with ID {id} not found.");

        await userCache.SetAsync(idString, userFromDb); 
        logger.LogInformation("Cache updated for user {Id}", id);

        return userFromDb;
    }

    // --- READ: Get All (Collection Cache-Aside) ---
    public async Task<List<User>> GetAllUsersAsync()
    {
        var users = await allUsersCache.GetAsync(AllUsersCacheKey);

        if (users is not null)
        {
            logger.LogInformation("Cache HIT for ALL USERS list.");
            return users;
        }

        logger.LogWarning("Cache MISS for ALL USERS list. Accessing DB (1000 records).");
        await Task.Delay(400); 

        var usersFromDb = Database.Values.ToList();

        // Saves the list of 1000 users in the cache for 5 minutes
        await allUsersCache.SetAsync(AllUsersCacheKey, usersFromDb, TimeSpan.FromMinutes(5));

        logger.LogInformation("Cache updated for ALL USERS list.");

        return usersFromDb;
    }

    // --- UPDATE: Double Invalidation (Individual Key + List Key) ---
    public async Task<User> UpdateUserAsync(int id, CreateUserDto userDto)
    {
        if (!Database.ContainsKey(id))
            throw new KeyNotFoundException($"User with ID {id} not found for update.");

        var updatedUser = new User(id, userDto.Username, userDto.Email);
        Database[id] = updatedUser; 

        logger.LogInformation("User {Id} updated in DB.", id);

        // 1. Invalidates the individual key, as it is now outdated
        await userCache.RemoveAsync(id.ToString());
        // 2. Invalidates the total list
        await allUsersCache.RemoveAsync(AllUsersCacheKey);

        logger.LogInformation("Individual user key and All Users list invalidated.");

        return updatedUser;
    }

    // --- DELETE: Double Invalidation (Individual Key + List Key) ---
    public async Task DeleteUserAsync(int id)
    {
        if (!Database.Remove(id))
            throw new KeyNotFoundException($"User with ID {id} not found for deletion.");

        logger.LogInformation("User {Id} deleted from DB.", id);

        // 1. Invalidates the individual key
        await userCache.RemoveAsync(id.ToString());
        // 2. Invalidates the total list
        await allUsersCache.RemoveAsync(AllUsersCacheKey);

        logger.LogInformation("Individual user key and All Users list invalidated after deletion.");
    }

    // --- Static "Database" Configuration (Simulation) ---
    private static readonly Dictionary<int, User> Database;
    private static int _nextId;

    static UserService()
    {
        string seedPath = Path.Combine(AppContext.BaseDirectory, "UserSeedData.json");

        if (File.Exists(seedPath))
        {
            string jsonString = File.ReadAllText(seedPath);
            var users = JsonSerializer.Deserialize<List<User>>(jsonString) ?? [];

            Database = users.ToDictionary(u => u.Id, u => u);
            _nextId = users.Count > 0 ? users.Max(u => u.Id) + 1 : 1;
        }
        else
        {
            Database = new Dictionary<int, User>();
            _nextId = 1;
        }
    }
}

Testing Performance and the Cache-Aside Pattern

In this section, we will prove that the architecture with the Distributed Redis Cache works, demonstrating the difference in response time between a Cache Miss (DB access) and a Cache Hit (Redis access).

For the tests, we will use the GET /api/users endpoint, which loads the complete list of 1000 users.

Test 1: Cache Miss (First Access)

Upon the first call to the endpoint, the cache is empty. The UserService is forced to simulate a database query for the 1000 records, which introduces a purposeful latency (400ms Task.Delay).

Test 2: Cache Hit (Subsequent Access)

The UserService saved the complete list of users to Redis after the first access. The second call (made immediately afterward) finds this data in the cache.

Cache Coherence Strategy (Double Invalidation)

To ensure that the list of users (/api/users) is always synchronized after a change, its implementation follows the Double Invalidation pattern.

Read Operation (GET /api/users)

The complete list of 1000 users (key: "all_users_list") is loaded from the database only if it is empty in Redis. It is saved with a short Time To Live (TTL) (5 minutes).

Write Operations (POST, PUT, DELETE)

Whenever the primary data source (the "database") is modified, the system triggers the invalidation of two cache keys.

This way, the list of all users is only rebuilt (with the 400ms latency cost) when there is a real necessity (right after a modification), ensuring data coherence without sacrificing performance for the vast majority of read requests.

Conclusion

The main focus of our guide was the functional integration, where we utilized essential interaction methods:

Cache-Aside Strategy: Ensuring that the most costly read operation (GetAllUsers) only accesses the database when necessary (Cache Miss).
Redis Methods in Practice: Each C.R.U.D. operation demonstrated the use of fundamental Redis commands: SET (implicit when saving data to the cache) and DEL (implicit in invalidating individual and list keys).
Proven Performance: The final test validated the reduction in response time from hundreds of milliseconds (DB access) to milliseconds (cache access).

Project: https://github.com/JoaoOliveira889/RedisCacheApi

User Secrets: The Right Way to Protect API Keys in .NET

Joao Oliveira — Sun, 12 Oct 2025 18:15:48 +0000

This tutorial demonstrates how to use the User Secrets Manager in an .Net Web API to store sensitive configuration data, such as API tokens, ensuring they are never accidentally committed to source control.

Step 1: Project Setup via CLI

Create a new Web API project

dotnet new webapi -n SecretsDemo
cd SecretsDemo

Enable the User Secrets storage for this project

dotnet user-secrets init

Output confirms UserSecretsId added to .csproj
A UserSecretsId is added to the project file.

The .NET user-secrets init command creates a unit ID for your project and adds it to the SecretsDemo.csproj file. this ID links your project to the local secrets.jsonfile on your machine.

Step 2: Define Configuration Structure

In the appsettings.json we have the credentials to access and authentication in a external service for save logs called "ExternalLogger", we need to connect the props

{
  "ExternalLogger": {
    "BaseUrl": "https://logservice.com/v1",
    "ApiKey": "",
    "MinimumLevel": "Information" 
  }
}

Step 3: Create the Option Class

In .NET standard approach is to map configuration sections to C# classes. This provides type safety and better structure.

Create a new folder named Configuration and add the following class:

Configuration/ExternalLoggerOptions.cs

namespace SecretsDemo.Configuration;

public class ExternalLoggerOptions
{
    // Define a constant for the section name to avoid magic strings
    public const string ExternalLogger = "ExternalLogger";

    public string BaseUrl { get; set; } = string.Empty;

    // This is the sensitive property that will be stored in User Secrets
    public string? ApiKey { get; set; } 

    public string MinimumLevel { get; set; } = "Warning";
}

Step 4: Implement the Demo Service

Create a service that consume the configuration. This service will contains a logic to check if the required secret is available.

Create a new folder named Services and add the following class:

Services/LoggerService.cs

namespace SecretsDemo.Services;

public class LoggerService(IOptions<ExternalLoggerOptions> options)
{
    private readonly ExternalLoggerOptions _options = options.Value;

    public string LogMessage(string message)
    {
        // Crucial validation logic: is the secret present?
        if (string.IsNullOrEmpty(_options.ApiKey))
        {
            // Message when the secret is missing
            return "ERROR: The ExternalLogger Token is missing. Please configure User Secrets!";
        }

        // Simulates real use where the token allows the operation
        var logDetails = $"Base URL: {_options.BaseUrl} | Level: {_options.MinimumLevel}";

        // ApiKey would be used here
        return $"Log of '{message}' sent successfully! Config: {logDetails}";
    }
}

Step 5: Register and Consume

Update the Program.csfile to register your configuration class and your new service with the DI container.

var builder = WebApplication.CreateBuilder(args);

builder.Services.Configure<ExternalLoggerOptions>(
    builder.Configuration.GetSection(ExternalLoggerOptions.ExternalLogger)
);

builder.Services.AddSingleton<LoggerService>();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
}

app.UseHttpsRedirection();

app.MapGet("/log", (LoggerService logger) => 
    Results.Ok((object?)logger.LogMessage("Test message from the API")));

app.Run();

Step 6: Test and Add the Secret

First run the application without the secret to test what happen when the secret is not found. This is the expected result:
ERROR: The ExternalLogger Token is missing. Please configure User Secrets!

Now add the Secret

dotnet user-secrets set "ExternalLogger:ApiKey" "xdfs989-fs990fs-fdas"

Run the application again and this is the expected result:
Log of 'Test message from the API' sent successfully! Config: Base URL: https://logservice.com/v1 | Level: Information

Step 7: Modifying, Deleting, and Viewing User Secrets

The .NET CLI provides three simple commands to manage your secrets stored in the local secrets.json file. Remember to execute these commands from your project's root directory.

Changing a Secret

dotnet user-secrets set "ExternalLogger:ApiKey" "abc-fsfsfsd-erwerw3"

Deleting a Specific Secret

dotnet user-secrets remove "ExternalLogger:ApiKey"

Viewing All Secrets

dotnet user-secrets list

Bulk Deletion

dotnet user-secrets clear

Conclusion

The User Secrets Manager is crucial development tool that prevents sensitive keys from entering source control. However, for Production environments, it is imperative to use a dedicated secret store like Azure Key Vault or AWS Secrets Manager.

GitHub Project Link: https://github.com/JoaoOliveira889/SecretsDemo

Forem: Joao Oliveira

Building an API in Go to Manage Files on Amazon S3

What This API Does

Architecture and Project Structure

How Each Part Connects

Directory Structure

Important Decisions in This Structure

Endpoints

Files

Buckets

Key Highlights and Technical Decisions

UUID v7 for file names

Streaming to avoid memory spikes

Real file type (MIME) validation

Timeouts as resource protection

Initializing the Project

Dependency Management

Domain Layer

Entities

Repository Interface

Infrastructure Layer

Technical Highlights in the Infrastructure Layer

Service Layer

Technical Highlights in the Service Layer

Concurrency with errgroup

Security: file type validation

Resilience: per-operation timeouts

Encapsulation and dependency injection

Handler Layer

Technical Decisions in the Handler

The Entry Point (main)

Technical Highlights in main

Security and configuration: managing credentials

.env file (local development)

Obtaining Credentials in the AWS Console (IAM)

Middlewares: observability and resilience

Structured logging

Timeout control

Technical Highlights in Middlewares

Unit Tests

Implementing Service Tests

Technical Highlights in Tests

Automation with GitHub Actions

Configuring the Workflow (go.yml)

Technical Highlights in Automation

Centralized Configuration

Semantic Errors

Technical Highlights in Error Handling

Containerization with Docker

Optimized Dockerfile

Orchestration with Docker Compose

Technical Highlights in Docker

Useful Commands

Build the image and start the container in the background

Conclusion

Project Links

References

Redis Implementing Pub Sub and Streams in NET 10

Concepts: Pub/Sub vs Stream

Environment Configuration

Docker Setup

Project Implementation

Model

Interface

Service

Workers

Program.cs

Comparison

Testing the endpoints

Pub/Sub Broadcast

Simulate a Pub/Sub without Worker

Stream Log

Peak History on Stream

Stream Process

Process Flow

Conclusion

Project Source Code

References & Further Reading

Enrich Logs with CorrelationID in .NET

Create a Middleware

Technical Highlights in `main`

`.env` file (local development)

Configuring the Workflow (`go.yml`)