Forem: Shobhit Bhatnagar

Understanding Django, Gunicorn, and Database Connections

Shobhit Bhatnagar — Thu, 16 Oct 2025 10:29:31 +0000

Today, I was investigating an error that drew my attention to Gunicorn workers and Django database connections. I realized this is an important topic for anyone building scalable Django applications.

Gunicorn Workers and Django Connections

Let's clarify a common misconception: each Gunicorn worker is a forked process of the master. This means the Django CONN_MAX_AGE parameter applies independently within each worker process.

CONN_MAX_AGE = 0 → A new database connection is created and dropped for every request.
CONN_MAX_AGE = None → Connections are kept open indefinitely, allowing Django to reuse them across all requests handled by that worker.
CONN_MAX_AGE > 0 → Connections are reused by the worker for the specified time (in seconds) and dropped after the timeout.

Important: Workers do not share database connections. Each worker maintains its own connection(s) based on CONN_MAX_AGE.

Ideal Configuration by Traffic

Case 1: Low Traffic

CONN_MAX_AGE = 0

Behavior: Each worker creates and drops connections per request.

Advantages:

Minimal number of DB connections
Fresh connection for each request
Works well for low-traffic or development environments

Disadvantages:

Performance overhead due to frequent connection creation
Increased network traffic
Scalability issues under higher load

Case 2: Medium/High Traffic

CONN_MAX_AGE ≈ 600 seconds

Behavior: Each worker maintains its connections for 10 minutes, creating new ones only after the timeout.

Advantages:

Reduced latency
Lower database overhead
Better connection management
Increased application resilience

Disadvantages:

Slightly higher memory/resource consumption per worker
Configuration complexity
Less effective on very low-traffic sites

Case 3: Very High Traffic

CONN_MAX_AGE = None

Behavior: Connections are kept open indefinitely. Workers do not create/destroy connections per request.

Advantages:

Reduced overhead on DB server
Improved performance
Efficient resource usage

Disadvantages:

Risk of deadlocks or long-held locks if not managed properly
Scalability challenges if worker count increases significantly
Requires careful handling in ASGI or async contexts

Worker Example

Suppose you have 2 Gunicorn workers:

Case 1: Each worker can have 1 active connection per request, frequently changing based on incoming requests.
Case 2: Each worker maintains 1 persistent connection per worker, refreshed every CONN_MAX_AGE seconds.
Case 3: Each worker maintains 1 persistent connection indefinitely, minimizing connection churn.

Common Misconception: More Workers = Better Performance

It is a myth that increasing the number of Gunicorn workers automatically improves performance. In reality:

More workers share the same CPU and memory resources, so each worker is still limited by the host's capacity.
Instead of increasing worker count, it's recommended to increase the number of running containers. This reduces per-container overhead and allows horizontal scaling, which is far more effective in real-world production environments.

The Worker Formula and Its Practical Pitfall

You may have heard the formula for calculating the number of workers:

Number of workers = 2 × N + 1
where N is the number of CPU cores.

For example, if your system has 8 vCPUs, you might assume you can safely run 17 workers.
But in real life, you shouldn't.

Instead of maxing out a single large instance, it's better to use multiple smaller instances - for example, servers with 1–2 vCPUs running 3–5 workers each.
If the load grows, scale horizontally by adding more containers or instances, rather than vertically increasing CPU and RAM on one machine.

Why You Should Avoid Vertical Scaling ?

Vertical scaling (increasing CPU/memory on a single host) has several drawbacks:

It's expensive — larger instances cost disproportionately more.
Even with more CPU and memory, you can still be bottlenecked at the network layer.
Network saturation often limits how many database connections or DNS resolutions can occur simultaneously.

In my recent investigation, we were using a high number of workers with CONN_MAX_AGE = 0, and encountered DNS resolution errors for the database hostname (DB host name not resolved).
The issue wasn't with the database or Django — it was due to network-level bottlenecks caused by too many workers constantly opening and closing connections.

After reducing the number of workers, the system stabilized and handled high load efficiently, even with the same configuration aside from CONN_MAX_AGE.

Best Practices

Use connection pooling tools like PgBouncer or AWS RDS Proxy. These handle connection management externally, improving scalability and reducing the overhead on your microservices.
Fine-tune CONN_MAX_AGE based on traffic patterns and worker count.
Monitor Gunicorn worker memory and connection usage to avoid unexpected bottlenecks.

References & Further Reading

Automate Your Go Project: Best Practices & CI/CD with GitHub Actions

Shobhit Bhatnagar — Sun, 05 Oct 2025 14:42:12 +0000

You've built something cool in Go (Golang) — maybe a library, a CLI tool, or an API.
Now you want to take it to the next level: package it properly and publish it so others can use it.

That's where Continuous Integration and Continuous Deployment (CI/CD) comes in. It takes your local code and turns it into a repeatable, automated release pipeline, making your Go project reliable and easy to maintain.

In this post, I'll walk you through how to build, test, and publish a Go package using GitHub Actions. We'll also use semantic versioning, linting, and a few good practices that help keep your releases clean and consistent.

What You'll Learn

Before we start, it'll help if you're already familiar with:

Basics of the Go programming language
How GitHub Actions work
What semantic versioning (semver) and semantic releases mean

By the end of this guide, you'll have a good understanding of how to automate the release process for your Go projects — the same way most open-source Go libraries are maintained.

Building a CI/CD-Ready Go Package

Here's a simple checklist before setting up automation:

Code & Repo Setup
- Start with solving a real problem.
- Keep your project clean and organized - Take ref from here
Documentation
- Document everything clearly
- What your package does - Mostly in README and in comments
- How to install and use it - Mostly in README
- Always add examples of How to use - Mostly in README and in comments
- Go docs auto creates docs using comments that are written in your code
- In addition, You can also create a doc.go in each package that can contains the docs
Linting & Formatting
- Before pushing changes, always format and lint your code
- Use githooks like precommit for auto linting and formatting on commits
- Or you can manually do it like
```
// Here we have used ./... to work on every go file in ./ directory
go fmt ./...   // Here fmt is used to format all the go files that are avaliable

go vet ./...   // Here vet examines your code and reports suspicious constructs
```
It's a small step that saves you from bigger code review headaches later.
Testing
- Named your test files as *_test.go
- Test your code before every commit using go test ./...
Build
- Build your module locally to make sure everything compiles go build ./...
- This also helps confirm your code is production-ready before tagging a release.
Release
- Once linting ✅, testing ✅, and building ✅ are done — you're ready to release. In a CI/CD setup, releases are usually automated with semantic tags like v1.0.0.

Setting Up the Repository

Let's get hands-on.

Create and initialize your repository:

mkdir <YOUR_PROJECT_NAME>
git init
mkdir src
cd src
go mod init github.com/<YOUR_USERNAME>/<YOUR_PROJECT_NAME>

Now, create your folder structure:

.github/workflows/*.yaml    // GitHub actions workflow configurations
.config/goreleaser.yaml     // GoReleaser configuration
src/
  cmd/
    main.go // entry point of your project
    doc.go  // package documentation
    main_test.go  // test cases for your code
  go.mod    // Your Go mod file contains module name, go version and packages you are using
README.md   // Documentation about repository i.e. your project 
LICENSE     // Optional if you want to add a LICENSE
.gitignore  // contains pattern and files that will be ignored by git
.pre-commit-config.yaml // pre-commit configuration and hooks

If you'd like to skip setup, you can clone my sample repository: github.com/ticatwolves/go-project-skeleton

Automating Build, Test, and Release with GitHub Actions

Here's where GitHub Actions works.
You can automate almost everything — formatting, testing, tagging, and even publishing.

A typical workflow includes:

Running go fmt, go vet, and go test on every pull request
Automatically tagging releases with semantic versions
Building the package and pushing it to the Go proxy

Simple and easy to adapt workflow for each pull request

name: Lint & Format

on:
  pull_request:
    branches:
      - main

# Grant the jobs required permissions for OIDC to work
permissions:
  id-token: write
  contents: write
  pull-requests: write

jobs:
  Linter:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.23
      - name: Install dependencies
        working-directory: ./src
        run: go mod tidy
      - name: Run go fmt
        working-directory: ./src
        run: go fmt ./...
      - name: Run go vet
        working-directory: ./src
        run: go vet ./...          
      - name: Update pull request
        uses: peter-evans/create-pull-request@v7
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          commit-message: "chore: format code"
          title: "[AUTO] Format code"
          body: "This PR automatically formats the code."
          branch: ${{ github.head_ref }}
          base: ${{ github.base_ref }}
          labels: |
            automated pr
  GoLangCI:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          # Required to report new issues on the PR
          fetch-depth: 0
      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.23
      - name: Run golangci-lint
        uses: golangci/golangci-lint-action@v8
        with:
          working-directory: ./src
  Test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.23
      - name: Test
        working-directory: ./src
        run: go test -v ./...

Simple and easy to adapt workflow for publishing changes to the package manager.

name: Release
on:
  push:
    branches: [ "main" ]
permissions:
  id-token: write
  contents: write
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.23
      - name: Test
        working-directory: ./src
        run: go test -v ./...    
      - name: Go Semantic Release
        id: semrel
        uses: go-semantic-release/action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
      - name: Fetch tags created by semantic release
        if: steps.semrel.outputs.version != null && steps.semrel.outputs.version != ''
        run: git fetch --tags

      - name: GoReleaser
        if: steps.semrel.outputs.version != null || steps.semrel.outputs.version != ''
        uses: goreleaser/goreleaser-action@v6
        with:
          distribution: goreleaser
          version: latest
          args: release --config .config/goreleaser.yaml --clean
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

In the above workflow we are using GoReleaser So here is the config file goreleaser.yaml.

version: 2
builds:
  - dir: "./src"
    main: ./cmd/main.go
    env:
      - CGO_ENABLED=0
    goos:
      - linux
      - windows
      - darwin
    goarch:
      - amd64
      - arm64
release:
  github:
    owner: <YOUR_USERNAME>
    name: <YOUR_PROJECT_NAME>

Publishing on pkg.go.dev

One last step — making your module discoverable.

Go's official package discovery site, pkg.go.dev, doesn't automatically index new repos. After your first release (say, v1.0.0), visit: https://pkg.go.dev/github.com/YOUR_USERNAME/YOUR_PROJECT_NAME

Then click “Request indexing” or “Fetch now” to get it listed.

Once done, your package becomes publicly available for everyone to import:

go get github.com/YOUR_USERNAME/YOUR_REPO@v1.0.0

Let's Talk

How do you handle CI/CD for your Go projects?
Have you automated your release process yet, or are you still doing it manually?

Drop your thoughts in the comments — I'd love to learn how others are approaching this!

References & Further Reading

Building, Testing, and Publishing Go Packages - Best Practices

Shobhit Bhatnagar — Thu, 02 Oct 2025 06:48:15 +0000

In my previous article, we created a self-deployable Go-based URL Shortener Service and deployed it using AWS SAM.

Now, let’s take things a step further by learning how to package and publish a Go module on the Golang package manager (pkg.go.dev) so that it’s available for the open-source community. 🚀

Getting Started

We’ll use the shrinkIt module from my GitHub repository. You can clone it from here:
👉 ShrinkIt Repository

Before publishing, let’s make sure the project is clean and production-ready.

Step 1: Verify Dependencies

Run the following command to clean up and verify module dependencies:

go mod tidy

This ensures your go.mod and go.sum files are correct and only contain the dependencies you actually need.

Step 2: Run Tests

Always test your module before publishing to ensure everything works as expected:

go test

If your tests pass, you’re ready to move forward.

Step 3: Commit Changes

Use Conventional Commits when pushing changes. This helps with semantic versioning, which is crucial when publishing Go packages.

git add .
git commit -m "feat: add new feature" 
git push origin main

Step 4: Tag Your Release

Now, assign a version tag to your commit. For example, version v0.1.0:

git tag v0.1.0
git push origin v0.1.0

You should now see the tag under the Tags section in your remote Git repository.

Step 5: Publish to the Go Package Manager

Finally, publish your package to pkg.go.dev using the Go proxy:

GOPROXY=proxy.golang.org go list -m github.com/ticatwolves/shrinkit@v0.1.0

👉 Note: You can change the GOPROXY depending on whether you want to test or publish to the production package manager.

Step 6: Use Your Package

Once published, your package is available for anyone to use in their projects:

go get github.com/ticatwolves/shrinkit@v0.1.0

Congratulations 🎉 — you’ve published your first Go package to the public!

What’s Next?

That's it for now! Join us in our next article, where we'll explore how to add a CI/CD pipeline for automated building, testing, and publishing of your Go packages.

💬 How do you handle versioning and publishing in your Go projects? Share your workflow in the comments — I’d love to learn from your experience!

📚 Further Reading & References

Shorten URL Service with Go and AWS SAM

Shobhit Bhatnagar — Sun, 28 Sep 2025 14:13:48 +0000

In my previous article I explained how to structure a Golang project for better readability and maintainability.

Now, let’s take it a step further by building a URL Shortener Service using Go and AWS SAM (Serverless Application Model).🚀

What is AWS SAM?

SAM (Serverless Application Model) is an Infrastructure as Code (IaC) tool provided by AWS. It simplifies deploying serverless applications by letting you define them in a template and deploying via CloudFormation

Key advantages:

✅ Easy deployment of serverless applications
✅ Built-in support for AWS Lambda, API Gateway, DynamoDB, etc.
✅ Package and publish apps to the AWS Serverless Application Repository
✅ CI/CD friendly

Flow Diagram

Technical Flow

Here’s how the URL shortener works under the hood:

Client Request -> The user submits a request to API Gateway.
Lambda Trigger -> API Gateway routes the request to a Lambda function written in Go.
Supported Operations - POST, GET.
POST Operations:
- The Lambda Parse the request payload.
- The handler generates a hash of the original URL.
- The hash and actual URL are stored in the database (DynamoDB).
GET Operations:
- It retrieves the original URL by using Hash in the request URL.
- Lambda returns an HTTP 302 redirect with the original URL in the Location header.
- Client browser follows the redirect and loads the original URL.

Source Code

I’ve open-sourced the complete implementation here:
👉 ShrinkIt – URL Shortener with Go + SAM

Building and Deploying with SAM

Step 1: Clone git clone https://github.com/ticatwolves/shrinkIt.git
Step 2: Build the Project make build
Step 3: Deploy the Project make deploy

Feel free to check it out, clone the repo, and try it yourself!

💬 Have you tried building serverless applications with Go before? Share your experience in the comments — I’d love to hear your thoughts!

GoLang Project Template

Shobhit Bhatnagar — Tue, 23 Sep 2025 14:02:12 +0000

Starting a journey with a new programming language is always challenging — and I've been there myself.

I began learning Go (Golang) when I was comparing it with Python for some tasks deployed on AWS Lambda. The cold start time with Python was too high, so I started looking for alternatives. That’s when I discovered Golang.

After learning the basics, the first big question I had was:

What is the best way to structure a Go project?
How do companies organize and maintain Golang repositories?

Key Learnings About Go Project Setup

Every Go file must start with a package name.
The package name doesn't need to match the file or folder name.
A project's go version and module is defined in the go.mod file, which should always be in the root directory.
The go.sum file is auto-generated when you run go mod tidy. It records dependency versions to ensure reproducible builds.

In the end, you'll have a standalone executable binary file that can be published on the Go package manager or deployed (for example, on AWS Lambda).

Clean Go Project Structure (Template)
When building maintainable applications, following a clean and well-organized folder structure is essential. Here's a recommended Golang project structure you can use as a template:

root
|
|- build/              # Packaging and CI/CD related files
|- cmd/                # Application entry point
|   └── main.go        # For single entry point
|   └── EntryPointA    # For multiple entry points
|        └── *.go 
|   └── EntryPointB    # For multiple entry points
|        └── *.go 
|- internal/           # Private application and library code(internal logic)
|   └── *.go           # write your business logic also we can split them into packages like api, database, auth, etc
|   └── packageA       # packageA code 
|        └── *.go
|   └── packageB       # packageB code
|        └── *.go
|- api/                # Add backend API
|   └── *.go
|   └── apiA           
|        └── *.go
|- pkg/                # Public utility functions and reusable code
|   └── *.go
|   └── util           # Add Util code like validator, generic code, 
etc.  
|        └── *.go
|- vendor/             # Add Vendor packages
|   └── packageName     
|- static/             # Static files (html, css, js, asset)
|   └── css            # CSS files 
|   └── js             # JS files
|   └── asset          # Other static files .png, .jpg etc  
|   └── templates      # HTML templates 
|- scripts/            # Automation and helper scripts
|- configs/            # Configuration files (YAML, JSON, TOML, etc.)
|- go.mod              # Module definition
|- go.sum              # Dependency lock file (auto-generated)

This folder layout is inspired by clean architecture principles and helps keep Go projects scalable, testable, and easier to collaborate on.

Why Follow This Folder Structure?

Improves readability and maintainability
Supports clean architecture and modular code
Makes it easy for new developers to understand the project
Ensures scalable backend development with Go

Conclusion

If you're just starting with Golang, this template will give you a solid foundation for structuring your projects. Following a clean folder structure and using Go best practices ensures your applications are easier to maintain and scale.

💬 What do you think of this Go project structure? Did I miss something important? Share your thoughts in the comments — I'd love to learn from your experience too!