Forem: Bijaya Prasad Kuikel

Kafka with Go Part 1 — Understanding Async Systems, Distributed Architecture, and Your First Kafka Publisher/Subscriber

Bijaya Prasad Kuikel — Tue, 12 May 2026 00:37:25 +0000

Introduction

Modern web applications are no longer just simple static websites. Years ago, many websites were mostly:

static pages
blogs
content websites
simple request-response applications

But modern software systems are very different. Today’s applications do things like:

video uploads
realtime notifications
analytics processing
AI inference
payment processing
image optimization
email delivery
realtime chat
activity tracking
stream processing

Modern backend systems are now closer to continuously running software systems rather than just: “serve HTML and return response.”
And this creates a very important architectural problem.

The Problem With Long Running Tasks

Suppose a user uploads a video. Your backend now needs to:

save the file
generate thumbnails
compress the video
notify followers
update analytics
scan for moderation

Some of these tasks may take:

several seconds
minutes
sometimes even longer

Now imagine if the user had to wait for ALL of this before receiving a response.

Client
   |
   v
Backend
   |
   +--> Compress Video
   +--> Generate Thumbnail
   +--> Send Notifications
   +--> Update Analytics
   |
   v
Response Returned

This creates a terrible user experience.

The application becomes:

slow
blocked
harder to scale

Why Async Processing Exists

This is exactly why asynchronous systems exist.

Instead of making users wait:

Client -> Backend -> Everything Happens Here

modern systems usually do this:

Client -> Backend -> Queue -> Worker

The backend quickly stores a job into a queue.

Then background workers process the heavy tasks separately.

Now:

API becomes fast
users get immediate response
heavy processing happens in background

This is one of the most important concepts in backend engineering.

Real World Examples

This architecture exists almost everywhere.

Sending Emails

User Signup
    |
    v
Queue Email Job
    |
    v
Email Worker Sends Email

Image Processing

User Uploads Image
    |
    v
Queue Resize Job
    |
    v
Image Worker Processes File

Payment Notifications

Order Created
    |
    v
Queue Notification Job
    |
    v
Notification Worker

What Is a Queue?

A queue is simply

A middle layer between producers and workers.

Instead of directly doing heavy work:

Backend -> Heavy Task

we place the task into a queue:

Backend -> Queue -> Worker

This gives us:

asynchronous processing
better performance
better scalability
loose coupling

Messaging Systems

To implement queues and async systems, we use messaging systems.

Technology	Common Usage
Redis Pub/Sub	Lightweight realtime messaging
RabbitMQ	Traditional queues
Apache Kafka	Distributed event streaming
NATS	Lightweight distributed systems
Google Pub/Sub	Managed cloud messaging

All of them solve similar problems differently.

Distributed Systems

As Systems Grow, Architecture Evolves. Initially, a single application may work perfectly fine.

This is called a:

Monolith architecture.

Monolith Architecture

+----------------------+
|      Monolith        |
|----------------------|
| Auth                 |
| Orders               |
| Payments             |
| Notifications        |
+----------------------+

Everything lives inside one application. This is actually completely normal. Most successful applications start this way.

But as systems grow:

traffic increases
teams grow
deployments become difficult
scaling becomes harder
failures affect entire application

Eventually systems evolve into:

Microservices architecture.

Microservices

Instead of one giant application, each service becomes independent.

+---------+
| Auth    |
+---------+

+---------+
| Orders  |
+---------+

+------------+
| Payments   |
+------------+

+----------------+
| Notifications  |
+----------------+

Benefits:

isolated deployments
independent scaling
smaller codebases
better team ownership

But now another important problem appears.

How Do Services Communicate?

Suppose:

Order service creates order
Payment service charges customer
Notification service sends email

How should they communicate?

Most beginners first think:

Order Service ---> HTTP ---> Payment Service

This works initially.

But distributed systems become difficult quickly.

Problems With Direct HTTP Communication

Imagine:

Order Service ---> Payment Service

What if:

payment service is down?
network becomes slow?
retries create duplicate requests?
traffic spikes suddenly?

Now systems become tightly coupled. One service failure can affect everything. And remember the problem we discussed earlier:

long-running tasks should not block users

That same problem exists here too.

Sync vs. Async Comparison

Feature	Direct HTTP (Synchronous)	Kafka (Asynchronous)
Coupling	Tight: Services must know each other's IP/URL.	Loose: Services only need to know the Topic name.
Availability	If the target service is down, the request fails.	If the target is down, Kafka stores the data until it returns.
User Experience	User waits for every sub-task to finish.	User gets a "Success" as soon as data hits Kafka.
Resilience	Hard to handle retries and traffic spikes.	Built-in: Kafka acts as a buffer during high traffic.

Suppose:

sending email becomes slow
payment provider becomes delayed
analytics system becomes overloaded

Should users wait? Of course not.

So even microservices need:

asynchronous communication
buffering
scalable messaging systems

Event-Driven Communication

Instead of directly calling services, now Order service simply publishes an event:

Order Service ---> Message Broker ---> Payment Service

order_created

It does NOT care:

who consumes it
how many consumers exist
whether consumers are temporarily offline

This creates:

loose coupling
better scalability
better fault tolerance

This Is Where Kafka Comes In

Apache Kafka became extremely popular because it solves these problems at very large scale.

Kafka is heavily used in:

analytics systems
payment pipelines
notification systems
activity tracking
realtime monitoring
distributed systems
stream processing

Companies use Kafka because it handles:

huge traffic
distributed systems
realtime event streaming
scalable consumers
durable event storage

Kafka Is Not Just a Queue

This is important. Kafka can absolutely work like a queue. But Kafka is much more than that.

Kafka is fundamentally:

A distributed event streaming platform.

Meaning:

events can be stored
replayed later
consumed by multiple services
processed at massive scale

This becomes extremely powerful in modern architectures.

Kafka in One Simple Sentence

Kafka is basically:

Producer ---> Kafka ---> Consumer

Producer sends events. Consumers receive events.

Kafka stores events safely in between.

Basic Kafka Architecture

+------------+
| Producer   |
+------------+
       |
       v
+----------------+
| Kafka Topic    |
+----------------+
       |
       v
+------------+
| Consumer   |
+------------+

Important Kafka Terms

Term	Meaning
Producer	Sends messages
Consumer	Reads messages
Topic	Message category
Broker	Kafka server
Event	Actual data/message

So now we will implement a very simple

Which Go Package Are We Using?

We will use:

github.com/segmentio/kafka-go

In the past, Sarama was the go-to choice, but it can be quite "heavy" for beginners. We are using kafka-go because:

Idiomatic Go: It feels like writing standard Go code.
Context Support: Built-in support for context.Context for clean shutdowns.
Simpler API: No complex interfaces to implement for a basic consumer.

Kafka Setup (Modern KRaft Mode)

Older Kafka versions required Zookeeper.

Modern Kafka supports:

KRaft mode

which removes Zookeeper completely.

We will use the modern setup.

Install Kafka (Mac)

Using Homebrew:

brew install kafka

Start Kafka:

kafka-server-start /opt/homebrew/etc/kafka/kraft/server.properties

Create Kafka Topic

Open another terminal:

kafka-topics \
  --create \
  --topic orders \
  --bootstrap-server localhost:9092

Verify:

kafka-topics \
  --list \
  --bootstrap-server localhost:9092

You should see:

orders

Running Kafka with Docker

Instead of installing Kafka directly on your machine, we can also run it using Docker.

This is often easier because:

setup is cleaner
no local Java/Kafka installation needed
easier to reset environments
works consistently across machines

1. Create a `docker-compose.yml`

In your project root, create a file named:

docker-compose.yml

Add the following content:

services:
  kafka:
    image: apache/kafka:4.2.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      # Configure Kafka to run in KRaft mode
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: controller,broker
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

2. Start Kafka

Run the following command:

docker-compose up -d

This will:

pull the Kafka image
start Kafka container
expose Kafka on port 9092

Verify container is running:

docker ps

You should see a container named:

kafka

3. Create the Kafka Topic

Even with Docker, we still need to create our Kafka topic.

Run:

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic orders \
  --bootstrap-server localhost:9092

4. Verify Topic Creation

Run:

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9092

You should see:

orders

Kafka is now ready for our Go producer and consumer.

Create Go Project

mkdir go-kafka-tutorial
cd go-kafka-tutorial

Initialize Go module:

go mod init github.com/<your-github-username>/go-kafka-tutorial

Install Sarama:

go get github.com/segmentio/kafka-go

Project Structure

go-kafka-tutorial/
├── producer/
│   └── main.go
├── consumer/
│   └── main.go
├── go.mod
└── go.sum

Writing Our First Producer

Create:

producer/main.go

Code:

package main

import (
    "context"
    "log"

    "github.com/segmentio/kafka-go"
)

func main() {
    // 1. Initialize the writer
    writer := &kafka.Writer{
        Addr:     kafka.TCP("localhost:9092"),
        Topic:    "orders",
        Balancer: &kafka.LeastBytes{},
    }

    // 2. Ensure the writer is closed at the end
    defer writer.Close()

    // 3. Write a message
    err := writer.WriteMessages(context.Background(),
        kafka.Message{
            Value: []byte("new order created!"),
        },
    )

    if err != nil {
        log.Fatal("could not write message:", err)
    }

    log.Println("message sent successfully to Kafka!")
}

Writing Our First Consumer

Create:

consumer/main.go

Code:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/segmentio/kafka-go"
)

func main() {
    // 1. Initialize the reader
    reader := kafka.NewReader(kafka.ReaderConfig{
        Brokers:  []string{"localhost:9092"},
        GroupID:  "order-group",
        Topic:    "orders",
        MinBytes: 10e3, // 10KB
        MaxBytes: 10e6, // 10MB
    })

    defer reader.Close()

    fmt.Println("Consumer started... waiting for messages")

    // 2. Loop to read messages
    for {
        m, err := reader.ReadMessage(context.Background())
        if err != nil {
            log.Fatal("could not read message:", err)
        }

        fmt.Printf("Received message: %s\n", string(m.Value))
    }
}

Running the Application

Start consumer first:

go run consumer/main.go

Now in another terminal:

go run producer/main.go

Consumer output:

received message: new order created

You just built your first Kafka publisher/subscriber system using Go.

Conclusion

In this part we learned:

why async systems exist
long running task problems
queues and background workers
distributed systems basics
monolith vs microservices
service communication problems
messaging systems
Kafka fundamentals
creating producer and consumer using Go

Most importantly, You now understand:

WHY Kafka exists.

That foundation matters much more than memorizing APIs.

#golang #go #concurrency #goroutines Let's master concurrency in go from absolute basics. Be a part of my journey.

Bijaya Prasad Kuikel — Sat, 21 Jun 2025 01:00:13 +0000

Bijaya Prasad Kuikel

Jun 20 '25

Mastering Concurrency in Go, Part 1: Understanding Concurrency vs Parallelism

#go #concurrency #parallelism #programming

Comments

5 min read

Mastering Concurrency in Go, Part 1: Understanding Concurrency vs Parallelism

Bijaya Prasad Kuikel — Fri, 20 Jun 2025 14:35:56 +0000

Background

Modern software doesn’t run on a single processor anymore. Your phone likely has 8 cores, and servers have dozens. Yet many programs are still written as if only one task happens at a time.
To build fast, scalable software, you need to master concurrency and parallelism—and Go makes these concepts simple, powerful, and fun.

🌍 The Bigger Picture: Why Concurrency Exists at All

Before we dive into Go or any code, let’s understand the why behind concurrency.

In the early days of computing, programs ran in a single line of execution — do one thing, then the next, and so on. This was fine when computers were slow, and users had simple needs. But as computers became faster and systems more complex, we hit a problem: waiting.

Waiting for input. Waiting for files. Waiting for a network response.

And during that wait? The CPU just sat idle — wasting time and power.

To solve this, computer scientists introduced the idea of doing multiple things seemingly at the same time — called concurrency. It let systems remain productive while waiting on slow tasks like I/O or user interaction.

Later, as CPUs got multiple cores, we also gained parallelism — actually doing things truly at the same time.

Modern systems combine both. And that’s where software needs to evolve too.

🧠 Why Should You Care?

Look around you — apps today are expected to:

Load instantly
Respond to clicks while doing work in the background
Fetch data from multiple APIs
Keep UIs smooth
Handle thousands of users without crashing

But under the hood, every program faces the same old enemy: doing one thing at a time is slow and limiting.

If your app:

Calls APIs
Reads from disk
Talks to databases
Streams data
Handles real-time users

… then your app is doing tasks that wait — a lot. And if you don’t handle this well, your app becomes slow, unresponsive, or just stuck.

Concurrency lets your program start a task, move on to others while waiting, and keep everything flowing. It’s the key to efficiency, responsiveness, and scalability.

⚙️ Why Go?

Concurrency is powerful but often painful in other languages:

Java’s threads are heavyweight and require complex locks, which are hard to get right.
JavaScript’s single-threaded event loop can lead to callback hell and debugging nightmares.

Go was designed with first-class concurrency in mind. It gives you:

Goroutines — Lightweight “mini-programs” you can launch with one keyword: go. Unlike threads, they’re cheap and easy to use.
Channels — A safe way to share data between tasks without messy locks.
Go’s runtime scheduler - A behind-the-scenes manager that juggles goroutines efficiently across CPU cores.

You get simplicity, safety, and performance — all without breaking your brain.

🧭 What This Series Covers

In this series, we’ll walk through:

The fundamentals of concurrency vs parallelism (this part)
How Go implements concurrency with goroutines and channels
What makes Go’s scheduler unique and powerful
Real-world problems you’ll face with concurrency — and how Go solves them
Building systems that scale: crawlers, servers, pipelines, and more

Concurrency vs. Parallelism

Let’s clear up the confusion between these two core ideas.

➤ Concurrency:

Managing multiple tasks that can run independently, even if they don’t execute simultaneously.

Think of a chef in a kitchen juggling three dishes—chopping veggies for one, stirring a sauce for another, and checking the oven. The chef switches tasks to keep everything moving, even with just one pair of hands.

➤ Parallelism:

Executing multiple tasks at the same time on different cores.

Now imagine three chefs, each cooking a different dish at the same time. That’s parallelism—true multitasking with multiple workers (cores).

Key takeaway: Concurrency is about orchestrating tasks to avoid wasted time. Parallelism is about executing tasks at once to maximize speed. Go’s runtime handles both, making your code efficient and scalable.

Why Concurrency Matters

Modern software:

Calls APIs and databases
Waits for network or disk I/O
Serves thousands of users

Concurrency lets you start tasks and move on, maximizing resource use without waiting idly. It’s about efficiency, not just raw speed.

Real-World Case: Downloading Images

Imagine downloading 100 images:

Sequentially: One after another — painfully slow.
Concurrently: Start all downloads at once, utilizing network downtime.
In Parallel: Process multiple downloads across CPU cores.

Here’s what this might look like in code (don’t worry, we’ll dive into goroutines soon):

// Sequential
for _, url := range imageURLs {
    downloadImage(url) // Waits for each download to finish
}

// Concurrent (with goroutines)
for _, url := range imageURLs {
    go downloadImage(url) // Starts all downloads at once
}

Go’s runtime decides which downloads run in parallel, making your code clean and scalable.

Traditional Models: Threads and Locks

In languages like Java or C++, concurrency often means threads and shared memory. You create threads, manage locks, and pray you avoid:

Deadlocks (threads stuck waiting for each other)
Race conditions (unpredictable results from shared data)
Context-switching overhead (threads are expensive)

This model is powerful but complex and error-prone.

Go’s Simpler Model

Go abstracts threads away with goroutines — lightweight, user-space functions you launch like this:

go doSomething()

Goroutines are:

Managed by the Go runtime, not OS threads
Start with a small stack (a few KB, growing as needed)
Cheap enough to run thousands without crashing

Go’s M:N scheduler maps many goroutines to a few OS threads, balancing concurrency and parallelism. It’s fast, increasingly preemptive, and improves with every release.

Sharing Memory by Communicating

Traditional concurrency: Share memory and coordinate with locks.

Go’s philosophy: Do not communicate by sharing memory; instead, share memory by communicating.

Go’s channels are concurrency-safe queues for passing messages:

ch := make(chan string)

go func() {
    ch <- "hello"
}()

fmt.Println(<-ch)

Channels reduce the need for locks, making code simpler and safer.

Go’s Modern Concurrency

Go’s runtime keeps getting better:

Faster goroutine scheduling
Lower overhead for sync.Mutex, WaitGroup
Improved tools like runtime/trace and pprof
Easier debugging of concurrent systems

Go’s concurrency is simple, scalable, and production-ready.

TL;DR

Concurrency: Structuring tasks to run independently.
Parallelism: Executing tasks simultaneously on multiple cores.
Go’s goroutines and channels make concurrency simple and safe.
Compared to threads or event loops, Go offers less complexity, more performance.
Go’s runtime evolves to stay cutting-edge.

What’s Next?

In Part 2: Goroutines Under the Hood, we’ll dive into the magic of goroutines:

How does Go’s scheduler juggle thousands of tasks?
Why can you run 10,000 goroutines on a laptop without crashing?
What’s a goroutine leak, and how do you avoid it?

Plus, we’ll build a concurrent web crawler to see goroutines in action.

Implementing cursor pagination in Golang: Go Fiber, MySQL, GORM from scratch

Bijaya Prasad Kuikel — Mon, 19 Dec 2022 12:39:40 +0000

Introduction

Pagination is the process of dividing a large number of records or items into smaller chunks, or pages. Pagination is useful as it allows users to view a large amount of data
in a more manageable way.

There are several different approaches to pagination, including numbered pagination, which displays page numbers that the user can click to move between pages, and infinite scrolling, which automatically loads new pages of data as the user scrolls down the page. Cursor pagination is another approach, which uses a cursor, or pointer, to track the user's position in the record set and allows them to move between pages using the cursor.

In this tutorial we will create a new REST API using Go, Fiber, GORM and MySQL from scratch.

Prerequisite

Please make sure to have following dependencies installed on your system

Go (1.16 / higher is required)
MySQL 8

Implementation:

Let's start by creating a new folder, mkdir go-bookie.
Now let's init a new go mod: Go mod init“github.com/sadhakbj/go-bookie”
Now we need to install go fiber as our dependency. Pleae check official documentation for all the detailed information about go fiber
- go get github.com/gofiber/fiber/v2
We also need few other dependencies so install them as well:

go get gorm.io/gorm
go get gorm.io/driver/mysql
go get github.com/google/uuid

Now let's create new files and folders:

touch main.go && mkdir models database helpers

As our app's name is bookie, we want to list the books data, so let's create a model book.go inside the folder models.

package models

import (
    "time"

    "github.com/google/uuid"
    "gorm.io/gorm"
)

type Book struct {
    ID          string `gorm:"primaryKey"`
    Title       string `json:"title" gorm:"type:varchar(191);not null"`
    Description string `json:"description" gorm:"type:text;not null"`
    Author      string `json:"author" gorm:"type:varchar(191);not null"`
    Price       uint   `json:"price" gorm:"not null"`
    CreatedAt   time.Time
    UpdatedAt   time.Time
}

func (book *Book) BeforeCreate(tx *gorm.DB) (err error) {
    book.ID = uuid.NewString()
    return
}

We are keeping the fields very simple, and we are using uuid as our primary key, using GORM, we will set the id to new uuid using the BeforeCreate hook
Now let's create a new file database.go inside the folder database and write a method to connect to the database:

package database

import (
    "log"

    "github.com/sadhakbj/bookie-go/models"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "gorm.io/gorm/logger"
)

// DB is a global variable that represents the database connection
var DB *gorm.DB

// InitDB initializes the database connection
func InitDB() {
    // Connect to the database
    var err error
    dsn := "root:root@tcp(127.0.0.1:3306)/bookies?charset=utf8mb4&parseTime=True&loc=Local"
    db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        log.Fatal(err)
    }
    log.Println("Connected Successfully to Database")
    db.Logger = logger.Default.LogMode(logger.Info)
    log.Println("Running Migrations")
    db.AutoMigrate(&models.Book{})
    DB = db
}

Now let's update our main file which is main.go to connect to the database, instantiate a new fiber app and add routes to it:

package main

import (
    "fmt"
    "log"
    "math/rand"
    "time"

    "github.com/gofiber/fiber/v2"
    "github.com/sadhakbj/bookie-go/controllers"
    "github.com/sadhakbj/bookie-go/database"
    "github.com/sadhakbj/bookie-go/models"
)

func main() {
    app := fiber.New()
    database.InitDB()

    app.Get("/books/seed", func(c *fiber.Ctx) error {
        var book models.Book
        if err := database.DB.Exec("delete from books where 1").Error; err != nil {
            return c.SendStatus(500)
        }
        for i := 1; i <= 20; i++ {
            book.Title = fmt.Sprintf("Book %d", i)
            book.Description = fmt.Sprintf("This is a description for a book %d", i)
            book.Price = uint(rand.Intn(500))
            book.Author = fmt.Sprintf("Book author %d", i)
            book.CreatedAt = time.Now().Add(-time.Duration(21-i) * time.Hour)

            database.DB.Create(&book)
        }

        return c.SendStatus(fiber.StatusOK)
    })

    app.Get("/books", controllers.GetPaginatedBooks)

    log.Fatal(app.Listen(":3000"))
}

We are keeping the things very simple over here, we are seeding the initial 20 data using a new HTTP endpoint: /books/seed and we have another endpoint: '/books' which gets the paginated data

Now, that we have 20 data in our database, let's move on to implementing the actual feature in the controllers folder, so let's create a new bookscontroller.go inside the folder controllers

package controllers

import (
    "fmt"
    "strconv"

    "github.com/gofiber/fiber/v2"
    "github.com/sadhakbj/bookie-go/common"
    "github.com/sadhakbj/bookie-go/database"
    "github.com/sadhakbj/bookie-go/helpers"
    "github.com/sadhakbj/bookie-go/models"
)

func GetPaginatedBooks(c *fiber.Ctx) error {
    books := []models.Book{}
    perPage := c.Query("per_page", "10")
    sortOrder := c.Query("sort_order", "desc")
    cursor := c.Query("cursor", "")
    limit, err := strconv.ParseInt(perPage, 10, 64)
    if limit < 1 || limit > 100 {
        limit = 10
    }
    if err != nil {
        return c.Status(500).JSON("Invalid per_page option")
    }

    isFirstPage := cursor == ""
    pointsNext := false

    query := database.DB
    if cursor != "" {
        decodedCursor, err := helpers.DecodeCursor(cursor)
        if err != nil {
            fmt.Println(err)
            return c.SendStatus(500)
        }
        pointsNext = decodedCursor["points_next"] == true

        operator, order := getPaginationOperator(pointsNext, sortOrder)
        whereStr := fmt.Sprintf("(created_at %s ? OR (created_at = ? AND id %s ?))", operator, operator)
        query = query.Where(whereStr, decodedCursor["created_at"], decodedCursor["created_at"], decodedCursor["id"])
        if order != "" {
            sortOrder = order
        }
    }
    query.Order("created_at " + sortOrder).Limit(int(limit) + 1).Find(&books)
    hasPagination := len(books) > int(limit)

    if hasPagination {
        books = books[:limit]
    }

    if !isFirstPage && !pointsNext {
        books = helpers.Reverse(books)
    }

    pageInfo := calculatePagination(isFirstPage, hasPagination, int(limit), books, pointsNext)

    response := common.ResponseDTO{
        Success:    true,
        Data:       books,
        Pagination: pageInfo,
    }

    return c.Status(fiber.StatusOK).JSON(response)
}

func calculatePagination(isFirstPage bool, hasPagination bool, limit int, books []models.Book, pointsNext bool) helpers.PaginationInfo {
    pagination := helpers.PaginationInfo{}
    nextCur := helpers.Cursor{}
    prevCur := helpers.Cursor{}
    if isFirstPage {
        if hasPagination {
            nextCur := helpers.CreateCursor(books[limit-1].ID, books[limit-1].CreatedAt, true)
            pagination = helpers.GeneratePager(nextCur, nil)
        }
    } else {
        if pointsNext {
            // if pointing next, it always has prev but it might not have next
            if hasPagination {
                nextCur = helpers.CreateCursor(books[limit-1].ID, books[limit-1].CreatedAt, true)
            }
            prevCur = helpers.CreateCursor(books[0].ID, books[0].CreatedAt, false)
            pagination = helpers.GeneratePager(nextCur, prevCur)
        } else {
            // this is case of prev, there will always be nest, but prev needs to be calculated
            nextCur = helpers.CreateCursor(books[limit-1].ID, books[limit-1].CreatedAt, true)
            if hasPagination {
                prevCur = helpers.CreateCursor(books[0].ID, books[0].CreatedAt, false)
            }
            pagination = helpers.GeneratePager(nextCur, prevCur)
        }
    }
    return pagination
}

func getPaginationOperator(pointsNext bool, sortOrder string) (string, string) {
    if pointsNext && sortOrder == "asc" {
        return ">", ""
    }
    if pointsNext && sortOrder == "desc" {
        return "<", ""
    }
    if !pointsNext && sortOrder == "asc" {
        return "<", "desc"
    }
    if !pointsNext && sortOrder == "desc" {
        return ">", "asc"
    }

    return "", ""
}

Here the main method: GetPaginatedBooks performs the database query to get the data and method: calculatePagination calculates the pagination information. The logic to calculate the pagination is really simple, we perform the query in the database with the limit of (provided limit+1), so if the count of the results > provided limit it means there is next / previous page.

As you can see, we are missing two helper files: common.go and pagination.go, so let's write the contents for them as well. (The first file is intended to include the common helper functions which can be used across the app while second one includes functionalities related to pagination):

//common.go
package helpers

func Reverse[T any](s []T) []T {
    for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {
        s[i], s[j] = s[j], s[i]
    }
    return s
}

package helpers

import (
    "encoding/base64"
    "encoding/json"
    "time"
)

type PaginationInfo struct {
    NextCursor string `json:"next_cursor"`
    PrevCursor string `json:"prev_cursor"`
}

type Cursor map[string]interface{}

func CreateCursor(id string, createdAt time.Time, pointsNext bool) Cursor {
    return Cursor{
        "id":          id,
        "created_at":  createdAt,
        "points_next": pointsNext,
    }
}

func GeneratePager(next Cursor, prev Cursor) PaginationInfo {
    return PaginationInfo{
        NextCursor: encodeCursor(next),
        PrevCursor: encodeCursor(prev),
    }
}

func encodeCursor(cursor Cursor) string {
    if len(cursor) == 0 {
        return ""
    }
    serializedCursor, err := json.Marshal(cursor)
    if err != nil {
        return ""
    }
    encodedCursor := base64.StdEncoding.EncodeToString(serializedCursor)
    return encodedCursor
}

func DecodeCursor(cursor string) (Cursor, error) {
    decodedCursor, err := base64.StdEncoding.DecodeString(cursor)
    if err != nil {
        return nil, err
    }

    var cur Cursor
    if err := json.Unmarshal(decodedCursor, &cur); err != nil {
        return nil, err
    }
    return cur, nil
}

We are using base64 encoding mechanism to encode a cursor value if there is pagination. It includes the id, created_at timestamp & the information about if the current cursor points to next or not. The cursor is generated as a string and is returned back to the user.
The user need to send cursor as a query parameter, which is decoded and the database query is performed accordingly.
Finally, we now need to create a file inside a folder common named response.go which includes a struct for the response dto:

package common

import "github.com/sadhakbj/bookie-go/helpers"

type ResponseDTO struct {
    Success    bool                   `json:"success"`
    Data       any                    `json:"data"`
    Pagination helpers.PaginationInfo `json:"pagination"`
}

So, yeah we are now ready with our implementation. Now, lets run the server: go run main.go
Firstly let's seed our data: http://localhost:3000/books/seed
Now access our HTTP endpoint: http://localhost:3000/books?per_page=2&sort_order=asc, which should return the result:

{
  "success": true,
  "data": [
    {
      "ID": "c55a8347-0b78-42f1-9b48-936c9c65361e",
      "title": "Book 1",
      "description": "This is a description for a book 1",
      "author": "Book author 1",
      "price": 81,
      "CreatedAt": "2022-12-17T16:14:47.952+09:00",
      "UpdatedAt": "2022-12-18T12:14:47.954+09:00"
    },
    {
      "ID": "6e4d9295-6b26-439c-928f-980b7361b662",
      "title": "Book 2",
      "description": "This is a description for a book 2",
      "author": "Book author 2",
      "price": 387,
      "CreatedAt": "2022-12-17T17:14:47.956+09:00",
      "UpdatedAt": "2022-12-18T12:14:47.954+09:00"
    }
  ],
  "pagination": {
    "next_cursor": "eyJjcmVhdGVkX2F0IjoiMjAyMi0xMi0xN1QxNzoxNDo0Ny45NTYrMDk6MDAiLCJpZCI6IjZlNGQ5Mjk1LTZiMjYtNDM5Yy05MjhmLTk4MGI3MzYxYjY2MiIsInBvaW50c19uZXh0Ijp0cnVlfQ==",
    "prev_cursor": ""
  }
}

As we can see we have pagination info and encoded next_cursor value, now let's try to get the next 2 items using the cursor: http://localhost:3000/books?per_page=2&sort_order=asc&cursor=eyJjcmVhdGVkX2F0IjoiMjAyMi0xMi0xN1QxNzoxNDo0Ny45NTYrMDk6MDAiLCJpZCI6IjZlNGQ5Mjk1LTZiMjYtNDM5Yy05MjhmLTk4MGI3MzYxYjY2MiIsInBvaW50c19uZXh0Ijp0cnVlfQ== the response is:

{
  "success": true,
  "data": [
    {
      "ID": "62d843d7-889c-46c8-9956-12527b2b1a0d",
      "title": "Book 3",
      "description": "This is a description for a book 3",
      "author": "Book author 3",
      "price": 347,
      "CreatedAt": "2022-12-17T18:14:47.957+09:00",
      "UpdatedAt": "2022-12-18T12:14:47.954+09:00"
    },
    {
      "ID": "a0746960-a9da-4f62-be04-98e49ace03ef",
      "title": "Book 4",
      "description": "This is a description for a book 4",
      "author": "Book author 4",
      "price": 59,
      "CreatedAt": "2022-12-17T19:14:47.958+09:00",
      "UpdatedAt": "2022-12-18T12:14:47.954+09:00"
    }
  ],
  "pagination": {
    "next_cursor": "eyJjcmVhdGVkX2F0IjoiMjAyMi0xMi0xN1QxOToxNDo0Ny45NTgrMDk6MDAiLCJpZCI6ImEwNzQ2OTYwLWE5ZGEtNGY2Mi1iZTA0LTk4ZTQ5YWNlMDNlZiIsInBvaW50c19uZXh0Ijp0cnVlfQ==",
    "prev_cursor": "eyJjcmVhdGVkX2F0IjoiMjAyMi0xMi0xN1QxODoxNDo0Ny45NTcrMDk6MDAiLCJpZCI6IjYyZDg0M2Q3LTg4OWMtNDZjOC05OTU2LTEyNTI3YjJiMWEwZCIsInBvaW50c19uZXh0IjpmYWxzZX0="
  }
}

As we can see in this case we have both next & prev cursor.

Conclusion

We implemented a cursor paginator using Go, Go Fiber, Go ORM and MySQL pretty easily.

Getting started with Vue 3 composition API in typescript with Laravel Mix from scratch

Bijaya Prasad Kuikel — Mon, 28 Jun 2021 10:28:26 +0000

Laravel is one of the most loved web frameworks by the developers. It makes development so easier and fun. VueJs is one of the most loved JavaScript and rapidly growing Web Framework. This article will cover steps on implement Vue 3 in TypeScript with Laravel Mix and Composition API from scratch.

Why TypeScript?

Many developers love TypeScript. The reason is it simplifies JavaScript code, as it adds the readability and writability and makes easier to debug. It gives us all the benefits of the modern JavaScript (ES6 & beyond) also can help us to avoid painful bugs that developers commonly run into when writing JavaScript by type checking the code.

Let’s get started:

We can start by scaffolding a new laravel app using the command:

laravel new laravue3-ts

Install typescript globally on your local machine:

yarn global add typescript

Now let’s install Vue 3 & Vue loader for webpack:

yarn add vue@next vue-loader@next @vue/compiler-sfc

Also we need to install ts-loader and typescript as dev dependencies:

yarn add ts-loader typescript --dev

Now, let’s generate a tsconfig.json file in the root directory with the following contents on it:

{
  "compilerOptions": {
    "target": "esnext",
    "module": "esnext",
    // this enables stricter inference for data properties on `this`
    "strict": true,
    "jsx": "preserve",
    "moduleResolution": "node"
  }
}

Now, let’s generate shims-vue.d.ts file inside resources directory which stores the type defination for .vue files with the following content:

declare module '*.vue' {
   import type { DefineComponent } from 'vue'
   const component: DefineComponent<{}, {}, any>
   export default component
}

Now let’s update our webpack.mix.js file:

const mix = require("laravel-mix");
mix.ts("resources/js/app.ts", "public/js").vue({ version: 3 });

Now, let’s add a few components insides resources/js/components directory:

Welcome.vue:

<template>
  Hello {{ name }} welcome to my app.
  <hr />
  <counter></counter>
</template>
<script lang="ts">
import { defineComponent, ref } from "vue";
import Counter from "./Counter.vue";
export default defineComponent({
  components: {
    Counter,
  },
  setup() {
    const name = ref<string>("Bijaya");
    return {
      name,
    };
  },
});
</script>

Counter.vue:

<template>
  <button @click.prevent="decrement">-</button>
  {{ count }}
  <button @click.prevent="increment">+</button>
</template>
<script lang="ts">
import { defineComponent, ref } from "vue";
export default defineComponent({
  name: "counter",
  setup() {
    const count = ref<number>(0);
    const increment = ():number => {
      return count.value++;
    };
    const decrement = ():number => {
      return count.value--;
    };
    return {
      count,
      increment,
      decrement,
    };
  },
});
</script>

Great, now that we have added the components, now lets rename the default resources/js/app.js to app.ts and update the contents to have following:

import { createApp } from "vue";
import Welcome from "./components/Welcome.vue";
const app = createApp({
    components: {
        Welcome,
    },
}).mount("#app");

Also within our welcome.blade.php update the content to have something like this:

<body>
  <div id="app"></div>
  <script src="{{ asset('js/app.js') }}"></script>
</body>

Compile the assets with the command:

yarn dev

npm run dev

Run Laravel server:

php artisan serve

and access: http://localhost:8000

YOU CAN WATCH MY VIDEO ON GETTING STARTED WITH VUE 3 IN TYPESCRIPT WITH LARAVEL MIX FROM SCRATCH HERE:

YouTube:

How to install & use ReactJS in TypeScript with Laravel 8 & Laravel Mix 6

Bijaya Prasad Kuikel — Mon, 22 Feb 2021 06:55:43 +0000

Laravel is one of the most loved web frameworks available. It makes development so easier and fun. React is one of the most loved JavaScript Library because of it’s simplicity. In this article we will learn to implement Reactjs in TypeScript with Laravel Mix (Laravel Mix is a tool for compiling and optimizing assets in a Laravel app).

Why TypeScript?

Let’s get started:

We can start by scaffolding a new laravel app using the command:

laravel new larareacts

Install typescript globally on your local machine:

yarn global add typescript

npm install -g typescript

Now let’s install React & react -dom with their types:

yarn add react react-dom @types/react @types/react-dom

Also we need to install ts-loader and typescript as dev dependencies:

yarn add ts-loader typescript @babel/preset-react --dev

Now we need to generate tsconfig.json file using the following command:
tsc --init --jsx react

Now let’s update our webpack.mix.js file:

const mix = require("laravel-mix");
mix.ts("resources/js/app.tsx", "public/js").react();

Now let’s create a index file inside: resources/js named index.tsx :

import React from "react";
import { render } from "react-dom";
import App from "./components/App";
render(<App />, document.getElementById("app"));

We need to create a root App component inside resources/js/components folder with the following contents:

import React, { useState } from "react";

export interface IUser {
    name: string;
    age: number;
}
const App = () => {
    const [users, setUsers] = useState<IUser[]>([
        {
            name: "Bijaya",
            age: 25,
        },
        {
            name: "Ram",
            age: 25,
        },
    ]);

    return (
        <div>
            <h1>Users list</h1>
            <ul>
                {users.map((user: IUser) => {
                    return (
                        <li key={user.name}>
                            {user.name} is {user.age} years old
                        </li>
                    );
                })}
            </ul>
        </div>
    );
};

export default App;

Also within our welcome.blade.php update the content to have something like this:

<body>
  <div id="app"></div>
  <script src="{{ asset('js/app.js') }}"></script>
</body>

Compile the assets with the command:

yarn dev

npm run dev

Run Laravel server:

php artisan serve

and access: http://localhost:8000

YOU CAN WATCH MY VIDEO ON INSTALLING REACT WITH TYPESCRIPT FROM SCRATCH:

YouTube:

Forem: Bijaya Prasad Kuikel

Kafka with Go Part 1 — Understanding Async Systems, Distributed Architecture, and Your First Kafka Publisher/Subscriber

Introduction

The Problem With Long Running Tasks

Why Async Processing Exists

Real World Examples

Sending Emails

Image Processing

Payment Notifications

What Is a Queue?

Messaging Systems

Distributed Systems

Monolith Architecture

Microservices

How Do Services Communicate?

Problems With Direct HTTP Communication

Sync vs. Async Comparison

Event-Driven Communication

This Is Where Kafka Comes In

Kafka Is Not Just a Queue

Kafka in One Simple Sentence

Basic Kafka Architecture

Important Kafka Terms

Which Go Package Are We Using?

Kafka Setup (Modern KRaft Mode)

Install Kafka (Mac)

Create Kafka Topic

Running Kafka with Docker

1. Create a docker-compose.yml

2. Start Kafka

3. Create the Kafka Topic

4. Verify Topic Creation

Create Go Project

Project Structure

Writing Our First Producer

Writing Our First Consumer

Running the Application

Conclusion

#golang #go #concurrency #goroutines Let's master concurrency in go from absolute basics. Be a part of my journey.

Mastering Concurrency in Go, Part 1: Understanding Concurrency vs Parallelism

Mastering Concurrency in Go, Part 1: Understanding Concurrency vs Parallelism

Background

🌍 The Bigger Picture: Why Concurrency Exists at All

🧠 Why Should You Care?

⚙️ Why Go?

🧭 What This Series Covers

Concurrency vs. Parallelism

➤ Concurrency:

➤ Parallelism:

Why Concurrency Matters

Real-World Case: Downloading Images

Traditional Models: Threads and Locks

Go’s Simpler Model

Sharing Memory by Communicating

Go’s Modern Concurrency

TL;DR

What’s Next?

Implementing cursor pagination in Golang: Go Fiber, MySQL, GORM from scratch

Introduction

Prerequisite

Implementation:

Conclusion

Getting started with Vue 3 composition API in typescript with Laravel Mix from scratch

Why TypeScript?

Let’s get started:

YOU CAN WATCH MY VIDEO ON GETTING STARTED WITH VUE 3 IN TYPESCRIPT WITH LARAVEL MIX FROM SCRATCH HERE:

How to install & use ReactJS in TypeScript with Laravel 8 & Laravel Mix 6

Why TypeScript?

Let’s get started:

YOU CAN WATCH MY VIDEO ON INSTALLING REACT WITH TYPESCRIPT FROM SCRATCH:

1. Create a `docker-compose.yml`