Forem: Rhaqim

Journal Entry: I Forgot to Clean Up After Myself

Rhaqim — Sat, 31 May 2025 11:25:53 +0000

TL;DR: Always call defer rows.Close() after querying your database.

Recently, I ran into a silent but critical problem in my Go application backed by PostgreSQL. Everything worked fine on the surface — requests were being handled, responses came back fast… until one day, I hit a wall.

New requests started hanging unexpectedly, and I noticed some routes weren’t returning data at all. But one route continued to work: the one fetching data from the cache instead of hitting the database.

Something wasn’t right.

🔍 The Clue: pg_stat_activity

I ran the following on my Postgres server:

SHOW max_connections;
SELECT COUNT(*) FROM pg_stat_activity;

And here was the result:

max_connections was set to 100
Active connections in pg_stat_activity? 67 😳

That number seemed way too high. On the client side (Go), I had:

db.SetMaxOpenConns(50)

So what was holding these connections open?

🧠 Aha Moment: I Forgot defer rows.Close()

After inspecting my codebase, I realized I had this kind of pattern:

rows, err := db.Query("SELECT * FROM rows_fetcher($1)", query_key)
if err != nil {
    return nil, err
}

for rows.Next() {
    // scan logic...
}

But no defer rows.Close(). 😱
Without explicitly closing the result set, the connection remains open and eventually floods the pool.

🔬 Digging Deeper: Seeing the Idle State

I ran:

SELECT pid, state, query, backend_start, xact_start, query_start 
FROM pg_stat_activity 
WHERE state = 'idle';

I noticed many of these had timestamps from weeks or even months ago!

🧹 Cleaning Up

I decided to take a few actions:

Update all database queries in my Go code to include:

defer rows.Close()

Use pg_terminate_backend(pid) to clean up old, stale processes manually.
Add SetConnMaxIdleTime to my DB setup:

db.SetConnMaxIdleTime(5 * time.Minute)

Consider setting PostgreSQL server config options like:

idle_in_transaction_session_timeout = 60000       # 1 min
idle_session_timeout = 300000                     # 5 min

✅ The Result After 24 Hours

I checked back the next day and ran:

SELECT COUNT(*) FROM pg_stat_activity;

To my surprise and relief, the number was down to 1.

One. From 67.

💡 Takeaways

Always close your rows after querying:
Use pg_stat_activity to spot stale connections.
Monitor timestamps, old queries might still be holding connections open.
Use connection pooling wisely (SetMaxOpenConns, SetConnMaxIdleTime).
Set database level timeouts to catch idle sessions that your app misses.

Speed vs Simplicity: Choosing the Right Cache

Rhaqim — Sat, 17 May 2025 14:41:16 +0000

Why I Switched from HashiCorp LRU to Ristretto for High-Performance Caching in Go

While working on Buckt, I implemented a caching layer to speed up repeated file reads and downloads. I started with HashiCorp's lru, which was simple and easy to integrate. But as the system scaled and concurrency increased, it became clear that it couldn't keep up.

This post highlights the issues I encountered and why Ristretto ended up being a much better fit.

❌ Issues with HashiCorp LRU

While the lru package is solid and predictable, I ran into a few limitations:

Blocking Writes - All Add() operations lock the cache, leading to bottlenecks under concurrent load.
Fixed Capacity, Not Cost-Based - It evicts based on item count, not memory usage — inefficient when storing large items like files.
No Native Metrics - Hit/miss tracking must be implemented manually.

✅ Why Ristretto?

Ristretto, created by Dgraph Labs, offers:

Non-Blocking Writes - Writes are buffered and processed asynchronously, preventing contention during high loads.
Cost-Aware Eviction - You can assign a "cost" (e.g., byte size), and the cache evicts based on total cost rather than item count.
TinyLFU Eviction Strategy - More efficient and accurate for real-world usage patterns than basic LRU.
Built-In Concurrency Support - Designed to scale with multiple goroutines hitting the cache simultaneously.

🔧 How I Use It

cache, _ := ristretto.NewCache(&ristretto.Config[string, []byte]{
    NumCounters: 1e7,     // Frequency-tracking keys
    MaxCost:     1 << 30, // 1GB total cost
    BufferItems: 64,      // Set buffer size
})

🚀 Improvements After the Switch

Requests are faster, especially repeated ones — no disk I/O.
Concurrency issues disappeared — no lock contention on write.
Memory usage is under control with cost-based eviction.

⚠️ Caveats

Avoid calling cache.Wait() in critical paths — it blocks until the write buffer is flushed.
Eviction is probabilistic, so results may vary slightly.
You must define the cost meaningfully for your use case.

🧪 Bonus: Cache Hit Tracking

type FileCache struct {
    cache *ristretto.Cache[string, []byte]
    hits  atomic.Uint64
    misses atomic.Uint64
}

func (fc *FileCache) Get(key string) ([]byte, bool) {
    val, ok := fc.cache.Get(key)
    if ok {
        fc.hits.Add(1)
    } else {
        fc.misses.Add(1)
    }
    return val, ok
}

🔚 Final Thoughts

HashiCorp’s LRU cache is great for simple use cases, but when performance and scalability matter (especially with concurrent file reads/writes) Ristretto is a better fit.

Highly recommend it for high-performance Go applications.

🔗 Resources

If you've used Ristretto let me know and in what way.

Journal Entry: When Overengineering Bit Me

Rhaqim — Wed, 14 May 2025 03:34:38 +0000

Overengineering is a silent productivity killer.

I recently found myself debugging an application that kept hanging. On the surface, everything looked fine: no panics, no major memory spikes just random, hard-to-reproduce blocking behaviour. It was frustrating.

Eventually, I discovered the root cause: I had hundreds of goroutines sleeping silently in the background.

Here’s what I had done:

After sending a verification email, I launched a goroutine to sleep for 10 minutes and then update the database to clear the token.
Each user action spun up a new goroutine.
I wasn’t tracking or cancelling these goroutines. No context, no lifecycle awareness.
As if that wasn’t bad enough, the goroutines were all hitting a relational database to update tokens that only lived for a few minutes.

At first, it felt like the “Go” thing to do. Using goroutines and sleep timers to delay token deletion. But in reality, I had:

Introduced memory bloat.
Increased DB load for ephemeral data.
Ignored lifecycle cleanup.

Surely there had to be a better and overall cleaner way to achieve this goal, so, I stepped back and I asked myself: “Do I really need to store short-lived verification tokens in a full-blown SQL database?”.
The answer was obvious. (The answer is no, by the way. In case you weren’t sure.
Don’t feel too bad.)

Here’s how I fixed the bug:

I moved token storage to Redis with a simple TTL.
Removed all background goroutines tied to token expiration.
Simplified the code and reduced the mental load of managing side effects.

And just like that, after rocking back and forth wide-eyed for days like a crazy person in front of my computer, the blocking issues disappeared. Redis cleaned up expired keys automatically, and my backend was leaner and more predictable.

💡 Lesson Learned

A major takeaway from this experience is that the simplest solutions are often the
most effective.
If you’re using goroutines for background jobs with sleep timers, ask yourself: Do I need this process to live beyond the request lifecycle?, Should this be a scheduled job instead?, Can Redis or another TTL-based store simplify this logic?

It’s fine to build fast but it’s much better to build wisely. That’s how you keep your
sanity intact.

Becoming DevOps

Rhaqim — Wed, 24 Jul 2024 15:45:33 +0000

I have no clue what I'm doing
I got a job a few months ago as a Golang Developer and I was ecstatic to finally work on a project that uses Go. For the past 3 years, I worked with Python and Node and had self-taught Go. I was eager to work on production-level code and not just my personal projects (of which I had built a lot), so when the opportunity came, I jumped at it. Due to some circumstances, I had to work on the project alone and I was left to manage a lot of the codebase, not just backend alone now but also frontend. The most troubling part, however, had to do with deployments.

All deployments were done manually, which meant that changes were slow and dependent on a developer. I had never really handled anything related to DevOps, but I knew there had to be an easier way to do this. But first, I needed to understand how the deployment even worked.

Given access to the server, I had no idea where to start or even what I was looking for. There was tech I had never worked with before like an Nginx server and podman which is similar to Docker (the only technology I am familiar with) and can work in tandem with it. There was a lot of work to be done and I lot I didn't understand so I got creative.

Noob DevOps
From my previous job, the developers were kind enough to explain how a lot of stuff on servers work, so I learned about SSH and I encourage everyone to learn about it as well. I knew security was a priority and gaining access to a server securely was possible with SSH.

Now that I was in the server though, I had no idea where to go next. I did remember there was a command called history that would show me a lot of the previous commands that had been typed, and I figured piecing them together would help me build a mental map of what was going on. Through history, I could recreate the manual deployments for a lot of the services. I read up on Podman and realized a lot of the commands were very similar to Docker, so that saved me a lot of time.

Nginx was a different beast entirely. I had never worked with it before; I had heard about it and was scared of it. I thought it was some mythical technology that only high-tech wizards could use. I was wrong, of course. It was set up already, so I didn't have to worry about that part, but I did need to understand the configurations and how sites-available links to sites-enabled. By the way, you can do this by running the command:

   sudo ln -s /etc/nginx/sites-available/meme /etc/nginx/sites-enabled/meme

This took me way longer than I would like to admit to figure out.

I figured out the syntax like server, location, alias (this one is really important), and then I set up my own service just to see that I could. It ended up being very important later on.

Once I got everything I wanted, I put them together in a bash script and tested that they all worked correctly. After a lot of trial and error, I got a working version. However, I still needed to SSH into the server to run these scripts for any change I made.

By the Power of Github Actions
A small job I did 2 years ago required me to use Github actions for a Slack channel, my manager at the time wasn't great so it didn't leave me with a great experience but I remembered how easy it was to setup and run. I read the documentation again and then went to work.

First I found a workflow that catered to what I was trying to do
Next I adjusted the YAML script, I configured them to run on the branches I wanted
I setup the secrets the workflow needed, they can be found in the settings on your repo
Lots and lots of tests After many a rotation on the axis I finally had what I wanted, now whenever I pushed new code to the configured branches I can rest assured that the code would be deployed.

Now my transformation was complete the noob had become slightly less noob

It was fun
Looking back, it was an exhilarating and educational experience. I transitioned from feeling completely lost to setting up a streamlined deployment process. I went from being a complete novice in DevOps to automating my deployments with GitHub Actions. The journey taught me a lot about perseverance, the importance of continual learning, and the satisfaction of overcoming technical challenges. While there were many frustrating moments, the triumphs made it all worthwhile.

Design Newsletter Emails in Golang

Rhaqim — Wed, 17 Apr 2024 19:52:36 +0000

Introduction

Have you ever wondered how newsletters from different companies and organisations work? Some companies use platforms like Mailchimp but if you're like me and want to understand what is going on and how to build yours so you have full control then follow along as we send a few custom newsletters to our subscribers.

The Breakdown

Before we start writing any code let us breakdown what happens when you receive a new email and what might be happening. When you receive a newsletter from any platform you're subscribed to, you notice that each one has a different design, logo, font. You can see the subject and a bold title and then we have the body and sometimes images. These are all contained in Templates.

Most times, these are crafted with HTML, CSS and Javascript (pretty vanilla, I know). The HTML files can be either a .html or .tmpl A sample template would usually start with the usual <!DOCTYPE html>. Here's a sample template:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>My Newsletter</title>
    <style>
        /* Add CSS styles here */
    </style>
</head>
<body>
    <header>
        <h1>Welcome to My Newsletter</h1>
    </header>
    <main>
        <!-- Newsletter content goes here -->
    </main>
    <footer>
        <p>&copy; 2024 My Company. All rights reserved.</p>
    </footer>
</body>
</html>

Dynamic templates

It would be a hassle, if we had to constantly edit the html file every time we wanted to send new content to our subscribers, luckily there's a way we can maintain the base feel of our newsletter while dynamically changing what our users see whenever they receive an email from us.

Using Go Templates
In Go, the html/template package allows us to create dynamic HTML content using template files. These templates can include placeholders or variables that are replaced with actual data when rendering the template.

Let's take a look at an example template file (newsletter_template.html):

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{{.Title}}</title>
    <style>
        /* Add CSS styles here */
    </style>
</head>
<body>
    <header>
        <h1>{{.Header}}</h1>
    </header>
    <main>
        <p>{{.Content}}</p>
    </main>
    <footer>
        <p>&copy; {{.Year}} My Company. All rights reserved.</p>
    </footer>
</body>
</html>

In this template, we've defined placeholders like {{.Title}}, {{.Header}}, {{.Content}}, and {{.Year}} that will be dynamically replaced with actual data when rendering the template.

Rendering the Template in Go
Now, let's see how we can render this template in a Go program:

package main

import (
    "html/template"
    "os"
)

func main() {
    // Read template file
    tmpl, err := template.ParseFiles("newsletter_template.html")
    if err != nil {
        panic(err)
    }

    // Define data for template
    data := struct {
        Title   string
        Header  string
        Content string
        Year    int
    }{
        Title:   "My Newsletter",
        Header:  "Welcome to My Newsletter",
        Content: "This is a test newsletter. Enjoy!",
        Year:    2024,
    }

    // Execute template
    if err := tmpl.Execute(os.Stdout, data); err != nil {
        panic(err)
    }
}

In this Go program, we parse the template file (newsletter_template.html) and define the data to be used when rendering the template. We then execute the template, passing the data, and the rendered HTML is written to the standard output (os.Stdout).

Benefits of Dynamic Templates
Using dynamic templates offers several benefits:

Flexibility: With dynamic templates, we can easily update the content of our newsletters without modifying the underlying HTML structure. This allows for quick iteration and experimentation with different content layouts.

Personalisation: Dynamic templates enable us to personalise the content of each newsletter based on user preferences, demographics, or behaviour. We can tailor the message to resonate with each subscriber, increasing engagement and conversion rates.

Automation: By integrating dynamic templates with our newsletter delivery system, we can automate the process of sending personalised emails to subscribers. This saves time and effort while ensuring a consistent brand experience for recipients.

Preparing the Email Setup

Now that we have our email template with all the data we want to send, the next step is to actually send it out to our subscribers. In this section, we'll explore how to accomplish this using Go and the gomail package.

Using the gomail Package
The Gomail is a package I developed that provides a simple and efficient way to send emails in Go. We'll use it to compose our email message, including the HTML content from our template, and send it to our subscribers. We'll need certain requirements in order to use the package:

Template Directory: To maintain organization and ease of management, it's helpful to store all our email templates in a dedicated directory. This directory will hold the HTML templates we've created for our newsletters. For example, we can create a directory named templates in our project directory and store all our template files there.

project_directory/
├── templates/
│   ├── newsletter_template.html
│   └── other_templates...

Mail Credentials: We need credentials from an email client, you can get them from your favourite providers here's how to get one for Google Gmail SMTP. Using Gomail we can add the credentials with the struct:

    package main

    import (
        "log"
        "github.com/rhaqim/gomail"
    )

    func main() {
        auth := gomail.EmailAuthConfig{
            Host:     "smtp.gmail.com",
            Port:     587,
            Username: "user",
            Password: "password",
            From:     "me@gmail.com",
        }

        templateDir := "templates"

        g := gomail.NewGoemail(auth, templatesDir)
    }

Note: I included the template directory and also created a new instance with the auth and templateDir

Sending the Email

Now that we have our email sender setup, we need to send the actual email. Gomail needs to know who you're sending the email to, could be one or more people, include a subject, what template in the template directory you want to use and then the body of the message. Putting it all together we have the following:

    package main

    import (
        "log"
        "github.com/rhaqim/gomail"
    )

    func main() {
        auth := gomail.EmailAuthConfig{
            Host:     "smtp.gmail.com",
            Port:     587,
            Username: "user",
            Password: "password",
            From:     "me@gmail.com",
        }

        templateDir := "templates"

        g := gomail.NewGoemail(auth, templatesDir)

        NewsletterApp(g)

    }

    func NewsletterApp(mail gomail.Gomail) {

        email := &gomail.Email{
            Recipients:       []string{"recipient1e@gmail.com", "recipiente2@gmail.com"},
            Subject:          "Hello",
            Body:             "Hello, this is a test email",
            TemplateFileName: "newsletter_template.html",
        }

        err := mail.SendEmail(email)
        if err != nil {
            log.Fatal(err)
        }
    }

Conclusion

This is an abstraction and the code for Gomail is publicly available. Feel free to fork and modify it for your own usecase. With the following you can design different templates and for different functions.

Multi-Select Component: React Native

Rhaqim — Fri, 01 Mar 2024 14:22:30 +0000

I started learning React Native a year ago to build a mobile app for a company I started with my friends, I wasn't a mobile developer at the time but lack of resources meant I had to be. Learning how it worked was fairly easy thanks to my background knowledge of React and Expo CLI. Working with the two meant picking up mobile development was somewhat easy, some what but not completely easy.

The Problem

React Native comes with a lot of helpful components that seem to have been built on HTML tags, so if you're familiar with HTML then you're halfway through becoming a React Native developer. One component in particular of interest is the <select></select> tag in HTML that doesn't seem to be present in the framework, if you're unfamiliar with how it works, its a little something like this:



<select id="cars" name="cars" multiple>
  <option value="volvo">Volvo</option>
  <option value="toyota">Toyota</option>
  <option value="mercedes">Mercedes</option>
  <option value="audi">Audi</option>
</select>

Note: I added the multiple so in order to select multiple options

This tag doesn't have a counterpart in React Native and if you're like me and don't like being dependent on people or packages then you would want something that came packaged with the framework, rather than having to install 3rd party libraries. Of course if you do, there's no judgement here and there are some pretty good options you could try like Nativebase. So now we don't want 3rd party libraries and React Native doesn't have a component, what do we do? we're developers, we make ours.

The Solution?

Thankfully React Native comes with a very useful component called FlatList, it's a component used to efficiently render a list of data e.g a list of users or clubs. It is recommended if the data size is large, where rendering all items at once might lead to performance issues. It only renders the items that are visible on the screen. Here's an example of how its used:



import { FlatList, Text, View } from 'react-native';

const data = [
  { id: '1', title: 'Volvo' },
  { id: '2', title: 'Toyota' },
  { id: '3', title: 'Mercedes' },
  { id: '4', title: 'Audi' },
  // Add more items as needed
];

const renderItem = ({ item }) => (
  <View style={{ padding: 10 }}>
    <Text>{item.title}</Text>
  </View>
);

const MyCarList = () => {
  return (
    <FlatList
      data={data}
      renderItem={renderItem}
      keyExtractor={item => item.id}
    />
  );
};

export default MyCarList;

It is a very powerful component I prefer using over the ScrollView to render a list of items. So, how does this almighty component help us select multiple items? It's fairly simple really, we create our own component that uses the FlatList to display our data.

MultiSelect Component

We're going to need one more component from React Native and that is the TouchableOpacity (when we select an item we want some feedback). The first thing we do is get our imports:



import React from 'react';
import { FlatList, Text, TouchableOpacity, View } from 'react-native';

Then we define our interface for this new component we are creating.



export type TouchableOpacityProps = DefaultTouchableOpacity["props"];

interface MultiSelectProps<T> extends TouchableOpacityProps {
  items: (string | T)[];
  valueKey?: keyof T;
  displayKey?: keyof T;
  selectedItems: string[];
  setSelectedItems: React.Dispatch<React.SetStateAction<string[]>>;
}

This extends the TouchableOpacity Props, this way we get the props associated with Touchable Opacity, at this point you're wondering why TouchableOpacity and not FlatList, it'll be clear as we go along.

Lets breakdown the MultiSelectProps:

MultiSelectProps<T>- this is the name of the prop but using typescript, we can set a generic for the type of data our FlatList would be working with.
items: (string | T)[] - This is a list of the items we will be displaying, as you can see it can either be a string or the T we defined as the generic
valueKey?: keyof T - This is the value we want when select an item e.g ID and it has to be a Key that exists on the T type -displayKey?: Keyof T - Similar to the valueKey, this is used to display content the user e.g name
selectedItems: string[] - All the items the user selects will be returned to this callBack function
setSelectedItems - In order to know what Items were selected we also need the state

Next we have our MultiSelect definition:



export function MultiSelect<T>(props: MultiSelectProps<T>): JSX.Element {
  const {
    items,
    valueKey,
    displayKey,
    selectedItems,
    setSelectedItems,
    ...otherprops
  } = props;

We need a way to handle the selected items the user clicks on:



  const handleSelect = (item: string | T) => {
    setSelectedItems((prev) => {
      const itemId = valueKey ? (item as T)[valueKey] : item;
      if (prev.includes(String(itemId))) {
        return prev.filter((i) => String(i) !== String(itemId));
      }
      return [...prev, String(itemId)];
    });
  };

The handleSelect function updates the selected items list by toggling the selection state of the provided item based on its ID or valueKey.

Finally we get to the part where we render the options given to us with the items:



  return (
    <FlatList
      data={items}
      keyExtractor={(item, index) =>
        valueKey ? String((item as T)[valueKey]) : String(index)
      }
      renderItem={({ item }) => (
        <TouchableOpacity
          onPress={() => handleSelect(item)}
          className="flex flex-row items-center justify-between w-full h-10"
          {...otherprops}
        >
          <Text>{displayKey ? (item as T)[displayKey] : item}</Text>
          <View
            className={`border border-black rounded-full w-4 h-4 ${
              selectedItems.includes(
                valueKey ? String((item as T)[valueKey]) : String(item),
              )
                ? "bg-black"
                : "bg-white"
            }`}
          />
        </TouchableOpacity>
      )}
    />
  );
}

Putting it all together we have:



import React from 'react';
import { FlatList, Text, TouchableOpacity, View } from 'react-native';

interface MultiSelectProps<T> {
  items: (string | T)[];
  valueKey?: keyof T;
  displayKey?: keyof T;
  selectedItems: string[];
  setSelectedItems: React.Dispatch<React.SetStateAction<string[]>>;
}

export function MultiSelect<T>(props: MultiSelectProps<T>): JSX.Element {
  const {
    items,
    valueKey,
    displayKey,
    selectedItems,
    setSelectedItems,
    ...otherprops
  } = props;

  const handleSelect = (item: string | T) => {
    setSelectedItems((prev) => {
      const itemId = valueKey ? (item as T)[valueKey] : item;
      if (prev.includes(String(itemId))) {
        return prev.filter((i) => String(i) !== String(itemId));
      }
      return [...prev, String(itemId)];
    });
  };

  return (
    <FlatList
      data={items}
      keyExtractor={(item, index) =>
        valueKey ? String((item as T)[valueKey]) : String(index)
      }
      renderItem={({ item }) => (
        <TouchableOpacity
          onPress={() => handleSelect(item)}
          className="flex flex-row items-center justify-between w-full h-10"
          {...otherprops}
        >
          <Text>{displayKey ? (item as T)[displayKey] : item}</Text>
          <View
            className={`border border-black rounded-full w-4 h-4 ${
              selectedItems.includes(
                valueKey ? String((item as T)[valueKey]) : String(item),
              )
                ? "bg-black"
                : "bg-white"
            }`}
          />
        </TouchableOpacity>
      )}
    />
  );
}

The MultiSelect component provides a versatile solution for creating multi-select interfaces in React Native applications. By accepting an array of items, along with optional valueKey and displayKey props for handling complex data structures, it offers flexibility in rendering and managing selections. The component efficiently utilizes FlatList for optimal performance, ensuring smooth interactions even with large datasets. With its customizable rendering and selection handling, the MultiSelect component simplifies the implementation of multi-select functionality across various use cases in React Native applications. An example of how to use the <MultiSelect />:



import React, { useState } from 'react';
import MultiSelect from './MultiSelect';

const App = () => {
  const [selectedItems, setSelectedItems] = useState<string[]>([]);

  // Example data
  const cars = [
    { id: 'volvo', name: 'Volvo' },
    { id: 'toyota', name: 'Toyota' },
    { id: 'mercedes', name: 'Mercedes' },
    { id: 'audi', name: 'Audi' },
    // Add more items as needed
  ];

  return (
    <MultiSelect
      items={cars}
      valueKey="id"
      displayKey="name"
      selectedItems={selectedItems}
      setSelectedItems={setSelectedItems}
      style={{ width: 200, maxHeight: 200 }}
    />
  );
};

export default App;

And here is how it looks on screen:

Feel free to abstract it as much as you like, add or take things out for your various use cases, ciao!