Forem: Swayam Maheshwari

From 8 CPUs to Efficiency: How a Single Unicode Character Doubled Our Bill

Swayam Maheshwari — Fri, 15 May 2026 07:53:01 +0000

In the world of cloud computing, auto-scaling is often viewed as a safety net. It’s the magic that keeps your app alive during a traffic surge. But what happens when your server scales to 8 CPUs not because of a surge in users, but because of a "poison pill" hidden in your database queries?

Last week, our team faced a production crisis: our CPU utilization hit 100%, our cloud costs spiked by 100% overnight, and the culprit was a single malformed Unicode character.

The Incident: The "Ghost" Traffic Spike

It started with an automated alert. Our AWS/GCP instances were hitting their limits, and the auto-scaler was aggressively spinning up 8-core machines.

When we checked our analytics, the math didn’t add up. We had a very low volume of active users—nowhere near enough to justify that kind of compute power. Yet, the SQL logs showed a different story: the database was gasping for air.

The Root Cause: The RegEx CPU Bomb

After digging into our PostgreSQL slow query logs, we found the bottleneck. It was a SELECT query used for our public sidebar data.

To prevent the app from crashing due to malformed JSON (caused by binary PDF data and null bytes), we were using a SQL-level fix:

regexp_replace("publishedEndpoint"::text, '\\u0000', '', 'g')

Why this killed our performance:

Linear Scans: For every single request, the database had to cast large JSON blobs into text.
Regex Overhead: Running a Regular Expression engine over "vast data" (like 2MB strings of PDF-polluted JSON) is extremely CPU-intensive.
Frequency: Because this was a sidebar query, it was being called constantly.

Essentially, we were asking our database to perform deep-cleaning surgery on thousands of rows of data every second.

How We Fixed It: A Three-Layered Strategy

We realized that "fixing it in the query" was a band-aid that had become a liability. We moved to a multi-layered architectural solution.

1. Breaking the Query

First, we decoupled the monolithic API. Instead of one massive query that fetched and cleaned everything, we broke it into two separate, optimized APIs. This reduced the "surface area" of the data being processed by the database engine at any one time.

2. The Redis Buffer

Why clean the same data twice? We implemented Redis to store the "sanitized" version of the sidebar.

The Flow: The first time a user requests the data, the server cleans the Unicode, formats the JSON, and stores it in Redis.
The Result: Subsequent requests are served in milliseconds directly from memory. The database never sees the request, and the CPU stays cool.

3. Edge Caching with Cloudflare

For our public APIs, we added a layer of protection at the "Edge." By configuring Cloudflare Cache, we ensured that public data is served from the CDN.

This means a user in London gets their data from a London server without ever hitting our origin database in the first place.

Final Lessons Learned

Sanitize at the Entry, Not the Exit: The best way to handle a \u0000 (null byte) error is to never let it reach the database. Sanitize your inputs in your Node.js/Sequelize logic before the INSERT.
SQL is not a Text Editor: While SQL can perform RegEx, it is not optimized for it at scale. If you find yourself using regexp_replace in a high-traffic SELECT, you are sitting on a performance time bomb.
Monitor Costs as a Metric: A spike in CPU is a technical issue; a 100% increase in billing is a business crisis.

By moving the heavy lifting away from the SQL engine and into Redis and Cloudflare, we were able to scale back down to our standard CPU usage, saving our performance and our budget.

Have you ever had a "poison pill" query crash your production? Let's discuss in the comments! 🚀

Why Did My Single-Node Redis Think It Was a Replica? A Debugging Deep Dive

Swayam Maheshwari — Thu, 14 May 2026 05:46:55 +0000

Swayam Maheshwari

May 14

The Mystery of the Redis Read-Only Error in a Single-Node Setup

#ai #productivity #devops #node

Comments

4 min read

The Mystery of the Redis Read-Only Error in a Single-Node Setup

Swayam Maheshwari — Thu, 14 May 2026 05:45:27 +0000

If you manage a realtime application, you know that Redis is often the beating heart of your infrastructure. Recently, our production application—which relies heavily on Redis for both backend caching and realtime collaboration (via Hocuspocus/Yjs)—experienced a bizarre and catastrophic outage.

Every few months, out of nowhere, Redis would randomly crash our system. The logs were flooded with a single, confusing error:

READONLY You can't write against a read only replica

The symptoms were severe: writes failed entirely, reads stopped working, and the entire realtime system came to a grinding halt. Restarting the Docker container fixed the issue immediately, but without a root cause, it was only a matter of time before it happened again.

Here is a step-by-step breakdown of how I investigated, debugged, and ultimately solved this elusive Redis bug.

Step 1: Evaluating the Infrastructure

Before diving into logs, I needed to confirm exactly what our architecture looked like.

Hosting: A single Google Cloud Platform (GCP) VM (t2d-standard-1 with Debian 12, 1 vCPU, 4 GB RAM).
Deployment: Redis running inside a Docker container.
Topology: A single Redis node. No Redis Cluster. No Sentinel. No intentional replicas.

This is where the mystery deepened. If there was only one Redis node, how could it possibly think it was a "read-only replica"?

Step 2: Checking the Current Redis State

My first move was to check the current role of the Redis instance. I connected to the server and ran:

redis-cli INFO replication

The output was telling:

role:master
connected_slaves:0
master_failover_state:no-failover

Redis was clearly functioning as a master with no connected replicas. Whatever had caused the READONLY error wasn't a permanent state change.

Step 3: Ruling Out the Red Herrings

When debugging distributed systems, it's easy to go down the wrong rabbit hole. Here is what I evaluated and quickly ruled out:

Redis Cluster & Sentinel Failovers: I wondered if an automated failover had demoted our primary node. However, since we weren't running Cluster or Sentinel mode, there was no orchestration tool present to trigger a failover or slot migration.
Redlock / Distributed Lock Split-Brain: While distributed locks can cause chaos, they don't change a server's replication role.
The "Read" Clue: If Redis had truly become a standard replica, reads should still have worked. The fact that reads and writes both failed suggested this wasn't just a simple case of a node functioning as a healthy replica.

Step 4: Investigating Memory and Resources

Could the server be buckling under memory pressure? I checked the system and Redis memory stats:

redis-cli INFO memory

The results were eye-opening, but not in the way I expected:

used_memory_human: 1.60M
used_memory_rss_human: 15.85M
total_system_memory_human: 3.83G

Our actual dataset was only about 672 KB! Redis was using a fraction of a percent of the VM's RAM. It wasn't an Out-Of-Memory (OOM) crash.

However, I discovered a massive production risk in our configuration:

maxmemory:0
maxmemory_policy:noeviction

With no memory limit and noeviction set, if Redis ever did fill up, it would refuse all writes. While this wasn't the root cause of the current bug, it was a ticking time bomb that needed immediate fixing.

Step 5: Piecing Together the Root Cause

With OOM and Cluster failovers ruled out, the evidence pointed toward a few highly probable culprits for a single-node setup:

Accidental REPLICAOF Execution: A rogue script, automation, or network blip might have accidentally sent a REPLICAOF host port command, temporarily turning the node into a replica.
Stale Node.js Client Connections: Our Node.js backend and Hocuspocus websocket server maintain long-lived TCP connections. If the network dropped or the Docker container glitched, the client connection pool might have ended up in a stale state, misinterpreting the connection status.
Docker/Network Instability: Temporary network partitions or disk IO blocks (during AOF/RDB saves) might have forced Redis into a protective mode that the application clients misinterpreted.

The temporary nature of the issue, combined with both reads and writes failing, strongly pointed to a combination of stale client connections combined with a transient Docker or network interruption. Restarting the container severed those dead connections and forced a clean reconnect.

Step 6: The Fix and Future-Proofing

To stabilize the system and ensure this doesn't happen again, I implemented a multi-layered fix.

1. Hardening the Memory Config

First, I patched the memory risk by adding proper limits to /etc/redis/redis.conf:

maxmemory 2gb
maxmemory-policy allkeys-lru

2. Disabling Dangerous Commands

To prevent any accidental role changes in our single-node setup, I locked down the replication commands in redis.conf:

rename-command REPLICAOF ""
rename-command SLAVEOF ""

3. Creating a Debug Playbook

I established a strict rule: Next time it fails, do not restart immediately. Instead, run these diagnostics to capture the exact failure state:

redis-cli INFO replication
redis-cli INFO stats
redis-cli CONFIG GET replica-read-only
docker logs <redis-container-name> --tail 200

4. Rethinking the Architecture

While a single Redis node is fine for basic caching, heavy realtime workloads (like Hocuspocus Pub/Sub) demand high availability. Our long-term fix isn't to overcomplicate things with Redis Cluster, but rather to migrate to a standard Primary + Replica + Sentinel setup. This will give us automatic failover and separate the realtime collaboration load from the standard cache.

Conclusion

Sometimes the most intimidating errors—like an impossible READONLY replica state on a single node—are symptoms of deeper infrastructural quirks rather than actual state changes. By methodically checking the actual Redis state, analyzing memory limits, and ruling out red herrings, we not only diagnosed the immediate issue but uncovered hidden risks that made our production environment infinitely stronger.