Real-Time API Abuse Detection Engine

Jasmin Tankić — Wed, 06 Aug 2025 23:21:54 +0000

This is a submission for the Redis AI Challenge: Beyond the Cache.

🚧 What I Built

API Abuse Detection Engine is designed to instantly detect, analyze, and mitigate cybersecurity threats. The solution handles various common API abuse patterns such as:

Brute-force attacks: Quickly identifies repeated failed login attempts across username, IP, and username+IP scopes.
Enumeration attacks: Spots attackers systematically probing endpoints by tracking distinct paths per IP per minute.
IP Recurrence / Rate-Limit Abuse: Detects aggressive requesters using a Redis-backed token bucket.
Replay attack Detection: Rejects re-submitted requests via idempotency/correlation keys with TTLs.

The system only stores confirmed threats in Redis, using quick “minute buckets” and unique lists to detect them almost instantly. It can quickly group threats by IP and show the most recent examples for fast investigation.

It also keeps fast-access “jars” of known malicious IPs, usernames, and fingerprints, so it can immediately escalate or block threats as soon as they appear.

🚀 Demo

ANY request to /api/ingest/** (GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS/TRACE)
- The engine extracts identity signals (username, client-ip, user-agent, correlation-id) from headers/query/body per config.
- Redis counters/sets/hashes are updated atomically; detectors return a verdict immediately.
- If threats are detected, the event is stored in Redis for analytics (body truncated & base64 if large; header allowlist applied).
- The response contains threats, recommendations, and details.
GET /api/dashboard/threats?from=2025-08-06T01:12:00Z&to=2025-08-06T01:20:00Z&samplesPerIp=3
- Returns totals and IP-grouped results for the window, with the newest samples per IP.
- GET /api/dashboard/threats/{id} returns full stored event details.

The engine also ships with an integrated, out-of-the-box dashboard that visualizes all detected attacks in real time and lets you drill down into the full details of each threat event.

GIT source code: https://github.com/jasmintankic/api-guard

🔰 How I Used Redis 8

Redis 8 is the core of both detection and analytics:

Primary Real-Time Store: Redis holds minute buckets, distinct sets, token buckets, and threat records. That gives me constant-time (O(1)) updates and lookups on the hot path.
Atomic Counters & TTL: INCR + EXPIRE enable rolling-window analytics without background jobs. Buckets just expire away.
Fast Distincts: Using Sets/HyperLogLog, I track unique endpoints per IP per minute to surface enumeration behavior instantly.
Replay Safety with SETNX: Idempotency/correlation keys are stored with TTL so replays are rejected immediately.
Token Bucket in a Hash + CAS: A per-principal token bucket lives in a Redis Hash (c = credits, ts = last update). I apply WATCH/MULTI/EXEC for optimistic concurrency so limits are precise under contention.
ZSET Time Indexes: Threat events are indexed by timestamp for millisecond-range queries. Grouping by IP retrieves newest samples without scanning.

🔋 Under the Hood: How Redis Powers Real-Time API Threat Detection

Brute-force Attack Detection:
Per-minute counters in Redis track failed attempts across username, IP, and username+IP. A 1-minute bucket granularity with a multi-minute window and TTL provides precise rolling windows. Threshold trips set cool-off locks in Redis to dampen bursts.

Enumeration Attack Detection:
For each IP and minute, I track distinct endpoints accessed using Sets/HyperLogLog. A high cardinality over a short window is a strong enumeration signal.

Replay Attack Prevention:
An idempotency/correlation key is written with SETNX and a short TTL. If the same key reappears, Redis rejects it at the door — no double processing.

IP Rate-Limit Abuse:
A token bucket per principal is stored in a Hash (credits + last timestamp). With WATCH/MULTI/EXEC, credits are replenished by elapsed time and atomically decremented per request, ensuring accurate limits even at high concurrency.

Traffic/DoS Anomaly Signal:
Each endpoint maintains per-minute counts. I keep an EWMA mean/variance in Redis and compute a z-score per minute; large spikes flag anomalies fast, without shipping data elsewhere.

Real-time Analytics:
Threat-positive events are stored as compact Hashes and indexed in ZSETs by time. Queries over [from,to] return totals and IP-grouped samples in milliseconds, regardless of raw traffic volume.

This approach lets me track:

Total threats in any timeframe.
Breakdown by IP with the newest, most relevant samples.
Drill-down to full event details by ID.

The high performance, flexibility, and atomic semantics of Redis 8 power the entire engine with minimal latency and maximum reliability.

🚢 Conclusion

At the core of this engine, Redis 8 acts as the brain that powers every detection and analytics decision. From counting requests and tracking unique endpoints, to blocking replays and indexing threats by time, Redis does all the heavy lifting in real time. Its atomic operations, TTL-based rolling windows, sets, hashes, and sorted sets keep detection instant and analytics lightning-fast, all within a single, high-performance system.
Redis isn’t just supporting the process, it is the process that makes real-time API security possible.

Forem: Jasmin Tankić

Real-Time API Abuse Detection Engine

🚧 What I Built

🚀 Demo

🔰 How I Used Redis 8

🔋 Under the Hood: How Redis Powers Real-Time API Threat Detection

🚢 Conclusion