Forem: Tasos Nikolaou

When Chrome Ate My RAM: Designing a Pressure-Aware Tab Orchestrator with Rust

Tasos Nikolaou — Wed, 01 Apr 2026 16:30:07 +0000

Chrome wasn't "crashing."

It was just...slowly suffocating my system.

Over time, RAM usage would creep up. Background tabs accumulated state. Other applications started freezing. The fan would spin up. And yet, nothing looked obviously wrong. No single tab was the culprit.

The problem wasn't too many tabs.
The problem was a lack of coordination between the browser and the system.
So I built something to experiment with that idea.

This article explains the architecture and reasoning behind a hybrid Chrome extension & Rust native host that manages tab lifecycle based on real system pressure and user context.

The Problem: Browser Entropy

Modern browsers are operating systems.

They manage:

Dozens of isolated processes
Background timers
Network activity
Memory-heavy applications (Jira, GitHub, Gmail, ChatGPT, Claude 😊 etc.)

Most tab suspension tools rely on a simple rule:

"If a tab hasn't been used in X minutes, suspend it."

That's convenient, but blind.

They don't know:

Whether the system is under memory pressure
Whether CPU is spiking
Whether you're on battery
Whether the tab is part of your active workflow

They operate on time, not state.

What I wanted was:

A deterministic, pressure-aware, context-sensitive lifecycle engine.

Not AI. Not cloud analytics. Just a well-structured system.

Design Goals

Before writing any code, I defined constraints:

Deterministic behavior (no black-box magic)
No cloud, no telemetry
Respect user intent (never discard active or pinned tabs)
Pressure-aware decisions
Context-aware heuristics
Clean separation of responsibilities

This last one became the most important architectural decision.

Architecture Overview

The system consists of two components:

Chrome Extension (MV3)
  - Tab activity tracking
  - Focus clustering
  - TTL gating & guardrails
        ↓ Native Messaging
Rust Native Host
  - System metrics (RAM, CPU, Battery)
  - Pressure scoring engine
  - Deterministic classification

Why split it?

Chrome extensions cannot access low-level system metrics like real memory pressure in a reliable way.

So I separated concerns:

The extension manages browser lifecycle.
The Rust native host understands system state.

They communicate through Chrome's Native Messaging API.

This keeps the system clean:

Browser logic stays in the browser.
System logic stays native.

The Pressure Engine (Rust)

Instead of checking raw RAM percentage, I built a weighted pressure scoring model.

The Rust host collects:

Total RAM
Used RAM
Free RAM
CPU usage
Battery state (if available)

From these, it computes:

pressure_score (0-100)
pressure_level (LOW / MEDIUM / HIGH)
pressure_reasons (RAM_HIGH, CPU_ELEVATED, ON_BATTERY, etc.)

RAM is the dominant signal.
CPU acts as a modifier.
Battery adds a small aggressiveness bias.

The goal is not to be perfect, it's to be consistent and explainable.

Instead of saying:

"System busy."

It says:

HIGH pressure because RAM_HIGH + ON_BATTERY.

That reason tagging matters for transparency.

Context Awareness - Focus Clustering

Not all inactive tabs are equal. A tab opened 30 minutes ago in your active workflow is very different from a forgotten tab in another window.

So I introduced Focus Mode.

Focus clustering is based on:

Same hostname as active tab
Recent activity window
Same window constraint
Cluster size cap

Tabs inside the active "cluster" use a longer TTL.Tabs outside the cluster expire faster under pressure. This makes the system:

Less disruptive
More aligned with user context
Less likely to discard something you'll immediately need

It's still deterministic, but just smarter.

Guardrails & Safety

Aggressive resource management can easily become destructive. So, strict guardrails were built in:

Never discard active tabs
Never discard pinned tabs
Never discard audible tabs
Respect protected domains
Enforce TTL minimums
Apply cooldown between prune cycles

This prevents oscillation and surprise behavior. The goal is not maximum efficiency. The goal is controlled stability.

Why Rust?

Rust was chosen for the native host because:

Memory safety
Explicit modeling
Strong type system
Clean modular architecture
Lightweight binary

The Rust side is structured into modules:

metrics, system state collection
battery, optional battery signal
pressure, scoring logic
protocol, native messaging transport
state, API contract

This makes the native host feel like a real subsystem, not a script.

What It Achieves

In practice, this system:

Reduces RAM pressure under load
Keeps active workflows intact
Makes browser behavior predictable
Avoids blind "time-based" suspension
Plays nicer with other system applications

It doesn't eliminate memory usage. It orchestrates it.

What I Learned

A few things stood out during this project:

1. MV3 Service Workers Have Quirks

Extension background scripts are ephemeral. State management must be deliberate.

2. Determinism Beats "Smartness"

Clear, explainable rules feel safer than opaque heuristics.

3. Separation of Concerns Changes Everything

Keeping system logic in Rust and browser logic in the extension made experimentation much easier.

4. Observability Matters

Reason tagging and structured logging made debugging and tuning far easier.

Future Directions

This project is still evolving. Some experimental directions:

Event-driven pressure signals (instead of polling)
Chrome process memory integration
Predictive return probability modeling
Offline data analysis of tab lifecycle patterns
Adaptive TTL tuning

The architecture supports these without becoming tangled.

That was intentional.

Conclusion

Chrome didn't have a bug. It was just operating without coordination. By introducing a pressure-aware, context-sensitive orchestration layer, the browser becomes less chaotic and more cooperative with the system.

This project started as frustration with RAM usage. It turned into an exploration of how browsers and operating systems can communicate more intelligently, without AI hype, and without cloud dependencies.

Just clean architecture and deterministic policy!

Checkout the project here:
Github: https://github.com/tase-nikol/tab-memory-orchestrator

Sometimes the problem isn’t that a system is broken
It’s that its parts aren’t talking to each other.

PromptCache Part II: When High Cache Hit Rates Become Dangerous

Tasos Nikolaou — Thu, 05 Mar 2026 07:50:27 +0000

A benchmark-driven look at semantic cache safety and intent isolation.

In the previous article, "Stop Paying Twice for the Same LLM Answer" - I introduced PromptCache as a semantic caching layer designed to reduce LLM cost and latency.

The premise was simple, if two prompts are semantically similar, we shouldn't pay for the answer twice. The results were compelling: high cache hit rates, significant cost reduction, lower latency. But after deploying and stress-testing the design, a more important question emerged:

What guarantees that a cache hit is actually correct?

Reducing cost is easy.
Ensuring safe reuse is harder.

This article documents the experiment that reshaped PromptCache's architecture, and why intent isolation became a non-negotiable design constraint.

Measuring Semantic Cache Safety in LLM Systems

Most semantic caches follow a simple pattern:

Embed the prompt
Retrieve the nearest cached embedding
If similarity ≥ threshold then reuse the answer

This works well for performance.
But it assumes something that isn't guaranteed:

That semantic similarity implies safe reuse.

To test that assumption, I built a controlled benchmark.

The Question

Can cosine similarity thresholding alone guarantee safe reuse? Or do we need structural isolation between tasks? To answer this, I defined a metric:

Unsafe Hit

A cache hit is unsafe if the returned answer belongs to a different task (intent) than the incoming request. This measures semantic collision, not embedding quality.

What Is Intent Isolation?

Intent isolation means:

Partition the semantic cache by task boundary before performing similarity search.

Instead of searching across all cached entries, we search only within the matching task. Similarity becomes a refinement step, not a boundary mechanism.

Figure 0 - Semantic Cache Search Space.
Without isolation, similarity search spans all tasks in one shared embedding space.

With isolation, search is restricted to the matching intent partition.

Experimental Setup

I evaluated semantic caching across:

Embedding model: all-MiniLM-L6-v2
Backends:
- In-memory brute-force cosine
- Redis (HNSW via RediSearch)
Workloads:
- Support queries
- RAG-style retrieval questions
- Creative prompts
Threshold sweep: 0.82 -> 0.92
~2400 requests per configuration

Two configurations were tested:

1. No Intent Isolation

All prompts shared the same semantic space.

2. Intent Isolation Enabled

Cache entries were partitioned by intent_id.

Each configuration was evaluated across identical prompt sequences to ensure comparability.
Unsafe hits were computed by comparing stored intent_id against request intent.

Result 1 - Hit Rate Looked Excellent

Without intent isolation:

Hit rate: ~97-99% With intent isolation:
Hit rate: 13-38% depending on threshold

Figure 1 - Hit Rate vs Threshold. Without intent isolation, semantic caching achieves ~98% hit rate. Enabling intent partitioning significantly reduces reuse density.

At first glance, the non-isolated configuration looks superior. But this metric is incomplete.

Result 2 - Unsafe Hit Rate Reveals the Problem

Without intent isolation:

Unsafe hit rate: ~95-100% With intent isolation:
Unsafe hit rate: 0%

Figure 2 - Unsafe Hit Rate vs Threshold. Similarity thresholding alone does not prevent cross-intent reuse. Nearly all cache hits become unsafe without intent partitioning.

This pattern was consistent across support, RAG, and creative workloads. In other words:

Almost every "successful" cache hit without isolation was incorrect.

This is not a marginal effect. It is structural cross-contamination.

Why Similarity Is Not Enough

Embedding similarity measures geometric proximity in vector space.
Intent boundaries are categorical.

Cosine similarity answers:

"Are these prompts semantically related?"

It does not answer:

"Are these prompts operationally interchangeable?"

Semantic closeness is continuous and task equivalence is discrete. Threshold tuning cannot convert a continuous metric into a categorical guarantee, but, partitioning can.

Result 3 - Backend Did Not Affect Correctness

Both Redis (HNSW) and the in-memory backend produced identical:

Hit rate curves
Unsafe hit rate curves

This was expected, due to the fact that both implemented cosine nearest-neighbor search with identical threshold logic. Correctness was dominated by key structure, not vector store implementation.

Backend choice affects:

Persistence
Multi-process access
Scalability
Latency under load

But correctness properties should not depend on storage details!

Cost Savings Followed Hit Rate

In this benchmark, each miss triggered a full LLM call with similar token usage.

As a result, cost_savings ≈ hit_rate, which confirms internal consistency. But cost reduction is meaningless if reuse is unsafe, meaning that correctness precedes optimization.

Production Implications

If you rely solely on similarity thresholding:

You will inflate hit rates
You will inflate cost savings
You may silently reuse incorrect answers

This is particularly dangerous in:

Multi-tenant systems
Support bots
RAG pipelines
Tool-driven workflows

The correct architectural pattern is:

Partition first.
Threshold second.

Similarity is a refinement mechanism, not a safety boundary.

Limitations

This was a controlled benchmark.

Dataset size was modest (~2-3k prompts)
Workloads were synthetic but structured
Extreme-scale recall behavior was not evaluated
Concurrency stress was not measured

The goal was to isolate semantic collision behavior, not benchmark vector database scalability.

Future work should explore:

Larger datasets
Cross-model embedding drift
Concurrency stress testing
Partial response reuse

Core Insight

The dominant factor in semantic cache correctness is not:

The embedding model
The vector database
The similarity threshold

It is key design.
Intent isolation is not an optimization.
It is a safety requirement.

Final Takeaway

A 98% cache hit rate looks impressive.
But without structural boundaries, it may be misleading.

If your semantic cache shows:

98% hit rate
98% cost savings

Ask yourself, how many of those hits are actually correct?
Optimization without isolation is probabilistic reuse. If you're building LLM infrastructure, this is not an academic nuance, but a production concern.

Similarity optimizes reuse.
Isolation guarantees correctness.

PromptCache Part I: Stop Paying Twice for the Same LLM Answer

Tasos Nikolaou — Tue, 24 Feb 2026 08:23:32 +0000

Designing a semantic cache layer for cost and latency optimization in LLM systems.

Most LLM cost isn’t spent on novelty.
It’s spent on repetition, requests that are semantically identical, but syntactically different.

PromptCache was built to eliminate that redundancy.

The Invisible Cost Leak in LLM Systems

If you’re running an LLM in production, you are almost certainly paying for this:

"How do I reset my password?"
"I forgot my password, what do I do?"
"Steps to reset account password?"
"Help me change password"

Different strings.
Same intent.
Same answer.
Different billable request.

Traditional caching doesn't help because:

"How do I reset my password?" != "Steps to reset account password?"

Exact match fails.
But meaning hasn't changed.
That's where semantic caching comes in.

The Theory: Why This Works

LLMs don't understand text as strings.
They convert text into vectors (embeddings).
Two sentences with similar meaning produce vectors that are close together in high-dimensional space.

Example (simplified):

"Reset my password"
      ↓
[0.12, -0.87, 0.44, ...]

"How do I change my password?"
      ↓
[0.11, -0.89, 0.41, ...]

These vectors are very close.

So instead of asking:

"Have I seen this exact string before?"

We ask:

"Have I seen something semantically similar before?"

If the similarity is high enough, we reuse the answer.
That's semantic caching.

How It Works in Practice

When a request comes in:

User Prompt
     ↓
Embedding
     ↓
Vector search in Redis
     ↓
High similarity?
     ↓
Yes → Return cached response
No  → Call LLM and store result

You're adding a semantic memoization layer in front of your LLM.

Real Results

In a support-heavy workload with repetitive queries:

~60% cache hit rate
~50% reduction in token usage
~40% lower API spend

Results vary by workload density and repetition patterns, but in structured environments, the impact is immediate.

Example Implementation

Here's a simplified example using Redis vector search:

from promptcache import SemanticCache
from promptcache.backends.redis_vector import RedisVectorBackend
from promptcache.embedders.openai import OpenAIEmbedder
from promptcache.types import CacheMeta

embedder = OpenAIEmbedder(model="text-embedding-3-small")

backend = RedisVectorBackend(
    url="redis://localhost:6379/0",
    dim=embedder.dim,
)

cache = SemanticCache(
    backend=backend,
    embedder=embedder,
    namespace="support-bot",
    threshold=0.92,
)

meta = CacheMeta(
    model="gpt-4.1-mini",
    system_prompt="You are a helpful support assistant.",
)

result = cache.get_or_set(
    prompt="How can I change my password?",
    llm_call=my_llm_call,
    extract_text=lambda r: r.output_text,
    meta=meta,
)

print(result.cache_hit)

That's it.

No orchestration framework required.

If you want to try this approach, I packaged it up here:

GitHub: https://github.com/tase-nikol/promptcache
PyPI: https://pypi.org/project/promptcache-ai/

Install:

pip install promptcache-ai

When This Works Best

Semantic caching is powerful when:

Prompts are repetitive
Temperature is low
Answers are stable
Volume is high

It won't help much for:

Highly personalized prompts
Creative writing
Rapidly changing context

In those cases, novelty dominates repetition, and caching provides diminishing returns.

The Bigger Insight

Most LLM systems are fundamentally stateless.
They recompute answers even when nothing meaningful has changed.

Semantic caching introduces selective memory, reusing intelligence only when it is economically justified.

Instead of optimizing prompts endlessly, sometimes the smarter move is optimizing infrastructure.

If you're building LLM systems in production, semantic caching is one of the highest-leverage optimizations you can add.

But optimizing cost raised a more uncomfortable question:
What guarantees that a cache hit is actually correct?

In the next article, we examine how high hit rates can silently mask semantic errors, and why PromptCache evolved beyond threshold tuning.

Intelligence is expensive.
Memory is cheap.
Use both wisely.

Building a Framework-Agnostic Health Check Library for Python Microservices

Tasos Nikolaou — Tue, 17 Feb 2026 21:31:51 +0000

From duplicated /health endpoints to a published PyPI package - an engineering deep dive.

The Problem: Death by Copy-Paste Health Checks

In a typical microservice architecture, health endpoints start simple:

GET /health
GET /ready

But over time, reality sets in.

Some services use:

Django + PostgreSQL + Redis + Celery
FastAPI + SQLAlchemy + Redis
BFFs that depend on upstream HTTP services
RabbitMQ + background workers
Async stacks mixed with sync stacks

Each service needs:

Liveness checks
Readiness checks
Dependency verification
Timeouts
Structured JSON output
Correct HTTP status codes

And before long, every service has its own slightly different HealthService.

Different thresholds.
Different response formats.
Different timeout logic.
Different readiness semantics.

That's when I realized:

Health checks are infrastructure. They should not be rewritten per service.

So I built PulseCheck - a framework-agnostic health and readiness library for Python.

Design Goals

Before writing a single line of code, I defined constraints:

Framework-agnostic core
Pluggable dependency checks
Async-first design (to support FastAPI)
Sync compatibility (for Django)
No forced dependency pollution
Kubernetes-friendly readiness semantics
Optional dependency extras
Clean, structured JSON output
Production-safe timeouts

This wasn't just about code reuse.

It was about architectural consistency.

Architecture: Core + Adapters

The key design decision was separation of concerns.

pulsecheck/
│
├── core/        ← Framework-agnostic health engine
├── fastapi/     ← FastAPI adapter
└── django/      ← Django adapter

1. Core Engine

The core layer contains:

Health registry
Health aggregation logic
Status combination rules
Dependency check base class
Timeout handling
Response schema

It has zero framework dependencies.

The core doesn't know what FastAPI or Django is.

2. Pluggable Checks

Each dependency is implemented as a check:

SQLAlchemyAsyncCheck
DjangoDBCheck
RedisAsyncCheck
RedisSyncCheck
RabbitMQKombuCheck
CeleryInspectCheck
HttpDependencyCheck

Each check:

Has a name
Has a timeout
Has a degraded threshold
Returns structured results

Example:

registry = HealthRegistry(environment="prod")

registry.register(SQLAlchemyAsyncCheck(engine))
registry.register(RedisAsyncCheck(redis_url))
registry.register(CeleryInspectCheck(celery_app))

No monolithic service class.
Just composition.

Async-First, Sync-Compatible

FastAPI is async.
Django is traditionally sync.

Instead of creating two engines, the core is async-first.

Sync checks are wrapped using:

asyncio.to_thread(...)

This gives:

Async compatibility
Non-blocking readiness
Unified aggregation logic

This avoids duplicating the health engine.

Readiness vs Liveness

This is often misunderstood.

Liveness:

"Is the process alive?"

Readiness:

"Can this service safely receive traffic?"

PulseCheck separates them cleanly:

registry.liveness()
await registry.readiness()

Readiness runs dependency checks.
Liveness does not.

This mirrors Kubernetes probe behavior.

Handling Degraded States

Health isn't binary.

Instead of just UP or DOWN, PulseCheck supports:

HEALTHY
DEGRADED
UNHEALTHY

If a dependency is slow but responding:

{
  "status": "DEGRADED",
  "response_time_ms": 750
}

This gives operational insight without triggering restarts.

Optional Dependencies Done Right

One of the most important design decisions was dependency management.

FastAPI projects already have FastAPI.
Django projects already have Django.

The library must not force unnecessary installations.

In pyproject.toml:

[project.optional-dependencies]
fastapi = ["fastapi>=0.100"]
django = ["Django>=4.2"]
redis_async = ["redis>=5.0"]
rabbitmq = ["kombu>=5.3"]
celery = ["celery>=5.3"]

Now:

FastAPI service:

pip install pulsecheck-py[fastapi,redis_async]

Django service:

pip install pulsecheck-py[django,redis_sync]

Clean. Explicit. Controlled.

Hiding Health Endpoints From Swagger

Health endpoints are infrastructure endpoints.

In FastAPI:

@router.get("/health", include_in_schema=False)

They exist.
They work.
They don't pollute public API docs.

Small detail. Big professionalism signal.

Testing Before Publishing

Before uploading to PyPI, I tested:

Editable installs (pip install -e .)
Wheel builds (python -m build)
Installation from built wheel
Installation from TestPyPI
Optional extras resolution
Fresh virtual environments

I also learned something important:

TestPyPI contains junk packages that can interfere with dependency resolution. Always use:

--extra-index-url https://test.pypi.org/simple/

Not --index-url.

Small ecosystem lesson.

Publishing to PyPI

Publishing was straightforward:

python -m build
python -m twine upload dist/*

Important rule:

You cannot overwrite a version on PyPI.

Every change requires a version bump.

This enforces discipline.

Engineering Lessons Learned

Design the API before writing implementation.
Keep core logic framework-agnostic.
Async-first design avoids duplication.
Optional dependencies prevent ecosystem pollution.
Health endpoints are infrastructure, not business logic.
Packaging and versioning require discipline.
Publishing is easier than maintaining.

Why This Matters

Microservices suffer from invisible duplication.

Health checks are often treated as boilerplate.

But consistency in infrastructure code improves:

Operational clarity
Monitoring integration
Kubernetes reliability
Onboarding speed
Codebase maintainability

PulseCheck turned copy-paste health logic into a reusable, composable abstraction.

Future Roadmap

OpenTelemetry hooks
Prometheus integration
Circuit-breaker awareness
Startup probe support
Health history tracking
Async worker health strategies

Final Thoughts

Publishing a library is not about writing code.

It's about:

API design
Dependency discipline
Versioning strategy
Documentation clarity
Ecosystem compatibility

PulseCheck started as internal cleanup.
It became a reusable infrastructure layer.

If you're duplicating health logic across services, consider abstracting it.

Your future self will thank you.

Links

PyPI: https://pypi.org/project/pulsecheck-py/
GitHub: https://github.com/tase-nikol/pulsecheck-py

If you'd like feedback on the architecture or want to contribute, feel free to reach out.

"PulseCheck is intentionally minimal today. But its architecture allows deeper observability and resilience integrations"