Forem: Rost

16 GB VRAM LLM benchmarks with llama.cpp (speed and context)

Rost — Sat, 04 Apr 2026 12:42:13 +0000

Here I am comparing speed of several LLMs running on GPU with 16GB of VRAM, and choosing the best one for self-hosting.

I have run these LLMs on llama.cpp with 19K, 32K, and 64K tokens context windows.

For the broader performance picture (throughput versus latency, VRAM limits, parallel requests, and how benchmarks fit together across hardware and runtimes), see LLM Performance in 2026: Benchmarks, Bottlenecks & Optimization.

The quality of the response is analysed in other articles, for instance:

I did run similar test for LLMs on Ollama: Best LLMs for Ollama on 16GB VRAM GPU.

In this post I am recording my attempts to squeeze as much performance in a sense of speed as possible.

LLM speed comparison table (tokens per second and VRAM)

Model	Size	19K VRAM	19K GPU/CPU	19K T/s	32K VRAM	32K Load	32K T/s	64K VRAM	64K Load	64K: T/s
Qwen3.5-35B-A3B-UD-IQ3_S	13.6	14.3GB	93%/100%	136.4	14.6GB	93%/100%	138.5	14.9GB	88%/115%	136.8
Qwen3.5-27B-UD-IQ3_XXS	11.5	12.9	98/100	45.3	13.7	98/100	45.1	14.7	45/410	22.7
Qwen3.5-122B-A10B-UD-IQ3_XXS	44.7	14.7	30/470	22.3	14.7	30/480	21.8	14.7	28/490	21.5
nvidia Nemotron-Cascade-2-30B IQ4_XS	18.2	14.6	60/305	115.8	14.7	57/311	113.6	14.7	55/324	103.4
gemma-4-26B-A4B-it-UD-IQ4_XS	13.4	14.7	95/100	121.7	14.9	95/115	114.9	14.9	75/190	96.1
gemma-4-31B-it-UD-IQ3_XXS	11.8	14.8	68/287	29.2	14.8	41/480	18.4	14.8	18/634	8.1
GLM-4.7-Flash-IQ4_XS	16.3	15.0	66/240	91.8	14.9	62/262	86.1	14.9	53/313	72.5
GLM-4.7-Flash-REAP-23B IQ4_XS	12.6	13.7	92/100	122.0	14.4	95/102	123.2	14.9	71/196	97.1

19K, 32K, and 64K are the context sizes.

The load above is a GPU Load.
If you see a low number in this column- that means model is running mostly on CPU and can not get any decent speed on this hardware. That pattern matches what people see when too little of the model fits on the GPU or when context pushes work back to the host.

Why context length changes tokens per second

As you move from 19K to 32K or 64K tokens, the KV cache grows and VRAM pressure rises. Some rows show a big drop in tokens per second at 64K while others stay flat, which is the signal to revisit quants, context limits, or layer offload rather than assuming the model is “slow” in general.

The models and quants I've chosen to test - are to run by myself and see do they give a good gain in a sense of cost/benefit on this equipment or not. So no q8 quants here with 200k context :) ...

GPU/CPU is a load, measured by nvitop.

llama.cpp when autoconfiguring the layers unloading to GPU is trying to keep 1GB free.
We manually specify this parameter via commandline param -ngl but I'm not finetuning it here,
just need to understand that if there is significant performance drop when increasing context window size from 32k to 64k - we can try to increase speed on 64k by finetuning number of unloaded layers.

Test hardware and llama.cpp setup

I tested the LLM speed on a PC with this config:

CPU i-14700
RAM 64GB 6000Hz (2x32GB)
GPU RTX-4080
Ubuntu with NVidia drivers
llama.cpp/llama-cli, no unloaded layers specified
Initial VRAM used, before starting llama-cli: 300MB

Extra runs at 128K context (Qwen3.5 27B and 122B)

Model	128K Load	128K: T/s
Qwen3.5-27B-UD-IQ3_XXS	16/625	9.6
Qwen3.5-122B-A10B-UD-IQ3_XXS	27/496	19.2

Takeaways for 16 GB VRAM builds

My current favorite Qwen3.5-27B-UD-IQ3_XXS is looking good on it's sweetspot 50k context (I am getting approx 36t/s)
Qwen3.5-122B-A10B-UD-IQ3_XXS is overtaking performance-wise the Qwen3.5 27B on the contexts above 64K.
I can push Qwen3.5-35B-A3B-UD-IQ3_S to handle context 100k tokens, and it fits into vram, so no performance drop
I will not use gemma-4-31B on 16GB VRAM, but gemma-4-26B might be medium-well..., need to test.
Need to test how well Nemotron cascade 2 and GLM-4.7 Flash REAP 23B work. will they be better then Qwen3.5-35B q3? I doubt but still, might test to confirm the suspicion.

RTX 5090 in Australia March 2026 Pricing Stock Reality

Rost — Wed, 01 Apr 2026 02:51:36 +0000

Australia has RTX 5090 stock.
Barely.
And if you find one, you will pay a premium that feels detached from reality.

For GPUs, CPUs, memory, AI workstations, and wider compute trends, see Compute Hardware in 2026: GPUs, CPUs, Memory & AI Workstations.

RTX 5090 stock availability in Australia

Let us be blunt: the RTX 5090 is not "out of stock everywhere", but it is effectively scarce.

Across major Australian retailers:

Some models are listed as in stock, but only a handful at a time
Most SKUs are sold out or flip between in stock and gone within days
High end variants dominate availability, not entry models

For example:

Scorptec shows multiple RTX 5090 listings, but most are sold out with only occasional stock appearing
Umart and Mwave still carry limited inventory, but only a few models are actually purchasable at any given time

This creates a strange situation:

Technically available
Practically hard to buy at will

Real wait times

If you are not lucky:

Restock cycles feel like 2 to 6 weeks
Popular models disappear within hours or days
Preorders are quietly back without strong visibility

This mirrors global trends where RTX 5090 stock vanished quickly after demand spikes in early 2026

RTX 5090 pricing in Australia right now

Here is where things get painful.

Typical street prices (March 2026)

From real retailer listings:

Entry level AIB cards: ~5999 AUD
Mid range premium cards: 6300 to 6500 AUD
High end OC or liquid cooled: 6500 to 7500 AUD

Even within a single retailer:

Same GPU silicon
Price swings of 1000+ AUD depending on cooling and branding

Compared to official pricing

Official starting price was about 4039 AUD
Real world pricing is now 50 to 80 percent higher

This is not normal inflation. This is structural scarcity plus margin stacking.

Why RTX 5090 is still expensive and scarce

1. AI demand is eating supply

Blackwell is not just a gaming GPU.

AI workloads
Local LLM inference
Enterprise spillover demand

Gamers are competing with developers and businesses.

2. GDDR7 and bleeding edge silicon constraints

5090 depends on:

New memory supply chains
Advanced packaging

These are not scaling fast enough.

3. AIB partner pricing strategy

Board partners learned from previous cycles:

Launch high
Stay high
Discount later (maybe)

Right now, there is zero incentive to reduce prices.

4. Australia tax and logistics penalty

Australia always pays more:

Import costs
Smaller allocation pools
Currency effects

So shortages hit harder locally.

Should you buy RTX 5090 in March 2026

Opinionated answer: only if you absolutely need it.

You are paying:

Early adopter tax
Supply chain tax
Hype tax

The only rational buyers:

AI developers needing local compute
High end creators
People upgrading from very old GPUs (like 20 series or earlier)

Everyone else is subsidising Nvidia margins.

RTX 5080 Super and RTX 6000 series expectations

RTX 5080 Super

Rumors and historical patterns suggest:

Release window: late 2026
Likely improvements:
- Higher clocks
- Faster GDDR7 bins
- Better efficiency

But do not expect miracles. It will be a refinement, not a revolution.

RTX 6000 consumer series

Looking further:

Expected timeframe: 2027
Likely direction:
- Stronger AI acceleration baked in
- Better perf per watt
- Possible shift toward chiplet style GPUs

The bigger question is pricing, not performance.

If Nvidia keeps current strategy:

RTX 6090 could launch even higher than current 5090 street prices

The uncomfortable prognosis

The GPU market has changed.

What used to be:

Launch spike
Then price normalization

Is now:

Launch spike
Then sustained high plateau

Unless one of these happens:

AMD delivers real high end competition
AI demand cools down
Supply massively increases

Do not expect RTX 5090 prices in Australia to drop meaningfully in 2026.

Related GPU and RAM pricing in Australia

Earlier checks on the same cards and retailers, plus paired RAM moves for build totals:

For global DDR5 pressure and how it stacks next to GPU line items, see RAM Price Surge: Up to 619% in 2025.

Bottom line

RTX 5090 is available in Australia, but barely
Prices are consistently 6000 to 7500 AUD
Supply is unstable and unpredictable
Waiting might save money, but not guaranteed

If you are asking whether now is a good time to buy:

It is not.

It is just the least bad time if you need the performance right now.

Remote Ollama access via Tailscale or WireGuard, no public ports

Rost — Tue, 31 Mar 2026 05:06:46 +0000

Ollama is at its happiest when it is treated like a local daemon: the CLI and your apps talk to a loopback HTTP API, and the rest of the network never finds out it exists.

By default, that is exactly what happens: the common local base address is on localhost port 11434.

This article is about the moment you want remote access (laptop, another office machine, maybe a phone), but you do not want to publish an unauthenticated model runner to the whole internet. That intent matters, because the easiest scaling move (open a port, forward it, done) is also the move that creates the mess.

A practical north star is simple: keep the Ollama API private, then make the private network path boring. Tailscale and WireGuard are two common ways to do that, and the rest is making sure the host listens only where it should and the firewall agrees.

Remote device
  |
  | (private VPN path: tailscale or wireguard)
  v
VPN interface on host (tailscale0 or wg0)
  |
  | (local hop)
  v
Ollama server (HTTP API on localhost or VPN IP)

Threat model and who should reach the API

How can Ollama be accessed remotely without exposing it to the public internet? The answer is less about a specific tool and more about being explicit on "who is allowed to connect" and "from where".

A useful mental model is three concentric rings:

Local only: only processes on the box can call the API.
LAN only: devices on the same local network can call the API.
VPN only: selected devices and users on a private overlay network can call the API.

The first ring is the default. Many guides (and tools like Postman) assume the base URL is localhost 11434, which is both convenient and a surprisingly strong safety boundary.

The reason the rings matter is that Ollama is commonly described as having no built-in authentication for its local HTTP API, meaning network exposure and access control become your job if you move beyond localhost.

The other reason is cost and abuse: even a "private" LLM endpoint is still an API endpoint. The OWASP API Security Top 10 calls out categories like security misconfiguration and unrestricted resource consumption; a model runner is practically a poster child for "resource consumption" if exposed casually.

So the basic threat model is not only "an attacker reads my data". It is also "someone can drive my CPU and GPU like a rented car" and "unintended users discover it and start building against it".

OLLAMA_HOST and bind semantics in 90 seconds

What does OLLAMA_HOST do and what is the safest default value? OLLAMA_HOST is the switch that controls where the Ollama server listens. In ollama serve, the environment variable is described as the IP address and port for the server, with a default of 127.0.0.1 and port 11434.

In plain terms, the bind address decides which networks can even attempt a TCP connection:

127.0.0.1 means localhost only.
A LAN IP (like 192.168.x.y) means the LAN can reach it.
0.0.0.0 means all interfaces (LAN, VPN, everything) can reach it unless a firewall blocks them.

That is why most "make it accessible" how-tos suggest switching from 127.0.0.1 to 0.0.0.0, but that advice is incomplete without an interface-aware firewall.

Here is the cheat sheet I keep in my head:

# Local only (baseline)
export OLLAMA_HOST=127.0.0.1:11434

# All interfaces (powerful, easy to regret)
export OLLAMA_HOST=0.0.0.0:11434

# VPN interface only (preferred when the VPN has a stable IP on the host)
export OLLAMA_HOST=100.64.0.10:11434   # example tailscale IP
export OLLAMA_HOST=10.10.10.1:11434    # example wireguard IP

# Different port (useful when 11434 is already taken)
export OLLAMA_HOST=127.0.0.1:11435

The "different port" case is explicitly discussed in the Ollama issue tracker as an example of using OLLAMA_HOST to alter the listen port.

One operational footnote that bites people: if Ollama runs as a managed service, setting environment variables in an interactive shell does not necessarily change the service configuration. This is why many "it worked in my terminal but not after reboot" stories end up in systemd unit overrides or service manager configuration.

Pattern A VPN first with Tailscale

Can Tailscale restrict access to only one service port on a machine? Yes, and that is a big part of why Tailscale is a good fit for "remote access without publishing".

Tailscale gives you a private network (a tailnet) with centrally managed access controls (ACLs). ACLs exist specifically to manage device permissions and secure the network.

No public port means no router choreography

The cleanest pattern is to avoid opening any internet-facing port for Ollama at all and treat the VPN as the only ingress. With Tailscale, devices attempt to connect directly peer-to-peer when possible, and can fall back to relay mechanisms when direct connectivity is not possible.

This is not magic security by itself, but it radically shrinks the blast radius compared to "I forwarded 11434 on my router".

Split horizon and naming with MagicDNS

A second question that shows up in real life is "do I connect via LAN IP when I am at home and via VPN IP when I am away". That is basically a split-horizon problem.

Tailscale MagicDNS helps by giving each device a stable tailnet hostname. Under the hood, MagicDNS generates a FQDN for every device that combines the machine name and your tailnet DNS name, and modern tailnet names end in .ts.net.

The opinionated take is that using a name is usually better than hard-coding an IP, because the name follows the device even if your tailnet IP changes. But it is also fine to be intentionally boring and keep a small hosts file or a single internal DNS record if you prefer. MagicDNS exists so you do not have to.

Direct port versus tailnet-only proxying

There are two common Tailscale ways to reach a service:

Direct port access, where the service listens on the tailnet interface and clients connect to that IP and port.
Tailscale Serve, where Tailscale routes traffic from other tailnet devices to a local service on the host.

Serve is explicitly described as routing traffic from other tailnet devices to a local service running on your device.

For Ollama, Serve can be attractive because it lets you keep Ollama on localhost and expose only a controlled ingress path through Tailscale. It also pairs naturally with HTTPS inside the tailnet if you want browser-friendly endpoints.

A related feature worth naming and then mentally parking is Funnel. Funnel is designed to route traffic from the broader internet to a service on a tailnet device and is explicitly for "anyone to access even if they do not use Tailscale". That is the opposite of this article.

Pattern B WireGuard for those who want the raw primitives

WireGuard is the underlying primitive that powers many VPN products, and it is deliberately minimal: you configure an interface, define peers, and decide what traffic is allowed to flow.

The WireGuard quick start shows the basic shape: create an interface such as wg0, assign IPs, and configure peers with wg.

The key concept for scoping access is AllowedIPs. In the Red Hat documentation, WireGuard reads the destination IP from a packet and compares it to the list of allowed IP addresses; if the peer is not found, WireGuard drops the packet.

For an Ollama host, the practical translation is:

Put the host on a private WireGuard subnet.
Bind Ollama either to localhost and forward to it, or bind directly to the WireGuard IP.
Only peers that have the correct keys and AllowedIPs can route traffic to that private IP.

This is fewer moving parts than a commercial overlay, but it also means you are responsible for key distribution, peer lifecycle, and how remote peers reach your network.

Firewall allow only VPN interface or tailnet

How can a firewall limit Ollama to only VPN interface traffic? The goal is to prevent accidental exposure even if the bind address becomes broader than intended.

The general pattern is:

Allow the Ollama TCP port only on the VPN interface (tailscale0 or wg0).
Deny the same port on everything else.
Prefer "default deny inbound" if you operate that way for the host.

Tailscale has explicit guidance on using UFW to restrict non-Tailscale traffic to a server, which is essentially the "lock down everything except the tailnet" approach.

One nuance that matters for Tailscale specifically is that host firewall expectations may not match reality if you assume UFW will block tailnet traffic. The Tailscale project has discussed that it intentionally installs a rule to allow traffic on tailscale0 and relies on an ACL-controlled filter inside tailscaled.

That is not an argument against a host firewall. It is an argument for being deliberate about which control plane is actually enforcing policy. If you want "only these devices can reach port 11434", Tailscale ACLs are designed for that job.

If you do want interface-level host controls anyway, the examples tend to look like this:

# UFW style logic (illustrative)
ufw allow in on tailscale0 to any port 11434 proto tcp
ufw deny  in to any port 11434 proto tcp

# Or for wireguard
ufw allow in on wg0 to any port 11434 proto tcp
ufw deny  in to any port 11434 proto tcp

Even if you rely primarily on VPN policy, the host firewall still provides a useful "seatbelt" against misbinding to 0.0.0.0 or unexpected service wrappers.

Optional reverse proxy only on VPN ingress

When is a reverse proxy useful for remote Ollama access? A proxy is useful when you want one or more of these properties:

A standard authentication gate (basic auth, OIDC, client certs).
TLS termination with a certificate clients trust.
Request limits and timeouts.
Cleaner URLs for tools that dislike raw ports.

This is where the "do not publish to the internet" intent should still stay true: the reverse proxy is reachable only via the VPN, not on the public WAN interface.

Is TLS needed when traffic already goes through a VPN? Not always for cryptography, but often for ergonomics. Tailscale points out that connections between nodes are already end-to-end encrypted, but browsers are not aware of that because they rely on TLS certificates to establish HTTPS trust.

If you are in the Tailscale world, you can enable HTTPS certificates for your tailnet, which requires MagicDNS and explicitly notes that machine names and the tailnet DNS name will be published on a public ledger (certificate transparency logs).

That public-ledger detail is not a reason to avoid TLS, but it is a reason to name machines like an adult (avoid embedding private project names or customer identifiers in hostnames).

This article intentionally does not include full reverse-proxy configuration (see your A1 article for that). The only idea that matters here is placement:

Ollama listens on localhost or VPN IP.
Reverse proxy listens on the VPN interface only.
Proxy forwards to Ollama.

Security checklist for remote Ollama API access

This is the checklist I use to keep "remote" from silently becoming "public".

Binding and reachability

Confirm the server listens where you think it listens. The documented default is 127.0.0.1 and port 11434, and OLLAMA_HOST changes that.
Treat 0.0.0.0 as a deliberate choice, not a convenience toggle.
Prefer binding to a VPN interface IP when it is stable and fits the topology.

Access control

If using Tailscale, implement ACLs that allow only the specific users or tagged devices to the Ollama port. ACLs exist to manage device permissions.
If using WireGuard, keep AllowedIPs tight and treat keys as the real identity boundary. WireGuard drops packets that do not match a valid peer AllowedIPs mapping.

Firewall

Add a host-level rule that allows the Ollama port only on tailscale0 or wg0 and blocks it everywhere else.
If you expect UFW to block tailnet traffic, verify how Tailscale interacts with your firewall. Tailscale has discussed allowing tailscale0 traffic and relying on ACL filtering inside tailscaled.

TLS and proxying

Use TLS when clients are browsers or when tooling expects HTTPS, even if the VPN already encrypts transport. Tailscale documents this gap between VPN encryption and browser HTTPS trust.
If you enable Tailscale HTTPS certs, remember the certificate transparency implication for hostnames.
If you add a reverse proxy, keep it VPN-only and use it for auth and limits, not for internet exposure.

Avoid accidental public exposure

Be wary of features explicitly designed to publish services to the internet. Tailscale Funnel routes traffic from the broader internet to a tailnet device, which is not the default-safe path for an Ollama API.
If anything ends up internet-reachable, do not leave an anonymous /api surface. At that point, the OWASP API "security misconfiguration" and "resource consumption" risk categories stop being theoretical.

Observability and damage control

Log requests at the ingress layer (VPN policy logs, proxy logs, or both).
Add request and concurrency limits if your proxy supports them, because model inference is a resource event, not a normal API call.

The consistent theme is boring on purpose: keep the Ollama API private by default, add a private path for remote access, then enforce that policy twice (VPN identity plus host firewall) so a single misstep does not turn into a public endpoint.

Structured Logging in Go with slog for Observability and Alerting

Rost — Sat, 28 Mar 2026 10:34:14 +0000

Logs are a debugging interface you can still use when the system is on fire.
The problem is that plain text logs age poorly: as soon as you need filtering,
aggregation, and alerting, you start parsing sentences.

Structured logging is the antidote.
It turns every log line into a small event with stable fields, so tools can search and aggregate reliably.
For how logs connect to metrics, dashboards, and alerting in the wider stack, see the Observability: Monitoring, Metrics, Prometheus & Grafana Guide.

What structured logging is and why it scales

Structured logging is logging where a record is not just a string, but a
message plus typed key value attributes. The idea is boring in the best way:
once logs are machine-readable, an incident stops being a grep contest.

A quick comparison:

Plain text (human-first, tool-hostile)

failed to charge card user=42 amount=19.99 ms=842 err=timeout

Structured (tool-first, still readable)

{"msg":"failed to charge card","user_id":42,"amount":19.99,"duration_ms":842,"error":"timeout"}

In production, it helps to think of logs as an event stream emitted by the
process, while routing and storage live outside the application. That mental
model pushes you toward writing one event per line and keeping events easy to
ship and re-process.

Slog in Go as a shared logging front end

Go has had the classic log package since forever, but modern services need
levels and fields. The log/slog package (Go 1.21 and later) brings structured
logging into the standard library and formalises a common shape for log
records: time, level, message, and attributes.
For a compact language and command refresher alongside this guide, see the Go Cheatsheet.

The key parts of the model are:

Record

A record is what happened. In slog terms, it contains time, level, message,
and a set of attributes. You create records via methods like Info and Error,
or via Log when you want to supply the level explicitly.

Attributes

Attributes are the key value pairs that make logs queryable. If you log the
same concept under three different keys (user, userId, uid), you get three
different datasets. Consistent keys are where the real value hides.

Handler

A handler is how records become bytes. The built-in TextHandler writes
key=value output, while JSONHandler writes line-delimited JSON. Handlers are
also where redaction, key renaming, and output routing tend to happen.

One under-rated feature is that slog can sit in front of existing code. When
you set a default slog logger, top-level slog functions use it, and the classic
log package can be redirected to it too. That makes incremental migration
possible.

Groups

Groups solve the "every subsystem uses id" problem. You can group a set of
attributes for a request (request.method, request.path) or namespace an entire
subsystem with WithGroup so keys do not collide.

A production shaped slog setup

The following setup hits the usual goals.
The examples use a small logx package; for where packages like that usually live in a real module, see Go Project Structure: Practices & Patterns.

one JSON event per line
logs written to stdout for collection
stable service metadata attached once
context-aware logging for request and trace IDs
central redaction for sensitive keys

package logx

import (
    "log/slog"
    "os"
)

var level slog.LevelVar // defaults to INFO

func New() *slog.Logger {
    opts := &slog.HandlerOptions{
        Level:     &level, // can be changed at runtime
        AddSource: true,   // include file and line when available
        ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
            // Centralised redaction: consistent and hard to bypass by accident.
            switch a.Key {
            case "password", "token", "authorization", "api_key":
                return slog.String(a.Key, "[redacted]")
            }
            return a
        },
    }

    h := slog.NewJSONHandler(os.Stdout, opts)

    return slog.New(h).With(
        "service", os.Getenv("SERVICE_NAME"),
        "env", os.Getenv("ENV"),
        "version", os.Getenv("VERSION"),
    )
}

func SetLevel(l slog.Level) { level.Set(l) }

A tiny detail with large consequences: the built-in JSON handler uses standard
keys (time, level, msg, source). When your log backend expects a different
schema, ReplaceAttr is the pressure-release valve that lets you normalise keys
without rewriting call sites.

Schema matters more than the logger

Most "structured logging" failures are schema failures.

Essential fields that keep paying rent

Every log backend will store a timestamp, level, and message. In practice, a
useful application schema often adds a small set of stable fields:

service, env, version
component (or subsystem)
event (a stable name for the thing that happened)
request_id (when a request exists)
trace_id and span_id (when tracing exists)
error (string) and error_kind (stable bucket)

Notice the pattern: these fields answer operational questions, not developer
curiosity.

Semantic conventions are a cheap consistency hack

If you already use OpenTelemetry, its semantic conventions provide a standard
vocabulary for attributes across telemetry signals. Even if you do not export
logs via OpenTelemetry, borrowing attribute names reduces the "what did we call
this field in service B" tax.

High cardinality and why logs get expensive

High cardinality means "too many unique values". It is fine inside a JSON
payload, but it becomes painful when a backend treats some fields as indexed
labels or stream keys. User IDs, IP addresses, random request tokens, and full
URLs tend to explode combinations.

The practical outcome is simple: keep labels and index keys boring (service,
environment, region), and keep high-cardinality fields inside the structured
payload for filtering at query time.

Correlation with request IDs and traces

Correlation is the point where logs stop being just text and start behaving
like telemetry.

Request ID as the lowest-friction correlation key

A request ID is the simplest bridge between an incoming request and everything
that happens because of it. It tends to work even without distributed tracing,
and it is still useful when traces are sampled.

A common pattern is to attach a per-request logger to the context:

package logx

import (
    "context"
    "log/slog"
)

type ctxKey struct{}

func WithLogger(ctx context.Context, l *slog.Logger) context.Context {
    return context.WithValue(ctx, ctxKey{}, l)
}

func FromContext(ctx context.Context) *slog.Logger {
    if l, ok := ctx.Value(ctxKey{}).(*slog.Logger); ok && l != nil {
        return l
    }
    return slog.Default()
}

Trace correlation with W3C Trace Context and OpenTelemetry

W3C Trace Context defines a standard way to propagate trace identity (for HTTP,
via traceparent and tracestate). OpenTelemetry builds on that so trace IDs and
span IDs can be extracted from context.

This middleware example logs both request_id and trace identifiers when
available:

package middleware

import (
    "crypto/rand"
    "encoding/hex"
    "net/http"

    "go.opentelemetry.io/otel/trace"
    "log/slog"

    "example.com/project/logx"
)

func requestID() string {
    var b [16]byte
    _, _ = rand.Read(b[:])
    return hex.EncodeToString(b[:])
}

func WithRequestLogger(base *slog.Logger) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            rid := r.Header.Get("X-Request-Id")
            if rid == "" {
                rid = requestID()
            }

            l := base.With(
                "request_id", rid,
                "method", r.Method,
                "path", r.URL.Path,
            )

            if sc := trace.SpanContextFromContext(r.Context()); sc.IsValid() {
                l = l.With(
                    "trace_id", sc.TraceID().String(),
                    "span_id", sc.SpanID().String(),
                )
            }

            ctx := logx.WithLogger(r.Context(), l)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
}

Once correlation fields exist, the log line becomes an index into other data.
The difference in a live incident is not subtle.

Turning structured logs into monitoring and alerting signals

Logs are great at answering "what happened". Alerting is usually about "how
often and how bad".

A practical approach is to treat certain log events as counters:

event=payment_failed
event=db_timeout
event=cache_miss

Many platforms can derive log-based metrics by counting matching records over a
window. Structured logs make that count resilient, because it is based on a
field value rather than a brittle text match.
When you are ready to visualise and explore those signals, Install and Use Grafana on Ubuntu: Complete Guide walks through a full Grafana setup you can point at common log and metrics backends.

This is also where log levels start to matter. Debug logs are often valuable,
but they are also where cost and noise hides. Using a dynamic level (LevelVar)
lets the system stay quiet by default, while still allowing targeted detail
when needed.

Closing thoughts

Structured logging in Go is no longer a library debate. The interesting part is
whether your log records are consistent, correlatable, and affordable to store.

When your logs carry stable fields like event, request_id, and trace_id, they
stop being "strings someone wrote" and start being a dataset you can operate.

Notes

The Go team introduced log/slog in Go 1.21 and emphasised that structured logs
use key-value pairs so they can be parsed, filtered, searched, and analysed
reliably, and also noted the motivation of providing a common framework shared
across the ecosystem.

The log/slog package documentation defines the record model (time, level,
message, key-value pairs) and the built-in handlers (TextHandler for key=value
and JSONHandler for line-delimited JSON), and documents SetDefault integration
with the classic log package.

For distributed correlation, the W3C Trace Context specification standardises
traceparent and tracestate propagation, and OpenTelemetry specifies that its
SpanContext conforms to W3C Trace Context and exposes TraceId and SpanId,
making log-trace correlation straightforward when a span is present.

For log storage cost and performance, Grafana Loki documentation strongly
recommends bounded, static labels and warns about high cardinality labels
creating too many streams and a huge index, which is directly relevant when
deciding what becomes a label vs what stays as an unindexed JSON field.

Ollama in Docker Compose with GPU and Persistent Model Storage

Rost — Fri, 27 Mar 2026 09:57:49 +0000

Ollama works great on bare metal. It gets even more interesting when you treat it like a service: a stable endpoint, pinned versions, persistent storage, and a GPU that is either available or it is not.

This post focuses on one goal: a reproducible local or single-node Ollama "server" using Docker Compose, with GPU acceleration and persistent model storage.

It intentionally skips generic Docker and Compose basics. When you need a compact list of the commands you reach for most often (images, containers, volumes, docker compose), the Docker Cheatsheet is a good companion.

When you want HTTPS in front of Ollama, correct streaming and WebSocket proxying, and edge controls (auth, timeouts, rate limits), see Ollama behind a reverse proxy with Caddy or Nginx for HTTPS streaming.

For how Ollama fits alongside vLLM, Docker Model Runner, LocalAI, and cloud hosting trade-offs, see
LLM Hosting in 2026: Local, Self-Hosted & Cloud Infrastructure Compared.

When Compose beats a bare metal install

A native install is frictionless for one developer on one machine. The moment you have any of the following, Compose starts to win on ergonomics:

A team setup benefits because the service definition is a file you can review, version, and share. A single-node server benefits because upgrades turn into an image tag bump and a restart, while your model storage stays put (as long as it is on a volume). Ollama also tends to live next to sidecars: a Web UI, a reverse proxy, an auth gateway, a vector DB, or an agent runtime. Compose is good at "one command to start the whole stack", without turning your host into a snowflake.

This approach aligns well with how the official Ollama container is designed: the image runs ollama serve by default, exposes port 11434, and is meant to keep state under a mountable directory.

A Compose skeleton that is actually useful for Ollama

Start with two decisions:

First, how you will pin versions. The Docker Hub image is ollama/ollama, so you can pin a specific tag in .env instead of relying on latest.

Second, where the model data will live. The official docs mount a volume to /root/.ollama so models are not re-downloaded each time the container is replaced.

Here is a Compose file that bakes those decisions in, and keeps the "knobs" close to the service:

services:
  ollama:
    image: ollama/ollama:${OLLAMA_IMAGE_TAG:-latest}
    container_name: ollama
    restart: unless-stopped

    # Keep it local by default, expose it later if you need to.
    ports:
      - "${OLLAMA_BIND_IP:-127.0.0.1}:11434:11434"

    # Persistent models and server state.
    volumes:
      - ollama:/root/.ollama

    environment:
      # The official image already defaults to 0.0.0.0:11434 inside the container,
      # but keeping it explicit helps when you override things later.
      - OLLAMA_HOST=0.0.0.0:11434

      # Service tuning.
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE:-5m}
      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-1}
      - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS:-1}

      # Optional, but relevant when a browser-based UI talks to Ollama directly.
      # See the Networking section for why this exists.
      - OLLAMA_ORIGINS=${OLLAMA_ORIGINS:-}

    # GPU reservation is a separate section below.
    # Add it only on hosts that actually have NVIDIA GPUs.

volumes:
  ollama: {}

A matching .env keeps upgrades boring:

# Pin the image version you have tested.
OLLAMA_IMAGE_TAG=latest

# Local by default. Change to 0.0.0.0 when you intentionally expose it.
OLLAMA_BIND_IP=127.0.0.1

# Keep-alive tweaks cold-start latency vs memory footprint.
OLLAMA_KEEP_ALIVE=5m

# Concurrency knobs.
OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_LOADED_MODELS=1

# Leave empty unless you are serving browser clients that hit Ollama directly.
OLLAMA_ORIGINS=

A small but important nuance: Ollama itself has a default host bind of 127.0.0.1:11434 in the general configuration, but the official container image sets OLLAMA_HOST=0.0.0.0:11434 so the service is reachable through published ports.

If you want a quick sanity check without involving any client SDKs, the Ollama API includes a "list local models" endpoint at GET /api/tags.

Persistent model storage and the least painful way to move it

If you only remember one thing, make it this: the container must have persistent storage, otherwise every rebuild is a re-download.

Ollama lets you choose the models directory using OLLAMA_MODELS. In the reference implementation, the default is $HOME/.ollama/models, and setting OLLAMA_MODELS overrides that.

Inside the official Docker image, $HOME maps naturally to the /root layout used by the documented volume mount (/root/.ollama), which is exactly why the official docker run examples mount that directory.

There are two storage patterns that tend to work well in practice:

A named Docker volume is simplest and portable. It is also easy to accidentally orphan, so it is worth naming it intentionally (for example ollama) and keeping it stable across Compose refactors.

A bind mount to a dedicated disk is better when model sizes start to dominate your root filesystem. In that case, you either mount the whole /root/.ollama to that disk, or you mount a custom directory and point OLLAMA_MODELS at it.

If you are actively reorganising storage, this is where an explicit "move models" playbook helps. See: move-ollama-models .

NVIDIA GPU support with Compose and the NVIDIA Container Toolkit

Ollama can use NVIDIA GPUs in Docker, but the image cannot magic a GPU into existence. The host needs working NVIDIA drivers and the NVIDIA Container Toolkit, and Docker must be configured to use it. The Ollama Docker docs explicitly call out installing nvidia-container-toolkit, configuring the runtime via nvidia-ctk runtime configure --runtime=docker, and restarting Docker.

On the Compose side, the clean, modern way is device reservations. Docker documents GPU access in Compose using deploy.resources.reservations.devices, with capabilities: [gpu], driver: nvidia, and either count (including all) or device_ids.

Add this to the ollama service when you are on an NVIDIA host:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

If you have multiple GPUs and want to keep Ollama on specific devices, switch from count to device_ids as documented by Docker (they are mutually exclusive).

You will sometimes see legacy Compose examples that use runtime: nvidia. That can fail on newer setups with errors like "unknown or invalid runtime name: nvidia", which is a strong hint that you should move to the supported device reservation pattern and make sure the toolkit is configured on the host.

A useful detail hiding in plain sight: the official ollama/ollama image sets NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=compute,utility. These are standard knobs recognised by the NVIDIA container runtime, and they are already present unless you overwrite them.

To confirm whether you are actually getting GPU inference (not just a container that starts), Ollama recommends using ollama ps and checking the "Processor" column, which shows whether the model is on GPU memory.

Platform reality check: Ollama notes that GPU acceleration in Docker is available on Linux (and Windows with WSL2), and not available on Docker Desktop for macOS due to the lack of GPU passthrough.

Networking choices: host vs bridge, ports, and CORS

Networking is where most "it runs but my app cannot connect" bugs come from.

Bridge networking with published ports

The default Compose network is a bridge network. In this setup, publishing 11434:11434 makes Ollama reachable from the host on port 11434, while other containers should talk to it using the service name ollama (not localhost). A lot of people trip on this because localhost inside a container means "this container", not "the Ollama container".

Ollama itself runs an HTTP server on port 11434 (the image exposes it), and the common convention is that clients use http://localhost:11434 on the host when ports are published.

Host networking

network_mode: host can be tempting on a single-node server because it removes port publishing and makes localhost semantics simpler. The trade-off is you lose the isolation and namespacing benefits of a bridge network, and you are more likely to hit port conflicts.

Exposing Ollama intentionally

Ollama on a normal install binds to 127.0.0.1 by default, and the documented way to change the bind address is OLLAMA_HOST.

In Docker, you have two layers:

Ollama bind address, controlled by OLLAMA_HOST (the container image defaults to binding on all interfaces inside the container).

Reachability from outside the container, controlled by Compose ports and the host firewall.

A pattern I like is "bind locally by default" via 127.0.0.1:11434:11434, then switch to 0.0.0.0:11434:11434 only when I have a reason to expose it.

Browser clients and OLLAMA_ORIGINS

If a browser-based UI or extension calls Ollama directly, you are in CORS territory. Ollama allows cross-origin requests from 127.0.0.1 and 0.0.0.0 by default, and you can configure additional origins using OLLAMA_ORIGINS.

This matters even on a single node, because "it works with curl" does not mean "it works from a browser app".

Upgrade and rollback patterns that fit a single-node server

Ollama evolves quickly. Your Compose file can make that a calm process instead of a late-night surprise.

Upgrade by bumping a tag, not by hoping "latest" behaves

The most practical upgrade strategy is to pin the image to a known-good tag in .env, and bump it intentionally. The image is published as ollama/ollama on Docker Hub.

Because model data and server state are stored under a mounted directory (/root/.ollama in the official docs), replacing the container does not imply re-downloading models.

Rollback is just switching the tag back

Rollback is the same mechanism in reverse: set the previous tag, recreate the container, keep the same volume. This is where pinning pays for itself.

Data migration is mostly about storage paths

Most "migrations" in a single-node setup are not about database schemas. They are about disk layout. If you change the models directory (via OLLAMA_MODELS) or move the mounted volume to a new disk, you are doing a data migration whether you call it that or not.

If you want a practical guide for reorganising the model directory on real machines, see: move-ollama-models .

A final note that is easy to miss: Ollama's API documentation explicitly says the API is expected to be stable and backwards compatible, with rare deprecations announced in release notes. That makes "upgrade the server, keep clients working" a reasonable default expectation for a single-node service endpoint.

Common failures: GPU permissions, driver mismatch, and OOM

This section is deliberately symptom-driven. The goal is not "every possible Docker error", only the failures that show up specifically in Ollama + GPU + persistent storage setups.

GPU visible on the host, missing in the container

If the host has a working NVIDIA driver but the container does not see a GPU, the common causes are:

The NVIDIA Container Toolkit is not installed or the Docker runtime is not configured via nvidia-ctk. Ollama's Docker docs call this out directly.

Compose is not reserving a GPU device. The supported way is deploy.resources.reservations.devices with the gpu capability as documented by Docker.

A legacy runtime: nvidia configuration is being used on a daemon that does not recognise it, producing "unknown or invalid runtime name: nvidia".

For validation, ollama ps gives you a pragmatic check: it shows whether a model is loaded in GPU memory.

Permission denied on GPU devices

The "permission denied" flavour of GPU failures typically points to environment constraints rather than Ollama itself. Examples include running rootless Docker, security policies, or device nodes not being exposed as expected. The Docker Compose GPU support docs are explicit that the host must have GPU devices and that the Docker daemon must be set accordingly.

When in doubt, reduce the variables: confirm the toolkit configuration (host), then confirm GPU reservation (Compose), then confirm GPU usage (ollama ps).

Wrong driver, wrong expectation

Ollama in Docker relies on the host driver stack. If the host driver is missing, too old, or misconfigured, you will see failures that look like "Ollama is broken" but are really "CUDA stack is not usable". The official docs place the container toolkit and Docker daemon configuration as prerequisites for NVIDIA GPU usage.

Out of memory: VRAM or RAM disappears fast

OOM is the most predictable failure mode for local inference, and it is usually self-inflicted by configuration.

Ollama supports concurrent processing through multiple loaded models and parallel request handling, but it is constrained by available memory (system RAM on CPU inference, VRAM on GPU inference). When GPU inference is used, new models must fit in VRAM to allow concurrent model loads.

Two configuration details are worth treating as first-class "server settings":

OLLAMA_NUM_PARALLEL increases parallel request processing per model, but required memory scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.

OLLAMA_KEEP_ALIVE controls how long models remain loaded (default is 5 minutes). Keeping models loaded reduces cold-start latency, but it also pins memory.

If you are stabilising a single-node service under load, the non-dramatic fixes usually look like:

Lower parallelism and context defaults before you change anything else.

Limit how many models are allowed to remain loaded concurrently.

Consider memory-reduction features like Flash Attention (OLLAMA_FLASH_ATTENTION=1) and lower precision K/V cache types (OLLAMA_KV_CACHE_TYPE) when your bottleneck is memory, not raw compute.

When it is not Ollama: choosing Docker Model Runner instead

Sometimes the "failure" is really a tooling mismatch. If your organisation already standardises on Docker-native artifacts and workflows, Docker Model Runner (DMR) can be a better fit than running Ollama as a long-lived service container.

Docker positions DMR as a way to manage, run, and serve models directly via Docker, pulling from Docker Hub or other OCI registries, and serving OpenAI-compatible and Ollama-compatible APIs.

It also supports multiple inference engines (including llama.cpp, and vLLM on Linux with NVIDIA GPUs), which can matter if you care about throughput characteristics, not just "run one model locally".

If you want a practical command reference and a deeper comparison angle, see: Docker Model Runner Cheatsheet: Commands & Examples.

Ollama behind a reverse proxy with Caddy or Nginx for HTTPS streaming

Rost — Fri, 27 Mar 2026 09:57:32 +0000

Running Ollama behind a reverse proxy is the simplest way to get HTTPS, optional access control, and predictable streaming behaviour.

This post focuses on Caddy and Nginx ingress for the Ollama API, not on client code.

If you already have Python or Go clients talking to Ollama, this post is the missing piece: ingress and transport for the same API.

For how Ollama fits alongside vLLM, Docker Model Runner, LocalAI, and cloud hosting trade-offs, see
LLM Hosting in 2026: Local, Self-Hosted & Cloud Infrastructure Compared.

For request examples and client code, see
Ollama CLI Cheatsheet.

For UI and multi-user layers, see
Open WebUI overview, quickstart and alternatives.

For the bigger picture on self-hosting and data control, see
LLM self-hosting and AI sovereignty.

For a reproducible single-node Ollama service in Docker Compose (persistent volumes, OLLAMA_HOST, NVIDIA GPUs, upgrades), see
Ollama in Docker Compose with GPU and Persistent Model Storage.

Why you should proxy Ollama instead of exposing port 11434

Ollama is designed to run locally first. Out of the box it binds to localhost on port 11434, which is great for a developer workstation and a not-so-subtle hint that the raw port is not meant to be internet-facing.

I treat port 11434 as an internal, high-cost API. If it is reachable from the public internet, anyone who finds it can burn your CPU or GPU time, fill your disk by pulling models, or just keep connections open until something times out. A reverse proxy does not make Ollama safer by magic, but it gives you a place to put the controls that matter at the edge: TLS, authentication, timeouts, rate limits, and logs.

This matters because the local Ollama API does not ship with a built-in authentication layer. If you expose it, you typically add auth at the edge or keep it private and reachable only over a trusted network.

The second reason is UX. Ollama streams responses by default. If the proxy buffers or compresses in the wrong spot, streaming feels broken and UIs look like they are "thinking" with no output.

Minimal architecture and binding strategy

A clean minimum looks like this:

Client (curl, Python, Go, UI)
        |
        | HTTPS (optional Basic Auth or SSO)
        v
Reverse proxy (Caddy or Nginx)
        |
        | HTTP (private LAN, localhost, or Docker network)
        v
Ollama server (ollama serve on 127.0.0.1:11434)

Two practical rules keep this boring in the best way.

First, keep Ollama private and move exposure to the proxy. If Caddy or Nginx runs on the same host, proxy to 127.0.0.1:11434 and do not change Ollama's bind address. If the proxy runs elsewhere (separate host, separate VM, or a container network), bind Ollama to a private interface, not 0.0.0.0 on the public NIC, and lean on a firewall.

Second, decide early whether browsers will call Ollama directly. If a browser-based tool hits Ollama from a different origin, you may need to deal with CORS. If everything is served from one domain via the proxy (recommended for sanity), you can often avoid CORS entirely and keep Ollama strict.

Reverse proxy configs for streaming and WebSockets

Ollama's API is regular HTTP, and its streaming is newline-delimited JSON (NDJSON). That means you want a proxy that can do three things well:

Do not buffer streaming responses.
Do not kill long-running requests just because the model took a while to speak.
If a UI uses WebSockets (some do), forward Upgrade cleanly.

You can keep this simple. In many cases, "correct WebSockets handling" is just having a config that is Upgrade-safe even if the upstream does not use WebSockets today.

Caddy Caddyfile example

Caddy is the "less config, more defaults" option. If you put a public domain name in the site address, Caddy will typically obtain and renew certificates automatically.

Minimal reverse proxy, HTTPS, and streaming-friendly settings:

# ollama.example.com A/AAAA -> your proxy host
ollama.example.com {

    # Optional Basic Auth at the edge.
    # Generate a password hash with:
    #   caddy hash-password --algorithm bcrypt
    #
    # basic_auth {
    #   alice $2a$12$REDACTED...
    # }

    reverse_proxy 127.0.0.1:11434 {

        # Some setups prefer pinning the upstream Host header.
        # Ollama's own docs show this pattern for Nginx.
        header_up Host localhost:11434

        # For streaming or chat-like workloads, prefer low latency.
        # NDJSON streaming usually flushes immediately anyway, but this makes it explicit.
        flush_interval -1

        transport http {
            # Avoid upstream gzip negotiation if it interferes with streaming.
            compression off

            # Give Ollama time to load a model and produce the first chunk.
            response_header_timeout 10m
            dial_timeout 10s
        }
    }
}

If you already have an SSO gateway (oauth2-proxy, Authelia, authentik outpost, etc), Caddy has an opinionated forward auth directive. The pattern is "auth first, then proxy":

ollama.example.com {
    forward_auth 127.0.0.1:4180 {
        uri /oauth2/auth
        # Copy the identity headers your gateway returns, if you need them.
        copy_headers X-Auth-Request-User X-Auth-Request-Email Authorization
    }

    reverse_proxy 127.0.0.1:11434
}

Nginx server block example

Nginx gives you a bit more rope. The upside is that the knobs are explicit, and it has built-in primitives for rate limiting and connection limiting. The footgun is buffering: Nginx buffers proxied responses by default, which is the opposite of what you want for NDJSON streaming.

This example includes:

HTTP to HTTPS redirect
TLS certificate paths (Certbot style)
WebSocket-safe Upgrade forwarding
Streaming-friendly proxy_buffering off
Longer timeouts than the default 60s

# /etc/nginx/conf.d/ollama.conf

# WebSocket-safe Connection header handling
map $http_upgrade $connection_upgrade {
    default upgrade;
    ""      close;
}

# Optional request rate limiting (IP-based)
# limit_req_zone $binary_remote_addr zone=ollama_rate:10m rate=10r/s;

server {
    listen 80;
    server_name ollama.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name ollama.example.com;

    ssl_certificate     /etc/letsencrypt/live/ollama.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.example.com/privkey.pem;

    # Optional Basic Auth at the edge.
    # auth_basic "Ollama";
    # auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        # Optional rate limit
        # limit_req zone=ollama_rate burst=20 nodelay;

        proxy_pass http://127.0.0.1:11434;

        # Match Ollama docs pattern when proxying to localhost.
        proxy_set_header Host localhost:11434;

        # WebSocket Upgrade handling (harmless if unused).
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Critical for NDJSON streaming.
        proxy_buffering off;

        # Prevent 60s idle timeouts while waiting for tokens.
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
}

If you want an SSO-style gate in Nginx, the equivalent pattern is auth_request. Nginx sends a subrequest to your auth service, and only proxies to Ollama when auth returns 2xx.

TLS automation and renewal gotchas

For TLS, the operational split is simple.

With Caddy, TLS is usually "part of the reverse proxy". Automatic HTTPS is one of its flagship features, so certificate issuance and renewal are coupled to keeping Caddy running, having working DNS, and exposing ports 80 and 443.

With Nginx, TLS is usually "a separate ACME client plus Nginx". The common failure mode is not crypto, it is plumbing:

Port 80 not reachable for HTTP-01 challenges.
Certificates stored in a container but not persisted.
Rate limits when doing repeated fresh installs or test deploys.

A subtle point that matters for long-lived services is that certificate lifetimes are short by design. Treat renewals as a background automation requirement, not an annual calendar event.

Authentication, abuse control, and verification

This is the part that makes an internet-facing LLM endpoint feel professional.

Authentication options, from blunt to elegant

Basic Auth at the proxy is blunt, but surprisingly effective for a private endpoint. It is also easy to apply to both HTTP requests and WebSocket upgrades.

If you want browser-friendly login flows, forward auth and auth_request are the common pattern. Your proxy stays stateless, and an auth gateway owns sessions and MFA. The trade-off is more moving parts.

If you are already running Open WebUI, you can also rely on its app-level authentication and keep Ollama itself private. The proxy then protects Open WebUI, not Ollama directly.

If you do not need public access at all, a network-only approach can be cleaner. For example, Tailscale Serve can expose a local service inside your tailnet without opening inbound ports on your router.

Abuse basics for an expensive API

Ollama is a powerful local API, and its surface goes beyond generation. It has endpoints for chat, embeddings, listing models, and version checks. Treat the whole API as sensitive.

Official API reference (endpoints and streaming): https://docs.ollama.com/api

At the proxy layer, there are three low-effort controls that reduce day-one pain:

Rate limiting per IP on generation endpoints.
Connection limits to stop a small number of clients holding everything open.
Conservative timeouts that match your model and hardware reality, not generic web defaults.

At the Ollama layer, it can also reject overload with 503 and has server-side knobs for queueing. Proxy rate limiting keeps you from getting there as often.

Verification checklist

Use the same checks you would use for any streaming API.

Basic connectivity and TLS
- curl -sS https://ollama.example.com/api/version
- curl -sS https://ollama.example.com/api/tags | head
Streaming works end to end (no buffering)
- curl -N https://ollama.example.com/api/generate -H "Content-Type: application/json" -d '{"model":"mistral","prompt":"Write 10 words only.","stream":true}'

If you are behind Basic Auth:

curl -N -u alice:REDACTED https://ollama.example.com/api/generate -H "Content-Type: application/json" -d '{"model":"mistral","prompt":"Write 10 words only.","stream":true}'

Browser UI sanity
- Load your chat UI and trigger a response.
- If the UI uses WebSockets, confirm you do not see 400 or 426 errors and the connection stays open during generation.

If the curl output only appears at the end, it is almost always buffering at the proxy. Re-check proxy_buffering off in Nginx, and consider forcing low-latency flushing in Caddy for the Ollama site block.

Neo4j graph database for GraphRAG, install, Cypher, vectors, ops

Rost — Tue, 24 Mar 2026 10:59:46 +0000

Neo4j is what you reach for when the relationships are the data. If your domain looks like a whiteboard of circles and arrows, forcing it into tables is painful.

Neo4j models that picture as a property graph and queries it with Cypher.

This guide covers what Neo4j is used for, ACID behaviour, Neo4j vs Amazon Neptune vs TigerGraph (and peers), GraphRAG with vector indexes, local and production install paths, ports and neo4j.conf, and copy-paste Cypher and Python patterns.

For broader context on data infrastructure choices, see the Data Infrastructure for AI Systems pillar.

What is Neo4j used for in production graph workloads

Neo4j is for connected data where you need to ask connected questions, repeatedly, under production constraints. That is the direct answer to what is Neo4j used for in most teams.

Property graph data model with nodes, relationships, and properties

Neo4j uses the property graph model: nodes represent entities, relationships connect nodes, and both can have properties. Labels and relationship types give structure without locking you into a brittle schema.

You can start with a thin model, ship value, and evolve the graph as new questions appear.

Cypher graph query language for pattern matching without join soup

Cypher is declarative and built around pattern matching. You describe subgraph shapes and let the planner execute them.

If SQL is about sets, Cypher is about subgraphs. That matters for multi-hop traversal, path queries, recommendations, provenance, and “who touched what via which system” questions.

Is Neo4j ACID compliant and why you should care

Is Neo4j ACID compliant? Yes. Creating or updating relationships touches coherent structure; the database keeps that consistent under failure and concurrency.

Design graph apps around strong transactional guarantees unless you are forced otherwise. That makes debugging and reasoning about behaviour much easier than assuming vague eventual consistency.

Neo4j vs Amazon Neptune vs TigerGraph: a senior engineer comparison

A “Neo4j vs X” question is usually “Which ecosystem will we live in for years?”

Short, opinionated view—about engineering time, not benchmark slides.

Product	Core model and query style	Where it wins	Where it bites
Neo4j	Property graph and Cypher	Strong ergonomics for connected data, mature tooling, graph plus vector retrieval	Graph modelling is a skill you must invest in
Amazon Neptune	Managed graph on AWS (Gremlin, openCypher, SPARQL for RDF)	AWS-centric contracts and operations	Query language mix can feel platform-driven
TigerGraph	GSQL and OpenCypher-related patterns	Analytics-style workloads and compiled query approaches	Different mental model; not drop-in Cypher everywhere
JanusGraph	Distributed graph with external storage backends	Open source with pluggable backends	You operate the backend stack
ArangoDB	Multi-model (documents, KV, graph)	One database for mixed shapes	Graph depth varies versus graph-first engines
Memgraph	Property graph, Cypher compatible	Streaming and fresh-data workflows	Engine behaviour differs; compatibility is not identity

What to decide before you pick a graph database

Pick query language and operations model first.

If your team wants Cypher and a graph-first workflow, Neo4j is a strong default. If you already have Gremlin expertise, Neptune or JanusGraph can fit. If you want one multi-model store, ArangoDB can reduce moving parts.

Be honest about operations. “We will run a distributed storage backend” is easy to say until you are paged about compactions or JVM pressure at 03:00.

Neo4j for RAG and GraphRAG: vector search plus graph context

Many RAG stacks start as vector search plus prompt. That works until you need provenance, entity resolution, multi-hop context, or disambiguation—then you risk rebuilding a knowledge graph in application code.

How does GraphRAG improve retrieval augmented generation? It uses the graph to pull structured context—entities, relationships, neighbourhoods—that similarity alone often misses, which helps grounding and trustworthiness.

Neo4j vector index for embedding similarity search

Can Neo4j do vector search for RAG? Yes. Neo4j supports vector indexes for similarity over embeddings (commonly HNSW-style approximate nearest neighbour search).

Vectors find “things that look similar”. They do not by themselves encode “how they relate” in your domain. Neo4j lets you combine similarity with traversals.

Using the SEARCH subclause for vector-constrained pattern matching

Neo4j’s SEARCH subclause lets you constrain a Cypher MATCH pattern using approximate nearest neighbour hits from a vector index. That is the ergonomic bridge for hybrid retrieval.

Practical pattern: vector retrieval for candidates, then graph expansion for context, filters, and explanation.

GraphRAG in Python with neo4j-graphrag

Neo4j’s neo4j-graphrag package for Python wires a driver, retriever, and LLM interface into a GraphRAG flow. You can still use external vector stores if you want to split responsibilities.

How to install Neo4j locally and in production

How do you install Neo4j locally? Match the option to your risk profile.

Install Neo4j with Docker for local development

Docker is the fastest path to a repeatable server.

# Minimal run. Data is NOT persisted between restarts.
docker run \
  --restart always \
  --publish=7474:7474 --publish=7687:7687 \
  neo4j:5

For real work, set an initial password and mount a data volume.

docker run \
  --restart always \
  --publish=7474:7474 --publish=7687:7687 \
  --env NEO4J_AUTH=neo4j/your_password \
  --volume=$HOME/neo4j/data:/data \
  neo4j:5

Docker Compose for a team-friendly setup

services:
  neo4j:
    image: neo4j:5
    ports:
      - "7474:7474"
      - "7687:7687"
    environment:
      - NEO4J_AUTH=neo4j/your_password
    volumes:
      - $HOME/neo4j/logs:/logs
      - $HOME/neo4j/config:/config
      - $HOME/neo4j/data:/data
      - $HOME/neo4j/plugins:/plugins
    restart: always

Neo4j Desktop

Neo4j Desktop is strong for prototyping and teaching—projects, GUI, local instances. For CI and integration tests, Docker usually wins.

Linux, Windows, or macOS servers

For long-running hosts, follow official OS install paths. You will eventually care about service management, logs, memory, backups, and upgrades.

Neo4j AuraDB (managed)

If you prefer shipping product to running databases, AuraDB is the managed Neo4j cloud option.

Kubernetes with Helm

If the platform is Kubernetes, use the Helm-based deployment and expose Bolt and HTTP through services. Only deploy databases on K8s if your organisation can run state reliably there.

Neo4j configuration essentials: ports, connectors, and neo4j.conf

Settings live in neo4j.conf (key=value, # comments). Strict validation helps catch typos before you serve traffic.

Default Neo4j ports and connectors

What are the default Neo4j ports? Bolt 7687, HTTP 7474, HTTPS 7473 by default. In production, expose only what you need; often Bolt on a private network and HTTP UI restricted.

Example hardening (adapt IPs and TLS to your environment):

server.bolt.listen_address=10.0.1.10:7687
server.http.listen_address=127.0.0.1:7474
server.https.enabled=true
server.https.listen_address=10.0.1.10:7473

Transaction settings that limit unbounded damage

Useful levers in reviews include db.transaction.timeout for runaway queries and db.transaction.concurrent.maximum to avoid thundering herds.

db.transaction.timeout=10s
db.transaction.concurrent.maximum=1000

Practical Cypher and vector index examples for RAG

Create a vector index and store embeddings

CREATE VECTOR INDEX doc_embeddings
FOR (d:Document) ON (d.embedding)
OPTIONS {indexConfig: {
  `vector.dimensions`: 1536,
  `vector.similarity_function`: "cosine"
}};

Vector retrieval then graph expansion

Vector search for candidate nodes.
Traverse for neighbours, provenance, and constraints.
Format context for the LLM with clear boundaries.

Example using SEARCH inside MATCH (syntax may vary slightly by Neo4j version—check the manual for your server version):

MATCH (d:Document)
  SEARCH d IN (
    VECTOR INDEX doc_embeddings
    FOR $queryEmbedding
    LIMIT 10
  ) SCORE AS score
MATCH (d)-[:MENTIONS]->(e:Entity)
RETURN d.id AS doc_id, score, collect(distinct e.name) AS entities
ORDER BY score DESC
LIMIT 5;

Minimal GraphRAG in Python

from neo4j import GraphDatabase
from neo4j_graphrag.retrievers import VectorRetriever
from neo4j_graphrag.embeddings import OpenAIEmbeddings
from neo4j_graphrag.llm import OpenAILLM
from neo4j_graphrag.generation import GraphRAG

driver = GraphDatabase.driver("neo4j://localhost:7687", auth=("neo4j", "password"))

embedder = OpenAIEmbeddings(model="text-embedding-3-large")
retriever = VectorRetriever(driver, "doc_embeddings", embedder)

llm = OpenAILLM(model_name="gpt-4o", model_params={"temperature": 0})

rag = GraphRAG(retriever=retriever, llm=llm)

response = rag.search(query_text="How do I do similarity search in Neo4j?", retriever_config={"top_k": 5})
print(response.answer)

Real-world Neo4j use cases: fraud, recommendations, and knowledge graphs

Fraud detection and risk graphs

Fraud is rarely one row. It is patterns across accounts, devices, IPs, merchants, identities, and time. Graphs express neighbourhoods and multi-hop paths without ten-way join mazes.

Recommendations with behaviour and explicit relationships

Production recommendations combine scored candidates with inventory, constraints, hierarchies, and explainability. Graphs help you return paths people can reason about.

Knowledge graphs for RAG and agents

RAG needs grounding; agents need memory, provenance, and constraints. A knowledge graph stores entities, relationships, sources, and embeddings in one model—natural fit for GraphRAG.

When should you choose Neo4j over Amazon Neptune or TigerGraph?

When should you choose Neo4j over Amazon Neptune or TigerGraph? Choose Neo4j for a Cypher-first graph and vector + traversal in one product. Choose Neptune when AWS and Gremlin or RDF lines up with your org. Choose TigerGraph when GSQL and analytics-style workloads are the primary bet.

Useful links

Netlify for Hugo & static sites: pricing, free tier, and alternatives

Rost — Tue, 24 Mar 2026 10:59:34 +0000

Netlify is one of the most developer-friendly ways to ship Hugo sites and modern web apps with a production-grade workflow: preview URLs for every pull request, atomic deploys, a global CDN, and optional serverless and edge capabilities.

This guide explains how Netlify works, how credit-based pricing affects real deployments, what you can do on the Free plan, and when an alternative like Vercel or Cloudflare Pages is a better fit.

For a broader view of static site deployment options, see the Web Infrastructure cluster.

What Netlify is used for

Netlify is a deployment platform (often described as WebOps or a modern JAMstack platform) that connects to your repository, runs a build, and publishes the output behind a global CDN. The practical outcome is a workflow where every change can be previewed, and production releases are repeatable, reversible, and fast.

If you run a Hugo-based technical blog, Netlify’s sweet spot is:

Static sites built by Hugo, Astro, Eleventy, and similar generators.
Single-page applications where the build produces static assets.
Sites with lightweight backend needs, implemented via serverless functions (APIs, webhooks, auth glue) or edge logic (routing, geo-based content, experiments).

The core deploy model in one sentence

Netlify deploys are atomic: a new deploy becomes live only after the whole new version is uploaded, so visitors do not see inconsistent intermediate states.

Why developers pick Netlify

Netlify’s popularity is less about “static hosting” and more about the workflow and platform primitives around it.

Deploy Previews for pull requests

Deploy Previews generate a unique preview URL for each pull or merge request in a connected Git repository. Reviewers can validate content, layout, and performance without publishing to production. That is how Deploy Previews work on Netlify in practice—per-PR preview environments with their own URLs and deploy contexts.

Branch deploys for long-lived environments

For stable environments like staging, qa, or release/*, Netlify supports branch deploys. Configure branch deploys for specific branches (or for all new branches) when you want a permanent staging URL independent of PR previews.

Serverless Functions for web apps

Netlify Functions run on-demand code without provisioning servers. A “static site” can still handle webhooks, small API endpoints, scheduled automation, and form-driven notifications. Functions deploy with your site, so previews and rollbacks apply to those endpoints too.

If your “dynamic” work is model inference (tokens, GPUs, long-running jobs) rather than short HTTP handlers, you will usually run a dedicated inference stack outside Netlify Functions.

Edge Functions for low-latency logic

Edge Functions move selected logic to the edge. Typical uses include geolocation-based content, redirects, auth checks, and response modification close to the user—useful for global audiences and first-hit performance.

Built-in forms and basic protections

For many Hugo sites, a contact form is the last reason to keep a separate server. Netlify Forms can handle submissions as part of the deploy pipeline, with spam-protection options.

Deploying a Hugo site on Netlify

What Netlify auto-detects for Hugo

When you link a repo, Netlify can detect Hugo and suggest defaults such as build command hugo and publish directory public.

Pin your Hugo version for repeatable builds

The most common CI failure is Hugo version drift. Pin the version with an environment variable.

A minimal netlify.toml pattern:

[build]
  command = "hugo"
  publish = "public"

[build.environment]
  HUGO_VERSION = "YOUR_HUGO_VERSION"

That pattern is central to the best way to deploy a Hugo site on Netlify—reproducible builds that match local development.

Make Deploy Previews render correctly

Deploy Previews use their own URLs. If your Hugo config relies on absolute URLs (canonical links, sitemap, assets), set the base URL during preview builds. Netlify exposes DEPLOY_PRIME_URL for this:

[context.deploy-preview]
  command = "hugo --gc --minify --buildFuture -b $DEPLOY_PRIME_URL"

[context.branch-deploy]
  command = "hugo --gc --minify -b $DEPLOY_PRIME_URL"

Themes and submodules

If you use a Hugo theme, treat it as a CI dependency—typically a Git submodule so Netlify can fetch it on build.

Netlify pricing and plan model

Separate two ideas:

Plan features (collaboration, security, team workflows).
Measured usage (what you consume while deploying and serving).

Credit-based plans

Many newer accounts use credit-based pricing. Credits cover production deploys, bandwidth, requests, function compute, form usage, and related consumption. Older blog posts that only discuss “build minutes” may be outdated for your account type—check Netlify’s own billing docs for your team.

Plans at a glance

Self-serve tiers are commonly listed as Free, Personal, Pro, and Enterprise, each with a monthly credit allowance (Free has a hard cap; paid plans can add credits).

How credits are consumed

Credits map to real cost drivers—how Netlify pricing works with credits in practice:

production deploys
bandwidth
web requests
serverless function compute
form submissions
optional platform features you enable

Treat credits as a monthly budget, not a single number you ignore until the dashboard complains.

Team seats vs reviewers

Netlify distinguishes people who manage and deploy projects from people who only review. Using reviewer roles for stakeholders can control cost without blocking feedback.

How much you can achieve on the Free plan

The Free plan is real for production, but only if you respect limits.

What you get on Free

Typical Free-tier benefits include custom domains and TLS, unlimited Deploy Previews (previews are the main collaboration win), and access to CDN, functions, and related primitives. The hard constraint is the monthly credit limit.

Quick mental models for planning

Many production deploys to main can burn credits quickly.
Viral traffic or large assets can dominate bandwidth.
Asset-heavy pages can increase request volume.
Serverless APIs add compute—track it if you add backends.

Realistic Free-plan scenarios

A — Hugo blog, few production releases, optimised images, moderate traffic

Usually a good fit. Previews absorb most review load; production deploys stay low.

B — Docs site with constant merges to main

Production deploys can consume the budget. Batching merges, leaning on PR previews, or controlling release timing helps.

C — Static front end plus a small API

Often workable, but watch function compute. Heavy work belongs elsewhere—same story as for GPU-backed inference workloads, where you monitor latency, cost, and production signals on the serving tier, not inside Netlify’s function sandbox.

What happens when you hit the limit

What happens when you run out of Netlify credits? On Free, Netlify aims to avoid surprise charges by enforcing the cap—projects can be paused until the next cycle or until you upgrade or add credits on an eligible plan. Verify the exact behaviour for your account in Netlify’s current billing documentation.

Netlify competitors and alternatives

How does Netlify compare with Vercel and Cloudflare Pages? Roughly:

Vercel — Strong for modern frontend apps and preview-centric workflows; evaluate usage-based scaling for your traffic profile.
Cloudflare Pages — Pairs static hosting with Cloudflare’s edge; often attractive when bandwidth and edge integration matter.
GitHub Pages — Minimal moving parts for simple static sites; stricter limits and fewer platform features.
Azure Static Web Apps — Fits teams already on Azure; path from static hosting to Azure Functions.
AWS Amplify Hosting — Makes sense when you want AWS-native integration and are comfortable with AWS billing models.

For CLI-first AWS workflows, see Deploy Hugo Site to AWS S3 with AWS CLI.

Final recommendations

Pick Netlify when you want Git-centric Deploy Previews, atomic deploys, rollbacks, and optional functions or edge logic—what Netlify is used for in most successful Hugo teams.

Before you rely on Free for production, estimate monthly production deploy count and bandwidth or request volume (especially for large media). If you outgrow the free budget, pricing becomes part of architecture—not an afterthought.

Is Netlify free for commercial use? Yes, within plan limits; high traffic or deploy-heavy workflows usually need a paid tier.

Useful links

Apache Flink on K8s and Kafka: PyFlink, Go, ops, and managed pricing

Rost — Tue, 24 Mar 2026 10:57:30 +0000

Apache Flink is a framework for stateful computations over unbounded and bounded data streams.

Teams adopt it for correct, low-latency streaming with event-time semantics (watermarks), fault tolerance (checkpoints), controlled upgrades (savepoints), and operational surfaces (metrics and REST).

This guide targets DevOps and Go/Python developers. It compares deployment models (self-managed vs managed), explains core architecture, covers Kubernetes (Helm and Operator) and standalone setups, contrasts Flink with Spark, Kafka Streams, Beam, and streaming databases, and shows PyFlink plus Go integration patterns including LLM and AI-oriented pipelines.

For broader context on data infrastructure patterns including object storage, databases, and messaging, see Data Infrastructure for AI Systems: Object Storage, Databases, Search & AI Data Architecture.

What is Apache Flink and why teams use it for real-time processing

Apache Flink is explicitly positioned as a stateful stream processing engine: you model your logic as a pipeline of operators and Flink runs it as a distributed dataflow with managed state and time semantics. In modern Flink documentation, the project describes itself as a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.

From a practical DevOps/software engineering perspective, Flink is a good fit when you need at least one of these properties:

If you need join/aggregate/enrich at low latency with correctness guarantees, you typically use Flink’s event-time processing, where “time” is when the event happened (not when it arrived), and watermarks communicate event-time progress through the pipeline.

If you need stateful computation at scale (rolling counters, sessions, fraud rules, feature engineering), Flink treats state as a first-class part of the programming model and makes it fault tolerant via checkpointing.

If you need operationally robust streaming (failures, rolling upgrades, restarts), Flink checkpoints state and stream positions so the job can recover and continue with the same semantics “as a failure-free execution”.

Typical use cases for DevOps, Go, Python, and AI teams

Flink is widely used for “data pipelines & ETL”, “streaming analytics”, and “event-driven applications” (the categories used by the Flink docs).

For a DevOps + Go/Python stack, typical patterns look like this:

A Go service produces events to Kafka; Flink consumes those events, performs stateful processing (e.g., dedupe, windowed aggregation, enrichment), then writes derived facts back to Kafka or a database. Flink’s operator and checkpointing mechanisms exist to make these stateful pipelines production-safe.

For ML/LLM teams, PyFlink explicitly calls out scenarios like “machine learning prediction” and loading machine learning models inside Python UDFs as a dependency-management motivation, which is a direct endorsement of “Flink job as online inference / feature engineering runtime” patterns.

Apache Flink architecture and core features

Apache Flink cluster architecture for production deployments

Flink’s runtime consists of two process types: JobManager and TaskManagers. The docs emphasise that clients submit the dataflow to the JobManager; the client can then disconnect (detached mode) or stay connected (attached mode).

The JobManager coordinates distributed execution: scheduling, reacting to task completion/failure, coordinating checkpoints, and coordinating recovery. Internally, it includes: ResourceManager (slots/resources), Dispatcher (REST + Web UI + per-job JobMaster creation), and JobMaster (manages one job).

The TaskManagers execute the operators/tasks, and exchange/buffer data streams. The smallest scheduling unit is the task slot; multiple operators can execute in one slot (operator chaining and slot sharing affect this).

Operator chaining and task slots for performance and cost control

Flink chains operator subtasks into tasks, where each task is executed by a single thread. This is described as a performance optimisation that reduces thread handover and buffering overhead, increasing throughput and decreasing latency.

Slots matter operationally because they are the unit of resource scheduling/isolation. Flink notes that each TaskManager may have one or more task slots; slotting reserves managed memory per slot, but does not isolate CPU.

Event-time processing, watermarks, and late data

Flink supports multiple notions of time—event time, ingestion time, processing time—and uses watermarks to model progress in event time.

To work with event time, Flink needs timestamps assigned to events and watermarks generated; the official “Generating Watermarks” documentation explains timestamp assignment and watermark generation as the core building blocks, with WatermarkStrategy being the standard way to configure common strategies.

Fault tolerance: checkpoints versus savepoints in real systems

Checkpointing exists because “every function and operator in Flink can be stateful”; state must be checkpointed to become fault tolerant. Checkpoints enable recovery of both state and stream positions so execution can resume with failure-free semantics.

Flink is very explicit that savepoints are “a consistent image of the execution state of a streaming job, created via Flink’s checkpointing mechanism”, used to stop-and-resume, fork, or update jobs. Savepoints live on stable storage (e.g., HDFS, S3).

The official “Checkpoints vs Savepoints” page frames the difference like backups vs recovery logs: checkpoints are frequent, lightweight, managed by Flink for failure recovery; savepoints are user-managed and used for controlled operations like upgrades.

Apache Flink deployment options and pricing plans

Free/self-managed Apache Flink option

The open-source Flink runtime is “free” in the licensing sense, but in production you pay for infrastructure and operational effort.

Flink is designed to integrate with common resource managers (e.g., YARN and Kubernetes) and can also run as a standalone cluster or as a library.

Self-managed cost drivers for Apache Flink

Compute and memory costs are driven by JobManager and TaskManagers, and by your parallelism/slot layout. Flink’s configuration documentation explicitly calls out jobmanager.memory.process.size, taskmanager.memory.process.size, taskmanager.numberOfTaskSlots, and parallelism.default as core knobs for distributed setups.

Local disk is a frequent hidden cost for stateful jobs. Flink notes that io.tmp.dirs stores local data including RocksDB files, spilled intermediate results, and cached JARs; if this data is deleted, it can force “a heavyweight recovery operation”, so it should live on storage that is not periodically purged.

Durable object/file storage cost is driven by checkpoint/savepoint directories. In Flink 2.x config, checkpoints and savepoints are configured via execution.checkpointing.dir and execution.checkpointing.savepoint-dir and accept URIs like s3://… or hdfs://….

Managed Apache Flink plans and typical billing models

Managed services reduce operational cost but add platform fees and constraints. The specifics are provider-dependent.

Amazon Managed Service for Apache Flink bills by KPUs (1 vCPU + 4 GB memory per KPU) and charges by duration and number of KPUs in one-second increments. AWS also charges an additional “orchestration” KPU per application and separate storage/backups fees.

Confluent Cloud for Apache Flink is usage-based and serverless: you create a compute pool, and you’re billed for CFUs consumed per minute while statements are running. The billing page includes an example CFU price of $0.21 per CFU-hour (region-dependent) and emphasises that you can limit spend via compute pool maximums.

Aiven and Alibaba Cloud are notable managed Flink providers in the market, but their public pricing and billing details vary by plan/region and may require calculators or sales contact; treat exact costs as unspecified unless you quote a region+plan from their current docs.

Ververica offers both self-managed and managed deployment options around Flink; public pages emphasise deployment choices and managed service positioning, while exact pricing is typically handled via “contact/pricing details” flows (so specific numbers are often unspecified publicly).

Deployment options table for Apache Flink in production

Deployment option	Best for	Operational complexity	Key benefits	Key risks / trade-offs
Standalone cluster (VMs/bare metal)	Small teams, fixed capacity	Medium–High	Full control; simplest mental model	HA, autoscaling, upgrades are DIY (more toil)
Kubernetes with Flink Kubernetes Operator	Most modern platform teams	Medium	Declarative deployments; lifecycle management via control loop; operator supports Application/Session/Job deployments	Kubernetes + operator expertise required
Native Kubernetes (without operator)	K8s teams wanting direct integration	Medium–High	Direct resource integration; dynamic TaskManager allocation/deallocation described in Flink-on-K8s docs	More bespoke automation than operator
YARN	Hadoop-centric platforms	Medium	Integrates with YARN resource mgmt	Hadoop stack complexity
AWS Managed Service for Apache Flink	AWS-native data stacks	Low–Medium	Managed orchestration + scaling options; predictable billing unit (KPU)	Platform coupling; extra per-app overhead KPU + storage fees
Confluent Cloud for Apache Flink	Kafka-first shops, SQL-first stream apps	Low	Serverless usage billing; CFU-minute accounting; compute pools to cap spend	CFU costs + Kafka networking costs; service-specific APIs
Ververica managed offerings	Enterprises needing Flink expert ops	Low–Medium	“Flink experts” managed service positioning	Pricing often not transparent (unspecified)

Managed providers and costs table

Prices change by region and time; if you need exact numbers for your region, treat this as a starting point and verify against the provider’s current pricing pages (unquoted regions are unspecified).

Provider	“Plan” shape	Billing unit	Example compute price	Notable additional cost drivers
Amazon Managed Service for Apache Flink	Managed runtime	KPU (1 vCPU + 4 GB)	Example shown: $0.11 per KPU-hour (US East N. Virginia)	+1 orchestration KPU per app; running storage; optional durable backups
Confluent Cloud for Apache Flink	Serverless SQL/processing	CFU-minute/CFU-hour	Example shown: $0.21 per CFU-hour (region varies)	Kafka networking rates still apply; compute pool max to cap spend
Ververica (managed)	Managed “Unified Streaming Data Platform”	Unspecified (public pages)	Unspecified	Platform features/SLAs; pricing typically via sales (unspecified)
Aiven for Apache Flink	Managed service	Hourly usage billing model (platform-wide)	Unspecified without plan/region	Plan tier + cloud region + add-ons (unspecified)
Alibaba Cloud Realtime Compute for Apache Flink	Managed/serverless	Hybrid billing (pay-as-you-go + subscription mix)	Unspecified without region/workspace details	CU-based limits and workspace model (details vary; unspecified here)

Apache Flink vs competitors comparison

Flink sits in a busy ecosystem. The “best” choice depends on latency, statefulness, operational preferences, and authoring model.

Competitor comparison table: Flink vs Spark vs Kafka Streams vs Beam and newer options

Tool	What it is	Streaming execution model	State & exactly-once story	Where it shines	Typical pain points
Apache Flink	Distributed stream processing engine for stateful computations	Continuous streaming + event time via watermarks	Checkpoint-based fault tolerance; savepoints for controlled upgrades	Low-latency stateful pipelines; complex event-time logic	Operating state, checkpoints, upgrades correctly takes discipline
Apache Spark Structured Streaming	Spark’s streaming engine built around DataFrames/Datasets	Default micro-batch model (with a continuous mode discussed separately)	Strong for analytical pipelines; state exists but often higher latency	Unified batch+stream APIs; Spark ecosystem	Micro-batch latency and “streaming as incremental batches” mental model
Kafka Streams	Library to build stream-processing apps on Kafka	Record-at-a-time processing	Supports exactly-once processing semantics (EOS)	Simple Kafka-native apps; embed in JVM service	JVM-only; less flexible for large distributed compute patterns
Apache Beam	Unified programming model + SDKs; executed via runners (Flink, Spark, Dataflow, etc.)	Depends on runner; Beam pipelines translate to runner jobs	Semantics depend on runner capability matrix (runner-specific)	Portability, multi-language pipelines; avoid engine lock-in	Operational tuning still ends up being runner-specific
Materialize	“Live data layer” / streaming SQL DB; incrementally updates results as data arrives	Continuous incremental view maintenance	Strong consistency claims in product docs (details are product-specific)	Serving fresh derived views to apps/AI agents	Different operational model than Flink jobs; not a general operator API runtime
RisingWave	Streaming database where stream processing is expressed as materialized views	Continuous materialised view maintenance	SQL-first; engine-specific semantics	SQL-centric streaming apps without building Flink jobs	Less flexible for arbitrary code-heavy pipelines

A useful heuristic: if you want a runtime for complex stateful streaming jobs with deep control over event-time, operator logic, and deployments, Flink is a primary candidate. If you want SQL-first incremental views for serving, streaming databases may be alternatives. If you want a library embedded in a service, Kafka Streams is competitive. If you want one portable pipeline definition across engines, Beam is compelling.

For cloud-native event-driven architectures using AWS, Building Event-Driven Microservices with AWS Kinesis covers Kinesis Data Streams patterns for real-time processing and service decoupling.

How to use Apache Flink in custom-made systems

This section is intentionally practical: configuration, deployment, and how your Go/Python services typically interact with Flink.

Recommended architecture pattern: Go services + Kafka + Flink + serving layer

Flink is often the “stateful middle” that turns high-volume events into durable signals (counters, sessions, anomalies, enriched records). Checkpoints and state backends are what make that middle reliable in production.

Standalone configuration example for Apache Flink 2.x

Important version note: starting with Flink 2.0, the supported configuration file is conf/config.yaml; the previous flink-conf.yaml is “no longer supported”.

A minimal (illustrative) conf/config.yaml for a small self-managed cluster:

# conf/config.yaml (Flink 2.x style)
rest:
  address: flink-jobmanager.example.internal
  port: 8081

jobmanager:
  rpc:
    address: flink-jobmanager.example.internal
    port: 6123
  memory:
    process:
      size: 2048m

taskmanager:
  memory:
    process:
      size: 4096m
  numberOfTaskSlots: 2

parallelism:
  default: 2

# Checkpointing defaults (jobs can still override in code)
state:
  backend:
    type: rocksdb
execution:
  checkpointing:
    dir: s3://my-bucket/flink/checkpoints
    savepoint-dir: s3://my-bucket/flink/savepoints
    interval: 60 s

# Avoid tmp dirs that get purged (RocksDB files, cached jars, etc.)
io:
  tmp:
    dirs: ["/var/lib/flink/tmp"]

Why these keys: Flink’s configuration reference explicitly documents the rest.* and jobmanager.rpc.* discovery details, the process memory keys, the slot/parallelism keys, and the default checkpoint settings including state.backend.type, execution.checkpointing.dir, execution.checkpointing.savepoint-dir, and execution.checkpointing.interval.

The io.tmp.dirs choice is operationally important because Flink uses it for local RocksDB files and cached artefacts; deleting it can cause heavyweight recovery.

Legacy standalone config example for Flink 1.x

If you are on Flink 1.x (still common in some managed environments), you’ll see flink-conf.yaml in the wild. This is legacy for Flink 2.x users.

# conf/flink-conf.yaml (legacy 1.x style; NOT supported in Flink 2.x)
jobmanager.rpc.address: flink-jobmanager
rest.port: 8081
taskmanager.numberOfTaskSlots: 2
parallelism.default: 2

# Legacy checkpoint keys differ by version; treat as illustrative.
state.backend.type: rocksdb
state.checkpoints.dir: s3://my-bucket/flink/checkpoints
state.savepoints.dir: s3://my-bucket/flink/savepoints

If you’re migrating, Flink provides a migration script (bin/migrate-config-file.sh) to convert flink-conf.yaml to config.yaml.

Kubernetes/Helm deployment with the Flink Kubernetes Operator

The Flink Kubernetes Operator acts as a control plane for Flink application lifecycle management and is installed using Helm.

From the official operator Helm docs, you can install either from the source tree chart, or from the Apache-hosted chart repository:

# install from bundled chart in source tree
helm install flink-kubernetes-operator helm/flink-kubernetes-operator

# install from Apache downloads Helm repository (replace <OPERATOR-VERSION>)
helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-<OPERATOR-VERSION>/
helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator

These exact commands are shown in the operator’s Helm installation documentation.

Example FlinkDeployment CR (illustrative)

This is a simplified example to show the integration points you’ll typically customise (image, resources, checkpoint locations, logging/metrics). The operator reconciles this desired state via its control loop.

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: realtime-sessions
  namespace: flink
spec:
  image: my-registry.example.com/flink/realtime-sessions:2026-03-06
  flinkVersion: v2_2
  serviceAccount: flink
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    state.backend.type: "rocksdb"
    execution.checkpointing.dir: "s3://my-bucket/flink/checkpoints/realtime-sessions"
    execution.checkpointing.savepoint-dir: "s3://my-bucket/flink/savepoints/realtime-sessions"
    execution.checkpointing.interval: "60 s"
    rest.port: "8081"
  jobManager:
    resource:
      cpu: 1
      memory: "2048m"
  taskManager:
    resource:
      cpu: 2
      memory: "4096m"
  job:
    jarURI: local:///opt/flink/usrlib/realtime-sessions.jar
    parallelism: 4
    upgradeMode: savepoint
    state: running

The upgradeMode: savepoint pattern is common when you want safe stateful upgrades; savepoints are designed for stop/resume/fork/update workflows and point to stable storage locations.

PyFlink development: realistic Kafka streaming job with checkpoints and RocksDB state

PyFlink is the Python API for Apache Flink and is explicitly pitched for scalable batch/stream workloads including ML pipelines and ETL.

Dependency packaging for PyFlink Kafka jobs

When you use JVM connectors (Kafka, JDBC, etc.) from PyFlink, you must ensure the relevant JARs are available to the job. Flink’s Python “Dependency Management” docs show three standard mechanisms:

Setting pipeline.jars (Table API), calling add_jars() (DataStream API), or CLI --jarfile at submission time.

PyFlink Kafka job example (DataStream API + event time + state + checkpointing)

This example reads JSON events from Kafka, assigns event-time timestamps (with bounded out-of-orderness), maintains a per-user rolling count in keyed state, and writes an enriched event to an output topic.

Notes:

KafkaSource is built via KafkaSource.builder() and requires bootstrap servers, topics, and a deserialiser.
Exactly-once Kafka sink configuration in PyFlink requires setting delivery guarantee and a transactional ID prefix.
Checkpoint defaults can be configured in Flink config (execution.checkpointing.*) and/or in code; the config keys are documented in the Flink configuration reference.

import json
from typing import Any

from pyflink.common import Duration, Types
from pyflink.common.serialization import SimpleStringSchema
from pyflink.common.watermark_strategy import WatermarkStrategy, TimestampAssigner
from pyflink.common.configuration import Configuration

from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.functions import KeyedProcessFunction, RuntimeContext
from pyflink.datastream.state import ValueStateDescriptor

from pyflink.datastream.connectors import DeliveryGuarantee
from pyflink.datastream.connectors.kafka import (
    KafkaSource,
    KafkaOffsetsInitializer,
    KafkaSink,
    KafkaRecordSerializationSchema,
)

class EventTimeFromJson(TimestampAssigner):
    """
    Extract event_time_ms from the JSON payload.
    Expect: {"user_id":"u1","event_time_ms":1710000000000,"event":"click",...}
    """
    def extract_timestamp(self, value: str, record_timestamp: int) -> int:
        try:
            obj = json.loads(value)
            return int(obj["event_time_ms"])
        except Exception:
            # fallback: use record timestamp (ingestion) if malformed
            return record_timestamp

class RollingCount(KeyedProcessFunction):
    def open(self, runtime_context: RuntimeContext):
        desc = ValueStateDescriptor("rolling_count", Types.LONG())
        self.count_state = runtime_context.get_state(desc)

    def process_element(self, value: str, ctx: 'KeyedProcessFunction.Context'):
        obj = json.loads(value)
        current = self.count_state.value()
        if current is None:
            current = 0
        current += 1
        self.count_state.update(current)

        # emit enriched event
        obj["rolling_count"] = current
        obj["event_time_ms"] = int(obj.get("event_time_ms", 0))
        yield json.dumps(obj)

def build_env() -> StreamExecutionEnvironment:
    # Cluster/job defaults (can also be set in config.yaml)
    cfg = Configuration()
    cfg.set_string("state.backend.type", "rocksdb")
    cfg.set_string("execution.checkpointing.dir", "s3://my-bucket/flink/checkpoints/realtime-sessions")
    cfg.set_string("execution.checkpointing.interval", "60 s")
    env = StreamExecutionEnvironment.get_execution_environment(cfg)

    # In PyFlink, connector jars must be available; use env.add_jars(...) if needed.
    # env.add_jars("file:///opt/flink/lib/flink-connector-kafka-<VERSION>.jar")

    # Enable checkpointing explicitly as well (jobs can override defaults)
    env.enable_checkpointing(60_000)
    env.set_parallelism(4)
    return env

def main():
    env = build_env()

    source = (
        KafkaSource.builder()
        .set_bootstrap_servers("kafka:9092")
        .set_topics("events.raw")
        .set_group_id("realtime-sessions-v1")
        .set_value_only_deserializer(SimpleStringSchema())
        .set_starting_offsets(KafkaOffsetsInitializer.earliest())
        .build()
    )

    watermark_strategy = (
        WatermarkStrategy.for_bounded_out_of_orderness(Duration.of_seconds(10))
        .with_timestamp_assigner(EventTimeFromJson())
    )

    stream = (
        env.from_source(source, watermark_strategy=watermark_strategy, source_name="kafka-events-raw")
        .key_by(lambda s: json.loads(s)["user_id"])
        .process(RollingCount(), output_type=Types.STRING())
    )

    record_serializer = (
        KafkaRecordSerializationSchema.builder()
        .set_topic("events.enriched")
        .set_value_serialization_schema(SimpleStringSchema())
        .build()
    )

    sink = (
        KafkaSink.builder()
        .set_bootstrap_servers("kafka:9092")
        .set_record_serializer(record_serializer)
        .set_delivery_guarantee(DeliveryGuarantee.EXACTLY_ONCE)
        .set_transactional_id_prefix("realtime-sessions-txn")
        .build()
    )

    stream.sink_to(sink)

    env.execute("realtime-sessions-pyflink")

if __name__ == "__main__":
    main()

The API calls above line up with PyFlink’s KafkaSource builder usage pattern and required fields.

For delivery guarantees, PyFlink’s KafkaSinkBuilder documentation explicitly says that for DeliveryGuarantee.EXACTLY_ONCE you must set the transactional ID prefix.

For timestamping/watermarking, Flink’s watermark documentation explains timestamp assignment and watermark generation as the mechanism to process event time, and PyFlink provides a WatermarkStrategy API mirroring this model.

Go integration: Kafka producer/consumer + Flink REST job submission

Go does not have a native Flink job authoring API like Java/Python, so Go systems typically integrate with Flink through:

Kafka (or other brokers) as ingestion/egress.
The Flink REST API for operational actions (uploading JARs, starting jobs, querying job status, triggering savepoints, rescaling).

For Kafka setup and local development patterns, see Apache Kafka Quickstart - Install Kafka 4.2 with CLI and Local Examples.

Go Kafka producer/consumer example (kafka-go)

package main

import (
    "context"
    "log"
    "time"

    "github.com/segmentio/kafka-go"
)

func main() {
    ctx := context.Background()

    // Producer: write raw events
    writer := &kafka.Writer{
        Addr:         kafka.TCP("kafka:9092"),
        Topic:        "events.raw",
        RequiredAcks: kafka.RequireAll,
    }
    defer writer.Close()

    err := writer.WriteMessages(ctx, kafka.Message{
        Key:   []byte("user:u1"),
        Value: []byte(`{"user_id":"u1","event_time_ms":1710000000000,"event":"click"}`),
        Time:  time.Now(),
    })
    if err != nil {
        log.Fatal(err)
    }

    // Consumer: read enriched events
    reader := kafka.NewReader(kafka.ReaderConfig{
        Brokers:  []string{"kafka:9092"},
        Topic:    "events.enriched",
        GroupID:  "go-debug-consumer",
        MinBytes: 1e3,
        MaxBytes: 10e6,
    })
    defer reader.Close()

    msg, err := reader.ReadMessage(ctx)
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("enriched key=%s value=%s\n", string(msg.Key), string(msg.Value))
}

This is “plumbing” code, but it’s the most common practical integration surface: Kafka topics are the boundary between Flink and custom services.

Flink REST API: upload and run jobs from Go

Flink’s REST API is part of the JobManager web server and listens on port 8081 by default (configurable via rest.port).

The official OpenAPI spec for the dispatcher includes /jars/upload and explicitly states:

JAR upload must be sent as multi-part data
ensure the Content-Type header is set to application/x-java-archive
provides a curl example using -F jarfile=@path/to/flink-job.jar

A practical Go snippet to upload a JAR:

package flink

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "mime/multipart"
    "net/http"
    "os"
)

func UploadJar(ctx context.Context, flinkBaseURL, jarPath string) (*http.Response, error) {
    f, err := os.Open(jarPath)
    if err != nil {
        return nil, err
    }
    defer f.Close()

    var body bytes.Buffer
    w := multipart.NewWriter(&body)

    part, err := w.CreateFormFile("jarfile", "job.jar")
    if err != nil {
        return nil, err
    }
    if _, err := io.Copy(part, f); err != nil {
        return nil, err
    }
    if err := w.Close(); err != nil {
        return nil, err
    }

    req, err := http.NewRequestWithContext(ctx, http.MethodPost, flinkBaseURL+"/jars/upload", &body)
    if err != nil {
        return nil, err
    }

    // Important: multipart boundary
    req.Header.Set("Content-Type", w.FormDataContentType())

    // Some clients also set "Expect:" similarly to the curl example in the spec.
    req.Header.Set("Expect", "")

    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    if resp.StatusCode >= 300 {
        return resp, fmt.Errorf("upload failed: %s", resp.Status)
    }
    return resp, nil
}

This code is guided by the REST API OpenAPI description for /jars/upload including its multipart requirement and curl reference.

To run a previously uploaded JAR, Flink exposes /jars/{jarid}/run and supports passing program args via query parameters (and/or JSON body).

Operationally valuable endpoints you’ll likely automate:

/jobs and /jobs/{jobid} to list and inspect job state
/jobs/{jobid}/savepoints to trigger savepoints (async trigger + polling)
/jobs/{jobid}/rescaling to trigger rescaling

Code snippets comparison table: PyFlink vs Go in a Flink-based platform

Concern	PyFlink (Python jobs)	Go (services around Flink)
Authoring Flink logic	Native authoring via DataStream/Table APIs; supports state + timers	No native Flink API; implement logic in Flink (Java/Python) and integrate externally
Connectors/dependencies	Must ship connector JARs via `pipeline.jars`, `add_jars`, or `--jarfile`	Not applicable (you’re not running inside Flink), but you manage Kafka/DB clients
Ingestion/egress	KafkaSource/KafkaSink builders in PyFlink	Kafka producer/consumer libraries; standard microservice patterns
Ops automation	Can call Flink REST endpoints too	Often owns automation: upload JAR, deploy, rescale, trigger savepoint via REST

DevOps guide: monitoring, scaling, backups, and CI/CD for Apache Flink

Monitoring Apache Flink in Kubernetes and on VMs

Flink supports exporting metrics by configuring metric reporters in the Flink configuration file; these reporters are instantiated on JobManager and TaskManagers.

For Prometheus, Flink exposes Prometheus-format metrics when configured with metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory in a supported Flink version environment.

You generally combine that with Kubernetes ServiceMonitors (Prometheus Operator) or with your managed monitoring stack.

Scaling: parallelism, slots, and operator-based autoscaling

Flink’s scheduling model defines execution resources via task slots, and each slot can run a pipeline of parallel tasks.

For manual scaling, the REST API provides a rescaling endpoint for a job (/jobs/{jobid}/rescaling) as an async operation.

If you’re on Kubernetes with the Flink Kubernetes Operator, the operator project advertises a “Flink Job Autoscaler” as part of its feature set, which is worth evaluating if your workloads vary substantially.

Backups and safe upgrades: checkpoints and savepoints

Checkpoints are for automated recovery and are managed by Flink; savepoints are for user-driven lifecycle operations (stop/resume/fork/upgrade).

From an SRE standpoint:

Use checkpoints for “keep the pipeline running through failures”.
Use savepoints for “deploy a new version without losing state”.

Flink’s REST API also supports triggering savepoints asynchronously, which is useful for GitOps-style “deploy → trigger savepoint → upgrade” workflows.

CI/CD: GitOps + Helm + REST job submission

For Kubernetes:

Keep the operator installation and your FlinkDeployment CRs in Git, deploy via Argo CD/Flux, and version container images per build. The operator Helm docs explicitly discuss “Working with Argo CD”.

For standalone/session clusters:

Use the Flink REST API JAR upload and run endpoints for immutable artefact deployments.

Also note a subtle but valuable security/ops toggle: web.submit.enable governs uploads via the Web UI, but the docs note that even when disabled, session clusters still accept job submissions through REST requests; this is relevant when hardening UI surfaces while retaining CI/CD automation.

LLM/AI integration patterns with Apache Flink for real-time pipelines

LLM systems are often only as good as their real-time context. Flink fits into LLM/AI stacks as the component that produces “always fresh” features, embeddings, and behavioural aggregates.

Real-time embeddings pipeline with Flink

A common pattern is:

ingest user actions/events,
aggregate sessions and preferences,
produce embedding-generation tasks,
write embeddings to a vector store and/or feature store.

PyFlink’s dependency management documentation explicitly calls out “machine learning prediction” and loading ML models inside Python UDFs (for remote cluster execution), which maps directly to “online inference inside Flink operators” approaches.

Online feature store updates for recommendation and ranking

Flink’s keyed state and checkpointing model is built to maintain operator state across events and recover it reliably. That’s a natural match for continuous feature computation (rolling rates, counts, time-decayed metrics) that downstream recommenders need.

Practical latency/consistency trade-offs for AI pipelines

If your architecture requires exactly-once semantics end-to-end (e.g., avoid duplicate feature updates or duplicate billing events), you’ll structure sinks and sources around checkpointing and transactional guarantees.

In Kafka-based stacks specifically:

Flink’s Kafka connector can deliver exactly-once guarantees when checkpointing is enabled and delivery guarantee options are configured.
Kafka Streams also supports exactly-once semantics (EOS), which is relevant if your “AI feature pipeline” is small enough to live inside application code rather than a Flink cluster.

Architecture view for “Flink as the real-time AI context builder”

This diagram is grounded in Flink’s core primitives: event-time processing (watermarks), state backends (state.backend.type and system-managed local state), and checkpoint/savepoint mechanisms for fault tolerance and operations.

IndexNow explained - notify search engines when you publish

Rost — Mon, 23 Mar 2026 08:51:42 +0000

Static sites and blogs change whenever you deploy. Search engines that support IndexNow can learn about those changes without waiting for the next blind crawl.

This page covers why that matters, what the protocol does, and how to wire it into a real workflow, including patterns you can reuse in your own automation or a small Go CLI.

Why use IndexNow on a static or Hugo site

If you deploy Hugo to S3 or similar you already ship HTML and a sitemap.xml. Crawlers will eventually read the sitemap but timing is not under your control. After a migration or a batch of new posts you care about fresh indexing more than "sometime next week."

IndexNow is a push channel. You POST a list of canonical URLs you care about. Participating engines (including Microsoft Bing and others listed on indexnow.org) can prioritize fetching those URLs. It does not replace good URLs redirects or internal linking but it closes the loop between git push and search engine awareness.

What IndexNow does

At a high level each submission is an HTTPS POST with JSON like this conceptually:

host - your site hostname (for example www.example.com)
key - your pre-generated secret string
keyLocation (optional) - full URL of the verification file if it is not at the default path
urlList - one or more absolute URLs on that host you want to signal

Engines reject bad keys wrong hosts or malformed payloads. Success is usually HTTP 200 or 202 depending on the endpoint.

You can read the full rules and partner list on the official site. The important mental model is domain ownership proof via a text file plus explicit URL list not keywords or page content.

How to prepare your site

Key file and hostname

Pick a key - a long random string (treat it like a secret).
Publish https://your-domain/<key>.txt with only the key as the body (one line).
Use the same key in your CLI or automation when you POST.
Submit only URLs on that host that you want recrawled (new posts updated pages or redirects targets).

After you move many URLs at once you may want to batch-notify paths. IndexNow accepts multiple URLs in one request subject to each engine’s limits.

Ways to submit URLs

Manual POST - fine for debugging use curl with JSON.
Plugins - some CMS and hosting panels include IndexNow toggles.
Your deploy script - after hugo and upload call a small binary with the list of changed URLs or your sitemap URL.

For a Hugo workflow the natural triggers are "after build" or "after sync to bucket." Pass full HTTPS URLs that match your live site including www vs apex if that is what you serve.

A small Go CLI (optional)

Features you might implement

A minimal Go command-line tool fits IndexNow well because the payload is a small JSON POST and you can wire it into deploy scripts. A typical design includes:

Single or multiple URLs as positional arguments
--sitemap to fetch a sitemap.xml and submit every <loc> (with optional --limit)
Several engines in parallel via --engines (for example indexnow for the global aggregator, or per-provider endpoints)
Flags or environment variables such as INDEXNOW_KEY, INDEXNOW_WEBSITE_URL, and INDEXNOW_ENGINES
Verbose output with -v for debugging 403 or 422 responses

Build with go build or go install, install the binary on your PATH, then call it after publish:

indexnow --key YOUR_KEY --website https://www.example.com https://www.example.com/new-post/

For a full site refresh after deploy you can pass --sitemap with your public sitemap URL. Document response codes and engine lists in your own README, and keep a publish-then-index shell snippet next to whatever triggers your static deploy.

The Best LLMs for OpenCode - tested locally post used "implement an IndexNow notifier in Go" as a coding benchmark - useful if you want to see how different models handle the same spec and structured tasks.

Practical tips

Prefer the indexnow engine target when you want one submission to fan out through the shared infrastructure (see searchengines.json and mirror that list in your own client if you support multiple endpoints).
429 means slow down. 403 usually means key or host mismatch. Fix the key file location or hostname first.
IndexNow does not replace 301 redirects when you rename paths. Notify after redirects are live.

Hosted email for custom domains compared - Workspace, Microsoft 365, Zoho, Proton, WorkMail

Rost — Mon, 23 Mar 2026 08:51:40 +0000

Putting email on your own domain sounds like a weekend DNS task. In practice it is a small distributed system with a twenty-year legacy.

MX routes inbound mail. SPF lists who may send for your domain. DKIM signs messages. DMARC ties the story together and tells receivers what to do when something does not line up. Skip or miswire any layer and you get silent failures, spam-folder burial, or both.

This article compares the five hosted options I see engineers actually choose - Google Workspace, Microsoft 365, Zoho Mail, Proton Mail, and AWS WorkMail - with rough USD prices, setup friction, and honest when to pick what advice.

Why providers beat rolling your own

You can run Postfix or Exim on a VPS. You should not unless you sell email infrastructure. IP reputation, PTR, bounce handling, backscatter, and greylisting eat weekends. Managed hosts amortize that pain across millions of mailboxes. For a personal domain or a small team, paying a few dollars per seat is cheaper than your hourly rate debugging why Gmail drops your replies.

Google Workspace

Price - roughly 6-12 USD per user per month depending on tier and region.

What you get - Gmail UI, strong spam filtering, predictable deliverability, shared calendar and drive if you use them, and admin tooling that "just works" for most SMB setups.

Verdict - The default when you want reliability over novelty. Setup is boring in the good way. If your only goal is "you@yourdomain.com works forever," start here.

Microsoft 365

Price - roughly 8-22 USD per user per month depending on bundle (often bundled with Office apps).

What you get - Outlook, deep enterprise features, Entra ID (formerly Azure AD) integration, and a natural fit if your company already lives in Excel, Teams, and SharePoint.

Verdict - Match made in heaven for Microsoft-heavy shops. Slightly heavier admin surface than Google for tiny teams, but reliability and deliverability are solid once DNS is correct.

Zoho Mail

Price - free tiers exist for tiny teams; paid plans often land around 1-5 USD per user per month.

What you get - Straightforward hosted mail, control panels that get the job done, fewer bells than the giants.

Verdict - Best when budget matters more than polish. Fine for side projects and low-stakes mail. Expect more rough edges in UX and occasionally messier deliverability stories than Workspace or Microsoft.

Proton Mail

Price - roughly 5-10 USD per user per month for paid plans with custom domains.

What you get - Privacy-first design, encryption oriented workflows, Swiss jurisdiction story that matters to some buyers.

Verdict - Choose Proton when privacy is a requirement, not a vibe. You may accept more integration friction (calendar, third-party clients, automation) than with mainstream suites.

AWS WorkMail

Price - list pricing is modest (on the order of 4 USD per user per month) before add-ons and data transfer.

What you get - Mailboxes in AWS, IAM-flavored admin, and hooks into the broader AWS universe.

Verdict - Paper looks cheap. Reality includes SES identities, receipt rules, routing surprises, and debugging sessions that belong in a ticket queue. Pick WorkMail only when AWS-native email is a product requirement. Otherwise you trade dollars for engineering hours you will not get back.

At a glance

Provider	Rough $/user/mo	Setup	Best for
Google Workspace	6-12	Easiest	Default choice, minimum drama
Microsoft 365	8-22	Easy-medium	Microsoft-centric orgs
Zoho Mail	1-5 or free tier	Medium	Tight budgets, side projects
Proton Mail	5-10	Medium	Privacy requirements
AWS WorkMail	~4 + AWS overhead	Hard	Deep AWS integration (rare cases)

Prices move with region, currency, and promotions - treat the table as order-of-magnitude, not a quote.

Choosing today

Want it to work and move on → Google Workspace.
Already on Office and Entra → Microsoft 365.
Money is tight and mail is not mission-critical → Zoho (eyes open on deliverability).
Threat model or policy demands privacy → Proton.
Need mail inside AWS for architectural reasons you can defend in a design review → WorkMail; otherwise skip.

Closing

Email rewards boring choices. The clever move is rarely "my own Postfix box at a discount VPS." It is picking a reputable host, nailing DNS (see the web infrastructure cluster for more on DNS), and spending your creativity on software that is not SMTP.

SGLang QuickStart: Install, Configure, and Serve LLMs via OpenAI API

Rost — Sun, 22 Mar 2026 12:19:25 +0000

SGLang is a high-performance serving framework for large language models and multimodal models, built to deliver low-latency and high-throughput inference across everything from a single GPU to distributed clusters.

For a broader comparison of self-hosted and cloud LLM hosting options — including Ollama, vLLM, llama-swap, LocalAI, and managed cloud providers — see the LLM hosting guide for 2026.

If you already have apps wired to the OpenAI API shape, SGLang is especially appealing because it can expose OpenAI-compatible endpoints for chat completions and completions, helping you migrate from hosted APIs to self-hosted models with minimal client-side changes. When you need to route requests across multiple backends (llama.cpp, vLLM, SGLang, etc.) with hot-swap and TTL-based unloading, llama-swap provides a transparent proxy layer that keeps a single /v1 URL stable while swapping upstreams on demand.

This QuickStart walks through installation (multiple methods), practical configuration patterns, and a clean "install → serve → verify → integrate → tune" workflow, with working examples for both HTTP serving and offline batch inference.

If you need multimodal support (text, embeddings, images, audio) with a built-in Web UI and maximum OpenAI API drop-in compatibility, LocalAI offers a broader feature set with more model format support.

What is SGLang for high-throughput LLM and multimodal model serving

At its core, SGLang is designed for efficient inference and scalable serving. The “fast runtime” stack includes RadixAttention for prefix caching, a zero-overhead CPU scheduler, speculative decoding, continuous batching, paged attention, multiple parallelism strategies (tensor, pipeline, expert, data parallelism), structured outputs, chunked prefill, and multiple quantisation options (for example FP4, FP8, INT4, AWQ, GPTQ).

It targets broad-platform deployment: NVIDIA GPUs, AMD GPUs, Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more.

PyPI requires Python >= 3.10. As of 20 March 2026, the published line included 0.5.9 (released 23 February 2026)—pin or check current versions when installing.

How to install SGLang on Linux GPU hosts with uv pip, source builds, or Docker

Install options include uv or pip, source builds, Docker images, Kubernetes manifests, Docker Compose, SkyPilot, and AWS SageMaker. Most walkthroughs assume common NVIDIA GPU setups; other accelerators have their own setup notes elsewhere.

Install SGLang quickly with uv or pip on Python 3.10+

For a straightforward local install, uv is usually the fastest path:

pip install --upgrade pip
pip install uv
uv pip install sglang

CUDA 13 notes

For CUDA 13, Docker avoids host-side PyTorch/CUDA mismatches. Without Docker: install a CUDA 13 PyTorch build, then sglang, then the matching sglang-kernel wheel from the published wheel releases (version must match the stack).

# 1) Install PyTorch with CUDA 13 support (replace X.Y.Z as needed)
uv pip install torch==X.Y.Z torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

# 2) Install SGLang
uv pip install sglang

# 3) Install the matching CUDA 13 sglang-kernel wheel (replace X.Y.Z)
uv pip install "https://github.com/sgl-project/whl/releases/download/vX.Y.Z/sglang_kernel-X.Y.Z+cu130-cp310-abi3-manylinux2014_x86_64.whl"

Install and run SGLang with Docker Hub images

For containerised deployments—or to sidestep host CUDA/PyTorch pairing—use the published Docker Hub images. A typical docker run mounts the Hugging Face cache and passes HF_TOKEN when pulling gated models.

docker run --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=<secret>" \
  --ipc=host \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 30000

For production-style images, latest-runtime drops build tools and dev dependencies, so the image stays much smaller than the default latest variant.

docker run --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=<secret>" \
  --ipc=host \
  lmsysorg/sglang:latest-runtime \
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 30000

Install from source and other deployment methods

To develop against SGLang or carry local patches, clone a release branch and install the Python package in editable mode:

git clone -b v0.5.9 https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python"

For orchestration, the repo includes Kubernetes manifests (single- and multi-node) and a minimal Docker Compose layout—reasonable starting points before custom wiring.

How to configure SGLang server arguments with YAML config files and environment variables

SGLang configuration is driven by server arguments and environment variables. Flags cover model selection, parallelism, memory, and optimisation knobs; the full set is listed with python3 -m sglang.launch_server --help.

Environment variables use two prefixes: SGL_ and SGLANG_ (many flags accept either CLI or env form—launch_server --help shows the mapping).

Some commonly relevant env vars include host and port controls such as SGLANG_HOST_IP and SGLANG_PORT.

Use a YAML config file for reproducible SGLang server launches

For repeatable deployments and shorter command lines, pass a YAML file with --config. CLI arguments override values from the file when both set the same option.

# Create config.yaml
cat > config.yaml << 'EOF'
model-path: meta-llama/Meta-Llama-3-8B-Instruct
host: 0.0.0.0
port: 30000
tensor-parallel-size: 2
enable-metrics: true
log-requests: true
EOF

# Launch server with config file
python -m sglang.launch_server --config config.yaml

A few configuration and tuning essentials to keep in mind:

SGLang's --model-path can point to a local folder or a Hugging Face repo ID, which makes it easy to switch between local weights and Hub-hosted models without changing your serving code.

For multi-GPU, enable tensor parallelism with --tp. If startup fails with “peer access is not supported between these two devices”, add --enable-p2p-check.

If serving hits OOM, reduce KV cache pressure with a smaller --mem-fraction-static (default is 0.9).

If long prompts OOM during prefill, lower --chunked-prefill-size.

How to run an OpenAI-compatible SGLang server and call it from the OpenAI Python client

A practical "happy path" workflow looks like this:

Install SGLang (uv/pip or Docker).
Start the server with your chosen model and port.
Verify basic serving via OpenAI-compatible endpoints.
Integrate your application by pointing the OpenAI SDK base_url at the local server.
Tune throughput and memory with server args once you have real traffic.

Send a local chat completion request to SGLang using OpenAI SDK

For OpenAI-compatible usage, two details matter:

The server implements the OpenAI HTTP surface and, when the tokenizer provides one, applies the Hugging Face chat template automatically. Override with --chat-template at launch if needed.

Point an OpenAI client at the server’s /v1 prefix (base_url → http://<host>:<port>/v1), then call client.chat.completions.create(...) as usual.

Start the server with either entrypoint: python -m sglang.launch_server still works, but sglang serve is the preferred CLI.

# Recommended CLI entrypoint
sglang serve --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000

# Still supported
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000

Then call it with the OpenAI Python client:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[{"role": "user", "content": "List 3 countries and their capitals."}],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

How to run batch inference with the SGLang Offline Engine API and native endpoints

SGLang supports multiple "API surfaces" depending on what you're building:

The /generate endpoint is the low-level runtime API. Prefer /v1/... OpenAI-compatible routes when you want chat templates and the usual client ecosystem handled for you.

Without any HTTP server, the Offline Engine runs inference in-process: suited to batch jobs and custom services. It supports sync/async and streaming/non-streaming combinations—pick the mode that matches the call pattern.

Example using the native /generate endpoint

Minimal pattern: run a server, then POST /generate with temperature and max_new_tokens (and any other sampling fields you need).

import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
    },
)

print(response.json())

temperature = 0 is greedy sampling; higher values increase diversity.

Example using the Offline Engine API for in-process batch inference

Typical flow: construct sgl.Engine(model_path=...), run llm.generate(...) over a batch of prompts, then llm.shutdown() to release GPU and other resources.

import sglang as sgl

llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")

prompts = [
    "Write a concise self-introduction.",
    "Explain what prefix caching is in one paragraph.",
]

sampling_params = {"temperature": 0.2, "top_p": 0.9}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("PROMPT:", prompt)
    print("OUTPUT:", output["text"])
    print()

llm.shutdown()