I Built a GPU Dataset for LLM Inference — Here’s What I Learned

Thurmon Demich — Tue, 28 Apr 2026 13:49:49 +0000

TL;DR: Most GPU advice for LLMs is either outdated or too generic. I started collecting real-world data (VRAM, model fit, tokens/sec), and the patterns are surprisingly consistent.

Why I built this

If you’ve tried running LLMs locally (Ollama, llama.cpp, vLLM), you’ve probably hit this problem:

“Can my GPU run this model?”
“Why does 13B barely fit but runs so slow?”
“Do I really need 24GB VRAM?”

The answers online are all over the place.

So I started putting together a small dataset based on:

community benchmarks
consistent VRAM constraints
repeated patterns across setups

👉 Repo: https://github.com/airdropkalami/awesome-gpu-for-llm

The pattern is simpler than people think

After aggregating the data, a few rules kept showing up.

🧠 Practical rules (that actually work)

7B models → run on most GPUs
13B models → need ~16GB for comfortable use
34B models → require 24GB-class GPUs
70B models → usually better on cloud

These aren’t theoretical — they show up consistently across different frameworks.

VRAM matters more than compute

One thing became obvious quickly:

If it doesn’t fit in VRAM, it doesn’t run.

You can optimize speed, but you can’t “optimize around” missing VRAM.

Real-world dataset (simplified)

Here’s a small snapshot from what I collected:

GPU	VRAM	Model	Quant	tok/s	Fit
RTX 4060	8GB	7B	Q4	~35	✅
RTX 4060	8GB	13B	Q4	~18	⚠️
RTX 4090	24GB	13B	Q4	~45	✅
RTX 4090	24GB	34B	Q4	~25	⚠️
RTX 3090	24GB	34B	Q4	~22	⚠️

👉 Full dataset (updated):
https://github.com/airdropkalami/awesome-gpu-for-llm/blob/main/benchmark/dataset.md

What surprised me

1. 13B is the real sweet spot

Not 7B. Not 70B.

13B gives the best balance between:

quality
speed
hardware cost

2. 24GB is a hard ceiling for most users

Once you go beyond that:

cost explodes
scaling becomes inefficient
cloud often makes more sense

3. Benchmarks don’t reflect real usage

A lot of GPU comparisons focus on:

FLOPS
synthetic benchmarks

But for LLMs:

VRAM > everything else

When NOT to buy a GPU

This is where most people overspend.

If you:

want to run 70B models
only experiment occasionally
don’t need local inference

👉 you’re better off using cloud GPUs.

If you want a deeper breakdown

I wrote more detailed guides here:

Why I’m sharing this

I’m still expanding the dataset, and I’m trying to keep it:

practical (not theoretical)
consistent across setups
easy to use for decision making

If you have real benchmark data or setups, feel free to contribute.

Final thought

The “best GPU” isn’t the fastest.

It’s the one that:

fits your model
matches your budget
and actually works in your setup

If you’re building with local LLMs, I’d love to know what GPU + model combo you’re running.

How to Choose the Right GPU for Local LLMs (Without Wasting Money)

Thurmon Demich — Mon, 27 Apr 2026 13:15:35 +0000

How to Choose the Right GPU for Local LLMs (Without Wasting Money)

TL;DR: Most people overspend on GPUs for local LLMs. If you match model size ↔ VRAM ↔ quantization, you can save hundreds (or thousands) and still get great results.

Why this matters

If you’re running local LLMs (Ollama, llama.cpp, vLLM, etc.), the biggest mistake I see is:

Buying a GPU that’s too powerful (and too expensive)
Or worse, buying one with not enough VRAM

Both lead to frustration.

This guide breaks down how to choose the right GPU for your actual workload — not just benchmarks.

Step 1 — Understand what actually limits you

For LLM inference, VRAM matters more than raw compute.

Rough VRAM requirements

Model Size	Typical VRAM (quantized)	Notes
7B	6–8GB	Entry-level, very easy to run
13B	10–16GB	Sweet spot for many users
34B	20–24GB	High-end consumer GPUs
70B	40GB+	Usually cloud or multi-GPU

If you remember one thing:

VRAM determines what you can run. Compute determines how fast it runs.

Step 2 — Pick your use case first (not the GPU)

Before looking at GPUs, define your goal:

1. Lightweight local assistant (7B–13B)

Coding assistant
Chatbot
RAG experiments

👉 You don’t need a flagship GPU.

2. Serious local inference (13B–34B)

Better reasoning
Higher quality outputs
More stable pipelines

👉 This is where most developers should aim.

3. Large models (70B+)

High-end research
Production-level inference

👉 Local becomes expensive very quickly.

Step 3 — Real GPU recommendations (2026)

Here’s a practical breakdown:

Best budget option

RTX 4060 / 4060 Ti (8–16GB)
Good for: 7B–13B models
Limitation: VRAM ceiling

Best overall value

RTX 4090 (24GB)
Good for: 13B–34B models
Why: Enough VRAM + strong performance

Used value pick

RTX 3090 (24GB)
Still extremely relevant for LLMs

High-end / no-compromise

RTX 5090-class
Only if budget is not a concern

Step 4 — When NOT to buy a GPU

This is where most people get it wrong.

If you:

Want to run 70B models
Don’t need constant local inference
Are just experimenting

👉 Use cloud GPUs instead

It’s often cheaper and far more flexible.

Step 5 — Common mistakes

❌ Mistake 1: Buying for benchmarks

Benchmarks ≠ your real workload.

❌ Mistake 2: Ignoring VRAM

You can’t “optimize around” missing VRAM.

❌ Mistake 3: Overbuying

A $1600 GPU for a 7B model is overkill.

❌ Mistake 4: Forcing everything local

Cloud exists for a reason.

Step 6 — Simple decision guide

If you just want a quick answer:

Beginner / budget → RTX 4060
Most users → RTX 4090
Tight budget but want 24GB → used 3090
Need 70B → go cloud

Want a deeper breakdown?

I put together a more detailed guide (including VRAM charts and specific model compatibility):

👉 https://bestgpuforllm.com/articles/best-gpu-for-ollama/
👉 https://bestgpuforllm.com/articles/how-much-vram-for-llm/

Final thought

The best GPU isn’t the most expensive one.

It’s the one that:

Fits your model size
Matches your budget
And doesn’t lock you into unnecessary cost

If you get those 3 right, you’re already ahead of most people building local AI setups.

Curious what setups others are running? Drop your GPU + model combo below — I’m collecting real-world configs.

Forem: Thurmon Demich

I Built a GPU Dataset for LLM Inference — Here’s What I Learned

Why I built this

The pattern is simpler than people think

🧠 Practical rules (that actually work)

VRAM matters more than compute

Real-world dataset (simplified)

What surprised me

1. 13B is the real sweet spot

2. 24GB is a hard ceiling for most users

3. Benchmarks don’t reflect real usage

When NOT to buy a GPU

If you want a deeper breakdown

Why I’m sharing this

Final thought

How to Choose the Right GPU for Local LLMs (Without Wasting Money)

How to Choose the Right GPU for Local LLMs (Without Wasting Money)

Why this matters

Step 1 — Understand what actually limits you

Rough VRAM requirements

Step 2 — Pick your use case first (not the GPU)

1. Lightweight local assistant (7B–13B)

2. Serious local inference (13B–34B)

3. Large models (70B+)

Step 3 — Real GPU recommendations (2026)

Best budget option

Best overall value

Used value pick

High-end / no-compromise

Step 4 — When NOT to buy a GPU

Step 5 — Common mistakes

❌ Mistake 1: Buying for benchmarks

❌ Mistake 2: Ignoring VRAM

❌ Mistake 3: Overbuying

❌ Mistake 4: Forcing everything local

Step 6 — Simple decision guide

Want a deeper breakdown?

Final thought