Quantization

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Aman Sachan

Apr 30

KVQuant: Run 70B LLMs on 8GB RAM with KV Cache Quantization

#python #llm #quantization

1 min read

Aman Sachan

Apr 30

KVQuant: Run 70B LLMs on 8GB RAM with 4-bit KV Cache Quantization

#python #llm #quantization #optimization

1 min read

Cover image for Traditional Quantization vs 1.58-Bit Ternary Models: A Practical Comparison

Alan West

Apr 18

Traditional Quantization vs 1.58-Bit Ternary Models: A Practical Comparison

#machinelearning #llm #quantization #ai

5 min read

Denis Lavrentyev

Apr 13

GIMP's Posterization: Simple Quantization vs. Median Cut for Better Visuals

#gimp #posterization #quantization #mediancut

8 min read

plasmon

Apr 8

Q4 KV Cache Fit 32K Context into 8GB VRAM — Only Math Broke

#llm #quantization #vram #localllm

8 min read

ANKUSH CHOUDHARY JOHAL

Apr 27

Postmortem: How a Quantization Error in Llama 3.2 7B Caused Incorrect Code Suggestions for 500 Users

#postmortem #quantization #errors #llama

13 min read

Scott Everitt

Mar 30

Building a Vector Database That Never Decompresses Your Vectors

#vectordatabase #quantization #turboquant #go

16 min read

TildAlice

Feb 22

TorchAO vs ONNX Runtime: 8-bit Quantization Benchmark

#quantization #llminference #pytorch #onnx

1 min read

Hector Li

Feb 11

Bringing 2-Bit Quantization to ONNX Runtime's WebGPU Backend

#onnxruntime #webgpu #2bit #quantization

5 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Forem

# quantization

KVQuant: Run 70B LLMs on 8GB RAM with KV Cache Quantization

KVQuant: Run 70B LLMs on 8GB RAM with 4-bit KV Cache Quantization

Traditional Quantization vs 1.58-Bit Ternary Models: A Practical Comparison

GIMP's Posterization: Simple Quantization vs. Median Cut for Better Visuals

Q4 KV Cache Fit 32K Context into 8GB VRAM — Only Math Broke

Postmortem: How a Quantization Error in Llama 3.2 7B Caused Incorrect Code Suggestions for 500 Users

Building a Vector Database That Never Decompresses Your Vectors

TorchAO vs ONNX Runtime: 8-bit Quantization Benchmark

Bringing 2-Bit Quantization to ONNX Runtime's WebGPU Backend