What Gemma 4's multi-token prediction head actually means for your eval pipeline

Marcus Chen — Tue, 07 Apr 2026 12:21:46 +0000

Gemma 4 dropped with a multi-token prediction (MTP) head and immediately every benchmark thread on r/LocalLLaMA and r/MachineLearning filled up with MMLU scores, HumanEval numbers, and throughput charts.

Most of those benchmarks are not measuring what the MTP head actually changes. Here's what's actually happening, and what it means if you're running your own eval pipeline.

What MTP actually is

Standard autoregressive generation predicts one token at a time. At each step, the model outputs a probability distribution over the vocabulary, samples a token, appends it, and repeats.

Multi-token prediction trains an additional head to predict multiple future tokens simultaneously. The core model still generates token-by-token at inference time, but the MTP head is used during training as an auxiliary loss — forcing the model to maintain internal representations that are useful several tokens ahead.

The practical effect at inference time (depending on how it's deployed): speculative decoding becomes more effective because the MTP head can propose candidate continuations that the main model is more likely to accept. This is where the throughput numbers come from.

Here's a simplified view of what speculative decoding with an MTP head looks like:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def speculative_decode_step(
    main_model,
    draft_model,  # or MTP head used as draft
    input_ids: torch.Tensor,
    gamma: int = 4,  # number of draft tokens to generate
    temperature: float = 1.0,
) -> torch.Tensor:
    """
    One round of speculative decoding.
    Draft model proposes `gamma` tokens, main model verifies.
    """
    device = input_ids.device
    draft_tokens = []

    # Generate gamma draft tokens
    draft_input = input_ids.clone()
    with torch.no_grad():
        for _ in range(gamma):
            draft_out = draft_model(draft_input)
            next_token_logits = draft_out.logits[:, -1, :]
            next_token = torch.argmax(next_token_logits, dim=-1, keepdim=True)
            draft_tokens.append(next_token)
            draft_input = torch.cat([draft_input, next_token], dim=-1)

    # Verify with main model
    candidate_ids = torch.cat([input_ids] + draft_tokens, dim=-1)
    with torch.no_grad():
        main_out = main_model(candidate_ids)

    # Accept/reject draft tokens
    accepted = 0
    for i in range(gamma):
        main_logits = main_out.logits[:, input_ids.shape[1] + i - 1, :]
        draft_token = draft_tokens[i]

        # Simple greedy acceptance check (real implementations use sampling)
        main_token = torch.argmax(main_logits, dim=-1, keepdim=True)
        if torch.equal(main_token, draft_token):
            accepted += 1
        else:
            break

    # Return accepted prefix + one correction token
    result_length = input_ids.shape[1] + accepted + 1
    return candidate_ids[:, :result_length]

The MTP head improves the acceptance rate in that inner loop. More accepted draft tokens per round = higher effective throughput.

Why your benchmark results are probably misleading

The throughput gains from MTP are real, but they're not uniform across tasks. The acceptance rate of speculative decoding depends on how predictable the output sequence is.

High acceptance rate (MTP helps a lot):

Code generation — syntax is highly structured
Structured data extraction — JSON, CSV, templated output
Formulaic text — boilerplate, standard contract language, templated responses

Lower acceptance rate (MTP helps less):

Open-ended generation with high entropy
Creative writing
Reasoning chains that make non-obvious inferential jumps
Adversarial inputs where the model is already uncertain

If your benchmark is mostly code or structured output tasks, your throughput numbers will look great. If your production use case is open-ended dialogue or reasoning-heavy tasks, the gains will be smaller.

What I actually measured

At Nexus, we maintain domain-specific eval suites for our enterprise automation use cases. I ran Gemma 4 through these last week. Three categories:

Structured extraction (contract parsing, form extraction)

# eval structure — simplified
eval_results = {
    "task": "structured_extraction",
    "model_variants": ["gemma4-base", "gemma4-mtp"],
    "metrics": {
        "gemma4-base":  {"throughput_tps": 847,  "f1": 0.923, "exact_match": 0.871},
        "gemma4-mtp":   {"throughput_tps": 1001, "f1": 0.924, "exact_match": 0.873},
    }
}
# ~18% throughput improvement, no quality regression
# This is the good case for MTP

Open-ended summarization

eval_results = {
    "task": "open_ended_summarization",
    "metrics": {
        "gemma4-base":  {"throughput_tps": 612, "rouge_l": 0.441, "topic_drift_rate": 0.031},
        "gemma4-mtp":   {"throughput_tps": 679, "rouge_l": 0.438, "topic_drift_rate": 0.047},
    }
}
# ~11% throughput improvement
# Small but consistent increase in mid-sentence topic drift
# ROUGE difference is within noise, but topic_drift_rate is reproducible

topic_drift_rate here is an internal metric — we flag spans where the model shifts semantic focus within a sentence boundary. It's a custom eval, not something you'll find in standard benchmarks.

Adversarial robustness suite

eval_results = {
    "task": "adversarial_robustness",
    "test_families": [
        "paraphrase_invariance",     # same meaning, different phrasing
        "format_variation",          # valid but unusual formatting
        "rare_edge_cases",           # valid but low-frequency inputs
        "ambiguity_resolution",      # genuinely ambiguous inputs
    ],
    "metrics": {
        "gemma4-base": {"overall_pass_rate": 0.847},
        "gemma4-mtp":  {"overall_pass_rate": 0.849},
    }
}
# Effectively identical — MTP doesn't help or hurt adversarial robustness

The adversarial result is the most important one for production deployments. Throughput gains are nice. Robustness is what keeps you off the incident page.

What this means for your eval pipeline

If you're evaluating Gemma 4 for a production deployment, here's what to actually do:

1. Build task-specific benchmarks, not generic ones

Generic benchmarks tell you how the model performs on generic tasks. Your use case is not generic.

class DomainEvalSuite:
    def __init__(self, task_name: str, test_cases: list[dict]):
        self.task_name = task_name
        self.test_cases = test_cases  # [{input, expected_output, metadata}]

    def run(self, model, tokenizer) -> dict:
        results = []
        for case in self.test_cases:
            output = self._generate(model, tokenizer, case["input"])
            score = self._score(output, case["expected_output"])
            results.append({
                "input": case["input"],
                "output": output,
                "score": score,
                "metadata": case["metadata"]
            })
        return self._aggregate(results)

    def _score(self, output: str, expected: str) -> float:
        # Implement task-specific scoring — not ROUGE, not BLEU
        # Exact match, F1 over extracted fields, custom rubric, whatever fits
        raise NotImplementedError

    def _aggregate(self, results: list) -> dict:
        scores = [r["score"] for r in results]
        return {
            "mean": sum(scores) / len(scores),
            "p10": sorted(scores)[len(scores) // 10],  # tail performance matters
            "fail_rate": sum(1 for s in scores if s < 0.5) / len(scores),
        }

P10 tail performance and fail rate matter more than mean score for production systems. Mean score hides your worst cases.

2. Separate throughput eval from quality eval

Don't conflate them. Run throughput benchmarks under controlled conditions (fixed prompt length, fixed output length, known hardware). Run quality benchmarks separately. Don't let throughput optimization choices degrade quality metrics.

3. Test MTP specifically for your task type

If your use case is structured output: MTP is probably worth it, measure the throughput gain and verify quality holds.

If your use case is open-ended generation: measure both throughput gain AND any quality regressions before assuming MTP is better.

4. Include an adversarial subset

At minimum, include:

Paraphrase variants of your test inputs (same intent, different wording)
Format variations (valid but unusual)
A few hand-crafted tricky cases specific to your domain

If your model passes the standard tests but fails on paraphrases of those same tests, you have a memorization problem, not a generalization problem.

The bottom line

Gemma 4 MTP is a real improvement for throughput on structured tasks. The benchmark numbers showing gains on MMLU/HumanEval are real but somewhat misleading — those tasks happen to be ones where MTP acceptance rates are high.

Build your own eval suite. Measure what matters for your use case. Check both throughput and quality. Include adversarial coverage.

The model is not the hard part.

I'm happy to go deeper on any of the eval methodology here — custom metrics design, adversarial suite construction, or the MTP acceptance rate analysis. Drop questions in the comments.

Mastering Local AI Agents for Everyday Programming in 2026

Marcus Chen — Fri, 03 Apr 2026 17:10:19 +0000

The landscape of software development is shifting beneath our feet. While large cloud-based LLMs have dominated headlines, 2026 is the year local AI agents have truly matured into indispensable tools for everyday programming.

By running autonomous, agentic workflows on our own silicon, developers are unlocking new levels of privacy, speed, and offline capability. In this post, we'll explore why local agents matter and how you can seamlessly integrate them into your coding routine.

Why Local Agents?

Cloud LLMs are powerful, but they have limitations:

Privacy: Not every codebase can or should be sent over the wire. Local agents keep proprietary logic strictly on your machine.
Latency: No network trips means near-instant feedback for lightweight refactors or shell queries.
Cost: Once you have the hardware, inferences are virtually free. This enables "infinite loop" agents that can continuously run tests and iteratively fix bugs in the background without racking up API bills.

Essential Workflows for Local Agents

1. The Autonomous Test-Fixer

Instead of manually deciphering stack traces, local agents can watch your test output. When a test fails, the agent isolates the failure, analyzes the relevant module, and proposes a fix.

# Example of a broken function
def calculate_discount(price, discount_percent):
    return price - (price * discount_percent) # Oops, forgot to divide by 100

A background local agent detects the AssertionError, understands the logic, and patches the math error before you even switch back to your editor.

2. PR Review and Digest

Local models like specialized code models can read your diffs before you commit. They act as a ruthless but helpful rubber duck, pointing out logical gaps or missing edge-case coverage without complaining about rate limits.

3. Deep-Dive Log Analysis

Sifting through thousands of lines of logs locally? A local agent can process massive log files right where they live, grepping for anomalies and synthesizing a human-readable summary.

Tools to Get Started

If you're looking to build your own local agentic stack, here are a few tools leading the charge:

Ollama / LM Studio: The backbone for running quantized models efficiently.
OpenClaw / Aider: Terminal-native agents that can directly edit your files and run shell commands.

Conclusion

The question is no longer if AI will write code, but where that AI lives. By embracing local agents, we get the best of both worlds: the intelligence of modern LLMs with the speed, privacy, and control of our own hardware.

Are you running any agents locally in your workflow? Let me know in the comments!

Forem: Marcus Chen