Forem: Nathan Maine

Gemma 4 After 24 Hours: What the Community Found vs What Google Promised

Nathan Maine — Fri, 03 Apr 2026 02:31:45 +0000

Google released Gemma 4 yesterday under Apache 2.0. The benchmarks looked incredible. The community went to work. Here's what we're actually seeing.

I spent the last 24 hours reading through forums, running my own fine-tuning experiments, and collecting reports from dozens of early adopters. This is a summary of the real-world findings, the open questions, and where I think this model family lands.

The Good News First

Apache 2.0 is a big deal. Previous Gemma releases used a custom Google license that technically allowed them to restrict usage. Apache 2.0 removes that uncertainty entirely. For anyone building commercial products on open models, this matters more than any benchmark number.

Multilingual quality is genuinely strong. Users testing German, Arabic, Vietnamese, and French are reporting that Gemma 4 outperforms Qwen 3.5 in non-English tasks. One user called it "in a tier of its own" for translation. Another said it "makes translategemma feel outdated instantly." For global enterprise deployments, this is a significant differentiator.

The ELO score tells a different story than benchmarks. The 31B model scored 2150 on LMArena, which puts it above GPT-OSS-120B and comparable to GPT-5-mini. But side-by-side benchmark tables show it roughly tying with Qwen 3.5 27B. The gap between ELO (human preference) and automated benchmarks suggests Gemma 4 produces responses that humans prefer even when raw accuracy is similar.

The E2B model is absurd. Multiple users confirmed that the 2.3B effective parameter model beats Gemma 3 27B on most benchmarks. A user running it on a basic i7 laptop with 32GB RAM reported it was "not only faster, it gives significantly better answers" than Qwen 3.5 4B for finance analysis.

The Problems Nobody Warned About

Inference Speed

This is the elephant in the room. Multiple users are reporting that Gemma 4's MoE model (26B-A4B) runs significantly slower than Qwen 3.5's equivalent:

One user: 11 tokens/sec on Gemma 4 26B-A4B vs 60+ tokens/sec on Qwen 3.5 35B-A3B on the same 5060 Ti 16GB
Another confirmed higher VRAM usage for context at the same quantization level
Someone running on a DGX Spark asked "why is it super slow?" with no clear answer yet

For the dense 31B model, users are reporting 18-25 tokens/sec on dual NVIDIA GPUs (5070 Ti + 5060 Ti), which is reasonable but not fast.

The speed gap against Qwen 3.5 is concerning for production deployments where latency matters.

VRAM Consumption

Gemma models have historically been VRAM-hungry for context, and Gemma 4 appears to continue this pattern. One user noted they could only fit Gemma 3 27B Q4 with 20K context on a 5090, while Qwen 3.5 27B Q4 fit with 190K context on the same card.

For the 256K context window to be useful in practice, you need significantly more VRAM than competing models at the same parameter count.

Fine-Tuning Compatibility

As someone who attempted QLoRA fine-tuning within hours of release, I can confirm the tooling is not ready. Three issues hit immediately:

HuggingFace Transformers didn't recognize the gemma4 architecture (required installing from source)
PEFT couldn't handle Gemma4ClippableLinear, a new layer type in the vision encoder (required a monkey-patch)
A new mm_token_type_ids field is required during training even for text-only data (required a custom data collator)

I've filed issues on both huggingface/peft and huggingface/transformers. Both received responses within hours, and a fix for the mm_token_type_ids issue is already in progress. Unsloth also has day-one support if you prefer that path.

The community question "how easy is it to fine-tune compared to Gemma 3?" currently has no good answer beyond "harder, but solvable."

Stability Questions

One user testing the non-quantized 31B in Google AI Studio reported "infinite loops and no possibility to read text from the image." Another found that the model jailbreaks with basic system prompts. A third reported Mac hard crashes when loading either the 31B or 26B in LM Studio.

These are early reports and may be resolved with updates, but they're worth noting for anyone considering production deployment.

The Benchmark Reality

The community quickly assembled side-by-side comparisons. Here's the consolidated picture:

Metric	Gemma 4 31B	Qwen 3.5 27B	Winner
MMLU-Pro	85.2%	86.1%	Qwen
GPQA Diamond	84.3%	85.5%	Qwen
LiveCodeBench v6	80.0%	80.7%	Tie
Codeforces ELO	2150	1899	Gemma
TAU2-Bench	76.9%	79.0%	Qwen
MMMLU	88.4%	85.9%	Gemma
HLE (no tools)	19.5%	24.3%	Qwen

Gemma 4 wins on competitive coding (ELO) and multilingual (MMMLU). Qwen 3.5 wins on most reasoning benchmarks. Neither is a clear overall winner.

The honest take from one top commenter: "Gemma 4 ties with Qwen, if not Qwen being slightly ahead. And Qwen 3.5 is more compute efficient too."

What the Community Is Waiting For

QAT versions. Gemma 3 QAT (quantization-aware training) models arrived weeks after the initial release. The community expects the same for Gemma 4, and these will likely improve quantized inference quality significantly.

Abliterated/uncensored versions. At least one already exists. Multiple users are requesting more. The Apache 2.0 license makes this fully legal now.

Larger models. There were rumors of a 120B model that didn't materialize. Several users expressed disappointment. A 100B+ MoE from Google could be transformative.

A 9-12B dense model. The gap between E4B (4.5B effective) and 26B MoE leaves a hole in the lineup. Gemma 3's 12B model was popular, and there's no direct upgrade path.

Where This Leaves Us

Gemma 4 is not the clear winner the benchmarks suggested. But it's not trying to be.

The real value proposition is the combination of:

Apache 2.0 (fully permissive, no restrictions)
Multilingual excellence (best in class for non-English)
Base models available (fine-tuning ready on day one)
Size diversity (2B to 31B covers edge to server)
Native system prompts and function calling (production-ready features)

For English-only, benchmark-optimized, speed-critical deployments, Qwen 3.5 is still the better choice. For multilingual, legally unrestricted, fine-tuning-focused use cases, Gemma 4 has a compelling argument.

The speed and VRAM issues need to be addressed. The fine-tuning tooling needs a week or two to catch up. And we need QAT quantizations before the smaller models can truly compete on efficiency.

But make no mistake, releasing a 31B dense model under Apache 2.0 that rivals models 4-10x its size on human preference benchmarks is a significant moment for open AI. Google is finally competing on openness, not just capability.

I'll be publishing our fine-tuning results (including the day-zero bug fixes) and benchmark comparisons as the training run completes. Follow along if you're interested.

Nathan Maine builds AI systems for regulated industries. He is currently fine-tuning Gemma 4 31B for domain-specific deployment and has filed bug reports on huggingface/peft and huggingface/transformers for day-zero compatibility issues.

Fine-Tuning Gemma 4 on Day Zero: 3 Bugs We Solved in 30 Minutes

Nathan Maine — Thu, 02 Apr 2026 20:35:03 +0000

Google released Gemma 4 today under Apache 2.0 — their most capable open model family. The 31B dense model scores ~1452 on LMArena with a 256K context window.

We wanted to fine-tune it immediately. QLoRA on a single NVIDIA B200. It broke three times before training started.

Here's what happened and how we fixed each one.

Bug 1: "Transformers does not recognize this architecture"

The first error hits before the model even loads:

ValueError: The checkpoint you are trying to load has model type `gemma4` 
but Transformers does not recognize this architecture.

Why: The latest stable Transformers release (5.4.0) shipped before Gemma 4 existed. The gemma4 model type only exists in the dev branch.

Fix: Install from source.

pip install git+https://github.com/huggingface/transformers.git

This gets you 5.5.0.dev0 which includes the Gemma4ForConditionalGeneration class.

Time to fix: 2 minutes.

Bug 2: "Target module Gemma4ClippableLinear is not supported"

After installing Transformers from source, the model loads fine. But when PEFT tries to apply LoRA:

ValueError: Target module Gemma4ClippableLinear(
  (linear): Linear4bit(in_features=1152, out_features=1152, bias=False)
) is not supported.

Why: Gemma 4 introduces a new layer type called Gemma4ClippableLinear for its vision and audio encoders. It wraps nn.Linear with optional input/output clamping for numerical stability. The catch: it inherits from nn.Module, not nn.Linear.

PEFT checks the type of every target module before applying LoRA. Since Gemma4ClippableLinear isn't nn.Linear, PEFT rejects it — even though we only want to apply LoRA to the text decoder layers, not the vision encoder.

The exclude_modules parameter doesn't help either. PEFT runs the type check before filtering, so excluded modules still need to be recognized types.

Installing PEFT from source doesn't help either — the support simply doesn't exist yet.

Fix: Monkey-patch Gemma4ClippableLinear to inherit from nn.Linear before loading the model.

import torch.nn as nn
from transformers.models.gemma4 import modeling_gemma4

class PatchedClippableLinear(nn.Linear):
    def __init__(self, config, in_features, out_features):
        nn.Linear.__init__(self, in_features, out_features, bias=False)
        self.use_clipped_linears = getattr(config, "use_clipped_linears", False)
        if self.use_clipped_linears:
            self.register_buffer("input_min", torch.tensor(-float("inf")))
            self.register_buffer("input_max", torch.tensor(float("inf")))
            self.register_buffer("output_min", torch.tensor(-float("inf")))
            self.register_buffer("output_max", torch.tensor(float("inf")))

    def forward(self, x):
        if self.use_clipped_linears:
            x = torch.clamp(x, self.input_min, self.input_max)
        out = nn.Linear.forward(self, x)
        if self.use_clipped_linears:
            out = torch.clamp(out, self.output_min, self.output_max)
        return out

modeling_gemma4.Gemma4ClippableLinear = PatchedClippableLinear

Place this before any AutoModelForCausalLM.from_pretrained() call. PEFT now sees the vision encoder layers as standard linear layers and proceeds normally.

Result: 534M trainable parameters (1.68% of 31.8B total).

Time to fix: 15 minutes (including reading the Gemma 4 source to understand the layer).

Bug 3: "mm_token_type_ids is required"

LoRA applies, data loads, training starts — and immediately crashes:

ValueError: `mm_token_type_ids` is required as a model input when training

Why: Gemma 3 required token_type_ids during training. Gemma 4 adds a second required field: mm_token_type_ids (multimodal token type IDs). The model validates their presence in the forward pass, even for text-only training. For text-only inputs, both should be all zeros.

Standard tokenizers and data collators don't produce mm_token_type_ids. You need a custom collator.

Fix: Add both fields during tokenization and build a custom data collator.

# During tokenization
def format_chat(example):
    text = tokenizer.apply_chat_template(
        example["messages"], tokenize=False, add_generation_prompt=False
    )
    tokenized = tokenizer(text, truncation=True, max_length=4096)
    tokenized["token_type_ids"] = [0] * len(tokenized["input_ids"])
    tokenized["mm_token_type_ids"] = [0] * len(tokenized["input_ids"])
    tokenized["labels"] = tokenized["input_ids"].copy()
    return tokenized

# Custom data collator
@dataclass
class GemmaCollator:
    tokenizer: object
    def __call__(self, features):
        max_len = max(len(f["input_ids"]) for f in features)
        pad_id = self.tokenizer.pad_token_id
        batch = {
            "input_ids": [],
            "attention_mask": [],
            "token_type_ids": [],
            "mm_token_type_ids": [],
            "labels": [],
        }
        for f in features:
            pad_len = max_len - len(f["input_ids"])
            batch["input_ids"].append(f["input_ids"] + [pad_id] * pad_len)
            batch["attention_mask"].append(
                [1] * len(f["input_ids"]) + [0] * pad_len
            )
            batch["token_type_ids"].append([0] * max_len)
            batch["mm_token_type_ids"].append([0] * max_len)
            batch["labels"].append(
                f.get("labels", f["input_ids"]) + [-100] * pad_len
            )
        return {k: torch.tensor(v) for k, v in batch.items()}

Important: set remove_unused_columns=False in your training config, or the trainer will strip mm_token_type_ids before it reaches the model.

training_args = SFTConfig(
    ...,
    dataset_text_field=None,
    remove_unused_columns=False,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    data_collator=GemmaCollator(tokenizer),
)

Time to fix: 5 minutes.

The Result

After all three fixes:

31B model training at 4.5s/step on a single NVIDIA B200 (192GB)
534M trainable parameters via QLoRA (1.68% of 31.8B)
GPU utilization: 89%, 38GB VRAM used
Estimated training time: ~7.5 hours for 3 epochs on 16K examples

Total time from "model released" to "training steps running": under 4 hours (including model download).

Key Takeaways

Day-zero fine-tuning requires bleeding-edge dependencies. Install Transformers and PEFT from source when working with newly released models.
Multimodal models have hidden requirements for text-only training. Both token_type_ids and mm_token_type_ids are validated even when no images or audio are involved.
PEFT's type checking happens before module filtering. Even if you exclude vision modules, they still need to be recognized types. Monkey-patching is a valid workaround until official support lands.
None of these are avoidable with experience. They're day-zero discovery problems. The difference is how fast you solve them.

Issues filed:

[huggingface/peft] Gemma4ClippableLinear not supported
[huggingface/transformers] mm_token_type_ids required for text-only fine-tuning

Both include workarounds and suggested fixes. PRs welcome.