Forem: ArshTechPro

Gemma 4: A Practical Guide for Developers

ArshTechPro — Fri, 03 Apr 2026 15:23:32 +0000

Google DeepMind released Gemma 4 on April 2, 2026. It is their most capable open model family to date, built from the same research behind Gemini 3, and shipped under the Apache 2.0 license. That means no usage caps, no restrictive policies, and full commercial freedom.

This article breaks down what Gemma 4 is, what it can do, and how to actually run it in your projects. No fluff. Just the parts that matter if you are building something.

What is Gemma 4

Gemma 4 is a family of open-weight multimodal models designed for reasoning, code generation, and agentic workflows. It comes in four sizes:

Model	Parameters	Context Window	Best For
E2B	2.3B effective (5.1B total)	128K tokens	Phones, Raspberry Pi, IoT
E4B	4.5B effective (8B total)	128K tokens	Edge devices, fast inference
26B A4B (MoE)	26B total, 4B active	256K tokens	Low-latency server inference
31B (Dense)	31B	256K tokens	Maximum quality, fine-tuning base

Each size comes in both a base variant and an instruction-tuned (IT) variant. For most developer use cases, you want the IT variant.

The "E" prefix on the smaller models stands for "effective parameters." These models use a technique called Per-Layer Embeddings (PLE) that feeds a secondary embedding signal into every decoder layer, which means the model activates fewer parameters at inference time, saving RAM and battery.

The 26B model is a Mixture of Experts (MoE) architecture. It has 26 billion total parameters but only activates about 3.8 billion during inference. This makes it fast while still scoring near the top of the Arena AI leaderboard.

What It Can Do

Gemma 4 is not just a text chatbot. Here is what the model family supports out of the box:

Text generation and reasoning. Multi-step planning, deep logic, math. The 31B model scores 85.2% on MMLU Pro and 80.0% on LiveCodeBench v6.

Vision. All four model sizes accept image and video input. The vision encoder supports variable aspect ratios and configurable token budgets (70, 140, 280, 560, or 1120 tokens per image). More tokens means more detail at the cost of more compute.

Audio. The E2B and E4B models accept audio input natively. They handle speech recognition and speech-to-translated-text across multiple languages.

Code generation. All models can generate, complete, and correct code. The 31B model is strong enough to function as an offline code assistant.

Function calling. Native support for structured JSON output, function-calling syntax, and system instructions. This is the foundation for building agents.

140+ languages. Pre-trained on over 140 languages with strong support for 35+.

Step 1: Pick Your Model

Start by deciding which model fits your hardware and use case.

If you are running on a phone, Raspberry Pi, or Jetson Nano: Use gemma-4-E2B-it or gemma-4-E4B-it. These are designed for edge devices and run offline with near-zero latency.

If you have a single GPU (A100 or H100): Use gemma-4-26B-A4B-it. The MoE model fits in one GPU and gives you excellent latency because it only activates 4B parameters per forward pass.

If you have two GPUs or want maximum quality: Use gemma-4-31B-it. This is the dense model. It needs tensor parallelism across two 80GB GPUs for full bfloat16 inference, but quantized versions run on consumer GPUs.

If you just want to try it out first: Open Google AI Studio at aistudio.google.com and select the Gemma 4 model. No setup required.

Step 2: Install Dependencies

Gemma 4 requires transformers version 5.5.0 or later. Install the core packages:

pip install -U transformers torch accelerate

If you plan to work with images, also install timm:

pip install -U timm

If you want 4-bit quantization to run larger models on smaller GPUs:

pip install bitsandbytes

Step 3: Run Inference with Transformers

The fastest way to get started is with the Hugging Face pipeline API.

Text-only generation

from transformers import pipeline

pipe = pipeline(
    task="any-to-any",
    model="google/gemma-4-E2B-it",
    device_map="auto",
    dtype="auto",
)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}],
    },
    {
        "role": "user",
        "content": [{"type": "text", "text": "Explain dependency injection in three sentences."}],
    },
]

output = pipe(messages, return_full_text=False)
print(output[0]["generated_text"])

Image + text (vision)

from transformers import pipeline

pipe = pipeline(
    task="any-to-any",
    model="google/gemma-4-E4B-it",
    device_map="auto",
    dtype="auto",
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://example.com/your-image.jpg"},
            {"type": "text", "text": "Describe what you see in this image."},
        ],
    },
]

output = pipe(messages, return_full_text=False)
print(output[0]["generated_text"])

Lower-level control with AutoModel

If you need more control over generation parameters, load the model and processor directly:

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "google/gemma-4-E4B-it"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
).eval()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Write a Python function that reverses a linked list."},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
input_len = inputs["input_ids"].shape[-1]
print(processor.decode(output[0][input_len:], skip_special_tokens=True))

Step 4: Enable Thinking Mode

Gemma 4 supports chain-of-thought reasoning. When enabled, the model outputs its internal reasoning before the final answer.

To turn it on, include the <|think|> token at the start of your system prompt:

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "<|think|>You are a helpful assistant."}],
    },
    {
        "role": "user",
        "content": [{"type": "text", "text": "What is 127 * 43?"}],
    },
]

The model will output a thinking block followed by the final answer. If you are using the processor.parse_response() method, you can separate the thinking from the content automatically.

To disable thinking, simply remove the <|think|> token from the system prompt.

Step 5: Serve It with vLLM

For production workloads, you will want to serve Gemma 4 behind an OpenAI-compatible API using vLLM.

Install vLLM

pip install -U vllm --pre \
    --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
    --extra-index-url https://download.pytorch.org/whl/cu129 \
    --index-strategy unsafe-best-match
pip install transformers==5.5.0

Start the server

For the 26B MoE on a single A100/H100:

vllm serve google/gemma-4-26B-A4B-it \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90

For the 31B dense model on two GPUs:

vllm serve google/gemma-4-31B-it \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90

For the E4B edge model:

vllm serve google/gemma-4-E4B-it \
    --max-model-len 131072

Query it

Once the server is running, hit it with a standard OpenAI-compatible request:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "google/gemma-4-26B-A4B-it",
        "messages": [
            {"role": "user", "content": "Explain quantum entanglement in simple terms."}
        ],
        "max_tokens": 512,
        "temperature": 0.7
    }'

This means you can swap Gemma 4 into any application that already talks to an OpenAI-compatible API. No code changes beyond the model name and endpoint URL.

Step 6: Run It Locally with Ollama

If you want to run Gemma 4 on your laptop without any server setup:

ollama run gemma4

That is it. Ollama handles downloading the quantized weights, setting up the runtime, and exposing a local API. This is the easiest path for local development and testing.

Step 7: Fine-Tune for Your Use Case

Gemma 4 is strong out of the box, but fine-tuning lets you specialize it for your domain. The recommended approach is QLoRA through the TRL library.

Install fine-tuning dependencies

pip install trl peft datasets bitsandbytes

Load with 4-bit quantization

from transformers import AutoModelForImageTextToText, BitsAndBytesConfig
import torch

model_id = "google/gemma-4-E2B"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

From here, you attach LoRA adapters using PEFT, prepare your dataset, and train with TRL's SFTTrainer. The E2B model can be fine-tuned on a free Google Colab T4 GPU. The larger models need proportionally more memory.

You can also fine-tune on Vertex AI or with Unsloth for additional optimizations.

Function Calling

Gemma 4 supports native function calling, which is what makes it useful for building agents. The model can output structured JSON that specifies which function to call and with what arguments.

Here is the general pattern:

Define your available functions in the system prompt as a JSON schema.
Send the user's message.
The model responds with a function call in structured JSON.
You execute the function and return the result.
The model uses the result to generate its final answer.

This works across all four model sizes. Combined with the long context windows (up to 256K tokens), you can pass entire codebases or document collections alongside your tool definitions in a single prompt.

Where to Get the Models

All model weights are available for download:

Hugging Face: huggingface.co/collections/google/gemma-4
Kaggle: kaggle.com/models/google/gemma-4
Ollama: ollama.com/library/gemma4
Google AI Studio (browser): aistudio.google.com

The Hugging Face model IDs you will use most often:

google/gemma-4-E2B-it (smallest, edge)
google/gemma-4-E4B-it (small, edge)
google/gemma-4-26B-A4B-it (MoE, fast server inference)
google/gemma-4-31B-it (dense, maximum quality)

Key Architecture Details (if you care)

A few things worth knowing about how Gemma 4 works under the hood:

Alternating attention. Layers alternate between local sliding-window attention (512-1024 tokens) and global full-context attention. This is how it stays efficient while still handling long context.

Dual RoPE. Standard rotary position embeddings for sliding-window layers, proportional RoPE for global layers. This is what enables the 256K context window without quality degradation at long distances.

Shared KV cache. The last N layers reuse key-value tensors from earlier layers instead of computing their own. This cuts both memory and compute during inference.

Vision encoder. Learned 2D position encoder with multidimensional RoPE. Preserves original aspect ratios. Token budgets are configurable from 70 to 1120 tokens per image.

Audio encoder. USM-style conformer architecture (same as Gemma-3n). Handles speech recognition and translation with up to 30 seconds of audio on the smaller models.

What Changed from Gemma 3

If you have used Gemma 3 before, here is what is different:

License. Gemma 3 used a custom Google license with restrictions. Gemma 4 uses Apache 2.0. This is a significant change for commercial use.
MoE model. The 26B A4B is the first Mixture of Experts model in the Gemma family.
Per-Layer Embeddings. The E2B and E4B models use PLE for better parameter efficiency.
Shared KV cache. New efficiency optimization not present in Gemma 3.
Audio input. The E2B and E4B models handle audio natively. Gemma 3 did not.
Roles. Gemma 4 uses standard system, user, and assistant roles in chat templates. Gemma 3 had a different role structure.

Summary

Gemma 4 gives you a complete open model stack: four sizes covering everything from phones to multi-GPU servers, multimodal input (text, image, video, audio), native function calling for agents, up to 256K context, and an Apache 2.0 license that lets you ship products without restrictions.

The fastest path from zero to running code:

pip install -U transformers torch
Load google/gemma-4-E2B-it with the pipeline API
Start prompting

Hermes Agent: A Self-Improving AI Agent That Runs Anywhere

ArshTechPro — Mon, 30 Mar 2026 14:20:05 +0000

Most AI agents today are chatbots with extra steps. You talk to them, they forget everything, and you start over next time. Hermes Agent, built by Nous Research, takes a different approach. It remembers what it learns, creates reusable skills from experience, and runs on your own infrastructure instead of someone else's cloud.

This article covers what Hermes Agent actually is, why it matters for developers, and how to get it running.

What Is Hermes Agent?

Hermes Agent is an open-source AI agent (MIT licensed) with a built-in learning loop. That phrase sounds like marketing, but it describes something specific: after completing a complex task, the agent can save the approach as a reusable "skill" for next time. It maintains persistent memory across sessions. It builds a model of who you are and how you work.

It is not a wrapper around a single API. You can plug in whatever LLM provider you want -- OpenAI, Anthropic, OpenRouter (which gives you access to 200+ models), or your own self-hosted endpoint running Ollama, vLLM, or SGLang. Switching providers is a single command. No code changes.

The project has about 8,700 stars on GitHub, 142 contributors, and 2,293 commits as of late March 2026. It is written primarily in Python.

Why Should a Developer Care?

There are many agent frameworks out there. Here is what makes Hermes Agent worth looking at.

It is not tied to your laptop. You can run the agent on a $5 VPS, inside a Docker container, over SSH to a remote server, or on serverless infrastructure like Modal or Daytona that hibernates when idle. You talk to it from Telegram, Discord, Slack, WhatsApp, Signal, or the terminal. The conversation continues across platforms.

It has real memory, not a hack. The agent maintains two small, curated files: MEMORY.md (for environment facts, conventions, and lessons learned) and USER.md (for your preferences and communication style). These are injected into the system prompt at session start. The agent also has full-text search over all past sessions stored in SQLite, so it can recall conversations from weeks ago.

It learns from its own work. After a complex task (typically 5+ tool calls), the agent can autonomously create a skill -- a structured markdown document with procedures, pitfalls, and verification steps. Next time a similar task comes up, it loads the skill instead of figuring things out from scratch. Skills can also self-improve during use when the agent discovers a better approach.

It supports MCP out of the box. You can connect any Model Context Protocol server by adding a few lines to the config file. This means the agent can interact with GitHub, databases, or any service that exposes an MCP endpoint.

It is research-ready. If you are working on training better tool-calling models, Hermes includes batch trajectory generation, Atropos RL environments, and trajectory compression. This is not just a user-facing product; it is also infrastructure for building the next generation of agent models.

Core Concepts

Before setting it up, it helps to understand the main building blocks.

The Agent Loop

The core of Hermes is AIAgent in run_agent.py. It handles provider selection, prompt construction, tool execution, retries, compression, and persistence. It is a synchronous orchestration engine -- one loop that drives everything.

Skills

Skills are on-demand knowledge documents stored in ~/.hermes/skills/. They follow a progressive disclosure pattern to minimize token usage:

Level 0: the agent sees a list of skill names and descriptions (about 3,000 tokens)
Level 1: the agent loads the full content of a specific skill when needed
Level 2: the agent loads a specific reference file within a skill

Each skill is a directory with a SKILL.md file and optional reference materials, templates, and scripts. The format uses YAML front matter for metadata and markdown for the actual instructions.

The agent creates skills automatically after complex tasks, but you can also write them by hand, install them from the Skills Hub (which aggregates multiple registries including skills.sh and GitHub repos), or share them with a team via external skill directories.

Memory

Memory is bounded and curated. MEMORY.md gets 2,200 characters. USER.md gets 1,375 characters. That is roughly 1,300 tokens total -- small enough that it does not bloat the context window, but large enough to hold 15-20 useful entries.

The agent manages memory itself. It adds entries when it learns something useful, replaces entries when information changes, and consolidates entries when memory gets full. There is also security scanning on memory entries to prevent prompt injection.

For deeper recall, the agent can search all past sessions using SQLite full-text search and LLM summarization. This is not in the system prompt -- it is on-demand, called when the agent needs to find something from a previous conversation.

Terminal Backends

Hermes supports six ways to execute commands: local, Docker, SSH, Daytona, Singularity, and Modal. Docker and SSH give you sandboxed execution. Daytona and Modal give you serverless persistence -- the environment hibernates when idle and wakes on demand.

The Messaging Gateway

The gateway is a long-running process that connects the agent to messaging platforms. You configure it with hermes gateway setup, start it with hermes gateway start, and then talk to the agent from your phone. The same slash commands work across all platforms.

How to Set It Up

Prerequisites

You need git installed. That is it. The installer handles Python, Node.js, and all dependencies.

Windows is not natively supported. Use WSL2 instead.

Step 1: Install

Run the one-line installer:

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

After it finishes, reload your shell:

source ~/.bashrc   # or source ~/.zshrc

Step 2: Configure a Provider

Run the model selection command:

hermes model

This walks you through choosing an LLM provider. Your options include:

Nous Portal -- subscription-based, zero-config, uses OAuth
OpenRouter -- multi-provider routing, supports 200+ models, needs an API key
OpenAI -- uses Codex models via device code auth
Anthropic -- Claude models directly, via Claude Code auth or an API key
Custom Endpoint -- any OpenAI-compatible API (Ollama, vLLM, SGLang, etc.)

There are also providers for Z.AI, Kimi/Moonshot, MiniMax, Alibaba Cloud, Hugging Face, and several others.

You can switch providers at any time by running hermes model again.

Step 3: Start Chatting

hermes

That is the entire startup command. You will see a welcome banner showing your active model, available tools, and installed skills. Type a message and press Enter.

Step 4: Set Up Sandboxed Execution (Recommended)

By default, the agent runs commands on your local machine. For safety, use a sandboxed backend:

hermes config set terminal.backend docker    # Docker isolation
hermes config set terminal.backend ssh       # Remote server

Step 5: Connect Messaging (Optional)

If you want to talk to Hermes from Telegram, Discord, Slack, or another platform:

hermes gateway setup    # Interactive configuration
hermes gateway start    # Start the gateway process

Useful Commands to Know

Once Hermes is running, these commands cover most of what you need day-to-day:

hermes              # Start a conversation
hermes -c           # Resume the last session
hermes model        # Switch LLM provider/model
hermes tools        # Configure enabled tools
hermes doctor       # Diagnose issues
hermes update       # Update to latest version
hermes gateway      # Manage messaging platforms
hermes skills search <query>   # Find skills to install

Inside a conversation, type / to see all available slash commands. A few highlights:

/model              # Switch models mid-conversation
/tools              # List available tools
/skills             # Browse and manage skills
/personality pirate # Try a fun personality
/save               # Save the conversation
/compress           # Compress context when it gets long

Adding MCP Servers

To connect external tools via MCP, add entries to ~/.hermes/config.yaml:

mcp_servers:
  github:
    command: npx
    args: ["-y", "@modelcontextprotocol/server-github"]
    env:
      GITHUB_PERSONAL_ACCESS_TOKEN: "ghp_xxx"

The agent will then have access to whatever tools that MCP server exposes.

Setting Up Scheduled Tasks

Hermes has a built-in cron scheduler that delivers results to any connected platform. You set up tasks in natural language:

> Every morning at 9am, check Hacker News for AI news and send me a summary on Telegram.

The agent creates a cron job that runs automatically via the gateway. No crontab editing required.

Contributing

If you want to contribute to the project, here is the development setup:

git clone https://github.com/NousResearch/hermes-agent.git
cd hermes-agent
git submodule update --init mini-swe-agent
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv .venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[all,dev]"
uv pip install -e "./mini-swe-agent"
python -m pytest tests/ -q

The mini-swe-agent submodule is a required terminal backend. If you want to work on the RL/training side, also initialize the tinker-atropos submodule.

What It Is Not

Hermes Agent is not a managed service. There is no hosted version you sign up for. You run it on your own machine or server, bring your own API keys, and own all the data. If that is what you want -- full control over an agent that gets better the more you use it -- this is worth trying.

It is also not a simple chatbot wrapper. The codebase includes multiple API modes, prompt caching and compression, gateway-specific session management, RL environment infrastructure, and an editor integration layer via ACP. It is a substantial project with real architectural depth.

TurboQuant: What Developers Need to Know About Google's KV Cache Compression

ArshTechPro — Sat, 28 Mar 2026 18:26:46 +0000

If you've ever run a large language model on your own hardware and watched your GPU memory vanish as the context window grows, TurboQuant is built for exactly that problem.

Published by Google Research on March 24, 2026 and headed to ICLR 2026, TurboQuant is a compression algorithm that shrinks the KV cache -- the biggest memory bottleneck during LLM inference -- down to 3-4 bits per element without any retraining or fine-tuning. The result is roughly a 4-6x reduction in KV cache memory with negligible quality loss.

This article breaks down what TurboQuant actually does, why it matters for anyone deploying or experimenting with LLMs, and how to start using community implementations right now.

The Problem: KV Cache Is Eating Your VRAM

When a transformer model generates text, it computes key and value vectors for every token in the context and stores them so it doesn't have to recompute them on subsequent steps. This is the key-value (KV) cache.

The issue is simple: it grows linearly with context length, and it stores everything in full precision (typically FP16). For an 8B parameter model at 32K context, the KV cache alone can consume around 4.6 GB of VRAM. Scale that to multiple concurrent users or longer contexts, and you're out of memory before the model weights themselves become the bottleneck.

Existing approaches to this problem -- like FP8 quantization in vLLM or the q4_0/q8_0 cache types in Ollama -- either don't compress aggressively enough or introduce quality trade-offs that are hard to predict. TurboQuant aims to do better on both fronts.

How TurboQuant Works (The Short Version)

TurboQuant is a two-stage compression pipeline. It doesn't need any training data, calibration, or model-specific tuning. It works on any vector, which means it slots into any transformer architecture.

Stage 1: PolarQuant (b-1 bits)

The first step is a random orthogonal rotation applied to each KV vector. This rotation spreads the energy of the vector uniformly across all coordinates, which transforms the problem: after rotation, each coordinate follows a predictable statistical distribution (approximately Beta or Gaussian depending on the head dimension). Because the distribution is known in advance, you can compute a mathematically optimal set of quantization buckets (using the Lloyd-Max algorithm) once, ahead of time. No per-model or per-dataset calibration needed. PolarQuant then converts coordinates into polar form -- radius and angle rather than Cartesian x/y/z -- which eliminates the costly per-block normalization constants that traditional quantizers need.

Stage 2: QJL Residual Correction (1 bit)

The second stage takes the tiny quantization error left over from Stage 1, projects it through a random Gaussian matrix using the Johnson-Lindenstrauss transform, and stores only the sign bit (+1 or -1) of each resulting value. This single-bit sketch acts as a bias correction that makes the inner product estimates (i.e., attention scores) mathematically unbiased. The overhead is just 1 extra bit per coordinate.

The combined result: b bits total per coordinate, with provably near-optimal distortion bounds and zero memory overhead from normalization constants.

Why This Matters for Developers

It's training-free and model-agnostic. TurboQuant doesn't require fine-tuning, calibration datasets, or model-specific configuration. The rotation matrix and codebook are derived from math, not data. Point it at any transformer's KV cache and it works.

Compression scales with context length. The benefit is proportional to how much KV cache you have. At 512 tokens the savings are modest (tens of megabytes). At 4K tokens you start saving over 1 GB. At 8K+ tokens the savings reach 2 GB or more on a single model -- and that's when it starts changing what you can actually run on your hardware.

It enables longer contexts on existing hardware. If you're hitting OOM at 16K context on a 16 GB GPU, TurboQuant can push that boundary significantly without buying new hardware.

Speed gains under memory pressure. When FP16 KV cache pushes your GPU into swap territory, inference speed collapses. Community benchmarks show TurboQuant maintaining 2-3x higher token throughput in these regimes because the compressed cache stays in fast GPU memory.

It applies beyond LLMs. The same algorithm works for vector search / nearest-neighbor retrieval, compressing high-dimensional embedding indices with state-of-the-art recall.

Getting Started: The pip-Installable Path

The fastest way to try TurboQuant today is the turboquant Python package, a community implementation that provides a drop-in replacement for HuggingFace's KV cache:

pip install turboquant

Three lines to compress your model's KV cache:

from transformers import AutoModelForCausalLM, AutoTokenizer
from turboquant import TurboQuantCache
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct",
    dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

# Create compressed cache -- that's it
cache = TurboQuantCache(bits=4)

inputs = tokenizer("Your prompt here", return_tensors="pt").to(model.device)
outputs = model(**inputs, past_key_values=cache, use_cache=True)

There's also a built-in OpenAI-compatible inference server:

turboquant-server --model Qwen/Qwen2.5-3B-Instruct --bits 4 --port 8000

And you can use the core quantizer directly on any vectors:

from turboquant import TurboQuantMSE

tq = TurboQuantMSE(dim=128, bits=4, device='cuda')
indices, norms = tq.quantize(vectors)       # vectors: (N, 128)
vectors_hat = tq.dequantize(indices, norms)  # reconstruct

The llama.cpp Path

If you're running models locally through llama.cpp, there are active community implementations integrating TurboQuant as a KV cache type. One notable fork (turboquant_plus) already works end-to-end on Apple Silicon with Metal GPU kernels:

# Server mode
./build/bin/llama-server \
  -m models/your-model.gguf \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -ngl 99 -c 262144 -fa on \
  --host 0.0.0.0 --port 8080

# CLI mode
./build/bin/llama-cli \
  -m models/your-model.gguf \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -ngl 99 -c 2048 -fa on \
  -n 100 -p "Hello world"

There's also an open feature request on the vLLM project to integrate TurboQuant as a native KV cache quantization option. Google's official implementation is expected around Q2 2026.

Practical Notes and Gotchas

A few things the benchmarks and community experiments have surfaced that the paper doesn't emphasize:

4-bit is the sweet spot for most use cases. At 4 bits, quality is essentially indistinguishable from FP16 on 3B+ parameter models. At 3 bits, you get more compression but quality starts degrading noticeably on models smaller than 8B.

Small models are more sensitive. On 0.5B-1.6B parameter models, quantization noise from TurboQuant can produce repetitive or degraded output, especially at 3-bit. If you're running something under 3B parameters, test carefully.

Keys and values have different sensitivities. Community experiments have found that value quantization tends to be the bottleneck -- 2-bit values cause significant cosine similarity degradation (around 0.94), while 4-bit values maintain 0.997. If you're tuning bit allocation, give values more bits than keys.

Short contexts don't benefit much. Below 1K tokens, the KV cache is small enough that compression savings are negligible and the overhead of rotation + quantization can even be a net negative. TurboQuant really shines at 4K+ tokens.

The residual window matters. Most implementations keep the most recent 128-256 tokens in full FP16 precision and only compress older tokens. This is important for output quality since attention focuses heavily on recent context.

Community Implementations at a Glance

Project	Language	Integration	Notes
`back2matching/turboquant`	Python	HuggingFace drop-in	`pip install turboquant`, includes OpenAI-compatible server
`tonbistudio/turboquant-pytorch`	Python/PyTorch	Standalone	From-scratch implementation with detailed validation
`0xSero/turboquant`	Python	vLLM adapter	Triton kernels, vLLM monkey-patch
`TheTom/turboquant_plus`	C/Python	llama.cpp + Metal	Apple Silicon optimized, 500+ tests
`RecursiveIntell/turbo-quant`	Rust	Standalone lib	Embedding + KV cache, no runtime dependencies
`ggml-org/llama.cpp#20969`	C	llama.cpp discussion	Multiple community PRs in progress

The Bigger Picture

TurboQuant is one piece of a larger shift happening in LLM deployment: making inference cheaper and more accessible without sacrificing quality. It pairs well with weight quantization (GPTQ, AWQ, GGUF formats), speculative decoding, and other serving optimizations. The combination of a 4-bit quantized model with a 4-bit TurboQuant KV cache means you can run meaningfully large models on consumer GPUs with long contexts -- something that was impractical a year ago.

Xcode 26.4 - Here Is What Actually Matters for Devs

ArshTechPro — Thu, 26 Mar 2026 19:25:27 +0000

Apple released Xcode 26.4 (build 17E192) on March 24, 2026. It ships with Swift 6.3, updated SDKs for iOS/iPadOS/tvOS/macOS/visionOS 26.4, and the largest Instruments update in the entire Xcode 26 cycle.

This post cuts through the release notes and organizes everything by what will actually affect your day-to-day work.

If You Read Nothing Else: The Sanitizer Regression

Address Sanitizer and Thread Sanitizer hang indefinitely on any OS 26.4 target when you build with Xcode 26.3 or older. Not crash. Not fail. Hang. Your CI jobs will silently block without producing any error output.

If your machines are running OS 26.4, upgrade Xcode to 26.4 immediately. There is no other workaround.

This is the single most urgent item in the release.

Instruments: Serious New Profiling Tools

Instruments got three headline features that are worth learning before your next performance investigation.

Run Comparison. You can now compare call trees across multiple profiling sessions. Open it from View > Detail Area > Compare With, select a previous run, and Instruments shows you which functions got faster or slower between sessions. Combined with Call Tree filtering (like "Charge to callers"), this makes before/after performance work much less manual.

Top Functions. A new top-level Call Tree mode that surfaces the most expensive functions across an entire trace, regardless of where they sit in the call hierarchy. If you have ever spent time drilling through nested call stacks to find the real bottleneck, this saves that effort.

Power Profiler per-core breakdown. On-device Power Profiler traces now show activity broken down by individual CPU core. Useful for understanding how your workload distributes across efficiency and performance cores on Apple Silicon.

Other Instruments changes worth noting:

xctrace import now supports --append-run to combine multiple files into one trace document
A new "Hide Inlined Functions" option charges inlined function samples to their parent, which cleans up noisy traces
Flame graph fixes for node selection, context menus, resizing, and color contrast
Applying a dSYM now resymbolicates all recorded runs, not just the current one
The "Compress Run Data" setting is gone; trace files compress automatically now

Swift 6.3

Xcode 26.4 bundles Swift 6.3. The release notes do not detail Swift language changes in depth within the Xcode notes themselves, but the compiler and runtime are updated. Check the separate Swift 6.3 release notes for language-level details.

C++ Standard Library: Some Huge Performance Numbers

If you maintain any C++ code in your project, the standard library improvements in this release are significant.

The std::ranges::copy, copy_n, copy_backward, move, move_backward, and rotate algorithms are optimized for std::vector::iterator. Apple reports improvements up to 2000x in applicable workloads. Even if your real-world gains are a fraction of that, it is worth knowing about if you do bulk vector operations.

Other highlights:

std::ranges::equal improved up to 188x
std::ranges::swap_ranges improved up to 611x
std::stable_sort now uses radix sort for floating-point types (up to 10x faster)
bitset::to_string is up to 16x faster for dense bitsets

Five new C++26 language features landed in Apple Clang:

Structured Bindings can introduce a Pack (P1061R10)
Structured binding declaration as a condition (P0963R3)
Variadic Friends (P2893R3)
constexpr placement new (P2747R2)
The Oxford variadic comma (P3176R1)

The library also adds std::flat_set, views::join_with, constexpr stable sorting, and Unicode 16.0.0 support in the formatting library, among 14 newly implemented C++ papers.

Localization: String Catalog Gets Practical

The String Catalog editor received seven improvements that remove friction from localization work.

Cut, copy, paste, duplicate now work on strings, both within a catalog and across catalogs. When you paste, you choose between adding a new key with all its translations or applying translations to an existing key.

One-click language removal is available directly in the editor. You pick whether to remove the language from just that catalog or the entire project.

Pre-fill translations from an existing language when adding a new supported language. Instead of starting from empty catalogs, you get a working baseline to edit.

BUILD_ONLY_KNOWN_LOCALIZATIONS build setting limits compiled localized content to the languages in your Project Editor. Languages outside that list are visually de-emphasized in String Catalogs.

Strings are no longer extracted from code comments by default. If you relied on this behavior, set LOCALIZED_STRING_CODE_COMMENTS to YES to restore it.

Known issue: removing a language from a String Catalog inside a Swift Package can cause it to reappear.

Testing: Breaking Change and Key Fixes

XCTest interoperability with Swift Testing is off by default now. If your CI relies on it, you need to explicitly set the SWIFT_TESTING_XCTEST_INTEROP_MODE environment variable to limited in your test plan. This will break any pipeline that depended on the previous implicit behavior.

Swift Testing now supports image attachments directly for CGImage, NSImage, UIImage, and CIImage. You can also set severity levels when recording an Issue.

Flaky test fix. A bug where unwaited notification expectations could fire in unrelated tests has been resolved. If you have been chasing intermittent test failures in a project that creates expectations without always waiting for them, this might be the cause.

Known limitations:

UIImage attachments do not work in Mac Catalyst test targets. Use UIImage.cgImage as a workaround.
If continueAfterFailure is false, a failure in an async test method, setUp, or tearDown skips remaining retries. Workaround: enable "Relaunch Tests for Each Repetition" in the test plan.
Swift Testing tests may crash on Rosetta run destinations on Apple Silicon. Use Xcode Universal to avoid this.

Build System and Package Manager

Setting SKIP_MERGEABLE_LIBRARY_BUNDLE_HOOK to YES on mergeable libraries that do not access resources through standard Bundle APIs avoids extra launch-time overhead.
swift test now correctly applies sanitizers when using --sanitize together with --filter.
Source editor tabs survive external tools (like git) removing and re-adding files while Xcode is running.
You can now enable package traits on dependencies directly from the Package Dependencies view.

Coding Intelligence Fixes

Two fixes in this area:

Externally configured MCP servers were being overwritten during Codex initialization. Fixed.
Multiple "Allow Connection?" dialogs that appeared when external development tools connected to Xcode are resolved.

System Requirements

Xcode 26.4 requires macOS Tahoe 26.2 or later on the host Mac.

On-device debugging is supported for iOS 15+, tvOS 15+, watchOS 8+, and visionOS. The macOS requirement applies only to the machine running Xcode, not to your deployment targets.

Quick Reference: Who Needs to Care About What

If you...	Pay attention to...
Run CI on OS 26.4 devices	Sanitizer hang fix -- upgrade Xcode immediately
Do performance profiling	Run Comparison, Top Functions, Power Profiler core breakdown
Maintain C++ code	Standard library algorithm performance improvements
Ship localized apps	String Catalog editor overhaul, `BUILD_ONLY_KNOWN_LOCALIZATIONS`
Use Swift Testing + XCTest	Interoperability is off by default now -- explicit opt-in required
Debug flaky tests	Unwaited notification expectation fix
Use AI coding features	MCP server overwrite fix, connection dialog fix

Where to Get It

The full release notes are at developer.apple.com/documentation/xcode-release-notes/xcode-26_4-release-notes.

DeerFlow 2.0: What It Is, How It Works, and Why Developers Should Pay Attention

ArshTechPro — Tue, 24 Mar 2026 09:24:15 +0000

ByteDance open-sourced DeerFlow 2.0 on February 27, 2026, and within 24 hours it was sitting at the top of GitHub Trending. The repository has since accumulated around 25,000 stars and 3,000 forks. That kind of adoption is worth understanding, not just celebrating. This article breaks down what DeerFlow actually is, how its architecture fits together, and what it means for you as a developer who builds with or on top of AI systems.

A Brief History: From Research Tool to Runtime

DeerFlow started as an internal deep-research framework at ByteDance, essentially a tool for automating information gathering and summarization. Version 1 did that job well enough that developers outside its intended scope started bending it to do other things: building data pipelines, spinning up dashboards, automating content workflows.

The ByteDance team noticed this and drew a conclusion: DeerFlow was not just a research tool. It was an execution harness waiting to be built properly.

So they did a full rewrite. Version 2.0 shares no code with v1. If you need the original deep-research framework, it lives on the 1.x branch, which still receives contributions. But active development has moved entirely to 2.0, and the architecture is fundamentally different.

What DeerFlow 2.0 Actually Is

DeerFlow calls itself a "SuperAgent harness." That label is doing real work, so it is worth unpacking.

Most agent frameworks give you an agent that produces text. You ask it to research a topic, and it hands you back a string. You ask it to write code, and it gives you a code block. The execution of that code, the transformation of that report into a slide deck, the deployment of that scaffolded application -- all of that is your problem.

DeerFlow closes that gap. It gives the agent an actual computer: an isolated Docker container with a full filesystem, a bash terminal, and the ability to read, write, and execute files. The agent does not suggest a bash command. It runs it. The agent does not sketch a web page. It builds and outputs one.

That is the core shift. DeerFlow is an execution engine, not just a reasoning layer.

The Architecture

DeerFlow is built on LangGraph and LangChain. Here is how the major pieces fit together.

The Lead Agent and Task Decomposition

When you give DeerFlow a complex prompt, such as "Research the top 10 AI startups in 2026 and build me a presentation," it does not try to handle this in a single linear pass.

A lead agent acts as the orchestrator. It breaks the prompt into structured sub-tasks, decides which tasks can run in parallel, spawns sub-agents to handle them, and then synthesizes the results into a coherent output.

Each sub-agent gets its own scoped context, its own tools, and its own termination conditions. Sub-agents run in parallel where possible. One might handle web scraping for funding data. Another might run competitor analysis. A third might generate charts. The lead agent pulls it all together at the end.

This is how DeerFlow handles tasks that take minutes to hours without hitting context limits or losing coherence.

The Sandbox

Each task runs inside an isolated Docker container. This container has:

A persistent filesystem (skills, workspace, uploads, outputs)
A bash terminal
The ability to execute Python scripts and arbitrary shell commands

This is not a simulation. The agent can create files, modify them, run scripts against them, and produce outputs you can actually download and use. That is the distinction from most agent frameworks, which essentially role-play execution.

Skills

DeerFlow's extensibility mechanism is called Skills. A Skill is a Markdown file that defines a workflow, describes best practices, and references supporting resources. The format is intentionally plain: if you can write Markdown, you can write a DeerFlow Skill.

DeerFlow ships with built-in Skills for:

Deep web research
Report generation
Slide deck creation
Web page generation
Image and video generation

You can add your own. A Skill tells the lead agent how to approach a category of task, what tools to use, and what the output should look like. The agent loads relevant Skills progressively, which keeps context consumption manageable.

Memory

DeerFlow has a persistent memory system. It tracks user preferences, writing styles, project structures, and other context across sessions. Memory updates happen asynchronously through a debounced queue so they do not block the main conversation thread.

The project recently added TIAMAT as a cloud memory backend, which suggests ByteDance is thinking beyond local development toward enterprise-scale persistence.

It is worth being clear-eyed about this: persistent memory in agent systems is still an unsolved problem in practice. Confidence scoring on stored facts sounds good in theory and fails in interesting ways in production. The architecture here is thoughtful, but you should verify memory behavior in your own workloads before depending on it.

Model Agnosticism

DeerFlow integrates with any OpenAI-compatible API. You can point it at GPT-4, Claude, Gemini, DeepSeek, or local models via Ollama without changing the agent logic.

The repository recommends Doubao-Seed-2.0-Code (ByteDance's own model), DeepSeek v3.2, and Kimi 2.5 for best results. That recommendation matters: DeerFlow's lead agent needs strong instruction-following and structured output capabilities to decompose tasks properly. Smaller local models will likely struggle with the orchestration layer even if they handle individual sub-tasks acceptably. If you are running local models, start with Qwen 3.5 or DeepSeek before reaching for something smaller.

Getting Started

DeerFlow requires Docker (for the sandbox), Node.js (for the frontend), and Python 3.11+ (for the backend). The general setup path looks like this:

# Clone the repository
git clone https://github.com/bytedance/deer-flow.git
cd deer-flow

# Copy environment config
cp .env.example .env
# Edit .env to add your API keys and model endpoint

# Install dependencies
pip install -r requirements.txt
npm install

# Start the service
docker compose up

The web interface will be available at localhost:3000. You interact with DeerFlow through chat, and it handles the orchestration behind the scenes.

For API integration, DeerFlow exposes a REST interface. You can send a task programmatically and poll for results or stream the output as the agent works through sub-tasks.

What You Can Actually Build With It

Here are some concrete use cases that community members have demonstrated:

Research and report generation. Give DeerFlow a topic and it will search the web, gather sources, generate charts from the data it finds, and produce a formatted report with citations. Not a text summary: an actual document.

Data pipeline automation. DeerFlow can receive a dataset, write Python scripts to clean and transform it, execute those scripts in the sandbox, and return the processed output. The sandbox means this runs isolated from your host environment.

Slide deck creation. Feed it a research brief and it will generate a slide deck, including sourced visuals and structured content. This is a built-in Skill.

Full-stack web application scaffolding. Developers have used DeerFlow to go from a prompt describing an application to a working codebase. The agent writes the code, runs tests, iterates on failures, and returns a working project directory.

Competitive analysis. Spawn multiple sub-agents to research different competitors in parallel, then converge the findings into a comparison document.

Things to Think Carefully About

Security and governance. DeerFlow executes code in Docker containers, fetches external content, and writes to a filesystem. For local development and experimentation, the defaults are fine. For any production or enterprise deployment, you need to think about container hardening, network egress restrictions, and supply chain analysis. The code is MIT-licensed and auditable, which is a genuine advantage here, but auditing takes time and effort.

ByteDance provenance. Depending on your organization or sector, the country-of-origin and ownership context may trigger an additional review process. This is not a technical concern about the code itself. It is an organizational one, and it varies by context. Be aware of it rather than dismissing it.

Local model limitations. If you plan to run DeerFlow with local models to avoid API costs, test the orchestration layer specifically. Task decomposition and sub-agent spawning require strong structured output capabilities. Many smaller models will not handle this reliably.

Memory reliability. The persistent memory system is architecturally thoughtful, but agent memory is still a difficult problem. Do not assume it will recall the right things at the right times without testing against your specific use cases.

How It Compares

DeerFlow occupies a different position than frameworks like LangGraph (which it uses internally), CrewAI, or Microsoft's AutoGen. Those tools give you building blocks. DeerFlow gives you a running system with opinions: a default execution model, built-in Skills, a sandbox, and a memory layer. You can extend it, but you are extending something that already works, not assembling from scratch.

That is a tradeoff. You move faster with DeerFlow's defaults. You also take on its constraints and assumptions. For teams that want to ship an agentic workflow quickly without building all the infrastructure themselves, DeerFlow's opinionated architecture is an asset. For teams with highly specific orchestration requirements, it may feel like friction.

NVIDIA: Training Billion-Parameter Models : A Developer's Guide to Megatron-LM

ArshTechPro — Sat, 21 Mar 2026 19:22:08 +0000

If you have ever tried to train a large language model on a single GPU and watched it crash with an out-of-memory error, you already know the problem. Models that matter today — the ones with tens or hundreds of billions of parameters — simply do not fit on one device. Megatron-LM is NVIDIA's answer to that problem, and it has been quietly powering some of the most serious LLM research and production training runs in the world.

This article walks you through what Megatron-LM actually is, how its internals work, and how to go from zero to a real training run.

What Is Megatron-LM, and Why Should You Care

The repository at github.com/NVIDIA/Megatron-LM contains two things that are worth keeping distinct in your head.

Megatron-LM is the research-oriented layer — it bundles ready-to-run training scripts for GPT, BERT, T5, LLaMA, and multimodal models. If you want to get something running quickly without writing glue code from scratch, this is your starting point.

Megatron Core is the composable library underneath. It exposes the GPU-optimized building blocks (attention layers, parallelism strategies, optimizers, dataset loaders) as importable Python modules so you can assemble your own training framework on top of them. If you are a framework engineer or ML infrastructure developer, this is the layer you actually care about.

The benchmark numbers give you a sense of the scale this is designed for. The codebase has been used to train models ranging from 2B to 462B parameters across thousands of H100 GPUs, achieving up to 47% Model FLOP Utilization. That number matters because MFU is how you measure how efficiently you are using the hardware you paid for.

Understanding the Parallelism — This Is the Core Idea

Before you touch a line of code, you need to understand how Megatron distributes a model that does not fit in memory. There are four strategies, and Megatron uses them in combination.

Tensor Parallelism (TP) splits individual weight matrices across GPUs. A single attention layer's weight matrix gets sliced column-wise or row-wise across multiple devices. During the forward pass, each GPU handles a slice of the computation, and a small all-reduce communication syncs the results. The math is elegant: for a linear layer Y = XA, you can split A column-wise so GPU 0 computes XA_0 and GPU 1 computes XA_1, then concatenate. This is why Megatron can fit a single transformer layer that is too large for one device.

Pipeline Parallelism (PP) assigns different layers to different GPUs. GPU 0 handles layers 0–7, GPU 1 handles layers 8–15, and so on. Data flows through the pipeline in micro-batches, and Megatron's interleaved scheduling keeps the GPUs from sitting idle while waiting for the previous stage to finish.

Data Parallelism (DP) is the familiar one: you run copies of the model on multiple GPUs with different data shards, then average the gradients. Megatron supports both standard DDP and a distributed optimizer that shards optimizer states to reduce memory further.

Context Parallelism (CP) is newer and specifically useful for long sequences. It distributes the sequence dimension across GPUs so you can train on sequences that would otherwise blow your memory budget. The recent Dynamic Context Parallelism feature pushes this further by adapting the parallel size per batch based on actual sequence lengths, yielding up to 1.48x speedup for variable-length training.

Expert Parallelism (EP) is relevant if you are training Mixture-of-Experts models like DeepSeek-V3 or Qwen3. Different experts in the MoE layer live on different GPUs.

In practice you combine these. A typical 70B model training run might use TP=4, PP=4, DP=8, giving you 128 GPUs working together. Megatron handles the communication scheduling so you do not have to wire it up yourself.

Project Structure at a Glance

The repository is well organized once you know where to look:

Megatron-LM/
├── megatron/
│   ├── core/               # The library — import this in your own code
│   │   ├── models/         # GPT, BERT, T5, multimodal
│   │   ├── transformer/    # Attention, MLP, layer building blocks
│   │   ├── tensor_parallel/
│   │   ├── pipeline_parallel/
│   │   ├── distributed/    # FSDP, DDP
│   │   ├── optimizer/
│   │   ├── datasets/
│   │   └── inference/
│   ├── training/           # High-level training scripts
│   └── post_training/      # Quantization, distillation, pruning
├── examples/               # Shell scripts for GPT, LLaMA, Mixtral, etc.
├── tools/                  # Data preprocessing utilities
└── tests/

The split between megatron/core/ and megatron/training/ mirrors the Megatron Core vs Megatron-LM distinction described above. If you are building something custom, spend most of your time in core/. If you are running experiments, the examples/ directory has working shell scripts you can adapt.

Getting It Running

Prerequisites

You need:

One or more NVIDIA GPUs (Ampere or later recommended; H100 for FP8 training)
CUDA 12.x
Python 3.10+
PyTorch 2.x
NCCL (usually installed with CUDA)

The easiest path is NVIDIA's NGC container, which ships with all dependencies pre-installed:

docker pull nvcr.io/nvidia/pytorch:24.01-py3

If you are not using a container, you will also need Transformer Engine for FP8 support and Flash Attention for efficient attention computation.

Installation

# Install Megatron Core with its language model dependencies
pip install --no-build-isolation megatron-core[mlm,dev]

# Clone the repo for examples and training scripts
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
pip install --no-build-isolation .[mlm,dev]

The --no-build-isolation flag matters here. Megatron builds some CUDA extensions during install and needs access to your environment's CUDA headers.

Your First Training Run

Once installed, the simplest way to verify everything works is:

# Run a distributed training loop on 2 GPUs with mock data
torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py

If you want to try something closer to a real use case, the LLaMA-3 example uses 8 GPUs with FP8 precision:

./examples/llama/train_llama3_8b_fp8.sh

These scripts handle the argument wiring for you. Once you understand what the flags mean, you can start modifying them.

Data Preparation

This is where most people get tripped up the first time. Megatron does not consume raw text files. It needs data preprocessed into a binary indexed format (.bin + .idx files) for memory-mapped, high-throughput loading during training.

Step 1: Format Your Data as JSONL

Each line is a JSON object with a text field:

{"text": "Your first training document goes here."}
{"text": "Each document is a separate JSON line."}
{"text": "The tokenizer will handle the rest."}

Step 2: Run the Preprocessor

python tools/preprocess_data.py \
    --input data.jsonl \
    --output-prefix /path/to/processed_data \
    --tokenizer-type HuggingFaceTokenizer \
    --tokenizer-model /path/to/your/tokenizer \
    --workers 8 \
    --append-eod

Key flags to know:

--output-prefix: The path prefix for the .bin and .idx output files. Pass this same prefix to --data-path in your training script.
--tokenizer-type: Use HuggingFaceTokenizer for any tokenizer that follows the Hugging Face interface. GPT2BPETokenizer is available for GPT-2's original BPE tokenizer.
--workers: Parallelizes tokenization. Set this to the number of CPU cores you can spare.
--append-eod: Adds an end-of-document token between documents. Almost always what you want.

This step can take a while for large datasets. The output is worth the wait — the indexed binary format lets training data loaders access tokens at random offsets in O(1) time, which is critical when you have trillions of tokens.

Writing a Custom Training Loop with Megatron Core

If you want to plug Megatron's parallelism into your own training code rather than using the provided scripts, here is the minimal structure. This is what examples/run_simple_mcore_train_loop.py does under the hood.

import torch
from megatron.core import parallel_state
from megatron.core.models.gpt import GPTModel
from megatron.core.transformer.transformer_config import TransformerConfig

# Initialize distributed and parallelism groups
parallel_state.initialize_model_parallel(
    tensor_model_parallel_size=2,
    pipeline_model_parallel_size=1,
)

# Define your model configuration
config = TransformerConfig(
    num_layers=12,
    hidden_size=768,
    num_attention_heads=12,
    use_cpu_initialization=True,
    pipeline_dtype=torch.float16,
)

# Build the model — Megatron handles sharding based on parallel_state
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
model = GPTModel(
    config=config,
    transformer_layer_spec=get_gpt_layer_local_spec(),
    vocab_size=50257,
    max_sequence_length=1024,
)

# From here, training looks mostly like standard PyTorch
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)

The key thing to notice is that after initialize_model_parallel, the model construction is aware of which GPU it is running on. When you call GPTModel(...), Megatron automatically places the right layers on the right devices based on your TP and PP settings. You do not have to manually slice weight matrices.

Parallelism Configuration in Practice

When you run the provided shell scripts, you will see flags like these:

--tensor-model-parallel-size 4 \
--pipeline-model-parallel-size 4 \
--num-layers 32 \
--hidden-size 4096 \
--num-attention-heads 32 \
--micro-batch-size 2 \
--global-batch-size 1024 \
--seq-length 4096 \

A few rules of thumb:

Tensor parallel size should evenly divide num-attention-heads and the hidden size's intermediate MLP size. For a model with 32 attention heads, TP sizes of 1, 2, 4, 8, or 16 all work.

Pipeline parallel size should evenly divide num-layers. 32 layers with PP=4 means 8 layers per pipeline stage.

Global batch size equals micro-batch-size * gradient-accumulation-steps * data-parallel-size. Megatron enforces this math and will error if your settings are inconsistent.

Communication overlap flags are worth enabling once things are working. These flags let Megatron overlap gradient reduction with backward computation:

--overlap-grad-reduce \
--overlap-param-gather \
--tp-comm-overlap

They typically improve throughput by 5–15% at large scale.

Checkpoint Conversion: Getting In and Out of the Megatron Format

One practical concern is that Megatron's checkpoint format is not the same as Hugging Face's. If you want to start from a pretrained HuggingFace model or publish your trained weights back, you need to convert.

The Megatron Bridge project handles this bidirectionally. It supports popular models and is the recommended path for production checkpoint management.

For LLaMA specifically, the Megatron-LM repository includes conversion scripts under tools/ that have been used for Llama, Mistral, and other Llama-derived architectures.

FP8 Training

If you are running on H100 or later GPUs, FP8 training is worth trying. It reduces memory usage and can increase throughput significantly because Tensor Cores on Hopper are much faster in FP8 than BF16.

Enabling it is mostly a matter of setting flags:

--fp8-format hybrid \
--fp8-amax-history-len 1024 \
--fp8-amax-compute-algo max

You also need Transformer Engine installed, which handles the actual FP8 kernels. The train_llama3_8b_fp8.sh example shows a working configuration if you want to see all the pieces together.

MoE Models

The MoE (Mixture of Experts) support in Megatron Core has been one of the most active development areas recently. DeepSeek-V3 and Qwen3-style MoE architectures are explicitly supported, and there is an active roadmap for further optimizations through 2025.

The key additional parallelism dimension for MoE is Expert Parallelism:

--expert-model-parallel-size 8 \
--num-experts 64 \
--moe-router-topk 2

With 64 experts and EP=8, each GPU holds 8 experts. During the forward pass, tokens are routed to their top-2 experts, and the MoE communication layer handles moving tokens across GPU boundaries as needed.

Summary

The thing that makes Megatron-LM worth learning is not any single feature — it is that the entire system is designed around the constraint that you are always working at the edge of what the hardware can do.

NemoClaw: NVIDIA's Open Source Stack for Running AI Agents You Can Actually Trust

ArshTechPro — Thu, 19 Mar 2026 11:30:55 +0000

AI agents have crossed a threshold. They're no longer chatbots that answer questions and forget you exist. The new generation can remember context across sessions, spawn sub-agents, write their own code to learn new skills, and keep executing tasks long after you close your laptop. Tools like OpenClaw have made it possible for a single developer to spin up an autonomous assistant that works like a small team.

That's exciting. It's also terrifying if you think about it for more than five seconds.

A long-running agent with persistent shell access, live credentials, and the ability to rewrite its own tooling is a fundamentally different threat model than a stateless chatbot. Every prompt injection becomes a potential credential leak. Every third-party skill the agent installs is an unreviewed binary with filesystem access. Every sub-agent it spawns can inherit permissions it was never meant to have.

The agents are ready. The infrastructure to trust them has been missing — until now.

What Is NemoClaw?

NemoClaw is an open source stack from NVIDIA that wraps OpenClaw (the popular always-on AI assistant) with enterprise-grade privacy and security controls. It's built on top of two key NVIDIA projects:

OpenShell — an open source runtime (part of the NVIDIA Agent Toolkit) that acts as a governance layer between your agent and your infrastructure. Think of it as a browser sandbox, but for AI agents. It controls what the agent can see, do, and where its inference requests go.

Nemotron — NVIDIA's family of open models that can run locally on your own hardware for enhanced privacy and cost efficiency.

The whole point: you get the productivity of autonomous agents without giving up control.

Why Should You Care?

If you're building with or deploying AI agents, you've likely hit the "trust trilemma." You need three things simultaneously: safety, capability, and autonomy. With existing approaches, you can only reliably get two at a time.

Safe + autonomous but the agent can't access the tools and data it needs → it can't finish the job.
Capable + safe but gated on constant approvals → you're babysitting it.
Capable + autonomous with full access → a long-running process policing itself, with guardrails living inside the same process they're supposed to guard.

That last scenario is the critical failure mode. Tools like Claude Code and Cursor ship with valuable internal guardrails, but those protections live inside the agent. A compromised agent can potentially override them.

NemoClaw solves this by moving the control point outside the agent entirely. The agent literally cannot override the security policies because they're enforced at the infrastructure level, not the prompt level.

How It Works Under the Hood

NemoClaw's architecture has four main components:

1. The Plugin (CLI)

A TypeScript CLI that orchestrates everything. You use nemoclaw commands on your host machine to launch, connect to, and manage sandboxed agents.

2. The Blueprint

A versioned Python artifact that handles sandbox creation, policy configuration, and inference setup. It follows a four-stage lifecycle: resolve the artifact → verify its digest → plan resources → apply through the OpenShell CLI.

3. The Sandbox

This is where the magic happens. It's not generic container isolation — it's purpose-built for long-running, self-evolving agents. The sandbox provides:

Landlock + seccomp + network namespace isolation — the agent runs in a locked-down environment at the OS level.
Filesystem restrictions — the agent can only read/write inside /sandbox and /tmp. Everything else is off-limits.
Network egress control — unauthorized outbound connections are blocked. If the agent tries to reach an unlisted host, OpenShell blocks it and surfaces the request for your approval.
Process protection — privilege escalation and dangerous syscalls are blocked at sandbox creation time.
Live policy updates — network and inference policies can be hot-reloaded at runtime as you approve new permissions.

4. The Inference Layer

Inference requests from the agent never leave the sandbox directly. OpenShell intercepts every call and routes it through a privacy router. This router decides — based on your policy, not the agent's preferences — whether a request goes to a local model (like Nemotron running on your GPU) or to a cloud-based frontier model.

The default setup routes through NVIDIA's cloud API using nvidia/nemotron-3-super-120b-a12b. Local inference via Ollama and vLLM is experimental but available.

Getting Started (One Command, Seriously)

Prerequisites

You need a Linux machine (Ubuntu 22.04+) with Docker, Node.js 20+, and at least 8 GB of RAM (16 GB recommended). macOS is supported via Colima or Docker Desktop on Apple Silicon.

Install and Onboard

curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

That's it. The script installs Node.js if needed, then runs a guided onboarding wizard that creates a sandbox, configures inference, and applies security policies. It'll prompt you for an NVIDIA API key (grab one free from their website).

When it finishes, you'll see something like:

──────────────────────────────────────────────────
Sandbox      my-assistant (Landlock + seccomp + netns)
Model        nvidia/nemotron-3-super-120b-a12b (NVIDIA Cloud API)
──────────────────────────────────────────────────
Run:         nemoclaw my-assistant connect
Status:      nemoclaw my-assistant status
Logs:        nemoclaw my-assistant logs --follow
──────────────────────────────────────────────────

Connect and Chat

# Open a shell inside the sandbox
nemoclaw my-assistant connect

# Launch the interactive TUI
sandbox@my-assistant:~$ openclaw tui

# Or use the CLI for a single message
sandbox@my-assistant:~$ openclaw agent --agent main --local -m "hello" --session-id test

Key Commands Cheat Sheet

Command	What It Does
`nemoclaw onboard`	Interactive setup wizard
`nemoclaw <name> connect`	Shell into a sandbox
`nemoclaw <name> status`	Check sandbox health
`nemoclaw <name> logs --follow`	Stream logs
`nemoclaw start` / `stop`	Manage auxiliary services
`openshell term`	Launch OpenShell TUI for monitoring

The Protection Model Explained

Here's what's actually enforced — and when:

Layer	What It Guards	When Applied
Network	Blocks unauthorized outbound connections	Hot-reloadable at runtime
Filesystem	No reads/writes outside `/sandbox` and `/tmp`	Locked at sandbox creation
Process	Blocks privilege escalation and dangerous syscalls	Locked at sandbox creation
Inference	Reroutes model API calls to controlled backends	Hot-reloadable at runtime

The critical design choice: filesystem and process restrictions are locked at creation time. The agent can't unlock them mid-session, even if compromised. Network and inference policies can be updated live — but only by you, from outside the sandbox.

When the agent hits a constraint, it can reason about why it's blocked and propose a policy update. You see the request in the OpenShell TUI and make the final call. Full audit trail of every allow and deny decision.

What Makes This Different from Just Using Docker?

Fair question. You could run an agent in a Docker container with restrictive policies. But NemoClaw/OpenShell gives you several things Docker alone doesn't:

Agent-aware policy engine — it evaluates actions at the binary, destination, method, and path level. An agent can install a verified skill but can't execute an unreviewed binary.
Privacy routing — inference is intercepted and routed based on policy, keeping sensitive context on-device when needed.
Live policy updates — approve new network destinations or inference providers without restarting the sandbox.
Skill verification — as the agent evolves and learns new capabilities, each new skill is subject to the same controls.
Operator approval flow — blocked actions surface in a TUI for human review, not just silent failures.

Hardware Support

NemoClaw runs on a range of NVIDIA hardware:

GeForce RTX PCs/laptops — your everyday dev machine with a GPU
RTX PRO workstations — for heavier local inference workloads
DGX Spark — NVIDIA's compact AI workstation
DGX Station — for enterprise-scale local deployment

Current Limitations (It's Alpha)

The project is in alpha, and the README says so clearly. A few things to be aware of:

Interfaces, APIs, and behavior may change without notice.
The openclaw nemoclaw plugin commands are under active development — use the nemoclaw host CLI as the primary interface.
Local inference (Ollama, vLLM) is experimental, especially on macOS.
On machines with less than 8 GB RAM, the sandbox image (~2.4 GB compressed) can trigger the OOM killer during setup. Adding swap helps.
Setup may require manual workarounds on some platforms.

Where to Go from Here

GitHub repo: github.com/NVIDIA/NemoClaw — 9k+ stars, Apache 2.0 licensed

MiroFish: The Open-Source AI Engine That Builds Digital Worlds to Predict the Future

ArshTechPro — Sat, 14 Mar 2026 23:41:35 +0000

MiroFish is an open-source AI prediction engine that takes real-world data (news, reports, even novels), spawns thousands of AI agents with unique personalities and memories, lets them interact in a simulated world, and produces a prediction report based on what emerges. Think of it as SimCity meets AI forecasting.

What Problem Does MiroFish Solve?

Traditional prediction models — whether statistical or ML-based — treat the world like a math equation. You feed in numbers, you get numbers out. But the real world doesn't work that way. People react to each other. Opinions shift. Coalitions form and break apart. A single tweet can change the trajectory of a news cycle.

MiroFish takes a fundamentally different approach. Instead of crunching numbers, it simulates the messy, social dynamics of the real world using thousands of AI agents that talk, argue, persuade, and evolve — just like people do.

The result? You get a prediction that accounts for group behavior, social contagion, and emergent patterns that traditional models simply can't capture.

How It Actually Works (The 5-Step Pipeline)

Here's the workflow, broken down without the buzzwords:

Step 1: Knowledge Graph Construction

You upload "seed material" — this could be a news article, a financial report, a policy document, or even the first 80 chapters of a novel (yes, they actually did this with Dream of the Red Chamber to predict its lost ending).

MiroFish uses GraphRAG (Graph-based Retrieval Augmented Generation) to parse your input and extract entities and relationships. Instead of treating your document as a flat bag of text, it builds a structured knowledge graph — who are the key players, how are they connected, what pressures exist, what institutions are involved.

This graph becomes the "reality" that the simulated world is built on.

Step 2: Environment Setup & Agent Creation

Based on the knowledge graph, MiroFish automatically generates agent personas. Each agent gets:

A unique personality and background
A distinct stance or perspective on the topic
Long-term memory (powered by Zep Cloud)
Behavioral logic that governs how they interact

An "Environment Configuration Agent" then sets up the simulation parameters — essentially deciding the rules of the world these agents will live in.

Step 3: Dual-Platform Parallel Simulation

This is where things get interesting. MiroFish runs simulations on two platforms simultaneously (think Twitter-like and Reddit-like environments). Dozens or hundreds of agents start interacting — posting, commenting, debating, forming opinions, influencing each other.

The simulation engine under the hood is OASIS (Open Agent Social Interaction Simulations), built by the CAMEL-AI team. OASIS can scale up to one million agents and supports 23 different social actions (following, commenting, reposting, etc.).

During the simulation, the system automatically tracks your prediction question and dynamically updates each agent's memory as events unfold.

Step 4: Report Generation

After the simulation ends, a dedicated ReportAgent steps in. This agent has access to a rich toolkit and interacts with the post-simulation environment to synthesize everything that happened. It analyzes how agents' opinions shifted, what coalitions formed, and what patterns emerged — then produces a structured prediction report.

Step 5: Deep Interaction

The report isn't the final product. You can:

Chat with any agent in the simulated world to understand their reasoning
Talk to the ReportAgent to ask follow-up questions or get alternative analyses
Inject new variables and re-run scenarios ("What if we change X?")

The Tech Stack

Here's what's under the hood:

Component	Technology
Backend	Python 3.11+
Frontend	Vue.js
Simulation Engine	OASIS (by CAMEL-AI)
Knowledge Graphs	GraphRAG
Agent Memory	Zep Cloud
LLM Support	Any OpenAI SDK-compatible model
Recommended LLM	Qwen-plus (via Alibaba's Bailian platform)
Package Manager	uv (for Python)

Getting Started (Self-Hosted Setup)

Note: MiroFish was developed and tested on macOS. Windows compatibility is still being tested.

Prerequisites

Node.js 18+
Python 3.11+
uv (Python package manager)

1. Clone and Configure

git clone https://github.com/666ghj/MiroFish.git
cd MiroFish

# Copy the example env file
cp .env.example .env

Edit the .env file with your API keys:

# LLM Configuration (any OpenAI SDK-compatible LLM)
# Recommended: Qwen-plus on Alibaba's Bailian platform
LLM_API_KEY=your_api_key
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_MODEL_NAME=qwen-plus

# Zep Cloud (for agent memory persistence)
# Free tier is enough for basic usage: https://app.getzep.com/
ZEP_API_KEY=your_zep_api_key

2. Install Dependencies

# One command to install everything (root + frontend + backend)
npm run setup:all

Or step by step:

# Node dependencies (root + frontend)
npm run setup

# Python dependencies (auto-creates virtual environment)
npm run setup:backend

3. Run It

# Start both frontend and backend
npm run dev

That's it. Your frontend will be at http://localhost:3000 and the API at http://localhost:5001.

You can also start them separately:

npm run backend   # Backend only
npm run frontend  # Frontend only

What Can You Actually Predict With This?

The team has demonstrated several use cases:

Public Opinion Simulation — Feed in a news event and simulate how public sentiment might evolve. The demo shows a prediction of how a university controversy might unfold across social media.

Financial Forecasting — Inject market signals and watch how simulated traders, analysts, and retail investors react to each other's moves.

Policy Impact Testing — Upload a policy draft and see how different stakeholder groups might respond, form alliances, or push back.

Creative Exploration — The team fed the first 80 chapters of a classic Chinese novel into MiroFish and had it predict the lost ending based on how the characters would behave. This is a fun one — it shows the engine isn't limited to "serious" forecasting.

Important Caveats

Let's be real about what this is and isn't:

It's not a crystal ball. The team hasn't published benchmarks comparing predictions against actual outcomes. The simulations illustrate plausible scenarios based on emergent agent behavior — they're not probability estimates.

LLM costs add up. Running hundreds of agents through multiple simulation rounds means lots of LLM API calls. The README recommends starting with fewer than 40 rounds to manage costs.

Agent bias matters. The OASIS research paper notes that LLM agents tend to be more susceptible to herd behavior than real humans. Simulated crowds can polarize faster than real ones.

It's early. Version 0.1.0 was released in December 2025. This is a v0 product — powerful in concept, but still maturing.

The Backstory

MiroFish was built by Guo Hangjiang, a senior undergraduate student in China. It topped GitHub's Global Trending list in March 2026 and has attracted investment from Shanda Group founder Chen Tianqiao. The project's predecessor, BettaFish (a multi-agent public opinion analysis tool), also hit #1 on GitHub Trending in late 2024.

The core simulation engine comes from OASIS, an open-source project by the CAMEL-AI research community that supports up to one million agent interactions and has been published in peer-reviewed research.

Why Developers Should Care

Even if you're not building a prediction engine, MiroFish is worth studying because it's a clean example of several patterns coming together:

GraphRAG for knowledge grounding — how to give agents structured context, not just raw text
Persistent agent memory — using Zep to let agents remember across simulation rounds
Multi-agent orchestration at scale — coordinating hundreds of autonomous agents in real-time
Emergent behavior as a feature — designing systems where the output isn't programmed but emerges from agent interactions

These are patterns you'll see increasingly in production AI systems, and MiroFish packages them in a way that's easy to study and experiment with.

Core ML vs Foundation Models: Which Should You Use?

ArshTechPro — Wed, 25 Feb 2026 12:16:03 +0000

With iOS 26.3 now in the wild, iOS developers have two powerful on-device AI frameworks to choose from: Core ML — Apple's veteran ML inference engine — and Foundation Models — the new framework that exposes Apple's ~3B parameter LLM from iOS 26 onwards.

They sound like they do similar things. They don't.

This article cuts through the confusion, explains what each framework is actually designed for, and gives you a clear decision framework so you can pick the right tool for the job — or know when to combine both.

What Each Framework Actually Is

Before comparing them, let's be precise about what they are.

Core ML has been available since iOS 11. It's a general-purpose inference engine — you bring a trained model (in .mlmodel or .mlpackage format), and Core ML runs it on-device using the best available hardware: Neural Engine, GPU, or CPU depending on the task. Core ML itself doesn't contain any models. You either train your own (with Create ML), convert from TensorFlow or PyTorch, or download pre-trained ones. It supports image classification, object detection, NLP, audio analysis, tabular data prediction, and more.

Foundation Models was introduced in iOS 26. It's an API that gives you direct access to Apple's own pre-trained ~3B parameter large language model — the same one behind Apple Intelligence. You don't bring a model. The model is already on the device (downloaded as part of enabling Apple Intelligence). The framework specialises in natural language: text generation, summarisation, structured data extraction, and tool calling.

One is a runtime that runs your models. The other is an API for a specific pre-built Apple model.

Device Availability: A Critical Difference

This is where the two frameworks diverge most sharply and it matters enormously for your architecture decisions.

Core ML works on every device that runs iOS 11 and later. On iOS 26, that means any iPhone 11 or newer (A13 chip and up). It runs on essentially your entire user base.

Foundation Models requires Apple Intelligence, which means iPhone 15 Pro/Max or any iPhone 16 or 17 model. An iPhone 14, 15 (standard), or anything older simply cannot use it, regardless of which iOS version it runs. On top of the device requirement, Apple Intelligence must be enabled by the user, 7 GB of free storage is required, and the on-device model needs to finish downloading after Apple Intelligence is first enabled.

If your feature needs to work for the majority of your users today, Core ML is the safer choice. If you're building a feature that's an enhancement for users on newer devices, Foundation Models is a compelling option.

What Each Framework Is Good At

Think of the two frameworks as operating in completely different problem spaces.

Core ML: Structured ML Tasks

Core ML excels when your task is well-defined, has a clear input/output structure, and can be solved with a trained model:

Image classification — "What object is in this photo?"
Object detection — "Where are the faces/products/items in this frame?"
Pose estimation — "Where are this person's joints?"
Audio classification — "Is this the sound of a dog or a car?"
Text classification / sentiment analysis — "Is this review positive or negative?"
Tabular prediction — "Based on these health metrics, what category does this fall into?"
Real-time video analysis — Frame-by-frame inference at high frequency

A key characteristic of Core ML tasks: you can define the exact output schema up front, and the model reliably produces a label, a bounding box, a confidence score, or a numeric prediction. Deterministic, fast, and predictable.

Foundation Models: Language and Reasoning Tasks

Foundation Models is designed for tasks that involve language understanding and generation:

Text summarisation — turning long content into concise summaries
Structured extraction — pulling structured data from unstructured text (e.g. extracting a name, date, and location from a messy user note)
Content classification using natural language — not just labels, but nuanced categorisation with explanation
Contextual suggestions — generating personalised recommendations based on user context
Tool-augmented reasoning — letting the model call into your app's data to answer user questions

Apple itself explicitly warns that the on-device 3B model is not designed for world-knowledge Q&A, code generation, or complex maths. It's optimised for task-oriented, app-integrated intelligence.

When to Use Core ML

Use Core ML when:

You need vision or audio inference. Core ML is the only on-device option for camera-based features — real-time object detection, face analysis, pose estimation, scene classification. Foundation Models cannot process images at all (no image input support as of iOS 26.3).

You need it to work on older devices. If your feature must work on an iPhone 12, 13, or 14, Core ML is your only on-device option. It runs on any device with iOS 11+.

You have a specific, narrow ML task. A model trained to classify 10 types of skin lesions, or to detect a specific product in a frame, will outperform a general 3B LLM on that narrow task — and at a fraction of the memory and compute cost.

You need deterministic, repeatable outputs. Core ML models return the same output for the same input, every time. Foundation Models, being a generative LLM, produces varied responses.

You're doing real-time inference. Core ML can process frames from a camera feed at 30+ fps. Foundation Models is not suited for frame-by-frame tasks.

When to Use Foundation Models

Use Foundation Models when:

Your task is fundamentally a language task. Summarising a document, extracting key facts from a user's note, generating a personalised caption — these are natural fits.

You want structured output from unstructured text. The @Generable macro lets you extract type-safe Swift structs directly from free-form input. No JSON parsing, no regex, no post-processing.

You need natural language reasoning with tool calling. Foundation Models can decide when to call into your app's data, fetch it, and incorporate it into a response. Core ML models can't reason about when or whether to request more context.

You want zero model maintenance. With Core ML, you own the model — you retrain it, update it, and deal with drift over time. With Foundation Models, Apple maintains the base model. You get improvements with OS updates for free.

Speed of integration matters. Building a Core ML-powered feature involves choosing a model architecture, training data, training, conversion, and integration. Foundation Models can be integrated in an afternoon.

The Combination Pattern

Here's something worth noting: the two frameworks aren't competitors. They're complementary layers, and the most powerful apps combine them.

Apple highlighted a real example of this with SwingVision, a tennis/pickleball coaching app. It uses Core ML to analyse video frames and extract structured data about a user's movement and technique. It then feeds that structured output as context into a Foundation Models session to generate natural language coaching feedback.

This is the pattern:

Core ML handles the perception layer — processing images, audio, or sensor data into structured signals
Foundation Models handles the reasoning and language layer — turning those signals into meaningful, natural language insights

A fitness app might use Core ML for pose estimation during a workout, then pass rep counts and form data to Foundation Models to generate a personalised summary. A cooking app might use Core ML to identify ingredients from a photo, then Foundation Models to suggest a recipe.

// Step 1: Core ML classifies the image
let classificationRequest = VNCoreMLRequest(model: mlModel) { request, _ in
    guard let result = request.results?.first as? VNClassificationObservation else { return }

    // Step 2: Feed the structured output into Foundation Models
    Task {
        let session = LanguageModelSession()
        let response = try await session.respond(
            to: "The user photographed: \(result.identifier) with \(Int(result.confidence * 100))% confidence. Suggest what they could make with this."
        )
        print(response.content)
    }
}

The Decision Framework

Here's a straightforward way to pick:

Is your task visual, audio-based, or real-time? → Core ML

Do you need it to work on iPhone 14 or older? → Core ML

Is your task text-only — generation, extraction, summarisation, or reasoning? → Foundation Models

Do you need tight control over the model's behaviour for a narrow domain? → Core ML with a custom-trained model

Do you want to ship an AI feature today with minimal setup on iOS 26 devices? → Foundation Models

Do you need both vision and natural language? → Core ML for perception + Foundation Models for language

Summary

Core ML and Foundation Models solve different problems. Core ML is a mature, flexible inference engine for deterministic ML tasks across all your users. Foundation Models is a purpose-built API for language tasks on Apple Intelligence-compatible devices, with almost zero setup cost.

Requirements:

Core ML: iOS 11+ (any iPhone 11+ for iOS 26)
Foundation Models: iOS 26 + Apple Intelligence-compatible device (iPhone 15 Pro or newer, all iPhone 16/17 models)

Xcode 26.4 Beta: Smaller Changes, Real Developer Impact

ArshTechPro — Mon, 23 Feb 2026 10:46:14 +0000

Apple quietly released Xcode 26.4 beta, bringing updated SDKs and a set of practical improvements — especially around testing and localization workflows.

Unlike Xcode 26.3, which introduced agentic coding, 26.4 is a refinement release. No big headline features — but a lot of small improvements that remove friction from day-to-day development.

Here’s what developers should actually care about.

Updated SDKs

Xcode 26.4 includes SDK updates for all platforms:

Swift Testing Keeps Getting Better

Swift Testing continues to evolve and Xcode 26.4 makes it much more practical for real-world debugging.

Attach Images to Tests

You can now attach images directly to Swift tests:

CGImage
NSImage
UIImage
CIImage

This is especially useful for:

Snapshot testing
UI verification
Rendering tests
Vision / image pipelines

Instead of logging text failures, you can now see exactly what went wrong.

For UI-heavy apps, this is a big quality-of-life improvement.

Severity Levels for Test Issues

Swift Testing now lets you record issues with a specific severity level.

Instead of everything being a hard failure, you can now distinguish between:

Warnings
Non-critical issues
Actual failures

This makes test reporting more realistic for large test suites.

Not every issue needs to break CI.

Better Error Reporting for Attachments

If Xcode fails to save a test attachment, it now shows up as a runtime issue.

Previously this could fail silently, which made debugging test infrastructure harder.

Now you'll know immediately if something went wrong.

UI Test Crash Reports Are Easier to See

When an app crashes during UI testing via:

XCUIApplication(bundleIdentifier:)
XCUIApplication(url:)

Xcode now:

Reports the crash as a warning
Attaches the crash log automatically

This removes a lot of manual digging through DerivedData.

Small change — big time saver.

Mixing XCTest and Swift Testing Is Safer

Many projects are gradually migrating from XCTest to Swift Testing.

Xcode 26.4 improves this transition.

If you call an assertion from the wrong framework:

XCTest inside Swift Testing
Swift Testing inside XCTest

Xcode now reports a runtime warning instead of failing silently.

This makes mixed test suites much easier to maintain.

Localization Improvements (Finally)

If you've used String Catalogs for localization, you've probably noticed that editing them could feel limited.

Xcode 26.4 fixes several long-standing pain points.

Removing Languages Is Now Easy

You can now remove languages directly from the String Catalog editor.

Even better, you can choose:

Remove from just this catalog
Remove from the entire project

Previously this required manual cleanup.

Now it's a one-click operation.

Pre-Fill Translations When Adding Languages

When adding a new supported language in Project Settings, Xcode can now:

Pre-fill translations using an existing language.

This is surprisingly useful when:

Creating regional variants
Bootstrapping translations
Working with external translators

Instead of starting from empty catalogs, you get a usable baseline.

Copy & Paste Support for String Catalogs

String Catalog editing finally behaves like a normal editor.

You can now:

Cut
Copy
Paste
Duplicate strings

And this works:

Within a catalog
Between catalogs

When pasting strings, you can choose:

Add as a new key with translations
Apply translations to an existing key

If you've ever reorganized large catalogs, this will save a lot of time.

Build and Compiler Improvements

Like most minor Xcode updates, 26.4 includes compiler and build system improvements:

Better diagnostics
Improved stability
Fewer unexpected build issues

These updates rarely make headlines but usually improve everyday development.

Large projects tend to benefit the most.

Summary

Xcode 26.4 isn’t a flashy release — but it improves areas developers use every day:

Better Swift Testing
Easier localization
Updated SDKs
More stable builds

The Testing and Localization improvements alone make this a worthwhile upgrade for many teams.

If Xcode 26.3 introduced new workflows, 26.4 makes the existing ones smoother.

How to Fall Back Gracefully When Apple Intelligence Isn't Available

ArshTechPro — Sat, 21 Feb 2026 00:24:36 +0000

Apple Intelligence is one of the most exciting things to happen to iOS development in years. The Foundation Models framework gives you direct access to an on-device LLM with zero API costs, zero network calls, and full privacy. But here's the hard truth: a huge chunk of your users can't run it.

If you just drop a LanguageModelSession() into your app without any checks, you'll ship broken experiences to a large portion of your user base. This article is about how to handle that properly — detecting unavailability, communicating it clearly to users, and falling back gracefully so your app stays useful for everyone.

Who Can't Use Apple Intelligence?

Let's be precise about this, because the numbers matter.

iOS 26.3 (released February 11, 2026) runs on iPhone 11 and later — that's any device with an A13 chip or newer. But Apple Intelligence requires an A17 Pro chip or newer, which means only:

iPhone 15 Pro / iPhone 15 Pro Max
iPhone 16 / 16 Plus / 16 Pro / 16 Pro Max
iPhone 16e
iPhone 17 / 17 Pro / 17 Pro Max / iPhone Air

So an iPhone 15 (standard) running iOS 26.3 supports all the Liquid Glass UI changes, but gets zero Foundation Models access. Same for anything older. On top of that, even eligible devices need:

Apple Intelligence enabled in Settings (it's opt-in)
7 GB of free storage on the device
Device and Siri language set to a supported language
The model fully downloaded (it downloads in the background after enabling)

Bottom line: even among users on supported hardware, not everyone will have Apple Intelligence ready to go. You cannot assume availability.

The Three Unavailability Cases

The Foundation Models framework gives you exactly three reasons why the model might not be available, surfaced through SystemLanguageModel.default.availability:

import FoundationModels

switch SystemLanguageModel.default.availability {
case .available:
    // Good to go
case .unavailable(.deviceNotEligible):
    // A13 or older chip — Foundation Models will never work here
case .unavailable(.appleIntelligenceNotEnabled):
    // Compatible device, but user hasn't turned on Apple Intelligence
case .unavailable(.modelNotReady):
    // Compatible + enabled, but model is still downloading
@unknown default:
    // Future-proof: handle any new cases Apple might add
    break
}

Each case needs a different response from your app. They are not all the same problem.

Building a Proper Fallback Strategy

Think of these three cases as three separate UX problems.

Case 1: Device Not Eligible

This is permanent. The user's hardware will never support Apple Intelligence. Don't show them a spinner. Don't show a "check back later" message. Show them a functional experience that doesn't rely on the model at all.

case .unavailable(.deviceNotEligible):
    // Serve a non-AI version of the feature
    showBasicTextSummarizer()

What "basic version" means depends on your feature. For a smart journaling app, you might skip auto-tagging and let the user tag manually. For a writing assistant, you might offer simpler preset templates instead of generated suggestions. The key is: the feature should still work, just without the AI enhancement.

Case 2: Apple Intelligence Not Enabled

This one is different — the hardware supports it, but the user hasn't opted in. You can prompt the user to enable it. But be thoughtful about how you do this. Don't block the UI. Don't show it repeatedly. Show it once, explain the benefit clearly, and deep link directly to the setting.

case .unavailable(.appleIntelligenceNotEnabled):
    showEnablementBanner(
        message: "Enable Apple Intelligence in Settings to unlock AI-powered suggestions.",
        settingsURL: URL(string: UIApplication.openSettingsURLString)!
    )
    // Still show the basic version of the feature below the banner

Apple Intelligence is enabled at Settings → Apple Intelligence & Siri. You can open Settings directly with UIApplication.openSettingsURLString, but you can't deep link to that exact screen — the user has to navigate there themselves.

Case 3: Model Not Ready

This is temporary. The model is downloaded in the background after a user enables Apple Intelligence, and it can take a while. The right response here is to wait and retry — not to permanently fall back.

case .unavailable(.modelNotReady):
    showLoadingState(message: "AI features are warming up. This only takes a moment.")
    scheduleAvailabilityCheck()

For the retry logic, you can periodically re-check SystemLanguageModel.default.availability. A simple approach is to use a Timer or Task with a delay:

func scheduleAvailabilityCheck() {
    Task {
        try? await Task.sleep(for: .seconds(10))
        await checkAndUpdateAvailability()
    }
}

Don't poll too aggressively — once every 10–30 seconds is fine while the user is actively in that screen.

Putting It Together: A Clean Architecture

Here's a practical pattern that keeps your feature code clean and handles all three cases from a single place.

import FoundationModels
import SwiftUI

enum AIAvailabilityState {
    case available
    case unsupportedDevice
    case notEnabled
    case modelLoading
}

@Observable
class AIFeatureManager {
    private(set) var state: AIAvailabilityState = .modelLoading

    init() {
        refreshAvailability()
    }

    func refreshAvailability() {
        switch SystemLanguageModel.default.availability {
        case .available:
            state = .available
        case .unavailable(.deviceNotEligible):
            state = .unsupportedDevice
        case .unavailable(.appleIntelligenceNotEnabled):
            state = .notEnabled
        case .unavailable(.modelNotReady):
            state = .modelLoading
            scheduleRetry()
        @unknown default:
            state = .unsupportedDevice
        }
    }

    private func scheduleRetry() {
        Task {
            try? await Task.sleep(for: .seconds(15))
            refreshAvailability()
        }
    }
}

And in your view:

struct SmartFeatureView: View {
    @State private var aiManager = AIFeatureManager()

    var body: some View {
        switch aiManager.state {
        case .available:
            AIEnhancedView()
        case .unsupportedDevice:
            BasicFallbackView()
        case .notEnabled:
            EnablePromptView {
                aiManager.refreshAvailability()
            }
        case .modelLoading:
            LoadingView(message: "AI features are getting ready...")
        }
    }
}

This keeps your views thin. Each state gets its own view. And when the model becomes available, refreshAvailability() updates the state and SwiftUI re-renders automatically.

What Your Fallback UI Should Actually Do

A fallback isn't just "hide the AI button." A good fallback means the feature still delivers value without the model. Here are patterns for common use cases:

Smart text summarization → Fall back to a character-count preview or a "show more/less" toggle. Not as smart, but still useful.

Auto-tagging / content classification → Fall back to a curated list of tags the user picks from manually. Or skip tagging entirely and search by keyword.

AI-generated suggestions → Fall back to a set of hand-written preset options. Less personalised, but still functional.

Contextual chat assistant → Fall back to an FAQ-style interface or a link to your help docs.

The goal is: a user on an iPhone 14 should open your app and find a working, useful feature — not a broken screen or a wall of text explaining why their device isn't good enough.

A Note on the `@unknown default`

Always include @unknown default in your switch statement. Apple's API is still relatively new and they may add new unavailability reasons in future OS versions. If you omit it, a new case could cause a compile-time warning and — more importantly — unexpected runtime behaviour. Treat any unknown case the same as deviceNotEligible: assume the model isn't coming, and serve the basic experience.

Testing Without an Eligible Device

Testing all three unavailability states in Simulator can be tricky. Here's what works in practice:

.deviceNotEligible: Use an iPhone simulator that's older than iPhone 15 Pro (e.g. iPhone 14).
.appleIntelligenceNotEnabled: On a supported simulator, go to Settings → Apple Intelligence & Siri and toggle it off.
.modelNotReady: Harder to simulate reliably. You can mock this in your AIFeatureManager for testing by injecting a fake availability value.

For unit testing, make SystemLanguageModel.default.availability mockable by abstracting it behind a protocol:

protocol LanguageModelAvailabilityChecker {
    var availability: SystemLanguageModel.Availability { get }
}

struct LiveChecker: LanguageModelAvailabilityChecker {
    var availability: SystemLanguageModel.Availability {
        SystemLanguageModel.default.availability
    }
}

struct MockChecker: LanguageModelAvailabilityChecker {
    var availability: SystemLanguageModel.Availability
}

Inject MockChecker in your tests, LiveChecker in production. This lets you write clean unit tests for every availability state without needing a physical device.

The Bigger Picture

The developers who ship great apps right now are the ones who treat Foundation Models as a progressive enhancement — something that makes the experience better for those who have it, without breaking anything for those who don't.

Build the baseline first. Then layer the intelligence on top.

Requirements: iOS 26+ · Xcode 26+

Apple Intelligence devices: iPhone 15 Pro/Max, all iPhone 16 and 17 models

Apple's Foundation Models Framework: Run AI On-Device With Just a Few Lines of Swift

ArshTechPro — Wed, 18 Feb 2026 15:43:44 +0000

Apple has quietly shipped one of the most significant frameworks for iOS developers in years. With iOS 26 the Foundation Models framework gives you direct access to Apple's on-device ~3B parameter large language model — the same one powering Apple Intelligence — right from your Swift code.

No API keys. No cloud costs. No internet required. And it's completely free.

Let's break down what this means and how to start building with it.

What Is the Foundation Models Framework?

The Foundation Models framework exposes Apple's on-device LLM to third-party developers. Unlike cloud-based models like ChatGPT or Claude that run on remote servers, Apple's model runs entirely on the user's device using Apple silicon (CPU, GPU, and Neural Engine).

This gives you:

Privacy by default — all data stays on-device
Zero latency from network calls — inference happens locally
Offline support — works without internet
Free inference — no per-token costs, no API billing
Deep Swift integration — the API feels native, not bolted on

The model specializes in language understanding, structured output generation, and tool calling. It's not designed as a general-knowledge chatbot, but rather as an engine for building intelligent features tailored to your app.

Getting Started: Your First On-Device AI Feature

Here's how simple it is to generate a response from Apple's on-device LLM:

import FoundationModels

let session = LanguageModelSession()
let response = try await session.respond(to: "What's a good name for a travel app?")
print(response.content)

That's it. Three lines of code and you're running AI inference on-device.

Streaming Responses

For a ChatGPT-like experience where text appears token by token:

let session = LanguageModelSession()
let stream = session.streamResponse(to: "Suggest 5 creative app names for a fitness tracker")

for try await partial in stream {
    print(partial.content, terminator: "")
}

This creates a much smoother UX instead of waiting for the entire response to complete.

Guided Generation: The Killer Feature

Here's where Foundation Models really shines compared to typical LLM APIs. Guided Generation lets you get structured, type-safe outputs directly as Swift types.

Instead of parsing messy JSON strings, you define your output structure using the @Generable macro:

import FoundationModels

@Generable
struct MovieRecommendation {
    let title: String
    @Guide(description: "A brief one-sentence summary")
    let summary: String
    @Guide(.anyOf(["PG", "PG-13", "R", "G"]))
    let rating: String
}

Then generate structured output:

let session = LanguageModelSession()
let movie: MovieRecommendation = try await session.respond(
    to: "Recommend an action movie from the 2020s",
    generating: MovieRecommendation.self
).content

print(movie.title)   // e.g., "Top Gun: Maverick"
print(movie.rating)  // "PG-13" — guaranteed to be one of the allowed values

The @Guide macro lets you:

Add natural language descriptions to guide the model
Constrain values to a specific set with .anyOf()
Control array lengths with count()
Enforce string patterns with regex

This is constrained decoding built directly into the framework — the model is literally forced to produce valid output matching your Swift types at the token level. No more hoping the model returns valid JSON.

Tool Calling: Give the Model Superpowers

The on-device model has a ~3B parameter size, so it doesn't know everything. But with Tool Calling, you can extend its capabilities by giving it access to your app's data and APIs.

Here's an example — a health coach that reads HealthKit data:

import FoundationModels

struct BloodPressureTool: Tool {
    let name = "getBloodPressure"
    let description = "Fetches the user's latest blood pressure reading"

    func call(arguments: EmptyInput) async throws -> String {
        // Fetch from HealthKit
        let systolic = 120
        let diastolic = 80
        return "Systolic: \(systolic) mmHg, Diastolic: \(diastolic) mmHg"
    }
}

let session = LanguageModelSession(tools: [BloodPressureTool()]) {
    """
    You're a health coach. Help users manage their health 
    based on their blood pressure data.
    """
}

let response = try await session.respond(to: "How's my blood pressure looking?")

The model will automatically decide when to call your tool, fetch the data, and incorporate it into a natural language response. This is incredibly powerful for building context-aware features.

Specialized Adapters: Content Tagging Out of the Box

Beyond the general-purpose model, Apple provides specialized adapters for specific tasks. The content tagging adapter is built-in and optimized for:

Topic tag generation
Entity extraction
Topic detection

let taggingModel = SystemLanguageModel(useCase: .contentTagging)
let session = LanguageModelSession(model: taggingModel)

@Generable
struct Tags {
    let topics: [String]
}

let result: Tags = try await session.respond(
    to: "Apple announced new MacBook Pro with M5 chip at their spring event",
    generating: Tags.self
).content

print(result.topics) // ["Apple", "MacBook Pro", "M5", "Product Launch"]

Custom Adapter Training: Teach the Model New Tricks

For advanced use cases, Apple provides a Python-based adapter training toolkit that lets you fine-tune the on-device model with your own data using LoRA (Low-Rank Adaptation).

When should you consider training a custom adapter?

The model needs to become a subject-matter expert for your domain
You need a specific output style, format, or policy
Prompt engineering isn't achieving the required accuracy
You want lower latency by reducing prompt length

Key things to know about adapters:

Each adapter is ~160MB in storage
Adapters are compatible with a single specific model version — you must retrain when Apple updates the base model
Deploy via the Background Assets framework (don't bundle in your app)
Requires Mac with Apple silicon and 32GB+ RAM, or Linux GPU machines
You need the Foundation Models Framework Adapter Entitlement for production deployment

Apple recommends exhausting prompt engineering and tool calling before jumping to adapter training. It's powerful but comes with ongoing maintenance.

Availability and Requirements

Before creating a session, always check availability:

let availability = SystemLanguageModel.default.availability

switch availability {
case .available:
    // Ready to use
case .unavailable(.deviceNotEligible):
    // Device doesn't support Apple Intelligence
case .unavailable(.appleIntelligenceNotEnabled):
    // User needs to enable Apple Intelligence in Settings
case .unavailable(.modelNotReady):
    // Model is still downloading
default:
    break
}

Requirements:

iOS 26, iPadOS 26, macOS 26, or visionOS 26
Apple Intelligence-compatible device (iPhone 15 Pro or later, M-series Macs/iPads)
Apple Intelligence must be enabled in Settings

Practical Use Cases

Here are some real-world features you can build today:

Smart Journaling: Auto-generate mood tags and summaries from journal entries using guided generation.

Recipe Parsing: Point the model at unstructured recipe text and extract structured ingredient lists and steps.

Customer Support: Build an in-app assistant that uses tool calling to access order history and FAQs without any cloud dependency.

Content Moderation: Use the content tagging adapter to automatically classify user-generated content.

Personalized Learning: Generate quiz questions based on study material, all processed locally.

Workout Insights: Combine tool calling with HealthKit to generate natural language summaries of fitness data.

Limitations to Keep in Mind

The Foundation Models framework is powerful, but it's not a silver bullet:

Context window is limited — the ~3B model can't handle massive prompts
Not for general world knowledge — it's optimized for tasks, not trivia
Apple explicitly warns against using it for: code generation, math calculations, or factual Q&A
Device-dependent — older devices can't run it, so always have a fallback
Adapter retraining — every OS update with a new model version means retraining your adapters

What This Means for iOS Development

The Foundation Models framework represents a fundamental shift. For the first time, iOS developers have access to a production-quality LLM with:

Zero marginal cost per inference
Native Swift API that feels like any other Apple framework
Type-safe outputs through guided generation
Built-in privacy without any extra work

This isn't just another AI API wrapper. It's a deeply integrated, first-party framework that makes on-device intelligence a realistic feature for apps of all sizes — from indie side projects to enterprise applications.

If you haven't started experimenting with it yet, now's the time. The barrier to adding AI features to your iOS app has never been lower.

Requirements: Xcode 26 + iOS 26 SDK + Apple Intelligence-enabled device

Documentation: Foundation Models | Apple Developer Documentation

Forem: ArshTechPro

Gemma 4: A Practical Guide for Developers

What is Gemma 4

What It Can Do

Step 1: Pick Your Model

Step 2: Install Dependencies

Step 3: Run Inference with Transformers

Text-only generation

Image + text (vision)

Lower-level control with AutoModel

Step 4: Enable Thinking Mode

Step 5: Serve It with vLLM

Install vLLM

Start the server

Query it

Step 6: Run It Locally with Ollama

Step 7: Fine-Tune for Your Use Case

Install fine-tuning dependencies

Load with 4-bit quantization

Function Calling

Where to Get the Models

Key Architecture Details (if you care)

What Changed from Gemma 3

Summary

Hermes Agent: A Self-Improving AI Agent That Runs Anywhere

What Is Hermes Agent?

Why Should a Developer Care?

Core Concepts

The Agent Loop

Skills

Memory

Terminal Backends

The Messaging Gateway

How to Set It Up

Prerequisites

Step 1: Install

Step 2: Configure a Provider

Step 3: Start Chatting

Step 4: Set Up Sandboxed Execution (Recommended)

Step 5: Connect Messaging (Optional)

Useful Commands to Know

Adding MCP Servers

Setting Up Scheduled Tasks

Contributing

What It Is Not

Links

TurboQuant: What Developers Need to Know About Google's KV Cache Compression

The Problem: KV Cache Is Eating Your VRAM

How TurboQuant Works (The Short Version)

Why This Matters for Developers

Getting Started: The pip-Installable Path

The llama.cpp Path

Practical Notes and Gotchas

Community Implementations at a Glance

The Bigger Picture

Xcode 26.4 - Here Is What Actually Matters for Devs

If You Read Nothing Else: The Sanitizer Regression

Instruments: Serious New Profiling Tools

Swift 6.3

C++ Standard Library: Some Huge Performance Numbers

Localization: String Catalog Gets Practical

Testing: Breaking Change and Key Fixes

Build System and Package Manager

Coding Intelligence Fixes

System Requirements

Quick Reference: Who Needs to Care About What

Where to Get It

DeerFlow 2.0: What It Is, How It Works, and Why Developers Should Pay Attention

A Brief History: From Research Tool to Runtime

What DeerFlow 2.0 Actually Is

The Architecture

The Lead Agent and Task Decomposition

The Sandbox

Skills

Memory

Model Agnosticism

Getting Started

What You Can Actually Build With It

Things to Think Carefully About

How It Compares