DiffusionGemma - How to Run Locally

DiffusionGemma 26B-A4B is Google DeepMind’s new open multimodal model, built on the Gemma 4 MoE architecture. With support for 256K context, 140+ languages, DiffusionGemma is designed for high-speed text generation across text, video and image inputs. DiffusionGemma can run locally on 18GB RAM, and fine-tuning is now supported via Unsloth.

Instead of standard token-by-token decoding, DiffusionGemma uses diffusion generation to produce outputs in parallel and gradually refine them into a final answer - similar to diffusion image models, but for text. Run the model via Unsloth Studio or llama.cpp. On a RTX 6000, DiffusionGemma can reach 2000+ tokens/s. GGUF: diffusiongemma-26B-A4B-it-GGUF

Run DiffusionGemmaFine-tune DiffusionGemma

Usage Guide

DiffusionGemma is designed for users who need faster generation than standard models. It is suited to fast local inference, long-context doc analysis, image/video understanding, OCR and document parsing, code generation, tool use, agentic workflows, and low-latency small-batch inference.

Unlike standard Gemma 4 models, DiffusionGemma requires a diffusion-aware inference runtime. Autoregressive settings such as temperature, top_p, and top_k alone are not enough to reproduce the recommended behavior without the required diffusion sampler.

Hardware requirements

It's generally best to have at least 18GB RAM to run the model in 4-bit precision. GGUF: diffusiongemma-26B-A4B-it-GGUF

Table: DiffusionGemma Inference GGUF recommended hardware requirements (units = total memory: RAM + VRAM, or unified memory).

Run DiffusionGemma Tutorials

It's best to use at least 4-bit precision so we'll use the 4-bit Q4_K_M quant which needs 18GB RAM. GGUF: diffusiongemma-26B-A4B-it-GGUF

🦥 Unsloth Studio Guide🦙 Llama.cpp Guide

🦥 Unsloth Studio Guide

DiffusionGemma can now be run and trained in Unsloth Studio, our new open-source web UI for local AI. Unsloth Studio lets you run models locally on MacOS, Windows, Linux and:

• Search, download, run GGUFs and safetensor models

• Self-healing tool calling + web search

• Code execution (Python, Bash)

• Automatic inference parameter tuning (temp, top-p, etc.)

• Fast CPU + GPU inference via llama.cpp

• Train LLMs 2x faster with 70% less VRAM

1

Install Unsloth

Ensure you use the latest v0.1.463-beta or 2026.6.6. Run in your terminal:

MacOS, Linux, WSL:

Windows PowerShell:

2

Launch Unsloth

MacOS, Linux, WSL and Windows:

Then open http://127.0.0.1:8888 (or your specific URL) in your browser.

3

Search and download DiffusionGemma

On first launch you will need to create a password to secure your account and sign in again.

Then go to the Studio Chat tab and search for DiffusionGemma in the search bar and download your desired model and quant.

4

Run DiffusionGemma

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our Unsloth Studio inference guide.

🦙 Llama.cpp Guide

For this tutorial, we will be utilizing the Dynamic 4-bit Q4_K_M quant which needs 18GB RAM and llama.cpp for fast local inference, especially if you have a CPU.

Chat with DiffusionGemma

Then run the below:

You will see:

And if you type a question like "Create a Flappy Bird Game", you will see steps:

Then afterwards you'll see the output:

You can continue conversing as well!

Change -n 2048 as the number of tokens you want to predict, so more will produce longer answers.

Live visualization of diffusion

To see diffusion actually live, use the below - specially enable --diffusion-visual:

You will again see:

And we get:

All parameters for llama.cpp using the branch:

• -n, --n-predict N - target tokens; derives --diffusion-blocks and grows -ub / -b / -c.

• -ngl 99 - offload all layers to the GPU (-ngl 0 for CPU-only).

• -cnv - multi-turn conversation mode.

• --diffusion-visual - live canvas denoising view.

• The Entropy-Bound sampler is on by default (--diffusion-eb auto). Tune it with --diffusion-eb-max-steps (default 48), --diffusion-eb-t-max / --diffusion-eb-t-min (0.8 -> 0.4), --diffusion-eb-entropy-bound (0.1), and --diffusion-eb-confidence (0.005).

• --diffusion-kv-cache {auto,on,off} - prompt prefix KV cache (auto = on for single GPU).

Fine-tune DiffusionGemma

You can now train and fine-tune DiffusionGemma directly with Unsloth. In our example, we demonstrate the impact of domain-specific training by fine-tuning the model on Sudoku. The base model initially performs poorly on Sudoku tasks, but after training on a targeted dataset, it learns how to actually solve sudoku and solves every example correctly.

You can use our Colab notebook (A100) to fine-tune DiffusionGemma with:

Google Colabcolab.research.google.com

Recommended Settings

Unsloth Studio automatically sets the best inference settings for your model. Use below if necessary:

Adaptive Stopping Trigger Conditions

Adaptive stopping should trigger only when both conditions are met:

At each denoising step, the sampler should select the lowest-entropy tokens whose mutual information bound remains: entropy_bound = 0.1. Non-selected tokens should be fully renoised before the next denoising step.

Thinking Mode

DiffusionGemma supports Gemma 4-style thinking mode. To enable thinking, add the thinking token at the start of the system prompt:

When thinking is enabled, the model may emit an internal reasoning channel followed by the final answer:

To disable thinking, remove the <|think|> token from the system prompt. When thinking is disabled, the model may still emit an empty thought channel:

For multi-turn conversations, do not include previous hidden thoughts in the conversation history. Only include the final assistant response before the next user turn.

DiffusionGemma Best Practices

Multimodal Prompting

DiffusionGemma supports interleaved multimodal inputs, including text and images. Video can be processed as sequences of image frames.

For best results with multimodal prompts, place image or frame content before text instructions. Example:

For document parsing, OCR, chart understanding, UI understanding, or small text extraction, use a higher visual token budget.

Supported visual token budgets:

For video-style inputs, DiffusionGemma can process up to 60 seconds when sampled at 1 frame per second.

Sampling Notes

DiffusionGemma is not a normal next-token-only model. It generates a block of tokens, called a canvas, by repeatedly refining noisy token predictions. The generation process works roughly as follows:

1. The encoder processes the prompt and builds a context cache.

2. The decoder receives a 256-token generation canvas.

3. The diffusion sampler iteratively denoises the canvas.

4. Confident tokens are selected and preserved.

5. Uncertain tokens are renoised and refined again.

6. Once the canvas is complete, it is appended to the context.

7. The model continues with the next canvas.

This block-autoregressive approach allows DiffusionGemma to generate many tokens in fewer forward passes than a standard autoregressive model.

Benchmarks

DiffusionGemma is optimized for speed and multimodal reasoning, though standard Gemma 4 is stronger on conventional reasoning benchmarks.

Vision benchmarks: