How to Install Gemma 4 Locally with Ollama (2026 Guide)

Running frontier AI models on local hardware eliminates subscription fees, guarantees zero cloud data leaks, and works without an active internet connection. Google’s Gemma 4 family represents a monumental shift in open-weights AI—delivering multimodal reasoning, up to 256K token context windows, and Apache 2.0 commercial freedom. Combined with Ollama, you can download and run Gemma 4 locally on macOS, Linux, Windows, or edge ARM devices in under two minutes.

What is Gemma 4 and Why Should You Run It Locally with Ollama?

Google Gemma 4 is DeepMind’s flagship 2026 open-weights model suite built on Gemini 3 architecture, offering multimodal reasoning, native function calling, and an Apache 2.0 license. Running Gemma 4 locally with Ollama provides complete data privacy, zero latency over API networks, and offline execution across Apple Silicon, NVIDIA GPUs, and edge single-board computers.

Unlike proprietary APIs that log inputs for training, local execution keeps source code, internal documentation, and personal data strictly on your device. Furthermore, Ollama encapsulates model management, quantization, GPU memory allocation, and an OpenAI-compatible REST server into a single binary.

flowchart LR
    A["Developer / IDE\n(Cursor, VS Code)"] -->|REST / API Call| B["Ollama Local Runtime\n(localhost:11434)"]
    B -->|Hardware Acceleration| C{"Execution Engine"}
    C -->|Metal / Unified RAM| D["Apple Silicon (M1-M4)"]
    C -->|CUDA / Tensor Cores| E["NVIDIA GPUs (RTX/H100)"]
    C -->|CPU / AVX-512| F["Edge & System RAM"]
    D --> G["Gemma 4 Weights\n(E2B, E4B, 12B, 26B, 31B)"]
    E --> G
    F --> G
    G -->|Zero Cloud Latency| A

Key Technical Upgrades in Gemma 4

• Permissive Apache 2.0 License: Google replaced the restrictive “Gemma Terms of Use” with Apache 2.0, allowing commercial modification, SaaS embedding, and redistribution.
• Multimodal Inputs: Native vision (image understanding, OCR, document analysis) across all models, plus native audio processing on E2B, E4B, and 12B variants.
• Expanded Context Windows: 128K tokens for edge models (E2B, E4B) and 256K tokens for workstation models (12B, 26B, 31B).
• Mixture-of-Experts (MoE) Efficiency: The 26B MoE variant routes tokens through specialized sub-networks, activating only 4B parameters during inference to maintain high tokens/second throughput.

What Are the Hardware Requirements for Running Gemma 4?

Gemma 4 hardware requirements range from 4 GB system RAM for edge-optimised E2B models up to 24 GB+ VRAM for unquantized 31B Dense workstation models. Apple Silicon Macs with unified memory run the 12B and 26B MoE models seamlessly out of the box, while mid-range NVIDIA GPUs excel using Q4_K_M quantized weights.

Before downloading weights, consult the hardware compatibility matrix below to match your system specs:

How Do You Install Ollama on macOS, Linux, and Windows?

To install Ollama, download the native background service manager for your operating system via Homebrew on macOS, curl installer script on Linux, or executable setup on Windows. Once installed, Ollama runs an automated system daemon listening on port 11434.

Execute the installation command matching your operating system:

macOS Installation

Install via Homebrew package manager:

# Install Ollama on macOS using Homebrew
brew install ollama

# Start the Ollama background process
ollama serve

Linux Installation

Execute the official multi-architecture shell script:

# Download and install Ollama system service on Linux
curl -fsSL https://ollama.com/install.sh | sh

# Verify systemd service status
systemctl status ollama

Windows Installation

Download OllamaSetup.exe from ollama.com. The installer configures NVIDIA CUDA and DirectML GPU drivers automatically.

Verify that the CLI utility is operational:

# Check installed Ollama CLI version
ollama --version

How to Download and Run Gemma 4 Models in Ollama?

To download and run Gemma 4, execute ollama run gemma4:<tag> in your terminal. Ollama checks local storage in ~/.ollama/models/, downloads missing model weights automatically from the registry, and initializes an interactive terminal session.

Choose the command matching your hardware setup:

# Run the E2B edge model (Raspberry Pi / Mobile / 4GB RAM)
ollama run gemma4:e2b

# Run the E4B edge model (Default 4B model / 8GB RAM)
ollama run gemma4:e4b

# Run the 12B unified multimodal model (16GB RAM / 10GB VRAM)
ollama run gemma4:12b

# Run the 26B MoE model (Fast desktop inference / 12GB VRAM)
ollama run gemma4:26b

# Run the 31B Dense flagship model (Workstation / 24GB VRAM)
ollama run gemma4:31b

During your first run, Ollama displays a progress bar indicating download speed and weight verification:

pulling manifest
pulling 8a379c6d3b4e... 100% ▕████████████████████████████████████████████████████████▏ 18.5 GB
verifying sha256 digest
writing manifest
removing any unused layers
success
>>> Send a message (?:help)

Which Gemma 4 Model Size and Quantization Variant Should You Choose?

Selecting the optimal Gemma 4 variant depends on your hardware VRAM budget and application requirements. Use gemma4:e2b or gemma4:e4b for offline edge IoT, gemma4:26b for high-throughput coding autocomplete, and gemma4:31b-q4_K_M for frontier-level reasoning tasks on consumer GPUs.

Follow this decision tree to identify your exact deployment candidate:

flowchart TD
    Start["Determine Hardware Resources"] --> CheckVRAM{"Available VRAM / Unified RAM?"}
    CheckVRAM -->|Less than 6 GB| Edge["Select gemma4:e2b or gemma4:e4b\n(128K Context, Vision + Audio, Edge/IoT)"]
    CheckVRAM -->|8 GB to 12 GB| Mid["Select gemma4:12b or gemma4:26b-q4_K_M\n(256K Context, Fast MoE Inference)"]
    CheckVRAM -->|16 GB to 24 GB+| High["Select gemma4:31b or gemma4:31b-q4_K_M\n(256K Context, Frontier Reasoning)"]
    
    Edge --> AppEdge["Best For: Raspberry Pi 5, Android, Jetson Orin"]
    Mid --> AppMid["Best For: M1/M2/M3 Macs, RTX 3060/4070 GPUs"]
    High --> AppHigh["Best For: M3/M4 Max, RTX 4090, H100 Workstations"]

Understanding GGUF Quantization Precision

Ollama uses GGUF quantization formats to fit large models into limited VRAM:

• gemma4:31b (FP16 / Unquantized): Requires ~62 GB VRAM. Zero quality loss.
• gemma4:31b-q8_0 (8-bit Quantization): Requires ~32 GB VRAM. Maintains 99.8% precision.
• gemma4:31b-q4_K_M (4-bit Medium Quantization): Requires ~18.5 GB VRAM. Reduces VRAM usage by 70% while maintaining 97.5% baseline accuracy. Recommended default for 24GB GPUs.

Pull a specific quantized variant using explicit tags:

# Pull the 4-bit quantized 31B workstation variant
ollama pull gemma4:31b-q4_K_M

How Do Gemma 4 Benchmarks Compare to Other Open LLMs?

On standardized benchmarks, Google Gemma 4 31B scores 89.2% on AIME 2026 mathematics, 80.0% on LiveCodeBench v6 coding, and 85.2% on MMLU Pro reasoning—ranking #3 overall among open-weights models on the LMSYS Arena AI leaderboard.

The empirical benchmark data below details performance across coding, mathematics, and reasoning compared to rival open weights:

How Does Gemma 4 Compare to DeepSeek R1 and Qwen 2.5 Coder?

Gemma 4 excels as a versatile, multimodal, Apache 2.0 licensed model supporting vision, audio, and edge deployments. Compared to DeepSeek R1 (specialized in deep reasoning) and Qwen 2.5 Coder (specialized in code synthesis), Gemma 4 offers the broadest architectural versatility and official native multi-modal capabilities.

Review the head-to-head comparison matrix below to determine the best model for your workload:

How Fast Does Gemma 4 Run Across Different Hardware (Tokens/Second)?

Gemma 4 token generation speed ranges from 4–8 tokens/second on single-board computers like Raspberry Pi 5 up to 90 tokens/second on high-end GPUs like the NVIDIA RTX 4090. Apple Silicon Macs generate 15–45 tokens/second using Metal unified memory acceleration.

Consult the real-world performance comparison table below to estimate generation speeds for your setup:

Note: Tokens/second speeds fluctuate based on input prompt length, active context window size, and thermal throttling.

How to Configure Hardware Acceleration for Apple Silicon, NVIDIA GPUs, and Edge Devices?

Ollama configures hardware acceleration automatically during initialization by detecting Metal on Apple Silicon Macs, CUDA on NVIDIA GPUs, or OpenCL/Vulkan on ARM edge devices. You can verify active GPU offloading using ollama ps or environment variables.

Follow these hardware-specific optimization steps:

1. Apple Silicon Macs (M1/M2/M3/M4)

macOS leverages Metal Unified Memory Architecture (UMA) automatically. To allow Ollama to allocate more than 75% of system memory to model weights, set the system environment flag:

# Allow Ollama to consume up to 90% of unified memory on macOS
export OLLAMA_MAX_LOADED_MODELS=2
sudo sysctl iogpu.wired_mem_limit_default=0

2. NVIDIA GPUs (Linux & Windows)

Ensure CUDA 12.x drivers and the NVIDIA Container Toolkit are installed. Inspect active VRAM offloading:

# Check if Gemma 4 layers are fully offloaded to GPU
ollama ps

Expected output showing 100% GPU offload:

NAME           ID           SIZE     PROCESSOR    UNTIL
gemma4:31b    3a4e91f0     18.5 GB  100% GPU     4 minutes from now

If VRAM is insufficient, split model layers between GPU VRAM and System RAM:

# Control GPU layer offloading explicitly
OLLAMA_NUM_PARALLEL=1 OLLAMA_GPU_OVERHEAD=512MB ollama run gemma4:26b

3. Edge Devices (Raspberry Pi 5 & Jetson Orin)

On single-board ARM systems running gemma4:e2b, optimize memory swap and thread execution:

# On Raspberry Pi 5 with 8GB RAM
ollama run gemma4:e2b

# On NVIDIA Jetson Orin Nano
ollama run gemma4:e4b

Set maximum CPU thread count for ARM Cortex cores:

# Optimize CPU thread allocation on ARM Linux
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_VRAM=0
ollama run gemma4:e2b

What Are the Most Useful Ollama CLI Commands for Gemma 4?

Ollama provides CLI subcommands to inspect loaded models, pull quantized weights, delete unused checkpoints, and execute single-pass prompts directly from the shell. Commands like ollama ps and ollama show give immediate visibility into VRAM usage and model parameter layers.

Reference this operational CLI cheat-sheet for daily model management:

# List all locally downloaded models and tags
ollama list

# Download a specific model tag without starting an interactive session
ollama pull gemma4:31b

# Show model parameters, template format, and license details
ollama show gemma4:31b

# List currently running models loaded into VRAM/RAM
ollama ps

# Unload a model from VRAM immediately
ollama stop gemma4:31b

# Delete downloaded weights to free up disk space
ollama rm gemma4:26b

# Run a one-shot, non-interactive prompt directly from terminal
ollama run gemma4:e4b "Summarize this log file in 3 bullet points"

How to Use the Ollama REST API, Python SDK, and Multimodal Features with Gemma 4?

Ollama exposes an OpenAI-compatible REST API at http://localhost:11434/v1 and a native endpoint at /api/generate. You can interact with Gemma 4 using standard cURL commands, Python SDKs, base64 image streams, or LangChain pipelines.

1. Multimodal Image Analysis in Python

Pass local screenshots, system architecture diagrams, or document scans directly to gemma4:12b for visual understanding:

import base64
import ollama

# Encode local image asset to base64
with open("architecture_diagram.png", "rb") as f:
    image_base64 = base64.b64encode(f.read()).decode("utf-8")

# Query Gemma 4 with image input
response = ollama.chat(
    model="gemma4:12b",
    messages=[{
        "role": "user",
        "content": "Explain the data flow in this architecture diagram.",
        "images": [image_base64],
    }],
)

print(response["message"]["content"])

2. Direct cURL Request to REST Endpoint

Query the local native REST server:

# Send a chat prompt to local Gemma 4 via cURL
curl http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{
  "model": "gemma4:31b",
  "messages": [
    { "role": "system", "content": "You are a senior Rust engineer." },
    { "role": "user", "content": "Write a thread-safe MPSC queue wrapper." }
  ],
  "stream": false
}'

3. Native Python SDK Streaming (ollama)

Execute a structured streaming call using the Python SDK:

import ollama

# Stream response chunks from Gemma 4 locally
stream = ollama.chat(
    model='gemma4:26b',
    messages=[
        {
            'role': 'user',
            'content': 'Explain Rust ownership and borrowing with code.',
        }
    ],
    stream=True,
)

for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

How to Create Custom Gemma 4 Agents Using Ollama Modelfiles?

You can package custom system instructions, default sampling parameters, and context limits into a standalone agent binary using an Ollama Modelfile. Running ollama create compiles the custom configuration into a reusable local model instance.

Create a Modelfile in your working directory:

# Base model weights
FROM gemma4:31b-q4_K_M

# Configure system parameters
PARAMETER temperature 0.2
PARAMETER num_ctx 32768
PARAMETER top_p 0.95

# Set persistent system prompt
SYSTEM """
You are an expert DevSecOps specialist.
Review all input code snippets specifically for OWASP Top 10 vulnerabilities, unsanitized inputs, and memory leaks.
Provide solutions formatted in clear, production-ready code blocks.
"""

Compile and build your custom agent:

# Create local custom agent model named 'security-gemma'
ollama create security-gemma -f Modelfile

# Run your custom agent immediately
ollama run security-gemma "Audit this SQL query: SELECT * FROM users WHERE id = '" + userInput + "'"

How to Use Gemma 4 for Function Calling and Structured JSON Output?

Gemma 4 natively supports structured JSON schema output and function calling parameters. Passing format="json" to Ollama forces the model to constrain token selection strictly to valid JSON objects matching your requested schema.

Execute structured output requests in Python:

import ollama

# Request enforced JSON schema output from Gemma 4
response = ollama.chat(
    model="gemma4:31b",
    format="json",
    messages=[{
        "role": "user",
        "content": """
        Extract user metadata from this text: 'Contact Rahul Sharma at rahul@example.com (Senior DevOps Engineer, Surat)'.
        Return JSON with keys: name, email, title, location.
        """,
    }],
)

print(response["message"]["content"])

Expected JSON output:

{
  "name": "Rahul Sharma",
  "email": "rahul@example.com",
  "title": "Senior DevOps Engineer",
  "location": "Surat"
}

How to Integrate Gemma 4 with Modern IDEs (VS Code, Cursor, Continue, Claude Code)?

To integrate local Gemma 4 into development workflows, configure your IDE extension (such as Continue.dev or Cursor) to point to base URL http://localhost:11434. This enables inline code completion, side-panel refactoring, and automated test generation without telemetry tracking.

1. Continue.dev Extension Configuration

Open ~/.continue/config.json and register Gemma 4 for chat and tab autocomplete:

{
  "models": [
    {
      "title": "Gemma 4 31B (Local)",
      "provider": "ollama",
      "model": "gemma4:31b",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Gemma 4 26B MoE Autocomplete",
    "provider": "ollama",
    "model": "gemma4:26b",
    "apiBase": "http://localhost:11434"
  }
}

2. Cursor IDE Integration

Navigate to Cursor Settings $\rightarrow$ Models $\rightarrow$ OpenAI API Key:

1. Turn off Override OpenAI Base URL toggle off/on.
2. Set Base URL to http://localhost:11434/v1.
3. Add model string gemma4:31b to the custom models list.

3. Piping CLI Input into Gemma 4

You can pipe file contents directly into local Gemma 4 for code review from bash:

# Pipe code directly into Gemma 4 for instant bug review
cat src/utils/parser.ts | ollama run gemma4:31b "Review this TypeScript code for bugs and memory leaks"

How Should You Prompt Gemma 4 for Coding, Explanation, and Code Review?

Gemma 4 performs best when provided with direct role framing, specific output constraints, and clear task boundaries. Structured prompts with explicit severity ratings for code review or specific target developer levels yield superior responses.

Use these production-tested prompt templates:

Prompt 1: Production Code Generation

You are an expert Python developer. Write a clean, production-ready function that parses an incoming JSON payload and validates email strings using regex. Include type hints, comprehensive docstrings, and handle edge cases cleanly.

Prompt 2: Technical Explanation

Explain the difference between Mutex and RwLock as if I am a senior backend engineer familiar with concurrency primitives in Go, but learning Rust. Keep the explanation concise and focus on trade-offs.

Prompt 3: Code Review & Security Audit

Review the code below for bugs, performance bottlenecks, and style violations. Categorize each finding by severity (Low, Medium, High) and provide a corrected refactored code block.

How Do You Troubleshoot Common Gemma 4 Ollama Errors?

Common errors when running Gemma 4 in Ollama include CUDA out-of-memory crashes, slow CPU fallback, and API connection failures. Resolving them requires selecting smaller quantized variants, verifying system daemon status, or tuning environment variables.

Refer to the troubleshooting guide below to diagnose runtime issues:

1. “CUDA out of memory” or Crash on Initialization

• Cause: Model weights exceed physical GPU VRAM.
• Solution: Stop running models with ollama stop gemma4:31b and pull a 4-bit quantized variant such as ollama run gemma4:31b-q4_K_M or step down to gemma4:26b.

2. Extremely Slow Inference (Less Than 2 Tokens/Second)

• Cause: Ollama failed to detect GPU drivers and fell back to CPU execution.
• Solution: Run ollama ps to verify processor type. On Linux/Windows, reinstall CUDA drivers and restart the daemon via sudo systemctl restart ollama.

3. “Error: connection refused to localhost:11434”

• Cause: Ollama background process is stopped.
• Solution: Start the server process manually by running ollama serve in a dedicated terminal window.

Summary

• 5 Model Sizes: E2B, E4B (edge/mobile, 128K context), 12B (multimodal), 26B MoE (4B active parameters), and 31B Dense (workstation, 256K context).
• Apache 2.0 License: Complete commercial freedom for production deployment, enterprise customization, and SaaS packaging.
• Frontier Performance: Flagship 31B model ranks #3 on the LMSYS Arena AI leaderboard with 89.2% AIME math and 80.0% LiveCodeBench scores.
• Zero Cloud Latency & Privacy: Runs 100% offline via Ollama on macOS, Linux, Windows, and ARM edge hardware.
• OpenAI API Compatibility: Integrates seamlessly into VS Code, Cursor, Continue.dev, and native Python SDK pipelines.

Frequently Asked Questions

Is Gemma 4 free for commercial use?

Yes. Google released the entire Gemma 4 model family under the permissive Apache 2.0 license. You can use Gemma 4 commercially to build private SaaS products, modify weights, train adapters, and redistribute applications without royalties or attribution restrictions.

How does Gemma 4 26B MoE compare to Gemma 4 31B Dense?

Gemma 4 26B Mixture-of-Experts contains 26 billion total parameters but only activates 4 billion parameters per token during inference. This results in significantly faster inference speeds (~65 tokens/sec on RTX 4090) while requiring less VRAM. Gemma 4 31B Dense evaluates all parameters continuously, yielding maximum reasoning and code generation quality.

Can I process images and audio locally with Gemma 4 in Ollama?

Yes. All Gemma 4 variants support vision inputs (image prompts, OCR, diagram reading). Additionally, the E2B, E4B, and 12B variants feature native audio processing, allowing you to feed audio files directly to local Ollama endpoints.

What is the default model pulled when running ollama pull gemma4?

Running ollama pull gemma4 pulls the gemma4:e4b model tag by default—a 4-billion parameter edge-optimized variant designed for fast inference on consumer hardware and laptops with 8GB RAM.

How much VRAM is required to run Gemma 4 31B?

The unquantized FP16 version of Gemma 4 31B requires over 60 GB VRAM. However, the Q4_K_M (4-bit quantized) version fits comfortably into 18.5 GB VRAM, making it fully operational on single consumer GPUs like the NVIDIA RTX 4090 or Apple Silicon Macs with 24GB+ unified memory.

What to Read Next

• How to Install Ollama and Run LLMs Locally — Complete installation and configuration guide for local LLMs
• Qwen Coder Cheatsheet — Benchmark and setup guide for the leading open coding model
• DeepSeek R1 Local Setup Guide — Running local reasoning models with Ollama