How to Run Gemma 4 Locally in VS Code with Ollama

You can use Gemma 4 in VS Code by running the model locally with Ollama and connecting it to the Continue extension. This setup works well for code questions and targeted edits without sending prompts to a hosted model—provided Continue is configured only for the local Ollama endpoint. Model output can still be wrong, so review every proposed change and run your tests before accepting it.

What you need

• VS Code on Windows, macOS, or Linux.
• Ollama running on the same computer.
• Enough free storage and memory for the selected model. The current Ollama files are several gigabytes, and inference needs additional working memory.
• The Continue extension published by Continue.

Do not rely on a fixed VRAM recommendation from another computer. Quantization, context length, GPU offloading, and other running applications all affect memory use. Start with a smaller model, confirm that it responds at an acceptable speed, and move up only if the system has headroom.

1. Install and test Ollama

Install Ollama using the instructions for your operating system. Then open a terminal—VS Code's integrated terminal works—and verify the installation:

ollama --version

Ollama normally serves its local API at http://localhost:11434. You can test the service from a browser or terminal. A local response indicating that Ollama is running confirms that VS Code extensions can attempt to connect to it.

2. Download a Gemma 4 model

Ollama currently offers Gemma 4 tags for E2B, E4B, 12B, 26B, and 31B variants. The default gemma4 tag corresponds to E4B, but specifying a tag makes the configuration easier to understand.

ollama pull gemma4:e4b
ollama run gemma4:e4b

Enter a short coding prompt to confirm that generation works, then type /bye to leave the terminal chat. If E4B is too slow or produces a memory error, try the smaller edge tag:

ollama pull gemma4:e2b
ollama run gemma4:e2b

The Gemma 4 tag list shows the current download size and context limit for each build. Larger models can improve difficult answers, but they also take more storage and memory; size alone does not guarantee better code for a particular project.

3. Test the optional Ollama desktop chat

Recent Ollama desktop builds may provide a chat interface. Select the downloaded Gemma 4 tag and ask for a small function. This is a convenient check, but it is not required—the terminal command above tests the same local model service.

4. Connect Continue to Ollama

1. In VS Code, open Extensions with Ctrl+Shift+X or Cmd+Shift+X.
2. Search for Continue and verify that the publisher is Continue before installing it.
3. Open the Continue panel. If its setup flow detects Ollama, select the local provider and your downloaded Gemma 4 tag.
4. If automatic detection fails, open Continue's local configuration and add the YAML configuration below.

name: Local Gemma
version: 1.0.0
schema: v1

models:
  - name: Gemma 4 E4B
    provider: ollama
    model: gemma4:e4b
    roles:
      - chat
      - edit
      - apply

Save the file and reload the Continue configuration. Replace gemma4:e4b with the exact tag shown by ollama list if you downloaded a different variant. A remote apiBase is unnecessary when Ollama and VS Code run on the same machine.

5. Use Gemma 4 for chat and edits

• Select a function and ask Continue to explain assumptions, edge cases, or likely errors.
• Request a small edit, inspect the diff, and accept only the changes you understand.
• Provide relevant files or selected code instead of indexing an unrelated large workspace.
• Ask for tests, then run them yourself; generated tests can repeat the same mistaken assumption as generated code.

Continue commonly maps chat and edit actions to Ctrl/Cmd+L and Ctrl/Cmd+I, but keyboard mappings can conflict with other extensions. Use the Command Palette and search for Continue if a shortcut does not work.

Use a separate model for autocomplete

Gemma 4 is a general chat and reasoning model, not a fill-in-the-middle autocomplete model. Continue's documentation recommends a small coding model for low-latency Tab suggestions and notes that thinking models are usually too slow for this role. You can keep Gemma 4 for chat and add a separate local autocomplete model:

ollama pull qwen2.5-coder:1.5b

  - name: Local code autocomplete
    provider: ollama
    model: qwen2.5-coder:1.5b
    roles:
      - autocomplete

Add that entry under the existing models: list, preserving YAML indentation. This division usually works better than assigning a Gemma 4 thinking model to every role.

Troubleshooting

Continue cannot find Ollama

Run ollama list and open http://localhost:11434. If neither works, start or restart Ollama. If Ollama works but Continue does not, verify the provider name, model tag, YAML indentation, and extension logs.

The model requires more memory

Switch from E4B to E2B, close memory-heavy applications, or reduce the configured context length. Continue can request a larger context than a simple terminal prompt, so a model that starts in ollama run may still fail in the extension.

Responses are slow

Run ollama ps while the model is active to inspect how it is loaded. Use a smaller tag if most inference falls back to the CPU or if the editor becomes unresponsive. For autocomplete, use the dedicated small coding model rather than Gemma 4.

Local does not appear private

Confirm that the model provider points to local Ollama and review the extension's telemetry, account, and indexing settings. A local inference endpoint prevents that model request from going to a hosted model, but other enabled extension features may have separate network behavior.

For current configuration fields, consult Continue's Ollama provider documentation. Recheck the documentation after extension updates because commands and configuration screens can change.