---
title: Running a Local LLM on Windows with Ollama
date: 2026-04-07T10:54:00-04:00
author: cc_admin
canonical_url: "https://caffeinecreations.ca/blog/running-a-local-llm-on-windows-with-ollama/"
section: Blog
---
- [AI](https://caffeinecreations.ca/blog/category/ai/)
# Running a Local LLM on Windows with Ollama
Large language models don't have to live in the cloud. With Ollama, you can run powerful open-weight models entirely on your own Windows machine. No API keys. No subscription fees. No data leaving your computer.
This post was inspired by [John F. Morton's post on SuperGeekery](https://supergeekery.com/blog/running-googles-gemma-4-locally-on-macos-with-ollama) about doing the same thing on macOS. I wanted to document the Windows-specific steps since there are a few extra things to work through.
Here's how I set it up on my Windows 10 machine with an AMD Ryzen 5 5600X and 64GB of RAM.
## Why Run a Local LLM?
Before getting into the setup, here's why you might want to do this:
**Privacy** — Client code, internal documents, sensitive data. None of it leaves your machine.
**No cost** — No per-token billing, no usage caps, no monthly surprises.
**Works offline** — Once a model is downloaded, it runs without an internet connection.
### Hardware Considerations
Your results will vary depending on your hardware. A few things to keep in mind:
**RAM is the big one.** Models run in RAM, so the more you have the better. 16GB is workable for smaller models, 32GB+ opens up much more capable ones.
**GPU acceleration** helps a lot. Ollama has excellent support for NVIDIA GPUs via CUDA. AMD GPU support on Windows via ROCm is still hit or miss, so if you have an AMD card like I do, expect to run on CPU for now.
**Don't let that discourage you.** With enough RAM, CPU inference is very usable, especially for smaller models.
## Step 1: Install Ollama
Download the Windows installer from [ollama.com](https://ollama.com/) and run it. The install is silent and Ollama will appear in your system tray when running.
Open PowerShell and verify the installation:
```
```bash
ollama --version
```
```
You should see a version number. If you get a "not recognized" error, make sure Ollama is running in the system tray first, then try again in a fresh PowerShell window.
## Step 2: Add Ollama to Your PATH Permanently
On Windows, Ollama doesn't always add itself to the system PATH automatically. To fix this permanently, open PowerShell as Administrator (right-click > Run as administrator) and run:
```
```bash
[System.Environment]::SetEnvironmentVariable("PATH", $env:PATH + ";C:\Users\YOUR-USERNAME\AppData\Local\Programs\Ollama", [System.EnvironmentVariableTarget]::Machine)
```
```
Replace YOUR-USERNAME with your actual Windows username.
While you're at it, set the OLLAMA\_HOST variable so Ollama accepts connections from Docker (needed later for the web UI):
```
```bash
[System.Environment]::SetEnvironmentVariable("OLLAMA_HOST", "0.0.0.0", [System.EnvironmentVariableTarget]::Machine)
```
```
You may need to restart your computer for these changes to take effect.
## Step 3: Pull Your First Model
Ollama makes downloading models simple. Start with a smaller model to make sure everything is working:
```
```bash
ollama pull llama3.2
```
```
Once downloaded, run it:
```
```bash
ollama run llama3.2
```
```
You'll get a >>> prompt and can start chatting right in the terminal. Type /bye to exit.
## Step 4: Pull a More Capable Model
With enough RAM, you can run significantly more capable models. I went with Gemma 4 26B from Google, which sits in a sweet spot of quality and speed
```
```bash
ollama pull gemma4:26b
```
```
This is around 18GB so give it some time to download. Once done:
```
```bash
ollama run gemma4:26b
```
```
The quality difference compared to the smaller model is significant. Well worth the download if your hardware can handle it.
## Step 5: Set Up Open WebUI
Chatting in the terminal works but a proper browser-based UI is much nicer. Open WebUI is the best option and since it runs in Docker, setup is straightforward if you already have Docker Desktop installed.
Run this in PowerShell:
```
```bash
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
```
```
Then open your browser and go to http://localhost:3000. Create a local account (nothing leaves your machine) and your Ollama models should appear in the model dropdown automatically.
## Step 6: Add VS Code Integration with Continue
The Continue extension connects Ollama directly to VS Code, giving you AI code completions and a chat panel without leaving your editor.
Install the **Continue** extension from the VS Code marketplace
Open Ctrl + Shift + P and run Continue: Open Config File
Replace the contents with:
```
```yaml
name: Local Config
version: 1.0.0
schema: v1
models:
- name: gemma4:26b
provider: ollama
model: gemma4:26b
roles:
- chat
- edit
- name: llama3.2
provider: ollama
model: llama3.2
roles:
- chat
- autocomplete
```
```
Save the file and your models will appear in the Continue panel. Use Ctrl + L to open the chat panel and Ctrl + I for inline code editing.
## Useful Ollama Commands
```
```bash
ollama list # see downloaded models
ollama pull mistral # download another model
ollama rm llama3.2 # remove a model
ollama ps # see what's currently running
```
```
### Wrapping Up
The whole setup from scratch takes about 30 minutes. Once it's running you have a fully private, free AI assistant that works whether you're online or off.
My plan is to use my local AI to help me understand contracts and other legal documents without worrying about sensitive data shared with another company. I'm also going to set this up on my laptop so I can use it when I'm on a plane without reliable internet.
[### Displaying Exif Data in Craft CMS Templates](https://caffeinecreations.ca/blog/displaying-exif-data-in-craft-cms-templates/)
[### Boilerplates in Web Development](https://caffeinecreations.ca/blog/boilerplates-in-web-development/)
[### How to Choose a CMS - Podcast Episode](https://caffeinecreations.ca/blog/how-to-choose-a-cms-podcast-episode/)