Getting a Local Model Running

Before any Python script can talk to a model, the model itself has to be running and listening for requests. This is the part that happens entirely outside your code — in the terminal, one time (or once per model). Here's what's actually happening at each step, and why.

Step 1: Install Ollama itself

curl -fsSL https://ollama.com/install.sh | sh

(On Windows/Mac, you'd download the installer from ollama.com instead — this command is Linux/Chromebook-Crostini specific.)

What this actually installs: Ollama isn't a model — it's a model server and manager. Think of it like Docker, but for AI models instead of containers: it handles downloading, storing, loading into memory, and serving models over a local API. Without it, you'd have to manually handle model file formats, quantization, and inference code yourself — Ollama abstracts all of that into one command-line tool.

Step 2: Start the server

ollama serve

Why this is a separate step from installing: Installing puts the program on your disk; serve actually starts it running and listening on a port (11434 by default). This is what your Python script's requests.post("http://localhost:11434/...") is talking to — without this running, there's nothing on the other end of that request, and you'd get a connection-refused error.

Why it often starts automatically: Most installers set Ollama up as a background service, so after a reboot you usually don't need to run this manually — it's already listening. You only need it explicitly if you check and find it's not running (which is exactly what caused one of your earlier errors).

Step 3: Pull a model

ollama pull qwen2.5:0.5b

What's actually being downloaded: A model "pull" fetches a large binary file — the model's weights (the billions of numbers that encode everything it learned during training) — from Ollama's model registry, similar to how docker pull fetches a container image. For a 0.5B parameter model, expect roughly 300-400MB; a 7B model is several gigabytes.

Why the tag (:0.5b) matters so much: The same base model name often has multiple size variants hosted under it. ollama pull qwen2.5 with no tag grabs a default size — which might be far bigger than your hardware can handle. Always specify the exact variant, especially on a 4GB machine.

Where it's stored: Downloaded models live in ~/.ollama/models on disk, cached permanently until you delete them — pulling the same model again later is instant.

Step 4: Confirm it's actually there

ollama list

Why check before writing any code: This shows you the exact model name/tag string as Ollama sees it. Copy this exactly into your Python script's MODEL variable — a typo here (missing the tag, wrong capitalization) causes a 404 from Ollama's API, not a helpful error message.

Step 5: Sanity-check it works, before involving Python at all

ollama run qwen2.5:0.5b

Why test in the terminal first: This drops you into a direct chat with the model, no code involved. If this works — you type something, you get a coherent reply — you've confirmed the model downloaded correctly and your hardware can actually run it. This isolates "is the model itself working" from "is my Python script working," which matters a lot when debugging later — if this terminal test fails, the problem is in Ollama/the model, not your code.

What "loading" actually means here: The first message after starting ollama run is often slower than the rest — that's the model's weights being read from disk into RAM. Subsequent messages in the same session are faster because it's already loaded.

Type /bye or Ctrl+D to exit this test chat.

Step 6: Now point your script at it

Once steps 1-5 all check out, your Python script (from the last post) just needs the exact model string from ollama list, and it'll work — because all your script does is send HTTP requests to a server that's already confirmed to be running with that exact model loaded.

Why the general pattern applies beyond Ollama

Every local-model runner — Ollama, LM Studio, llama.cpp — follows this same shape: install the runner → download the weights → start a server → confirm it responds → point your code at it. The specific commands differ, but the mental model doesn't. Once you understand why each of these five steps exists, switching tools later is mostly just learning new command names for the same underlying steps.

---