How to Run a Local LLM with Ollama

Ollama is a local runtime that downloads and runs open-weights large language models on your own machine: no cloud account, no API key, no per-token bill. Running one takes a single command, ollama run qwen3.5. The model downloads on first use, the server exposes an OpenAI-compatible HTTP API on localhost:11434, and everything runs entirely offline. As of August 2026, the practical local-LLM stack is Ollama as the runtime, a chosen open-weights model (Qwen 3.5, Gemma 4, OpenAI's gpt-oss, or the old Llama 3.1 workhorse), and a hardware floor of 16GB unified memory on a Mac or 12GB VRAM on a discrete GPU. A 16GB card like the RTX 5080 is the comfortable mainstream. I'll walk install, model selection, the hardware floor, and the cases where local is actually better than cloud (and the cases where it isn't).

Local AI became a real choice in 2026 for three reasons. Open-weights models have caught up to last year's mid-tier cloud models on many practical tasks: OpenAI now publishes its own open-weights line, and Google's and Alibaba's open families sit near the top of the blind-vote leaderboards. The hardware to run them keeps getting cheaper per token. And the privacy story is honest: no token ever leaves the machine. For specific workflows (offline use, sensitive data, high-volume batch where token costs matter), local is the better answer. For everything else, cloud still wins on quality and capability.

Jump to:

• Install Ollama
• Pull and run a model
• The hardware floor
• Picking a model: Qwen, Gemma, gpt-oss, Llama
• Calling Ollama from JavaScript and Python
• When local beats cloud (and when it doesn't)
• Quantization: trading quality for memory
• FAQ

Install Ollama

macOS:

brew install ollama
ollama serve  # starts the local server on :11434

Alternatively, download the desktop app from ollama.com; it bundles a chat GUI and manages the server for you.

Linux:

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama   # the installer registers and starts the systemd service

Windows: download the installer from ollama.com. Ollama uses CUDA on NVIDIA cards, ROCm (or Vulkan) on AMD, and falls back to CPU otherwise.

Verify with ollama --version (Ollama is on the 0.x series; 0.32.x is current as of August 2026).

Two behavior changes worth knowing if you last touched Ollama in 2025. Since 0.32 (July 2026), running bare ollama with no arguments opens an interactive agent session (chat, coding, delegated work; some features, like web search, sign into Ollama's cloud) instead of printing help; the plain-chat path in this guide, ollama run <model>, works exactly as before. And the registry now lists models with cloud tags (gpt-oss:120b-cloud-style names): those execute on Ollama's hosted service, not your machine, so they are outside the scope of a local setup. On Apple Silicon, Ollama now pairs an MLX engine with the classic llama.cpp path (support is per model; look for -mlx tags), which is where the recent Mac-side speed gains come from.

Pull and run a model

The simplest happy path:

ollama run qwen3.5

This downloads Qwen 3.5 9B (a 6.6GB pull) and drops you into an interactive chat. Type a message; the model responds streaming. Exit with /bye.

ollama run llama3.1 still works, and Llama 3.1 8B (4.9GB) remains the single most-pulled model in the registry, but it is a mid-2024 model at this point: fine for basic chat, summarization, and RAG, weak by 2026 standards for tool use. Ollama 0.32 even shows a deprecation warning before using Llama 3.x, Qwen 2.5, or Mistral 7B as agent models.

Note that ollama run llama3.3 is a different beast again: Llama 3.3 ships in a single 70B size only, so that command pulls a roughly 43GB download and needs serious hardware. For laptop-tier use you want one of the small pulls below, not a 70B.

For specific model variants:

ollama pull gemma4:12b        # 7.6GB; multimodal (text, image, audio), 256K context
ollama pull gpt-oss:20b       # 14GB; OpenAI's open-weights reasoning model
ollama pull qwen3.5:35b       # 24GB; MoE, ~3B active parameters per token
ollama pull qwen3-coder:30b   # 19GB; coding and agentic work
ollama pull deepseek-r1:8b    # 5.2GB; compact reasoning distill
ollama pull llama3.3          # 70B only, ~43GB; needs serious hardware

List installed models:

ollama list

Remove a model:

ollama rm llama3.1

The hardware floor

The practical hardware floor for a local LLM is 16GB of unified memory on a Mac or 12GB of VRAM on a discrete GPU; below that, even a small model plus its context struggles to fit. What each common setup can run comfortably:

The RTX 5080 ships 16GB of GDDR7 at 960GB/s of memory bandwidth, and memory bandwidth is the main thing token generation speed scales with: an 8B-9B model at 4-bit decodes at roughly 100-150 tokens per second on this card (ballpark, at short context), and gpt-oss:20b (a 14GB pull with only ~3.6B active parameters per token) fits and flies. It is also the card a 2026 gaming-plus-AI build most often ends up with, which is why it gets its own note here. What does not fit: 24B-27B dense models at Q4 are 15-17GB pulls that spill past 16GB once the context cache is added, and speed falls off a cliff as layers offload to system RAM. Two buying caveats as of August 2026: street prices sit well above the $999 MSRP (price trackers put the August median around $1,400, a side effect of the memory supply crunch), and the long-rumored 24GB RTX 5080 Super remains unannounced and unshipped. If the box is purely for LLM inference rather than gaming, a used 24GB RTX 3090 still buys more VRAM per dollar.

The Apple Silicon advantage is still real in 2026, but it is no longer unique. Unified memory means the GPU can address most of system RAM, so a MacBook Pro with 48GB holds the 24GB-class pulls (Qwen 3.5 35B) that need a 32GB RTX 5090 to fit comfortably on the discrete-GPU side. What changed this year is that x86 caught up on the concept: 128GB unified-memory boxes built on AMD's Ryzen AI Max+ 395 (Framework Desktop, GMKtec EVO-X2 and friends, roughly $2,000) and NVIDIA's DGX Spark ($4,699 after February's memory-shortage price hike, full CUDA) hold gpt-oss-120b entirely in memory. That class of machine, not a stack of GPUs, is now the budget path to 100B-class models at home.

CPU-only inference works but is slow: 1-3 tokens per second on a typical 8B model versus 30-150 tokens/sec on GPU. CPU is fine for "I'm trying it out" but not for real workflows.

Picking a model: Qwen, Gemma, gpt-oss, Llama

The open-weights landscape turned over almost completely between mid-2025 and mid-2026. The families that matter now:

Qwen 3.5 and 3.6 (Alibaba): the general-purpose default. Qwen 3.5 (February 2026, Apache 2.0) is multimodal, carries a 256K context, and spans 0.8B to a 122B MoE; the 9B dense (6.6GB), the 27B dense (17GB), and the 35B-A3B MoE (24GB; 27B-class quality with ~3B active parameters per token, so it decodes like a small model) are the sweet spots. Qwen 3.6 followed with 27B and 35B sizes tuned for agentic coding; if you have room for a 17-24GB pull, take 3.6.

Gemma 4 (Google): April 2026, Apache 2.0 this time. The 12B "unified" variant (added in June) takes text, images, and audio in one 7.6GB pull, which makes it the best laptop all-rounder; the 31B debuted around third among all open models on the LMArena text leaderboard. There is also a 26B MoE with ~4B active parameters.

gpt-oss (OpenAI): OpenAI's open-weights line (August 2025, Apache 2.0). gpt-oss:20b is a reasoning MoE that fits 16GB cards and 16GB Macs; gpt-oss:120b (65GB) is the strongest thing you can run on a 96GB-128GB box. Both are strong at tool use and ship in their native MXFP4 quantization.

DeepSeek-R1 and V4: the R1 distills (1.5B to 70B; deepseek-r1:8b is a 5.2GB pull) remain the compact local reasoning staple: slower per response, because they generate reasoning chains, but they solve harder problems. DeepSeek V4 (April 2026, MIT license) is open-weights too, but V4-Flash is a 284B MoE and V4-Pro a 1.6T MoE: datacenter territory, and on Ollama V4-Flash is offered as a cloud model rather than a local pull. The much-rumored R2 never shipped.

Llama 3.1 / 3.3 (Meta): the 2024 workhorses. Still the most-pulled models in the registry as of August 2026, still fine for chat, summarization, and RAG. But Meta has shipped nothing new since Llama 4 Scout and Maverick (April 2025), whose MoE sizes start at 109B and never displaced 3.x locally, and Ollama now warns before using 3.x as an agent model. A safe default that is visibly aging.

Mistral Small (Mistral): Small 3.2 (24B, a 15GB pull) is the current Ollama option and is particularly strong on European-language tasks. Mistral Small 4 (March 2026, Apache 2.0) rebooted the line as a sparse 119B MoE (about 6.5B active) that folds reasoning, multimodal understanding, and agentic coding together, but it has not landed in the Ollama registry, and at that size it is unified-memory-box territory anyway; grab it from Hugging Face if you want it today.

For code-heavy workflows, Qwen3-Coder 30B (19GB, MoE) is the open-weights default; it replaced Qwen2.5-Coder, which Ollama now flags as deprecated for agent use.

Run ollama show <model> after pulling to see the exact parameter count, quantization, and context length; the model's registry page on ollama.com lists the download size up front, before you commit to the pull.

Calling Ollama from JavaScript and Python

Ollama exposes an OpenAI-compatible HTTP API. Use any OpenAI SDK pointed at localhost:11434.

JavaScript:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama",  // ignored, but the field is required
});

const response = await client.chat.completions.create({
  model: "qwen3.5",
  messages: [{ role: "user", content: "Hello, world." }],
});

console.log(response.choices[0].message.content);

Python:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="qwen3.5",
    messages=[{"role": "user", "content": "Hello, world."}],
)
print(response.choices[0].message.content)

Streaming, tool use, and structured outputs all go through the same request shapes as cloud OpenAI (the compatibility layer covers the Chat Completions surface; not every OpenAI endpoint is implemented). The native Ollama API at /api/generate and /api/chat is available too, with slightly different request shapes.

When local beats cloud (and when it doesn't)

Local is the right choice when:

• Privacy is non-negotiable. Medical, legal, financial data that cannot leave the network. No cloud-provider trust required.
• You're offline. Travel, airplane, ship, anything without reliable internet.
• High-volume batch. Processing 100K documents overnight: the per-token cost drops to electricity, and on hardware you already own and keep busy that lands 1-2 orders of magnitude below cloud API pricing.
• You want predictable latency. Local model is local-network latency; no API rate limits, no provider outages.
• You're learning or experimenting. No API budget needed.

Cloud is the right choice when:

• Best-in-class quality matters. Claude Opus 5 and GPT-5-class models still outperform every open-weights model on the hardest reasoning and agentic tasks, though the gap in 2026 is the narrowest it has ever been.
• You need very long context, reliably. Open models now ship 128K-256K token windows (Qwen 3.5 and Gemma 4 both run 256K), but the KV cache for a long window eats VRAM fast, so on consumer hardware you rarely get to use them in full. Cloud models handle 200K-1M contexts without you thinking about memory.
• You want zero ops. Cloud APIs require no hardware management, no model updates, no driver issues.
• You need agentic tool use at scale. Cloud models are still more reliable at correct tool selection across complex toolsets, though gpt-oss and the current Qwen and Gemma generations have closed much of the gap.

The honest 2026 answer: most production AI workloads still run on cloud LLMs, but the local slice is growing. A 27B-35B open model now covers a real share of everyday tasks, and 128GB unified-memory boxes put 100B-class models within a home budget. Local is no longer a toy; it still isn't a wholesale replacement.

Quantization: trading quality for memory

A "70B parameter" model at full precision needs 140GB of memory (2 bytes per parameter). At 4-bit quantization, it needs ~40GB. At 2-bit, ~20GB. Less memory means smaller hardware can run bigger models.

Ollama's default tag is Q4 (4-bit quantization) for most GGUF-era models, which gives the best size-quality trade-off in practice. Going lower (Q2, Q3) saves memory but visibly degrades output quality. Going higher (Q5, Q6, Q8) improves quality but takes more memory; the gains above Q4 are small.

To run a specific quantization: ollama pull llama3.3:70b-instruct-q4_K_M or :70b-instruct-q8_0.

Newer models increasingly ship in native low-precision formats rather than post-hoc quants: gpt-oss comes in MXFP4, and the registry now carries mxfp8 and nvfp4 variants for recent NVIDIA cards plus -mlx variants for Apple Silicon. The advice is unchanged either way: take the default tag, and worry about quantization only if you hit a memory ceiling and need to fit a bigger model.

What to do next

For combining local LLMs with the broader AI stack:

• How to Build an LLM Agent with Tool Use: Ollama supports tool use; the agentic loop is identical to cloud.
• How to Build RAG with Embeddings and Vector Search: local embedding models (BGE, E5) pair well with local LLMs for full-stack on-device RAG.
• How to Get Reliable JSON from an LLM: Ollama supports structured outputs via the format parameter.

For the choice between local and cloud for specific workloads:

• How to Choose Between Claude Haiku, Sonnet, and Opus: covers the cloud-tier decision; pair with this article for the local-vs-cloud decision.

External references: Ollama documentation, LMArena leaderboard for current open-vs-closed model rankings.

FAQ