Last updated
Ollama belongs in a different category from almost everything else on this site. It is not an assistant, and comparing it with ChatGPT or Claude is a category error in the same way as comparing a database server with a spreadsheet. Ollama is a runtime: it downloads open-weight models, manages them, and serves them from your own machine over a local API. What you do with that is up to whatever you point at it.
Its contribution was making this boring. Running an open model locally used to mean Python environments, driver versions, and model-format conversions performed by hand. Ollama reduced it to pulling a model and running it, in roughly the spirit that container tooling once did for deployment. Everything interesting about it follows from that, including the parts that disappoint people.
The first question about any local model is not how smart it is. It is whether it fits. A model's weights have to be held in memory to be used, and if they fit in GPU memory the model runs at the speed the hardware allows. If they do not, work spills onto the CPU and system RAM and throughput falls off sharply — not slightly slower, but slow enough to change what the tool is for. This is why hardware advice in this space sounds monotonous: capability is gated by memory capacity more than by raw compute.
Practically, small models in the range most people run first are comfortable on ordinary consumer hardware, mid-sized models want a reasonably capable discrete GPU or a machine with a large unified memory pool, and the largest open-weight releases are out of reach for typical personal machines regardless of patience. Apple Silicon is unusually well suited here because the GPU addresses the same large memory pool as the CPU, which sidesteps the discrete-VRAM ceiling that limits many consumer graphics cards. One detail that catches people out: the weights are not the only thing consuming memory. Long contexts require additional memory that grows with the amount of text in play, so a model that loads comfortably can still run out of room on a long document. The case for running models on your own machine and the hardware side of it go further into the trade-offs.
Models in Ollama's library are distributed in quantized form, meaning the weights are stored at reduced numerical precision so a model that would not otherwise fit does. This is the single most important thing to understand about local inference, because it is a quality setting that most users never realise they set. Heavier quantization shrinks a model and speeds it up, and it degrades output — usually gracefully at first, and then not. The useful heuristic is that a larger model quantized more aggressively often beats a smaller model at higher precision, but the crossover point depends on the model family and on your task, and the only way to find it is to try the variants on work you actually care about rather than on a puzzle you found online.
Be honest about the trade first: open-weight models you can run on consumer hardware generally trail frontier hosted models on hard reasoning, long-horizon coding, and instruction-following precision. There is no prompt that closes that gap and no upgrade button, because the ceiling is your hardware. What you get instead is three things a hosted API cannot offer at any price. Nothing leaves the machine, which is a categorically stronger guarantee than a vendor's retention policy, however well written — the relevant comparison is what guarantees are actually available from hosted providers, and "the request never happened" wins all of them. It works with no network, which matters for air-gapped environments, travel, and anything where connectivity is unreliable. And the marginal cost of a token is zero, which changes what you are willing to build: retry loops, bulk processing, and chatty agent designs that would be reckless against a metered API become unremarkable when the only meter is electricity.
Ollama serves a local HTTP endpoint and provides an OpenAI-compatible interface, which is why the ecosystem around it grew so fast: a large amount of existing software can be pointed at a local endpoint by changing a base URL and a model name. Command-line and editor assistants take Ollama as a backend — Aider is the common pairing, since it already treats the model as a swappable component, and running it against a local model turns a per-token cost into a fixed one. A Modelfile lets you pin a system prompt and generation parameters to a named model so a particular configuration becomes reproducible rather than something you retype. The practical shape that works for most people is not all-local or all-cloud but a split: local models for high-volume, privacy-sensitive, or repetitive work, and a frontier hosted model for the hard problems.
Do not do this if you need the best available answer and have no constraint forcing the work onto your own hardware — you will spend a weekend to arrive somewhere a subscription would have taken you in a minute. Do not do it on underpowered hardware and conclude the technology is bad; you have measured your laptop, not the field. Do not choose it for a production service without accounting for what you are now operating, because a local endpoint is a service with capacity limits, upgrade work, and someone responsible when it stops. Do not assume "open weights" means "open source" — several widely used model licences carry restrictions on commercial use and redistribution, and that is a question for whoever signs off on your dependencies, not an afterthought. And do not adopt it purely to save money without doing the arithmetic: the cost of inference only favours local hardware above a certain sustained volume, and below it the hardware sits idle while a metered API would have cost less than the electricity.
Where a client agreement, a regulatory rule, or an internal policy forbids sending source code or documents to a third party, local inference is not a preference but the only configuration that complies. The argument is unusually easy to make to a security reviewer because there is no outbound request to assess.
Tools that treat the model as a swappable component can be pointed at a local endpoint instead of a cloud API, which converts a usage-based bill into a fixed hardware cost. Pairing with Aider is the common example, and it changes how freely you iterate once every retry is free.
Pulling several model families and switching between them with one command makes genuine comparison practical, which used to be a research chore involving conversion scripts and dependency conflicts. This is the fastest way to find out which open model is actually adequate for a specific job.
Classification, tagging, extraction, and first-pass summarisation over large document sets are tasks where a smaller model is usually good enough and the volume is what costs money. Running these locally removes the per-token cost from the design entirely, which is often what makes the project viable.
Air-gapped networks, field work, flights, and unreliable connectivity all rule out hosted assistants by default. A local runtime keeps working because it never needed the network in the first place, which is also what makes it a reasonable foundation for applications that must degrade gracefully.
Ollama is free and open-source software with no subscription tiers, usage limits, or per-token billing — there is no plan to compare because inference is not being sold to you. That does not make it costless. The expense moves to hardware capable of holding a useful model in memory, to the electricity to run it, and to your own time configuring and maintaining it, none of which appear on an invoice. The comparison that matters is total cost over a sustained workload rather than price per request: a metered API is cheaper for intermittent use of a frontier model, while local hardware wins when volume is high and consistent, when the task does not demand frontier capability, or when a compliance requirement has already eliminated the hosted option and the real alternative is having no AI assistance at all.
Yes — open-source software with no subscription, no usage caps, and no per-token charge. The real cost is the hardware needed to run models at a speed you will tolerate, plus the time to set it up and keep it current.
Memory capacity is the binding constraint. Smaller models run acceptably on ordinary modern machines, mid-sized ones want a capable GPU or a large unified memory pool, and the biggest open-weight releases are impractical on typical personal hardware. If a model does not fit in GPU memory, it will still run, but slowly enough that the experience changes character.
Generally not on hard reasoning, long coding tasks, and precise instruction-following, and it is worth being blunt about that rather than discovering it mid-project. Ollama's advantages are privacy, offline operation, and cost control, not leaderboard performance. Many people run both and route work by difficulty.
Some output quality, in exchange for fitting in memory and running faster. The degradation is gradual up to a point and then noticeable. A bigger model at heavier quantization frequently outperforms a smaller model at lighter quantization, but where that crossover sits depends on the model and the task, so test the variants on your own work.
Usually. It serves a local HTTP endpoint with an OpenAI-compatible interface, so a great deal of existing software can be redirected to it by changing a base URL and a model name. That compatibility is the main reason the surrounding ecosystem grew as quickly as it did.
Once a model is downloaded, yes, completely. You need connectivity to pull models and updates, and nothing after that. This is what makes it usable on air-gapped networks and the reason it appears in environments where a hosted assistant was never an option.
No, and conflating them creates real legal exposure. Several widely used model licences place conditions on commercial use or redistribution that an OSI-approved licence would not. Read the licence for the specific model you intend to ship with rather than assuming the category is permissive.
Only if the arithmetic supports it. Sustained high volume, tasks that do not need frontier capability, and hard privacy requirements all favour local. Intermittent use of a genuinely difficult task favours the API, because idle hardware is pure cost while a metered call is not. Most teams end up splitting the work rather than choosing a side.
Full review coming soon.