LLM runtimes and inference

Inference: the process#

Inference is not a piece of software. It is the process of running a trained model forward: you hand it an input, it computes an output. Training is where the model learns its parameters; inference is every use of the model afterwards.

"You are a helpful assistant. What is 2+2?"
                    │
                    ▼
                LLM model
                    │
                inference
                    │
                    ▼
                   "4"

Concretely, one round of inference means:

• Load the weights into memory — all of them, and they stay there.
• Tokenize the input into the integer ids the model was trained on.
• Run the layers — a long chain of matrix multiplications and attention operations over those tokens.
• Produce a distribution over the vocabulary: a probability for every possible next token.
• Sample one token from it, according to your temperature and top-p settings.
• Repeat, with the new token appended, until the model emits a stop token or hits your limit.

That last line is the one people underestimate. A model does not compose a reply and hand it over. It re-runs the entire network once per token. A five-hundred-word answer is roughly seven hundred passes over the weights.

Which is why memory bandwidth, not raw compute, is usually what limits local inference: the machine spends its time moving gigabytes of parameters from memory to the processor, over and over, for every single word.

Two phases, two bottlenecks#

Inference splits into two phases with completely different performance characteristics, and knowing which one you are watching explains most of what feels strange about local models.

Prefill processes your prompt. Every token of it can be computed in parallel, so the hardware runs wide and — on a GPU at least — saturates its arithmetic units, making prefill compute-bound. This is the pause before the first word appears: long prompt, long pause.

Decode generates the reply, one token at a time. Each token depends on the one before it, so there is nothing to parallelize; the machine reads the whole model to produce a single token, and is limited by memory bandwidth. This is the steady stream you watch, and it is what “tokens per second” usually means.

  PREFILL                     DECODE
  ───────                     ──────
  All prompt tokens           One token at a time
  at once                     each needs the last
      │                            │
  compute-bound               bandwidth-bound
      │                            │
  "time to first token"       "tokens per second"

The bridge between them is the KV cache: prefill stores the attention keys and values for every prompt token so decode does not recompute them. It buys enormous speed and costs real memory, growing linearly with context length. We put the arithmetic for that in Local LLM fundamentals.

The inference engine#

An inference engine is the software that actually performs the computation. It is the layer where the maths lives.

Its responsibilities are narrow and deep:

• Kernels — the hand-optimised routines that multiply matrices and compute attention on a specific processor.
• Backends — CUDA for NVIDIA, Metal for Apple, Vulkan and ROCm for AMD, plain vectorised CPU code as a fallback.
• Quantized arithmetic — how to do useful maths on 4-bit weights without unpacking everything back to floats.
• Memory layout — where the weights, the KV cache and the activations live, and which parts are offloaded to the GPU.
• Batching — combining work so the hardware is not left idle.

llama.cpp, MLX, vLLM, TensorRT-LLM and ONNX Runtime are all engines. They take a model and some hardware and produce tokens.

          Inference engine
                 │
        ┌────────┴────────┐
        │                 │
      Model           Hardware
        │                 │
     Qwen 3          GPU / CPU
        │
        ▼
    Inference

What an engine deliberately does not do is find your model, pick a quantization, remember your settings, or expose a stable API for other programs. It expects to be handed a file and a configuration.

The runtime#

A runtime is the broader environment that manages running models. It contains an engine, and adds everything the engine leaves out.

                    LLM runtime
                         │
       ┌─────────────────┼─────────────────┐
       │                 │                 │
Model management    Inference          API server
       │                 │                 │
  Download           llama.cpp          HTTP API
  Storage            CUDA / Metal       Streaming
  Quantization       CPU / GPU          Sessions
  Loading            KV cache           Queueing

The analogy that makes it stick is a language runtime. The Python runtime is not the same thing as the bytecode interpreter inside it; it is the interpreter plus module loading, memory management, the standard library and the process model.

Python runtime            LLM runtime
     │                         │
     └── executes              └── executes
         Python code               LLM inference

This is why Ollama is properly called a runtime rather than an engine: the inference happens in a backend it delegates to, and what Ollama contributes is the model library, the sensible defaults, the background service and the API. It is also why the boundary is soft — llama.cpp ships llama-server and a model downloader, which makes it an engine that has grown a runtime around itself.

The useful test is what the layer is responsible for:

Serving: one model, many users#

A serving engine is a runtime specialised for one situation: a single model, a GPU or several, and a queue of simultaneous requests. It is a distinct layer because concurrency creates problems that single-user inference simply does not have.

Two of its techniques are worth understanding, because they explain the whole category.

Paged attention — vLLM calls its implementation PagedAttention — manages the KV cache the way an operating system manages memory: in fixed-size blocks, allocated on demand. Naively, each session reserves cache for its maximum possible context, so ten users at 32k reserve ten full allocations and mostly waste them. Paging hands out blocks as each conversation actually grows, which often means several times as many sessions on the same card.

Continuous batching attacks the other waste. Static batching groups requests, runs them together, and waits for the slowest to finish before starting the next group — so a batch containing one long answer leaves the GPU largely idle. Continuous batching drops finished sequences out of the batch and slots new requests in at every step.

                       ┌── User 1
                       ├── User 2
   Your API ──→ vLLM ──┼── User 3
                       ├── User 4
                       └── User 5
                             │
                        NVIDIA GPUs

Both only pay off under concurrency. For one person, one model, one machine, they buy nothing — which is why “vLLM is much faster” is true in a data centre and irrelevant on a laptop.

The stack, drawn#

Putting all four software layers together:

┌───────────────────────────────────────────────┐
│               Your application                │
│           chat · RAG · agent · IDE            │
└───────────────────────┬───────────────────────┘
                        │ OpenAI-compatible API
          ┌─────────────┴─────────────┐
          ▼                           ▼
┌───────────────────┐       ┌───────────────────┐
│    LLM runtime    │       │  Serving engine   │
│ models · config   │       │ batching · paging │
│ API · sessions    │       │ scheduling        │
│ ┌───────────────┐ │       │ ┌───────────────┐ │
│ │   Inference   │ │       │ │   Inference   │ │
│ │    engine     │ │       │ │    engine     │ │
│ └───────┬───────┘ │       │ └───────┬───────┘ │
└─────────┼─────────┘       └─────────┼─────────┘
          │                           │
          └─────────────┬─────────────┘
                        ▼
                 CPU · GPU · NPU

The two middle boxes are alternatives, not a sequence: a runtime and a serving engine are the same layer solving the same problem for different numbers of users. A desktop client like LM Studio or Jan sits in the top box, and usually bundles its own runtime rather than talking to somebody else’s.

A video game is the cleanest analogy for the three terms. Inference is the frame being rendered. The inference engine is the graphics engine that renders it. The runtime is the whole environment managing the game, the engine, assets, memory and input.

Where the familiar names fit#

With the layers in place, the tools everyone argues about sort themselves out. They are not four answers to the same question; they occupy different rows of the same stack. Asking whether Ollama is better than llama.cpp is close to asking whether a car is better than its engine.

One wrinkle worth knowing, because most explanations are a version behind. Ollama and LM Studio both began life as llama.cpp front ends, and that is still the mental model people carry. LM Studio still runs llama.cpp, plus MLX on Apple Silicon. Ollama has since built an engine of its own on GGML — the tensor library llama.cpp itself is written on — and routes newer models through that, falling back to llama.cpp elsewhere. The layering did not change; the engine under the runtime did.

• llama.cpp — the engine the local ecosystem grew out of, directly or through GGML. Runs quantized GGUF models on CPU, CUDA, Metal, Vulkan and ROCm, and exposes every knob: threads, GPU layers, context, batch size, the CPU/GPU split. New architectures land here first. Demanding, and maximally portable.
• Ollama — the runtime, and effectively Docker for local models. ollama run qwen3 pulls the weights, picks defaults for your hardware, and leaves a service on localhost:11434 speaking an API most local AI tooling already understands. That ubiquity is why it dominates application development; the cost is less low-level control.
• LM Studio — the desktop client. Search, download, read model cards, drag the offload and context sliders, chat, watch tokens per second, and start a local API server, without touching a terminal. The best tool in the ecosystem for deciding which of six models feels better on your machine.
• vLLM — the serving engine, for when the machine is really a GPU server with a request queue. Paged attention and continuous batching, as above.

Two rows are worth reading twice. HTTP API is uniform — every one of them will serve your application, so that is not the axis to choose on. And portability is where vLLM pays for its throughput: it is the only column here that assumes a particular kind of machine.

Which makes the practical choice short: Ollama to build against, LM Studio to explore with, llama.cpp to embed, vLLM to serve. The full ranking is below, once the rest of the field is on the table.

The fastest-runtime myth#

The layer model also disposes of the most common argument in local AI. You will read all three of these, often in one thread:

• “Ollama is slower than llama.cpp.”
• “LM Studio is faster than Ollama.”
• “Just use llama.cpp, it’s the fastest.”

If two tools run the same model through the same kernels on the same hardware, the arithmetic is identical — the inference is the same inference, and a wrapper cannot make a matrix multiplication faster. The differences people measure come from the configuration around it:

• a different engine version underneath
• a different backend, or a CPU-only build where a CUDA build was expected
• a different quantization of nominally the same model
• different context size, batch size or number of offloaded layers
• different thread counts, memory settings or cache behaviour
• prefill time counted as decode time, or application overhead counted as inference

Carefully matched comparisons of llama.cpp, Ollama and LM Studio tend to land much closer together than headline numbers suggest. The dramatic “I switched and gained 40%” reports are usually real — but what they measured was a configuration change wearing a runtime’s name.

The wider landscape#

The four above are the ones you meet first. The rest of the ecosystem sorts into the same layers.

Desktop clients

All-in-one graphical tools: one-click downloads, a chat window, automatic hardware configuration.

• LM Studio — the leading visual client on Windows, macOS and Linux.
• Jan — open source, modular, offline-first; the closest thing to a local ChatGPT replacement.
• AnythingLLM — built around private retrieval: turns your own documents into a local vector knowledge base.
• GPT4All — Nomic’s ecosystem, tuned to run acceptably on ordinary CPUs with no discrete GPU.
• Msty — a power-user client with side-by-side model comparison and multi-chat workspaces.
• text-generation-webui — the deeply configurable web UI, with every parameter exposed.
• Backyard AI (formerly Faraday) — specialised in offline character roleplay and creative writing.

Runtimes and developer tools

Terminal-driven, scriptable, easy to drop into a codebase.

• Ollama — the popular container-style runner and model manager.
• llamafile — Mozilla’s trick of collapsing weights and runtime into one executable that runs across operating systems.
• LocalAI — a self-hosted, drop-in replacement for the OpenAI API specification.
• llama-server — llama.cpp’s own HTTP server, if you want the engine without a wrapper.

Serving engines

Built for throughput and concurrency rather than for one person at a laptop.

• vLLM — the de facto standard for high-throughput serving.
• SGLang — optimised for complex, structured, multi-turn programs with heavy prefix reuse.
• TGI — Hugging Face’s production-oriented serving backend.
• TensorRT-LLM — NVIDIA’s open-source but NVIDIA-only library, compiling models for peak performance on RTX and data-centre GPUs.

Inference engines

The libraries that perform the computation underneath everything above.

• llama.cpp — the cross-platform C/C++ engine behind most local, quantized inference.
• MLX — Apple’s array framework, written for unified memory and the Metal GPU on M-series chips.
• ExLlamaV2 — a fast inference path built specifically for consumer NVIDIA cards.
• ONNX Runtime — the cross-framework engine, common where models ship as ONNX rather than GGUF.

Adjacent: training and fine-tuning

Frequently listed alongside runtimes, but solving the opposite problem — making a model rather than running one.

• Unsloth — fine-tuning open models locally with sharply reduced memory and time cost.
• Oumi — an end-to-end platform for training, fine-tuning and evaluation pipelines.

What to use, ranked#

Everything above, reduced to a lookup. Find the row that describes what you are doing; the first column is where to start, and the next two are the ones worth trying if it does not suit you.

Three choices, not one#

One last confusion the layers dissolve. Running a model locally is three independent decisions that get collapsed into one:

1. The model. Qwen, Llama, Gemma, Mistral, DeepSeek, Phi.
2. The format and quantization. Q4_K_M, Q5_K_M, Q8_0, MLX 4-bit, BF16.
3. The runtime. llama.cpp, Ollama, LM Studio, MLX, vLLM.

These are close to orthogonal. The same model runs under all of those runtimes; the same runtime serves a dozen quantizations; and the quantization you can afford depends on your memory budget, not your choice of tool. “Which runtime is best” is a question about the third axis, asked as though it settled the other two.

The decision that actually determines whether the thing works is the second one, and it is settled by arithmetic — parameters, bits per weight, KV cache, bandwidth. That arithmetic is written out in Local LLM fundamentals.

The layer above#

Every layer described so far assumes you have already answered the question it cannot answer: what should I run? An engine expects a model file. A runtime expects a model name. Neither knows what your machine can carry.

That gap is a layer of its own — a hardware-aware runtime manager: a program that detects the available hardware, evaluates which models and inference backends that hardware supports, and configures the best local AI environment for that specific machine. It is what ModelFit is.

                  ModelFit
                      │
              Hardware detection
                      │
         ┌────────────┴────────────┐
         │                         │
  Hardware profile           Model registry
         │                         │
         └────────────┬────────────┘
                      ▼
             Compatibility engine
                      ▼
             Model recommendation
                      ▼
               Benchmark engine
                      ▼
              Runtime selection
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
   llama.cpp        Ollama          MLX
       │              │              │
       └──────────────┼──────────────┘
                      ▼
                 Local model

It reads your CPU, RAM, GPU and VRAM; works out which backends the machine actually supports; applies the memory and bandwidth arithmetic to every model in its registry; picks the quantization that fits; measures real throughput instead of trusting a spec sheet; and tells you what to run and where.

The runtimes are good. Working out which model, at which quantization, on which backend, for your machine is the part still being done by hand.