The companion chapter covers what the categories mean and why anyone publishes a model for free. This one is practical: getting one, running it, and changing what it does.

Hugging Face is where essentially all of this lives — a public repository of models, datasets and the libraries to use them. It is to models roughly what npm or PyPI is to packages: not the only place, but the one everything defaults to.

Each model has a model card, and reading it before downloading is the difference between a working afternoon and a wasted one. What to actually look at:

• The licence, and whether it is the permissive kind. Some repositories require you to accept terms before the download unlocks.
• Base or instruction-tuned. A base model only continues text; ask it a question and it may write more questions. An instruct or chat variant has been tuned to follow instructions. Almost always you want the instruct variant, and this is the single most common first-time mistake.
• Context length — how much text it can consider at once.
• The prompt template. Models are trained with specific markers around the system, user and assistant turns. Use the wrong template and quality drops sharply for no visible reason. Modern tooling usually applies it for you; when output is mysteriously poor, check this first.
• Whether the numbers were reported by the publisher. They usually were.

You will also see the weights split across numbered shards with an index file — that is just chunking for large downloads, and the tooling reassembles them.

Four tools cover almost every situation, and choosing wrongly is the usual cause of a frustrating start.

# --- five minutes to a working local model -----------------------
curl -fsSL https://ollama.com/install.sh | sh

ollama run qwen3:8b            # pulls ~5 GB, then you are talking to it

# it also serves an OpenAI-compatible endpoint on localhost:11434,
# so most existing client code works by changing two lines:

python3 - <<'PY'
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")
r = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Summarise this in one line: ..."}],
)
print(r.choices[0].message.content)
PY

# --- production serving, same weights, very different engine -----
pip install vllm

vllm serve Qwen/Qwen3-8B \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --tensor-parallel-size 2          # split across 2 GPUs

# Also OpenAI-compatible. The difference from Ollama is not the API --
# it is continuous batching, which is what makes concurrent users viable.

A note on where the memory actually goes. The weights are a fixed cost. The KV cache — the model's running notes on the conversation so far — grows with context length and with the number of simultaneous users, and on a busy server it can rival the weights. That is the number that surprises people in production, and it is covered properly in Inference economics.

Mixture-of-experts models break the arithmetic in a useful direction. A model described as having, say, 100 billion parameters may activate only a few billion per token — so it is fast like a small model while still needing memory like a large one. Read the model card for both numbers; the headline is usually the total.

This is the ladder. Work down it, and stop at the first rung that solves your problem — the costs rise by orders of magnitude and the top two rungs are almost never the right answer for a first attempt.

# --- LoRA: what fine-tuning actually looks like now --------------
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                 # rank -- how much capacity the adapter has
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],   # which layers get an adapter
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 8,034,000,000 || trainable%: 0.05

# 0.05% of the parameters. That is why this fits on one GPU, why the
# result is a few megabytes, and why you can keep several adapters
# for different tasks and swap them at serving time.

Stated plainly, because the decision is usually made on price and the price is not the whole cost.

• The safety layer is now yours. A provider's endpoint sits behind filtering, abuse detection and a policy team. A downloaded model has whatever refusal behaviour its training left it with, and that can be removed by anyone with modest resources. If your application is user-facing, you are building the guardrails.
• Upgrades stop being free. A better model appearing means an evaluation, a migration and a redeployment — not a changed string in a config file.
• You need evaluations more, not less. With a hosted model you can at least assume the vendor tested it. Here, the only evidence that your quantised, adapted model still works is evidence you generated. See Evaluation.
• The economics have a crossover point, and it is a real one. Self-hosting is fixed cost against an API's variable cost, so it wins at sustained high volume and loses badly at low or spiky volume — an idle GPU bills exactly the same as a busy one. Inference economics has the arithmetic.

And the reasons that have nothing to do with cost, which are frequently the real ones: data that legally cannot leave a network; a model you need to still exist and behave identically in five years; latency that a round trip cannot meet; or a language or domain no commercial provider will prioritise. Any one of those justifies the work on its own, and none of them shows up in a price comparison.