How large language models are built

This article follows one large language model from raw data to a released product. Each stage solves a problem the previous stage left open. Knowing the stages explains many things users notice: why models know so much, why they still invent facts, and why two models trained on similar data behave differently.

The pipeline at a glance

flowchart LR
  data["Collect and clean data"] --> pre["Pre-training: predict the next token"]
  pre --> sft["Instruction tuning"]
  sft --> pref["Preference and reinforcement learning"]
  pref --> test["Safety testing and evaluation"]
  test --> serve["Release and inference"]

Pre-training produces a base model that knows a great deal but only continues text. The later stages, together called post-training, turn it into an assistant that answers questions, follows instructions, and refuses harmful requests. Pre-training uses most of the compute; post-training shapes most of the behavior people see.

Collecting and preparing data

A frontier model learns from trillions of tokens of text. The main sources are:

• Web pages, often starting from public crawls such as Common Crawl.
• Books, scientific papers, and encyclopedias.
• Source code from public repositories.
• Licensed data from publishers and other partners.
• Synthetic data generated by earlier models, increasingly used for math, code, and reasoning.

Raw web data is mostly unusable. Teams remove duplicates, spam, and boilerplate, filter out low-quality and harmful text, remove personal data where they can, and balance the mix of languages and topics. Data quality matters as much as quantity: a smaller model trained on carefully filtered data can beat a larger one trained on everything.

The data also defines the knowledge cutoff. Whatever happened after the data was collected is unknown to the model unless an application provides it later.

Which data may be used for training is contested. Authors, news publishers, and artists have sued AI companies over copyright, and the EU AI Act requires providers of general-purpose models to publish a summary of their training data.

Tokenization

Before training, all text is split into tokens by a tokenizer. The tokenizer is built from the data itself: frequent words become single tokens, rare words are split into pieces. A typical vocabulary has tens of thousands to a few hundred thousand tokens.

The tokenizer explains some odd model behavior. A model that sees “strawberry” as two or three tokens does not see the individual letters, which is why early models miscounted them. Languages that are rare in the training data are split into more tokens, so they cost more and fit less into the context window.

Pre-training

Pre-training teaches the model one task: given a sequence of tokens, predict the next one. The model sees a piece of text, guesses the next token, compares its guess with the real token, and adjusts its parameters slightly. This repeats over the whole dataset.

To predict the next word of a physics textbook, a legal contract, or a Python file well, the model has to pick up grammar, facts, styles, and patterns of reasoning. Nobody writes these into the model; they are a side effect of getting the predictions right.

Nearly all large language models use the Transformer architecture. Its key mechanism, attention, lets every token look at every earlier token and decide which ones matter for the prediction. In “The trophy did not fit in the suitcase because it was too big”, attention helps the model connect “it” to “trophy”. A model stacks dozens of these attention layers.

Pre-training a frontier model runs for weeks or months on clusters of tens of thousands of GPUs or TPUs. Research on scaling laws showed that quality improves predictably with more parameters, more data, and more compute, as long as the three grow in balance. That predictability is why companies invest billions in larger training runs.

Many recent models use a mixture of experts design: the model contains many sub-networks, and only a few of them are active for each token. It has the knowledge capacity of a very large model at the running cost of a much smaller one.

Instruction tuning

A base model continues text. Ask it “What is the capital of France?” and it may continue with more quiz questions instead of answering. Instruction tuning, also called supervised fine-tuning, fixes this. The model is trained further on tens of thousands to millions of example conversations: an instruction and a high-quality answer, written by people or generated and checked with the help of other models.

After this stage the model answers questions, follows formatting requests, and holds a conversation. It has learned the role of an assistant.

Preference and reinforcement learning

Instruction tuning shows the model good answers but does not tell it which of two answers is better. The next stage does.

In reinforcement learning from human feedback (RLHF), people compare several answers to the same prompt and pick the better one. A separate reward model learns to predict these preferences, and the language model is trained to produce answers that the reward model scores highly. This makes answers more helpful, better structured, and more likely to refuse clearly harmful requests. Variants replace part of the human ratings with AI feedback guided by written principles, or skip the separate reward model.

A newer form of reinforcement learning produced reasoning models. The model is given problems with checkable answers, such as math problems or code with tests. It writes a long chain of thought before answering and is rewarded when the final answer is correct. Over many rounds it learns to plan, check its own work, and try another approach when stuck. DeepSeek’s R1 paper in early 2025 showed publicly how far this method could go.

Post-training also explains why models are sometimes overconfident or overly agreeable: they were rewarded for answers people liked, and people tend to like confident, agreeable answers.

Safety testing and evaluation

Before release, providers test the model on public benchmarks and internal evaluations. They also run red-teaming: specialists try to make the model help with cyberattacks, weapons, fraud, or other harm, and the results feed back into training and safeguards. Several labs publish a system card for major models that describes these tests, the known risks, and the mitigations. Some governments run their own tests through AI safety or security institutes.

Release and inference

The finished model is deployed for inference on large server clusters, or its weights are published for others to run. Several techniques make it cheaper to serve:

• Distillation trains a smaller model to imitate a larger one. Many small and fast models are made this way.
• Quantization stores parameters with fewer bits, for example 4 or 8 bits instead of 16. The model needs less memory and runs faster, with a small loss of quality.
• Caching reuses the computation for text that many requests share, such as a long system prompt.

A released model does not keep learning from conversations. Improvements arrive as new versions, trained again through all or part of this pipeline.