A working notebook · 12 stops · ~35 min

One Token at a TimeHow large language models actually work

A language model does exactly one thing: it looks at some text and scores every possible next piece of text. Everything impressive you have seen it do comes from repeating that step, very fast, with a very large function. This page takes that function apart, in the order a word travels through it.

Top 4 candidates at each step. The pick is sampled, so it is not always the favourite.
00THE LOOPtext in → one token out

The whole game, in one picture

Strip away the product and a large language model (LLM) is a function. In goes a sequence of tokens (chunks of text). Out comes a list of probabilities, one for every token the model knows, saying how likely each is to come next. That is all a single run produces.

To write a paragraph, you run it in a loop: pick one token from that list, stick it on the end of the input, and run the whole thing again. A 500-word answer is roughly 650 runs of the same function, each one seeing everything that came before, including its own previous output.

text so far "The sky over" tokens [791, 13180, 927] the model billions of numbers probabilities " the" sample one token, append it, run again repeat until a stop token or a length limit

Three facts fall straight out of this picture, and they explain most of the behaviour people find strange:

It never plans the whole answer up front. Whatever "planning" happens is encoded in the internal state the model computes each step. There is no hidden draft being revealed.

It cannot take a token back. Once a word is sampled it becomes part of the input. A bad early pick is something the rest of the answer has to live with.

The same input can give different outputs. The loop samples from probabilities rather than always taking the top choice, unless you tell it otherwise. You can tune this in stop 07.

01TOKENIZEstring → [n] integers

The model never sees letters

Neural networks do arithmetic, so text has to become numbers first. The obvious options are both bad. One number per character makes sequences painfully long. One number per word needs an enormous dictionary and still breaks on every typo, name and new word.

The compromise used almost everywhere is subword tokenization, usually a method called byte-pair encoding (BPE). It starts with single characters and then learns, from a pile of training text, which neighbouring pairs show up together most often. The most frequent pair gets glued into a new token. Then it counts again and glues the next most frequent pair. Repeat a hundred thousand times and common words become one token, while rare words get built from a few familiar pieces.

The panel below runs the real algorithm in your browser. Its training text is this page, so the vocabulary it learns is shaped by what I happened to write about. Drag the merges slider and watch the same sentence compress as the vocabulary grows.

BPE workbench · trained on this pagelearning…
The dot (·) is a leading space. Real tokenizers glue the space onto the front of the next word, so " cat" and "cat" are different tokens with different IDs.
The strawberry problem

Ask a model how many r's are in "strawberry" and it may say two. It is not bad at counting. A production tokenizer chops it into a few pieces, something like str + aw + berry, and the model receives only their IDs. (This page's little tokenizer, which has never seen the word, gives [ ].) It is being asked about letters it never saw. Whatever it knows about spelling, it had to learn indirectly from text where people talk about spelling. Modern models often get this right now, partly through training on exactly these questions, partly by spelling the word out first so that each letter becomes its own token.

Tokenization leaves fingerprints on everything downstream. Prices for API use are per token. Context limits are counted in tokens. Text in languages under-represented in the tokenizer's training data costs more tokens per sentence, so it is slower, more expensive and fits less into the window. Numbers get chopped into arbitrary chunks, which is part of why raw arithmetic was historically shaky. In English, a useful rule of thumb is that one token is about three quarters of a word.

02EMBED[n] → [n × d]

Every token becomes a point in space

A token ID like 13180 is just a row number. The first real step looks up that row in a big table, the embedding matrix, and gets back a list of numbers: a vector. In a mid-sized model each vector has a few thousand numbers (the model's width, written d). Nobody chooses those numbers. They start random and get nudged during training until tokens used in similar ways end up near each other.

"Near each other" is meant literally. Treat each vector as a point in a space with thousands of dimensions, and directions in that space start to mean things. The classic demonstration is word arithmetic: take the vector for king, subtract man, add woman, and the closest word to where you land is queen.

Below is a toy version with 11 dimensions instead of thousands, flattened to 2D so you can see it. Hover a word to see its vector as a strip of cells. Then build an equation.

Embedding space · 11-d toy, flattened to 2-dteal − · vermilion +
− + ≈ ?
hover a word
dims: royal · male/female · young · animal · food · city/country · FR · JP · AU · feline/canine · noise
The dashed arrows show the trick: "man to king" and "woman to queen" are the same displacement. A consistent direction in space has come to mean "royal".
An honest caveat

I labelled the toy dimensions so you can read them. Real dimensions are unlabelled, and a concept is usually spread across many of them at once. The tidy king/queen arithmetic comes from older word-level models (word2vec, 2013). In an LLM the starting embedding belongs to a token, which might be half a word, and it carries little meaning on its own. The rich meaning gets built up layer by layer, from context. A "bank" token starts identical in "river bank" and "bank loan"; the layers that follow are what pull them apart.

03POSITION[n × d] + where

Word order has to be added back in

Here is a surprising design fact. The attention mechanism in the next stop treats its input as an unordered set. Left alone, it would see "dog bites man" and "man bites dog" as the same three vectors in a bag. Order has to be injected on purpose.

The original 2017 transformer added a fixed wave pattern to each token's vector: a unique fingerprint for each position. Most current open models use a cleverer scheme called rotary position embedding (RoPE). Instead of adding anything, it rotates parts of each vector by an angle proportional to the token's position. Think of each token carrying a set of clock hands that have been turned according to where it sits.

The payoff shows up when two tokens are compared. The comparison (a dot product) depends on the angle between their hands. Rotate both by the same amount and that angle does not change. So the model ends up sensitive to how far apart two tokens are, not to their absolute address. Try it.

Rotary positions · one 2-d slice of a vector15° per position
A real vector is split into hundreds of such pairs, each rotating at a different speed. Fast pairs track nearby words; slow pairs track long distances, like the second, minute and hour hands on a clock.
04ATTENTION[n × d] → [n × d] · mixes across tokens

Attention: tokens reading each other

So far each token is a vector that knows only about itself and its position. Attention is the one place in the model where information moves between tokens. It is the reason the word "it" can end up carrying information about the animal it refers to.

Each token produces three vectors from its own embedding, using three learned matrices:

Query: what this token is looking for. Key: what this token can be found by. Value: what this token hands over if it is chosen.

Every token compares its query against the keys of all earlier tokens (a dot product per pair). High score means "relevant to me". The scores go through softmax, which turns them into weights that add up to 1. The token then takes a weighted average of the chosen tokens' values and adds that to its own vector. That is attention, entire. The cleverness is all in the learned matrices.

A model has many attention heads per layer, each with its own matrices, running side by side. Different heads learn different jobs. Click any word to make it the query. Switch heads to see different jobs. Then change the last word.

Attention lab · hand-built heads, real softmaxhead
last word:

The grid is the full attention pattern: one row per query token, one column per key. Darker means more weight. The hatched triangle is the causal mask: a token may only look backwards, because when the model is generating, the future does not exist yet.
Notice what the mask does to "it"

Many diagrams show "it" pointing confidently at "animal". In a model that writes left to right, that is not quite possible. When "it" arrives, the word "tired" has not been written, so "it" can only split its attention between the two candidates. The ambiguity gets resolved later, at "tired", which can look back at both "it" and "animal" and tie them together. Information about meaning often lands on later tokens, not the one you would expect.

Why the scores get divided by √d

A dot product adds up d products, so as vectors get wider the scores get bigger, purely from arithmetic. Softmax exaggerates big differences, so wide models would end up putting nearly all weight on one token, and learning stalls because the gradients through softmax vanish. The fix in the original paper was one line: divide every score by the square root of d. Below, the vectors are random numbers generated in your browser. Widen them and switch the scaling off.

Score scaling · 1 query vs 6 keysd = 64
05FEED-FORWARD[n × d] → [n × 4d] → [n × d] · per token

The feed-forward layer: where much of the knowledge sits

Attention moves information between tokens. The feed-forward network (also called the MLP) then works on each token alone. It widens the vector, typically to four times its size, passes it through a simple nonlinearity, then squeezes it back down.

That sounds dull, but it holds roughly two thirds of the model's parameters. A good mental model, backed by interpretability research, is a giant lookup of patterns. Each of the thousands of hidden units (call them neurons) checks whether the incoming vector matches some pattern. If it does, the neuron fires and writes something useful back into the vector. "This looks like a question about a capital city" might fire and push towards city names. Many facts the model "knows" seem to be stored this way.

The toy below has eight neurons. Pick a context, see which fire, and switch some off to see what the output loses. (Real neurons are rarely this clean; many respond to several unrelated things at once. More on that in stop 11.)

Pattern detectors · 8 toy neuronsclick a neuron to switch it off
what the neurons push the next token towards
The orange underline is how strongly each neuron fires for this context. Switching one off is what researchers call ablation: a standard way to test what a part of a model is responsible for.
06RESIDUAL STREAMattention + MLP = one block · × L blocks

The residual stream: a shared notepad

One attention layer plus one feed-forward layer make a transformer block. A model stacks dozens of identical-looking blocks: 32 in an 8-billion-parameter model, 80 or more in large ones. Each block has its own learned weights.

The way blocks connect is the most useful idea on this page. A block does not replace the token's vector. It adds to it. Each token carries a running vector, the residual stream, and every attention layer and every MLP reads from that stream and writes a small update back into it. It works like a shared notepad passed down a line of specialists, each adding a note.

That has a neat consequence: you can stop at any layer and read the notepad through the model's final output step, to see what it would predict if it ended there. This trick is called the logit lens. The panel below fakes a tiny six-block model on the prompt "The capital of France is". Slide the lens through the layers, then knock a layer out.

Logit lens · stream at the token " is"norm
prediction if the model stopped here
Tick boxes knock a layer out. Try removing block 3's attention: the model never copies "France" into this position, so block 4's fact lookup has nothing to fire on. Later layers depend on what earlier ones wrote.

Two supporting parts keep this stack trainable. The residual connection itself (the "add, don't replace" rule) gives the training signal a clean path back through 80 layers, which is a big reason deep models work at all. And layer normalization rescales the vector before each layer reads it, so its size stays in a sensible range no matter how many updates have piled up. Without it, numbers drift too large or too small and training falls apart.

At the very end, the final vector for the last token is compared against every token in the vocabulary (another big matrix, often the embedding table reused). That gives one raw score per token, called a logit. Which brings us to choosing one.

07PREDICT & SAMPLE[d] → [V] logits → probabilities → 1 token

From scores to a single word

Logits are raw scores, anything from very negative to very positive. Softmax turns them into probabilities. Then a sampler picks one. The model's job ends at the probabilities; the sampler's settings belong to whoever is running it, and they change the personality of the output a lot.

Temperature divides the logits before softmax. Below 1 it sharpens the distribution towards the favourite. Above 1 it flattens it, giving long shots a chance. At 0 you always take the top token (called greedy decoding). Top-k keeps only the k best candidates. Top-p (nucleus sampling) keeps the smallest set whose probabilities add up to p, so it adapts: tight when the model is sure, loose when it is not.

Sampler · "For breakfast I usually have ___"entropy
Practical defaults: low temperature (0 to 0.3) for code, extraction and factual answers. Around 0.7 to 1 for writing and brainstorming. Above about 1.3, coherence drops quickly.

Now run the loop

Below is a complete generator: it predicts, samples, appends and repeats. The model inside is deliberately primitive. It counts which word follows each pair of words on this page (a trigram model) and nothing more. There is no attention and no understanding, yet the loop around it is exactly the one an LLM runs.

Generator · trigram model, trained on this page
Watch how often a candidate has 100% probability. With so little data, the model has seen most word pairs only once, so it can only parrot the page back. That is memorisation. Scale, and an architecture that can generalise, are the difference between this and a real LLM.
08TRAININGloss = −log p(actual next token)

How the numbers get there: play the model

Every one of those billions of numbers starts random. Training shows the model real text, one position at a time, and asks: what comes next? The model outputs its probabilities, then the real next token is revealed. The loss is the negative log of the probability it gave to the right answer. Confident and right costs almost nothing. Unsure costs a bit. Confident and wrong costs a lot.

Then calculus (backpropagation) works out, for every single number in the model, which direction would have lowered that loss, and each number is nudged a tiny step that way. Do this for trillions of tokens and the numbers settle into something that predicts text well. Predicting text well, it turns out, requires picking up grammar, facts, styles and some reasoning along the way.

Your turn. Pick the word you think comes next and how sure you are. Your remaining confidence is spread over the other options. Watch what the loss does when you are confidently wrong.

You are the modelround 1 / 6
For scale: a model guessing uniformly over 4 options scores ln 4 ≈ 1.39 per round. Guessing uniformly over a 100,000-token vocabulary scores about 11.5. Good LLMs average around 2 on general web text.

The three stages of a chat model

StageWhat it learns fromWhat you get
PretrainingTrillions of tokens of web pages, books, code, papers. Next-token prediction only.A "base model". It continues documents. Ask it a question and it may reply with more questions, because that is what a list of questions looks like.
Supervised fine-tuningTens of thousands of example conversations written or curated by people.A model that has learned the shape of a helpful reply: answer the question, follow the instruction, stop.
Preference and reinforcement tuningComparisons of pairs of answers (RLHF), checkable tasks like maths and code with a right answer, and written principles.Better judgement, tone and refusals, and, in "reasoning" models, the habit of working through a problem step by step before answering.

Almost all of the knowledge comes from pretraining. The later stages mostly change behaviour: what the model does with what it already knows.

09SCALEparams ≈ 12 · L · d² + V · d

Where the billions of parameters live

"Parameters" are the learned numbers. For a standard transformer you can estimate the count from three settings: number of blocks L, width d, and vocabulary size V. Each block holds about 4d² numbers in attention (the query, key, value and output matrices) and 8d² in the feed-forward layer (two matrices of d × 4d). Add the embedding table and you are done. Pick a preset; the formula lands close to the published numbers.

Parameter calculator
124M
Rules of thumb used here: 2 bytes per parameter at 16-bit precision, and roughly 2 floating-point operations per parameter for every token generated.

Two consequences worth holding onto. First, width is squared, so doubling d roughly quadruples the size, which is why big models are mostly wide. Second, every generated token touches every parameter, so a 70-billion-parameter model does about 140 billion arithmetic operations per token. Speed and price follow directly.

One way around that is mixture of experts: replace each feed-forward layer with many smaller ones and route each token through only two or so. The model can hold far more knowledge while each token only pays for a fraction. Several leading models are now built this way.

10FIELD NOTESmechanism → practice

What changes once you know this

Each of these follows from something earlier on the page. Use them when you use the tools.

  1. Your prompt is the start of a document. The model continues the most plausible text. Write the opening of the document you want: give the context, the role, the format, and an example of a good answer. Vague input gets the average of the internet.
  2. Examples beat descriptions. Attention is very good at spotting a pattern earlier in the context and copying it forward (researchers found dedicated "induction heads" that do exactly this). Two worked examples often do more than a paragraph of instructions.
  3. Let it think on the page. Each token gets the same fixed amount of computation. Asking for reasoning before the answer gives the model more tokens, and so more computation, to spend on the problem. Reasoning models are trained to do this by default.
  4. Fluent is not the same as true. Training rewards plausible continuations. When the model has no strong signal, the most plausible text is still confident-sounding text. That is what a hallucination is. Ask for sources, then check them.
  5. It does not look things up. Unless it is given a search or tool, everything comes from patterns fixed at training time, with a cutoff date. Paste in the document you care about rather than hoping it remembers.
  6. The context window is working memory, and it is uneven. Everything the model knows about your conversation is in the tokens it can see. In very long inputs, material in the middle tends to get less use than the beginning and end. Put key instructions at the start, and restate critical constraints near the end.
  7. It has no memory between chats. "Memory" features work by quietly pasting saved notes back into the context. Same mechanism, different packaging.
  8. Character-level tasks need help. Counting letters, reversing strings, exact rhymes, precise arithmetic: anything below the token level is hard to see. Ask it to spell things out, or to write and run code.
  9. Different answers each time is normal. That is sampling. For repeatable output, lower the temperature. For variety, raise it, or ask for several options and choose.
  10. Tokens are the unit of cost. Long pasted documents, verbose answers and non-English text all cost more. "Answer in three bullets" is cheaper and usually better.
11THE CASTreference

The cast of characters

Keep this open while reading papers or posts. Every term in plain language.

12GO DEEPERsources & next steps

Where to go next

In rough order of difficulty. If you only do one thing, watch Karpathy build a small GPT from scratch; after this page, you will recognise every part.

About the demos

The tokenizer, the √d scaling, the sampler and the generator run real algorithms on real numbers in your browser. The embedding map, attention heads, neurons and logit lens are hand-built toys: their numbers were chosen to show a mechanism clearly, not learned from data. Real models do these things, but messier, and across thousands of dimensions.