From scratch12 minBuilding a GPT with only NumPy sounds like a flex. The educational punchline is meaner: without pretraining data, your beautiful architecture just emits haunted word salad.Read now →

He Built a GPT From Scratch With Six LEGO Pieces and a Pile of NumPy. The Last Step Was Pure Gibberish
Strip a GPT down and there are shockingly few parts: embeddings, attention, layer norm, a feed-forward net, and two layers that bookend it all. Here is the whole architecture explained as a LEGO castle, one block at a time, and why the finished model said nothing at all.
Contents
- The surprisingly short parts list
- The LEGO game plan
- Step zero: turning text into numbers
- Block 1: embeddings, or meaning as coordinates
- A 30-second history of how GPT was born
- Block 2: attention, the part that matters most
- Block 3: layer normalization
- Block 4: the feed-forward network
- The residual connection trick
- Blocks 5 and 6: back to words
- The gibberish finale
- Where the real difficulty hides
- Why build it the hard way
- Why this matters
- Try it yourself
- Risks and counterpoints
- FAQ
- The bottom line
The surprisingly short parts list
Here is a fact that deflates a lot of mystique: the architecture of a GPT, the kind of model behind modern chatbots, has shockingly few distinct parts.
A feed-forward layer. Two layer normalizations. One attention layer. A softmax. That is essentially the whole repeating block.
The letters stand for generative pre-trained transformer. And the core insight of one from-scratch build is that you can assemble the entire thing like a LEGO set, using nothing but NumPy, the basic numerical library in Python. No TensorFlow. No PyTorch. Just the math, coded by hand.
That constraint is brutal in practice, because it means writing not only the forward pass but the backward pass, the calculus that lets a model learn. But it is also the best possible way to see what a GPT really is.
The LEGO game plan
The plan is almost insultingly simple:
- Build each layer individually, like separate LEGO bricks.
- Stack the bricks on top of each other.
There is no step three.
You start at the bottom of the castle, with the input, and work up to the top, the output. And the single most important thing to understand before any of it: a GPT is just a next-token predictor. You feed it text, it does a great deal of math, and it outputs a prediction for the next token. Everything else is detail.
Step zero: turning text into numbers
You cannot do math on the word "cat." So the first move is converting text into numbers called tokens.
This process is tokenization. A sentence like "I love green code videos" gets chopped into pieces and each piece becomes a number. The exact scheme has many variations, but the idea is constant: text in, a sequence of integers out. Those integers are the raw material everything else operates on.
Block 1: embeddings, or meaning as coordinates
The first real LEGO brick is the embedding layer. It takes those token numbers and converts them into more numbers, specifically a vector, a list of numbers, for each token.
Why? Because an embedding encodes meaning as a position in a high-dimensional space. The classic demonstration: take the embeddings for "king," "queen," "man" and "woman." Subtract "man" from "king" and add "woman," and you land near "queen." The geometry of the space actually carries relationships.
A small model like GPT-2 uses 768 dimensions for each embedding. You cannot picture 768 dimensions, but the principle is the same as the two-dimensional cartoon: similar meanings sit near each other.
The code for this block is easy. You initialize one big matrix where each row is the embedding for one token. That is it. The first brick snaps into place.
A 30-second history of how GPT was born
Before stacking more bricks, a quick origin story, because it explains the shape of everything.
- Around 2014, researchers mostly used RNNs and LSTMs, older architectures that were awkward to train.
- In 2017, a team at Google essentially said: delete most of that, keep the useful part. They called it attention, wrapped in a design called the transformer. It worked better and reshaped the field.
- In 2018, OpenAI simplified further, keeping one slice of the transformer and calling it a GPT.
The lineage matters because it tells you where to focus. The thing everyone kept, through every round of deletion, was attention.
Block 2: attention, the part that matters most
If you understand this one block, you basically understand GPT.
Recap the pipeline so far: text becomes tokens, tokens become embeddings. Now you have a sequence of embeddings, one per word, but they do not yet know about each other. They need to communicate. Think of it as the embeddings wanting to catch up on gossip.
That is what scaled dot-product attention does. The equation looks scary; the idea is not.
- Each embedding is passed through three separate trainable matrices, producing a query, a key and a value.
- Think of queries as questions and keys as answers. For the word "videos," the query might ask "what kind of videos?" The keys for "green" and "code" hold the answer.
- To find matches, you take the dot product of one word's query with another's key. A strong match scores high. You do this for every pair, producing a grid of attention scores.
- You divide by the square root of the key dimension, which is the "scaled" part, and then run a softmax so each row sums to one.
- Finally you bring in the values. Queries and keys decide where a word should focus; values are the actual content passed along. Multiply the normalized scores by the values, add them up, and each embedding becomes a new, context-aware embedding.
In reality you do this several times in parallel with different matrices, asking different questions at once. That is multi-head attention.
The whole point: attention lets information from other tokens flow into each token, so every word becomes aware of its surroundings. Explaining it takes a minute. Coding it by hand in NumPy, backward pass included, took the builder two full days. The LEGO picture is clean; the real work is a castle where every brick is on fire and one mistake fills everything with NaNs, the "not a number" errors that signal the math has broken.
Block 3: layer normalization
The next brick, layer normalization, is a love-hate relationship.
It is wonderful because it normalizes the features, keeps training stable and makes the model easier to optimize. It is painful because, done from scratch, the backward pass is genuinely hard to derive.
The nice thing about layer norm is that you can sprinkle it in wherever you want. The standard practice, used here, is to place it before the attention and feed-forward layers, because that arrangement has been shown to make training more stable. Another brick down.
Block 4: the feed-forward network
If attention is the communication phase, the feed-forward network is the computation phase.
It is modest: sandwich a non-linear activation function between two linear layers. A linear layer is just a matrix multiplication. The convention is to expand to four times as many hidden neurons as inputs, then apply a ReLU activation, which crushes negative values to zero and leaves positive ones unchanged.
That completes one full transformer block: layer norm, attention, layer norm, feed-forward. And here is the key: that block is meant to be stacked, over and over. The little "Nx" in architecture diagrams means "repeat this N times." A real GPT is many of these blocks piled up.
The residual connection trick
One detail inside the block is quietly one of the most important ideas in modern AI: the residual connection.
Those little arrows in diagrams mean you add a layer's input directly to its output. You sum the attention inputs with the attention outputs, and the same for the feed-forward layer.
Why does this matter so much? Because it lets each layer start out as the identity function, essentially "do nothing," and then gently learn how to tweak its behavior during training. That is what makes it possible to stack dozens or hundreds of layers without the model collapsing. Explained in a sentence, it sounds trivial. It is arguably what made very deep networks trainable at all.
Blocks 5 and 6: back to words
After all that, you have beautiful, context-aware embeddings. But embeddings are not words. You need to turn them back into a prediction.
A final linear layer converts each embedding from its hidden dimension to the size of the vocabulary, which might be around 50,000. Notice it is almost the mirror image of the embedding layer: the first block turned tokens into embeddings, this one turns embeddings back into a raw score for every possible token. Their weight matrices are nearly inverses of each other.
Those raw scores are arbitrary, though. They do not sum to one, so they are not probabilities. The fix is a final softmax, which squashes them into a proper probability distribution over the next token.
And that is the castle. Input text, six conceptual building blocks, output a probability for every possible next word. Entirely in NumPy.
The gibberish finale
So you run it. You give it a phrase like "green colored videos are ___" and ask for the next word. Append the prediction, feed it back in, repeat, and in principle the model talks.
The first real generation came out as pure gibberish.
That is not a bug. It is the whole lesson. The architecture was complete and correct, but it had never been trained on anything. A GPT with a perfect structure and no learning is just a very elaborate random-number generator. All the magic people associate with these models, the apparent knowledge and fluency, lives in the training, not the architecture.
The build in question was framed as the start of a series, with training, a custom tokenizer, and progressively larger models to come, culminating in a real training run. But the single most clarifying moment is this one: the castle stands, and it says nothing, because nobody has taught it yet.
Where the real difficulty hides
The LEGO metaphor is honest about the concepts and quietly dishonest about the labor. It is worth separating the two, because the gap is where most people underestimate modern AI.
The forward pass is the easy half. Running data through the model, multiply, normalize, attend, multiply again, is the part that matches the clean diagrams. If that were all, a from-scratch GPT would be an afternoon.
The backward pass is the other half, and it is where the suffering lives. To learn, a model must compute how every weight contributed to its error and nudge each one in the right direction. In a framework, this "backpropagation" is automatic; you never see it. From scratch in NumPy, you derive and code the gradient for every single block by hand, including the genuinely fiddly calculus of layer normalization and attention. A single sign error anywhere and training silently diverges into those NaN values, with almost no clue about which brick broke.
This is why the builder described attention taking two days and layer norm eating an afternoon just to understand before implementation. It is also why frameworks exist. The reason PyTorch and TensorFlow dominate is not that the forward math is hard. It is that they handle the backward math, the memory, the hardware acceleration and the numerical stability so you can think about the model instead of the derivatives.
Understanding that split changes how you read AI news. When a lab talks about training runs costing millions, they are not paying for the six blocks. They are paying for the backward pass, run over trillions of tokens, millions of times, on enormous clusters. The architecture is nearly free. The learning is the expense.
Why build it the hard way
Why do this at all when you could import a library in three lines?
Three reasons the builder gave, and they are good ones.
- Because it is cool. Curiosity is a legitimate motive.
- To answer "how hard is this, really?" Building from scratch is the only way to find out.
- To raise money. This particular project was tied to a cancer-research fundraiser, with code releases and personal stakes attached to donation milestones. (That is the creator's stated motivation; we mention it as context, not endorsement.)
But the deeper reason is educational. Nothing kills magical thinking about AI faster than coding a matrix multiply that the whole thing depends on.
Why this matters
- Demystification is empowerment. Once you see that a GPT is six blocks and a predictor, it stops being sorcery and becomes engineering you can reason about.
- The architecture is not the magic; the data is. The gibberish ending is the clearest possible proof. Capability comes from training, which is why data and compute dominate the modern AI economy.
- Attention is the center of gravity. Through every simplification from 2014 to today, attention survived. Understanding it is understanding the field.
Try it yourself
If this sparked something, a sensible path:
- Learn tokenization first. It is the least glamorous and most skipped step.
- Implement attention by hand, once. Even a tiny version. It is the brick that teaches the most.
- Use NumPy before a framework. Writing the backward pass yourself builds intuition no library will.
- Start microscopic. A character-level model on a tiny dataset will actually train on a laptop.
- Expect gibberish, then train. Treat the first incoherent output as the milestone it is.
Risks and counterpoints
- From-scratch is for learning, not production. Real systems use optimized frameworks for good reasons: speed, stability and gradients that actually work.
- A toy model is not a small ChatGPT. Scale, data quality and training technique separate an educational build from a usable assistant by an enormous margin.
- Understanding architecture is not understanding behavior. Knowing the blocks does not tell you why a trained model says a given thing.
- Hand-derived math is error-prone. The NaN-filled castle is real. Expect painful debugging.
FAQ
What does GPT stand for?
Generative pre-trained transformer. It generates text, it is pre-trained on large amounts of data, and it uses the transformer architecture.
What are the main parts of a GPT?
An embedding layer, attention, layer normalization, a feed-forward network, and a final linear layer plus softmax. Attention and feed-forward blocks repeat many times.
Why did the from-scratch model output gibberish?
Because it had the correct architecture but no training. All the apparent knowledge in these models comes from training on data, not from the structure alone.
Do I need PyTorch or TensorFlow to understand GPTs?
No. Building a tiny version in plain NumPy is slower and harder, but it is the clearest way to learn what the frameworks are doing for you.
What is attention, in one sentence?
A mechanism that lets each word pull in relevant information from the other words, so every token becomes aware of its context.
The bottom line
A GPT is not an incomprehensible black box. It is six conceptual building blocks stacked into a next-token predictor, and you can build the whole thing by hand if you are willing to suffer through the math.
The most important moment in the entire build is the anticlimax: a perfectly assembled model that produces nothing but noise. That is the reminder worth keeping. The architecture is just the castle. The magic is in teaching it to speak.
Sources
- Google Research, Attention Is All You Need
- OpenAI, Improving language understanding with unsupervised learning (GPT)
- Jay Alammar, The Illustrated Transformer





