Demystifying BERT Under the Hood
How a language model works out what a word means from the words around it.
The context decides the meaning.
BERT reads a sentence in both directions at once, so every word is understood through its full context.
Read as US citizens travelling to Brazil, the opposite of what was asked.
Understood as visa rules for Brazilians visiting the United States.
The Revolution Behind BERT & Modern LLMs
Self-attention lets a model read every word at once. BERT and GPT are both built on it.
“Attention Is All You Need” (2017)
Replaced older recurrent networks with self-attention mechanisms to process input sequences in parallel, laying the foundation for modern AI.
Encoder Transformer
Processes the entire input sequence simultaneously to deeply understand and contextualize text.
Decoder Transformer
Optimized for autoregressive text generation, predicting subsequent tokens sequentially.
Where the [MASK] Came From
BERT’s training task is a reading test from 1953.
- 1953The cloze test
Wilson Taylor removes words from a passage and asks readers to fill in the gaps.
- 2018BERT’s training
Google hides words behind
[MASK]and trains the model to recover them.
The cat sat on the ? and fell asleep.
You used the words on both sides of the gap. BERT does the same.
How BERT is trained
BERT learns language by filling in hidden words, using clues from both sides.
- Hide 15% of words behind a
[MASK] - BERT guesses each from words on both sides
- Learns grammar, facts & context on its own
Transformer Architecture is used for deep bidirectional understanding
From Sentence to Prediction
Follow one sentence through each layer until BERT picks the missing word.
Words snap into reusable pieces
to build every word
A few hundred Lego shapes build anything — you never need a custom brick. (Here hug·##ging splits into two pieces, while his and favorite are common enough to be single pieces — the ## means “glue me to the piece before.”)
Each piece becomes 768 numbers
Two very different movies land almost on top of each other — the computer can’t tell them apart.
One extra dimension sends The Dark Knight into the dark, gritty corner while The Avengers stays bright and comic-book.
Humans can’t picture a 768-D graph, but the computer reads it like a colour code — “#412” tells you nothing, while R240·G90·B20 says orange. BERT builds one of these 768-number fingerprints for every single token, so similar words land near each other.
Every word asks “who matters to me?”
Like twelve people proofreading one page — the editor watches grammar, the lawyer watches risk, the designer watches rhythm. Each head notices a different link, then they pool their views.
How a word decides who matters
Every word searches every other word, then takes in information from the best matches.
Every word gets its own Q, K and V by multiplying its 768-number embedding by three learned matrices. BERT runs this for all words at once, in 12 heads per layer. Numbers are illustrative.
Keep the signal strong & balanced
First add the original back so nothing is lost — then normalise, like a sound engineer riding the faders: nothing too loud, nothing gone. Every layer stays stable.
Each word thinks it over
contextual
vector
room to
think
weak signals
dropped
sharper
word
Having heard the room, each word steps aside on its own — expanding to think it over, then compressing back down, refining what it just learned.
From vector to word-scores
BERT hands over the blank’s 768-number vector. The LM (Language Model) Head compares it against every word BERT knows.
One dot product per word → a raw logit (score) for all 30,522 words.
The best-fit word wins
Softmax turns the logits into probabilities that add to 100%. Winner bank, 71%.