“You shall know a word by the company it keeps!”
e.g. “We went to the bank” vs. “the river bank”
Same word, different meaning, decided entirely by context.
Bidirectional Encoder Representations from Transformers reads text in both directions simultaneously to capture deep contextual meaning.
US citizens travelling to Brazil. The exact opposite direction.
Visa requirements for Brazilian citizens visiting the United States.
One word, two opposite meanings
An intuitive, visual walkthrough of BERT — from raw text all the way down to self-attention layers.
Input any sentence of your own and watch it move through every architectural layer, live.
Reads text bidirectionally in one pass, building contextual meaning.
BERT is Encoder-only.Generates text sequentially one token at a time using masked attention.
ChatGPT is Decoder-only.Combines both: encodes input context and decodes it to output text.
T5 is Encoder-Decoder.[MASK]Transformer Architecture is used for deep bidirectional understanding
A few hundred Lego shapes build anything — you never need a custom brick. (Here hug·##ging splits into two pieces, while his and favorite are common enough to be single pieces — the ## means “glue me to the piece before.”)
Two very different movies land almost on top of each other — the computer can’t tell them apart.
One extra dimension sends The Dark Knight into the dark, gritty corner while The Avengers stays bright and comic-book.
Humans can’t picture a 768-D graph, but the computer reads it like a colour code — “#412” tells you nothing, while R240·G90·B20 says orange. BERT builds one of these 768-number fingerprints for every single token, so similar words land near each other.
Like twelve people proofreading one page — the editor watches grammar, the lawyer watches risk, the designer watches rhythm. Each head notices a different link, then they pool their views.
First add the original back so nothing is lost — then normalise, like a sound engineer riding the faders: nothing too loud, nothing gone. Every layer stays stable.
contextual
vector
room to
think
weak signals
dropped
sharper
word
Having heard the room, each word steps aside on its own — expanding to think it over, then compressing back down, refining what it just learned.
BERT hands over the blank’s 768-number vector. The LM (Language Model) Head compares it against every word BERT knows.
One dot product per word → a raw logit (score) for all 30,522 words.
Softmax turns the logits into probabilities that add to 100%. Winner bank, 71%.