The model that taught computers to read both directions at once

orig. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” · Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Large Language Models Intermediate 4 min read Written, reviewed by Marginalia Editorial

In the margin

Heads up

For a few years, this one model quietly powered a huge chunk of Google Search and a wave of language tools.

What's going on

Earlier models read text left to right. BERT reads the whole sentence at once, looking both ways, which helps it understand how words depend on each other. It learns by playing fill-in-the-blank on huge amounts of text: hide a word, guess it from the context. After that general training, it can be quickly adapted to specific jobs like answering questions or judging sentiment.

Why it matters

BERT made the now-standard recipe popular: train one big model on lots of text, then fine-tune it for each task. That recipe is behind most modern language tools. It also went straight into real products, including search, soon after release.

Source

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Google AI Language

View on arXiv ↗ Read the PDF ↗

Still fuzzy on a sentence?

Paste it and we'll explain it even more simply.

Test your understanding

Member notes Sign in ↗

ME Marginalia Editorial TEAM

The editorial team picked this paper because it is worth your time, and an AI drafted this note. We check each one and rewrite it when the draft misses the mark.