Text & Language — Interactive Lab

A hands-on tour of the machinery that turns raw text into structure and meaning — tokenizers, inverted indexes, edit-distance grids, language models, and similarity metrics, built from scratch and run in front of you. Three layers: The machine dives that step you through a full text pipeline one operation at a time, Math focus dives that derive the scoring formulas, and Concept cards for the vocabulary. Click, drag, and run each idea to see it work. A sibling to the ML, Deep-Learning, Time-Series, Data-Engineering, and Databricks labs. Press Esc anytime for this menu.

The machine — text pipelines, one operation at a time

Math focus — the formulas behind the methods, derived step by step

Concept cards — focused & interactive

No concepts match your search.

Interactive · byte-pair encoding from scratch

The tokenizer, one merge at a time

How raw text becomes the integer tokens a model sees. Press Next to normalize the text, split it into words and then symbols, count adjacent pairs, learn a sequence of merges (byte-pair encoding) that grow the vocabulary, then encode an unseen word into subword tokens and read off its ids.

stage: raw

① text → words

② symbols → merges

③ encode → ids

corpus + pair frequencies

encode an unseen word

Interactive · inverted index + BM25 from scratch

The search engine, one posting at a time

How a search engine finds and ranks documents. Press Next to analyze a corpus into tokens, build an inverted index, analyze a query, fetch the matching postings, compute idf from document frequencies, score each candidate with BM25 (term-frequency saturation + length normalization), and rank the results.

stage: corpus

① corpus → analyze

② index → query

③ score → rank

index / postings / scores

BM25 scoring

Interactive · noisy-channel spelling correction

The spell checker, one candidate at a time

How autocorrect picks the right word. Press Next to take a misspelling, generate every word one edit away, filter to real dictionary words, identify each edit, weigh a language-model prior against an error-model channel, combine them into a posterior, and choose the correction — the noisy-channel model behind spell checkers.

stage: input

① the misspelling

② candidates

③ rank → correct

candidate generation & scoring

noisy-channel model

Interactive · Levenshtein dynamic programming

Edit distance, one cell at a time

How far apart are two strings? Press Next to set up the two words, initialize the base case, learn the recurrence, fill the dynamic-programming grid cell by cell, read the distance in the corner, backtrace the cheapest path, and recover the alignment of edits.

stage: strings

① strings + base

② recurrence + fill

③ distance + alignment

the DP grid

recurrence & alignment

Interactive · the angle between two document vectors

Cosine similarity, one term at a time

How similar are two documents, regardless of length? Press Next to turn each document into a vector over a shared vocabulary, take their dot product, measure each magnitude, divide to get the cosine, and read off the angle between them — the similarity metric behind search and clustering.

stage: documents

① documents + vocab

② vectors + dot

③ magnitudes + cosine

the computation

cos = a·b / (|a|·|b|)

Interactive · tf-idf weighting from counts to vectors

The TF-IDF vectorizer, one weight at a time

How raw word counts become weighted vectors that know which words matter. Press Next to count term frequencies, measure each term’s document frequency, turn that into an idf weight (rare = high), multiply to get tf-idf, and normalize each document to a unit vector — ready for cosine similarity and search.

stage: corpus

① corpus + tf

② df + idf

③ tf-idf + vectors

the term–document matrix

idf weights & formula

Interactive · regex → NFA → simulate

The regex engine, one state-set at a time

How a regular expression actually matches text. Press Next to compile the pattern ab*c into an NFA, take the ε-closure of the start state, then feed the input one character at a time — watching the set of active states advance through the b* loop until an accepting state is reached. Tracking a set (Thompson’s method) means matching in one linear pass, with no backtracking.

stage: pattern

① pattern + NFA

② simulate

③ result

the NFA

active states

Interactive · prefix tree + ranked completions

The autocomplete, one node at a time

How a search box finishes your word. Press Next to load a small dictionary, build a trie where words sharing a prefix share a path, type a prefix, walk down to its node, collect every word in that subtree, rank them by frequency, and show the top suggestions.

stage: dictionary

① dictionary

② walk the prefix

③ suggestions

the trie

completions

Interactive · longest common subsequence

The diff, one cell at a time

How a diff finds what changed between two versions. Press Next to line up the two sequences, fill the LCS table, read the length of the longest shared subsequence, backtrace it, and turn the path into a diff of keeps, insertions and deletions.

stage: sequences

① sequences

② LCS table

③ diff

the LCS grid

the diff

Interactive · bigram model + sampling

The n-gram generator, one token at a time

How a Markov model generates text. Press Next to read a corpus, count bigrams, normalize them into next-word probabilities, view the transition table, sample the next word from a distribution, walk the chain, and read the generated text.

stage: corpus

① corpus + counts

② probabilities

③ sample + generate

transition table

sampling & output

Interactive · Knuth-Morris-Pratt

The string matcher, one comparison at a time

How KMP finds a pattern in linear time. Press Next to set up the pattern and text, build the LPS table (the prefix function), scan the text comparing characters, use the table to shift on a mismatch without moving the text pointer back, and find the match.

stage: pattern + text

① pattern + text

② LPS table + scan

③ match

the scan

LPS table

Interactive · Naive Bayes with Laplace smoothing

Naive Bayes, one log-probability at a time

How Naive Bayes classifies text. Press Next to read training documents, compute class priors, count words per class, turn them into smoothed likelihoods, take a test document, sum the log scores per class, and predict the most probable label.

stage: training

① training + priors

② likelihoods

③ score + predict

priors & likelihoods

log scores

Interactive · BIO sequence labeling

The NER tagger, one token at a time

How named entities are extracted. Press Next to take a sentence, compute per-token features, assign BIO tags (B-/I-/O), group consecutive tags into spans, and read off the typed entities.

stage: sentence

① sentence

② features + BIO

③ spans + entities

tokens → BIO tags

entities

Interactive · logits to a distribution

Softmax, one temperature at a time

How raw scores become a probability distribution. Press Next to start from logits, divide by a temperature, exponentiate, and normalize into probabilities — then use the temperature button to watch the distribution sharpen or flatten.

stage: logits

① logits

② exponentiate

③ normalize + temperature

logits → probabilities

softmax & entropy

Interactive · LDA via collapsed Gibbs sampling

The topic model, one resample at a time

How latent topics emerge from raw co-occurrence. Press Next to load documents, posit K topics, give every word a random topic, build the counts, resample one word’s topic from the LDA probability, run many sweeps, and read the discovered topics.

stage: documents

① documents + K

② assign + resample

③ topics

topic assignments

resample & topics

Interactive · skip-gram embeddings in 2-D

Word2Vec, one update at a time

How words become vectors. Press Next to take co-occurrence pairs, scatter random vectors, run the skip-gram score, take a softmax + loss, update the vectors so co-occurring words move together, train to convergence, and find a word’s nearest neighbor.

stage: corpus

① corpus + vectors

② skip-gram + update

③ train + nearest

2-D embedding space

objective & neighbors

Interactive · MinHash + LSH near-duplicate detection

The dedup, one signature at a time

How near-duplicates are found at scale. Press Next to take documents, build shingle sets, compute the true Jaccard, hash them into MinHash signatures, estimate the similarity from matching positions, band them with LSH, and surface the near-duplicate.

stage: documents

① documents + shingles

② Jaccard + MinHash

③ estimate + LSH

signatures

similarity & LSH

Interactive · PMI for collocations

PMI, one pair at a time

How to detect phrases statistically. Press Next to read a corpus, count words and bigrams, form the marginal and joint probabilities, compute PMI for each pair, and rank them to surface the collocations.

stage: corpus

① corpus + counts

② probabilities

③ PMI + rank

association table

PMI formula

Interactive · bits, surprisal, Shannon entropy

Entropy, one symbol at a time

How information content is measured. Press Next to take a distribution, compute each symbol’s surprisal (−log2 p), weight it by its probability, sum to the entropy, and compare against the uniform maximum.

stage: distribution

① distribution

② surprisal

③ entropy

probability · surprisal · contribution

entropy in bits

Interactive · scoring a language model

Perplexity, one token at a time

How well does a model predict text? Press Next to take a test sequence, read the model’s probabilities, turn each into surprisal, average to cross-entropy, exponentiate to perplexity, and compare a good model against a weak one.

stage: test sequence

① test + probs

② surprisal + CE

③ perplexity

per-token probability & surprisal

cross-entropy & perplexity

Interactive · Laplace / add-k smoothing

Smoothing, one estimate at a time

How models avoid assigning zero probability. Press Next to see raw counts, the MLE estimate, the zero problem for an unseen word, the Laplace add-one fix, the general add-k, and how mass is redistributed from seen to unseen events.

stage: counts

① counts + MLE

② the zero problem

③ Laplace / add-k

probability estimates

formulas & the unseen word

Interactive · rank-frequency & vocabulary growth

Zipf & Heaps, one curve at a time

Two empirical regularities of text. Press Next to read a corpus, rank words by frequency, see Zipf’s law on a log-log plot, watch new words appear as you read more (Heaps), plot the vocabulary-growth curve, and interpret both.

stage: corpus

① corpus + ranks

② Zipf (log-log)

③ Heaps (growth)

rank-frequency / vocab growth

the two laws

Interactive · latent semantic analysis

LSA, one singular value at a time

How latent topics fall out of linear algebra. Press Next to build a term–document matrix, factor it with the SVD, inspect the singular values, truncate to the top few, and view documents in the resulting latent space where topics separate.

stage: matrix

① term–document matrix

② SVD + singular values

③ latent space

matrix / singular values / latent space

A = UΣVᵀ

Approximate matching

Hamming distance

When two strings are the same length, the simplest way to measure their difference is to count the positions where they disagree. That count is the Hamming distance. Flip the bits below and watch it move.

Try it: click any bit in your row to flip it. Differing columns light up; the distance is how many.

Only defined for equal-length strings — it has no notion of insertions or deletions.

Used for error-detecting codes and binary fingerprints like SimHash, where near-duplicates have small Hamming distance.

Approximate matching

Edit distance, live

Edit distance is the fewest single-character insertions, deletions, and substitutions that turn one string into another. Type a word and watch the distance to kitten — and the exact operations — update on every keystroke.

Try it: edit the word — the target is kitten.

= keep · ~ substitute · + insert · − delete. Each non-keep costs 1.

The Edit Distance machine shows the dynamic-programming grid that computes this in one pass.

Approximate matching

Jaccard similarity

To compare two sets, divide the size of their overlap by the size of their union. That ratio — the Jaccard index — runs from 0 (nothing shared) to 1 (identical). Toggle which items belong to A and B and watch it respond.

Try it: click the A and B boxes next to each item to add or remove it from that set.

J(A,B) = |A ∩ B| / |A ∪ B|. All-shared → 1, disjoint → 0.

Treat a document as the set of its word-shingles and Jaccard becomes a similarity score — the quantity MinHash estimates cheaply.

Representation

Bag of words

The simplest way to turn text into numbers: forget the order and just count how often each word appears. The result is a bag of words — a vector of counts. Shuffle the sentence and the bag is unchanged; that is exactly the information it throws away.

Try it:
and click any word above to drop it from the sentence.

Order is discarded; only multiplicity remains. “dog bites man” and “man bites dog” share a bag.

Bag of words is the substrate for TF-IDF and many classifiers — simple, strong, and order-blind.

Representation

One-hot vs dense vectors

How do you hand a word to a model? The naive way is one-hot: a vector as long as the vocabulary, all zeros except a single 1. It is huge and says no two words are alike. A learned dense vector is short, and its numbers place similar words near each other.

Try it: click a word to select it. Drag the slider to grow the vocabulary and watch the one-hot vector stretch while the dense vector stays small.

vocabulary size 16

One-hot length = vocabulary size; every pair is equally distant (no similarity).

Dense embeddings are learned: here cat and dog sit close, as do run and sit — the geometry carries meaning.

Similarity & distance

Cosine vs Euclidean

Two ways to compare vectors. Euclidean distance is the straight line between their tips — it cares about length. Cosine similarity is the angle between them — it cares only about direction. Drag the purple vector and watch both react; text similarity almost always uses cosine.

Try it: drag anywhere in the plane to move the purple vector. The mint one is fixed.

cosθ = (a·b)/(‖a‖‖b‖): 1 = same direction, 0 = perpendicular, −1 = opposite.

Double a vector’s length and cosine is unchanged but Euclidean distance grows — why direction-based cosine suits documents of different sizes.

String & character fundamentals

Characters & code points

A string looks like letters, but underneath it is numbers. Each character is a Unicode code point; UTF-8 then encodes that number as one to four bytes — ASCII in one, accented letters in two, many symbols in three.

Try it: type any text — letters, accents, symbols, emoji — and watch each code point and its UTF-8 bytes.

Code point: the number Unicode assigns (U+0041 = “A”). UTF-8 is variable-length.

Notice the count: characters ≠ bytes. Slicing on bytes can cut a character in half.

String & character fundamentals

Strings as sequences

A string is an ordered sequence of characters with positions. You address a character by its index from zero, take a slice with a half-open range, and read its length. Drag across the characters to select a slice.

Try it: drag across the boxes to choose a slice. Half-open s[a:b] includes a, excludes b.

Zero-based indexing; slice length is exactly b−a.

Half-open ranges tile cleanly: s[0:k] and s[k:n] cover the whole string with no overlap.

String & character fundamentals

Hashing a string

A hash folds a whole string into one fixed-size number. A polynomial rolling hash takes each character in turn: multiply the running value by a base and add the next code, modulo a large number. Same string, same hash; order matters.

Try it: type a string and drag the base. “cat” and “act” land on different hashes.
base 31

h = (h·base + code) mod M, left to right. The multiply makes position matter.

Powers dictionaries, sets, and substring search (Rabin–Karp). Collisions are possible but rare with a good hash.

String & character fundamentals

The three edit operations

To measure how different two words are, count the single-character edits to turn one into the other. There are exactly three: insert, delete, substitute. Edit them yourself and watch the running count — these are the atoms of edit distance.

Try it: click a letter to cycle it (substitute), a × above to delete, or a + below to insert.

~ substitute · − delete · + insert. Each costs 1.

The minimum number of such edits between two strings is the Levenshtein distance.

Tokenization & segmentation

Whitespace, subword, character

Before a model sees text it is cut into tokens. Split on spaces and rare words break the vocabulary; split into characters and sequences grow long; subword tokenization keeps common words whole and breaks rare ones into reusable pieces.

Try it: type a sentence and compare the three tokenizations and their counts.

Whitespace chokes on unseen words · character never does but is long · subword balances both.

The trade-off is vocabulary size against sequence length.

Tokenization & segmentation

Sentence segmentation

Splitting text into sentences sounds like “cut at every period” — until abbreviations and initials get in the way. A period both ends sentences and marks abbreviations. Click each period to decide, and watch the sentences regroup.

Try it: click a period to toggle it between a true sentence end (green) and an abbreviation dot (pink).

Clues a real segmenter uses: next-word capitalization, known abbreviations, numbers.

The sentence count updates as you reclassify each dot.

Tokenization & segmentation

N-grams

An n-gram is a contiguous run of n tokens. Slide a window of width n across the text and each position emits one gram. Set n and move the window yourself to see unigrams, bigrams, and trigrams appear.

Try it: set n and slide the window.
n = 2

A length-L sentence yields L − n + 1 n-grams.

Bigger n captures more context but explodes in count and starves on data — why smoothing exists.

Tokenization & segmentation

Why subwords win

A whitespace vocabulary can never list every word — names, typos, compounds appear forever. Subword tokenization fixes a vocabulary of frequent pieces and spells any word from them, so nothing is ever truly unknown. Type a word and watch it decompose.

Try it: type any word — even invented ones — and see it broken into known subword pieces (no <UNK>).

Shared pieces tie words together: happiness, happily.

BPE / WordPiece learn the pieces by merging frequent adjacent pairs — what the Tokenizer machine builds.

Normalization & linguistics

Case folding & normalization

Two strings can look identical yet be unequal to a computer — different case, or “é” written as one code point versus an “e” plus a combining accent. Edit both fields and watch when they match raw, after case folding, and after Unicode normalization.

Try it: edit either string.

Case folding unifies case; NFC composes characters so the two spellings of é become identical.

Without it, an index can store “café” twice and neither copy finds the other.

Normalization & linguistics

Stemming vs lemmatization

To treat “runs,” “running,” and “ran” alike, reduce each to a base form. Stemming chops suffixes by rule — fast, sometimes wrong. Lemmatization returns the true dictionary form, even for irregulars. Type a word and compare.

Try it: type a word and see the crude stem next to the dictionary lemma.
try: running, ran, better, mice, happily

Stemmer strips endings; lemmatizer looks up the lemma.

When the two disagree, it is usually an irregular form the rule-based stemmer cannot know.

Normalization & linguistics

Stopwords

A few words — “the,” “of,” “is” — dominate any text yet carry little topical meaning. Classic pipelines drop these stopwords. Type your own sentence, and click any word to add or remove it from the stoplist.

Try it: type a sentence, then click words to toggle them as stopwords (struck = removed).

Removing stopwords shrinks the index and keeps common words from dominating similarity.

Not always wise — “to be or not to be” is nearly all stopwords. Transformers keep them.

Normalization & linguistics

Morphology

Words are built from morphemes — the smallest meaning-bearing units. A root carries the core; prefixes and suffixes bend it. Assemble a word from pieces and watch its structure — the same idea that makes subword tokenization work.

Try it: click a prefix, a root, and one or more suffixes to build a word. Click again to remove.

Root holds meaning; prefixes/suffixes modify it. Inflection marks grammar; derivation makes new words.

Subword tokenizers rediscover these pieces statistically.

Pattern matching & algorithms

Exact match: the naive scan

The simplest substring search: line the pattern up at a position, compare left to right, and on a mismatch shift one step and start over. Drive the alignment yourself and watch it re-compare characters — the waste that smarter algorithms remove.

Try it: step the alignment across the text.

On a mismatch the naive scan shifts by one and forgets what it learned.

Worst case O(n·m); KMP reuses partial matches to never re-read text.

Pattern matching & algorithms

Regular expressions

A regular expression describes a set of strings: character classes say what kind, quantifiers how many, anchors where. Type a pattern and a test string, and the engine highlights every match live.

Try it: edit the pattern and the text. Matches highlight as you type.
patterntext

\d digit · [A-Z] capital · . any · + one-or-more · * zero-or-more · ? optional.

Highlighted spans are real matches from the live regex engine. Try [A-Z][a-z]+ or \w+.

Pattern matching & algorithms

Finite automata

A finite automaton has a few states and labeled transitions. Feed it characters and each moves it between states; if it ends in an accepting state, the string is in the language. This DFA accepts strings ending in “ab.” Type a string and step it through.

Try it: type a string of a’s and b’s, then step it through the machine.

Start at q0; each character follows its arrow. q2 (double ring) is accepting.

One left-to-right pass, O(n). Regex engines compile patterns into automata like this.

Pattern matching & algorithms

Wildcards & globbing

Globbing is the wildcard matching shells use for filenames: * is any run of characters, ? is exactly one. It is simpler than full regex and compiles straight into one. Type a glob and watch which files match.

Try it: edit the glob pattern and see matches update.
try: data?.csv, *.*, report.*

* any number of characters · ? exactly one.

A glob becomes a regex: *.txt → ^.*\.txt$. Matches below are computed that way.

Representation

Embedding geometry

Learned word vectors put meaning into directions. Differences between related words line up, so you can do arithmetic on them: take king, subtract man, add woman, and you land next to queen. Click three words and watch the analogy resolve.

Try it: click word A, then B, then C — the lab finds the nearest word to A − B + C.

The gender direction (man→woman) and the royalty direction are consistent across pairs.

Real embeddings have hundreds of dimensions; the same vector arithmetic powers analogy and semantic search.

Similarity & distance

Dot product

The dot product multiplies two vectors into a single number: a·b = aₓbₓ + aᵧbᵧ. Geometrically it is the length of b’s shadow on a, times the length of a. Positive means they point the same way, zero means perpendicular, negative means opposed. Drag b.

Try it: drag the purple vector b. The amber bar is its projection onto the fixed mint vector a.

a·b = ‖a‖‖b‖cosθ. Same direction → large positive; perpendicular → 0; opposite → negative.

The dot product is the workhorse behind cosine similarity, projections, and every neuron’s weighted sum.

Similarity & distance

Vector norms

A norm measures a vector’s size, and there is more than one. L2 is the straight-line length; L1 sums the absolute components (city-block distance); L∞ takes the largest component. Drag the vector and compare all three.

Try it: drag the vector tip and watch the three norms.

L1 = |x|+|y| · L2 = √(x²+y²) · L∞ = max(|x|,|y|). Always L1 ≥ L2 ≥ L∞.

L2 underlies cosine and Euclidean distance; L1 drives sparse (lasso) models; L∞ bounds the worst component.

Information retrieval

The inverted index

Search engines do not scan every document at query time. They build an inverted index once — a map from each term to the list of documents that contain it (its postings). A query just looks up those lists and combines them. Type a query and watch.

Try it: type one or more query terms; toggle how they combine.

AND intersects postings (all terms) · OR unions them (any term).

Postings are kept sorted so intersection is a fast merge — the core trick of web-scale search.

Information retrieval

Precision & recall

When a system returns results, two questions matter: of what it returned, how much was relevant (precision)? And of all that was relevant, how much did it find (recall)? They trade off. Click documents to mark them retrieved and watch both move.

Try it: click documents to toggle whether the system retrieved them. Relevant ones are outlined.

Precision = TP / retrieved · Recall = TP / relevant · F1 = their harmonic mean.

green true positive · pink false positive · amber missed (false negative).

Probabilistic & statistical

N-gram language model

A bigram model learns, from a corpus, how likely each word is to follow the last one. To generate text it just keeps sampling the next word from that distribution. Build a sentence yourself — click the next word from the live probabilities, or let it sample.

Try it: click a bar to append that word, or sample one at random.

P(next | last) comes straight from corpus bigram counts.

Real language models extend this to long contexts and learned probabilities — but the loop is the same: predict, pick, repeat.

Probabilistic & statistical

Bayes’ rule for text

A naive-Bayes spam filter starts from a prior belief and updates it with evidence. Each word in a message nudges the odds toward spam or ham by how much more likely it is under one than the other. Toggle which words appear and watch the posterior move.

Try it: click words to include them in the message. Each one shifts P(spam) by its likelihood ratio.

posterior odds = prior odds × ∏ P(word|spam)/P(word|ham). Independence is the “naive” assumption.

Words leaning spam push right; ham-leaning words push left. The prior is 50/50 here.

Classification & sentiment

Linear decision boundary

A linear classifier splits the plane with a straight line: everything on one side is class A, the other side class B. Drag the line’s endpoints to separate the two clouds of points and watch the accuracy climb — the same thing training does automatically.

Try it: drag either endpoint of the line to separate the blue and pink points.

Points are classified by which side of the line they fall on; misclassified points get a ring.

Training finds the weights automatically; perceptron, logistic regression, and linear SVM all draw a line like this.

Classification & sentiment

Sentiment lexicon

The simplest sentiment analyzer keeps a dictionary of word polarities and adds them up: “great” is +3, “terrible” is −3. A negator like “not” flips the next word. Type a sentence and watch each word’s contribution and the running total.

Try it: type a sentence using words like great, love, bad, terrible, not.

Each word’s score is summed; not flips the next scored word.

Fast and transparent, but blind to context and sarcasm — why learned classifiers eventually win.

Sequence labeling & parsing

BIO tagging for NER

Named-entity recognition is framed as tagging every token. The BIO scheme marks each one B (begins an entity), I (inside the same entity), or O (outside). Contiguous B…I runs become spans. Click tokens to tag them and watch the entities get extracted.

Try it: click a token to cycle its tag: O → B-PER → I-PER → B-ORG → I-ORG → B-LOC → I-LOC.

B begins an entity, I continues it, O is outside. A span is one B plus any following I of the same type.

Casting NER as per-token tagging lets sequence models (CRFs, BiLSTMs, transformers) learn it.

Sequence labeling & parsing

Part-of-speech tagging

Every word plays a grammatical role — noun, verb, adjective. POS tagging labels each one, and context decides: the same word can be a noun in one sentence and a verb in another. Tag the words yourself, then reveal the gold answer.

Try it: click a word to cycle its part of speech, then reveal the correct tags.

DET determiner · NOUN · VERB · ADJ adjective · ADV adverb · ADP preposition.

Ambiguity is the hard part: “book a flight” (verb) vs “read a book” (noun). Context disambiguates.

Topic & unsupervised

Document clustering (k-means)

Unsupervised clustering groups documents with no labels. k-means repeats two steps: assign each point to its nearest centroid, then move each centroid to the mean of its points. Drag the centroids or step the algorithm and watch the clusters settle.

Try it: drag the diamond centroids, or step the algorithm. Points recolor to their nearest centroid.

Assign → nearest centroid; update → centroid becomes the mean of its assigned points; repeat to convergence.

Documents become points via their vectors (TF-IDF or embeddings); k-means then finds topics as clusters.

Topic & unsupervised

Topic mixtures

A topic model says each document is a mixture of topics, and each topic is a distribution over words. Build a document by clicking words, and watch its topic proportions emerge from which topics those words belong to.

Try it: click words to add them to your document. The mixture bars show its topic proportions.

Each word leans toward a topic; the document’s mixture is the blend of its words’ topics.

LDA learns both the topic–word distributions and each document’s mixture at once, unsupervised.

Embedding era

Keyword vs semantic search

Keyword search matches the exact words you type — so “puppy” misses a document about a “canine.” Semantic search compares meaning vectors, so paraphrases match even with no shared words. Type a query and compare the two rankings.

Try it: type a query like puppy, kitten, motor, or dinner.

Keyword scores exact word overlap; semantic scores cosine of meaning vectors.

Green marks documents semantic search finds that keyword search misses — the synonyms and paraphrases.

Embedding era

Attention weights

Attention lets each token look at every other and decide how much to borrow from each. For a chosen query token, it scores every token by similarity and softmaxes those scores into weights that sum to one. Pick a token and adjust the temperature to sharpen or flatten its focus.

Try it: click a token to make it the query; drag temperature to sharpen (low) or flatten (high) the weights.
T = 1.0

weight(q,k) = softmax( q·k / T ). Low T → peaky (attend to one); high T → uniform.

Stacking many such attention heads is the core of the transformer.