Search by meaning

Word embeddings give each word one vector. Search needs more: one vector for a whole sentence, a question, or a page, so that “how can I save cash” lands near “putting a little money aside every month” even though they share no words. That is what a sentence embeddingSentence embeddingA single vector representing a whole sentence or passage, produced by a model trained so that texts with similar meaning get similar vectors.Open in glossary model does.

One vector per sentence

A sentence embedding model is a transformer that reads the whole text, produces a vector for every token, and then pools them into one vector, often by averaging (some models use the first or last token’s vector instead). The model is trained so that texts with similar meaning get similar vectors, typically by showing it pairs that belong together (a question and its answer, a sentence and its paraphrase) and pulling those vectors together while pushing unrelated pairs apart. This kind of training is called contrastive learningContrastive learningTraining that pulls the embeddings of matching pairs together and pushes non-matching pairs apart, so the model learns a useful space without category labels.Open in glossary.

The demo below runs all-MiniLM-L6-v2, a widely used open model from the sentence-transformers project. It is small, six transformer layers, and was fine-tuned with a contrastive objective on more than a billion sentence pairs. It turns any text into 384 numbers, scaled to length 1, so cosine similarity is simply the dot product. It was trained on texts up to 256 tokens long, and longer texts are cut off (this browser version cuts at 512), which is one reason long documents are split into chunks before embedding.

Two ways to search

Semantic searchSemantic searchSearch that ranks documents by how similar their meaning is to the query, usually by comparing sentence embeddings with cosine similarity.Open in glossary embeds every document once, embeds each query when it arrives, and ranks documents by cosine similarity. Keyword search, the approach behind classic search engines, ranks documents by the words they share with the query. The standard keyword method is BM25BM25A standard keyword ranking formula. It scores documents by the query words they contain, weighting rare words more and limiting the reward for repeated words.Open in glossary, which rewards documents containing the query’s words, gives more weight to rare words than to common ones, and gives less and less extra credit each time a word repeats.

Neither method is simply better. Embedding search handles paraphrases and synonyms, but it can miss exact strings such as product codes, names, and rare technical terms, and it will always return something, even for a query nothing in the collection answers. Keyword search is precise and easy to explain but blind to meaning. Production search systems often run both and combine the scores, an approach called hybrid search.

Searching millions of vectors

The demo compares the query with every sentence, which takes no time for twelve sentences. For millions of documents, systems store embeddings in a vector databaseVector databaseA database built to store embeddings and quickly find the ones most similar to a query vector, usually with an approximate nearest neighbor index.Open in glossary with an approximate nearest neighbor index. A popular one, HNSW (hierarchical navigable small world graphs, Malkov and Yashunin), links each vector to a few of its neighbors in a layered graph and finds a query’s neighbors by walking the graph, checking only a tiny fraction of the stored vectors. The result is almost always the same as an exhaustive search, at a fraction of the cost.