Memory, Knowledge and Retrieval

Models do not remember anything on their own. Each call starts empty, and whatever the model appears to know either came from training or was placed into the prompt by the surrounding system. This area covers both halves of that job: retrieval, which finds the right material at the moment it is needed, and memory, which decides what a system keeps about a person, a project, or its own past work.

It matters because this is the difference between software that repeats itself and software that accumulates. It is also where most accuracy complaints are genuinely solved. When an assistant answers confidently but wrongly, the fix is usually not a better model but better grounding, sharper chunks, or a retrieval step that quietly stopped returning the right document. Anyone evaluating a product should ask what it retrieves and what it stores long before asking which model sits underneath.

Start here

How the pieces fit together

Retrieval is a pipeline, and any stage in it can be the one that fails. Documents are ingested, text is extracted, that text is split into chunks, each chunk is turned into a vector and stored, a query is compared against the store, results are reordered, and the survivors are injected into the prompt. Memory sits on top of retrieval rather than beside it. Short-term memory is the current conversation, working memory is what the system is actively holding for the task at hand, and long-term memory is what gets written down and fetched later, which makes it a retrieval problem again.

Where to start

Read embeddings first, because the whole field rests on the idea that meaning can be measured as distance. Vector stores and similarity follow naturally from that. Then read chunking, which is the least glamorous and most consequential decision in any retrieval system: chunks that are too small lose the context that made them meaningful, and chunks that are too large dilute the match. Retrieval augmented generation and the pipeline around it come next. Grounding and citation come last, because they are what make an answer checkable rather than merely plausible.

Memory is a policy, not a feature

Deciding what a system remembers is a product decision with consequences you cannot patch later. Facts a user stated once may have been wrong or may since have changed, so a memory layer needs a way to supersede rather than only append. Resolving entities matters more than it looks: two spellings of the same company become two competing memories, and the system will use whichever surfaces first. Set an explicit rule for what gets written, what expires, and what a person can inspect and delete, then treat anything failing those rules as retrieval rather than memory.

What people get wrong

The most common error is assuming meaning-based search covers everything. It handles paraphrase well and exact identifiers badly, which is why product codes, names, and error strings need keyword matching running alongside it. The second is treating a similarity number as a confidence rating. It ranks candidates against each other and says nothing about whether any of them answers the question, so a top result comes back even when nothing relevant exists. The third is treating memory as an accumulator. Storing every exchange makes retrieval worse over time, because relevance drowns in volume and stale facts start outranking current ones.

Every term in Memory, Knowledge and Retrieval

Agentic Retrieval: Agentic retrieval lets a model decide for itself whether to search, what to search for, and when it has enough evidence, instead of running one fixed retrieval before every answer. The model can issue several searches in sequence, refine queries based on what came back, and choose among different sources or tools.
Approximate Nearest Neighbor Search: Approximate nearest neighbor search is a family of algorithms that find vectors close to a query vector without guaranteeing that the very closest ones are returned. By tolerating a small chance of missing a true neighbor, these methods answer queries over millions of vectors in milliseconds instead of comparing the query against every stored record.
BM25: BM25 is a ranking function that scores how well a document matches a keyword query, based on how often query terms appear in the document, how rare those terms are across the collection, and how long the document is. Developed from probabilistic retrieval research in the 1990s, it remains the default lexical baseline in search engines and retrieval benchmarks.
Chunk Overlap: Chunk overlap is the practice of repeating a portion of text at the end of one chunk and the start of the next, so a passage split across a boundary still appears intact somewhere. Overlap is usually expressed as a token or character count, commonly ten to twenty percent of chunk size. It costs storage and duplicate results.
Chunking: Chunking is the process of splitting documents into smaller passages before they are embedded and indexed. Chunk size and boundary choice determine what a retriever can return, since retrieval operates on whole chunks. Passages that are too large dilute meaning and waste context, while passages that are too small lose the surrounding information needed to interpret them.
Citation: A citation is a reference attached to a generated statement that identifies the source passage supporting it, usually as a document title, link, or passage identifier. Citations let a reader verify a claim, and they let a system be audited when an answer later turns out to be wrong.
Context Injection: Context injection is the practice of inserting relevant information into a model's prompt before it generates a response, such as retrieved passages, user profile facts, the current date, or tool schemas. Because the model only sees what the prompt contains, injection is how an application determines what the model knows for that call.
Context Precision: Context precision measures what share of the retrieved passages placed in a prompt are actually relevant to the question, and whether the relevant ones appear near the top. It penalizes padding a prompt with loosely related material. High recall with low context precision means the answer is buried in noise the model must ignore.
Contextual Retrieval: Contextual retrieval prepends a short, generated description of a chunk's surrounding document to that chunk before it is indexed. The added sentence or two states what the chunk is about and where it sits, so an isolated passage no longer loses the context that made it meaningful. Both embedding and keyword indexes are built over the enriched text.
Conversation Summarization: Conversation summarization compresses earlier turns of a dialogue into a shorter running summary so a session can continue past the limits of what fits in a single prompt. The summary replaces the omitted turns, usually while the most recent messages are kept verbatim. Every summarization pass discards detail permanently.
Corpus: A corpus is the complete body of documents a retrieval system searches over. It defines the boundary of what can be found: nothing outside it is retrievable, however capable the model. Corpus composition, freshness, coverage, and duplication shape retrieval quality more than most tuning decisions applied downstream.
Cosine Similarity: Cosine similarity measures the angle between two vectors, producing a value from negative one to one where one means the vectors point in the same direction. It ignores vector length and compares direction only, which is why it is the standard way to compare text embeddings whose magnitude carries little meaning.
Dense Retrieval: Dense retrieval finds relevant text by comparing embeddings, so a query matches passages whose meaning is close even when no words overlap. Both query and documents are converted into fixed length vectors by a neural model, and similarity is computed in that shared space. It is the retrieval style that most vector database workflows are built around.
Document Ingestion: Document ingestion is the pipeline that takes source files and records, extracts their text and metadata, splits them into passages, generates embeddings, and writes everything into a searchable index. It is the process by which raw material becomes retrievable, and it runs repeatedly as sources are added, changed, or removed.
Embedding: An embedding is a list of numbers that represents a piece of content, such as a sentence, image, or record, in a way that places related items close together in a shared coordinate space. A model produces the vector, and the distance between two vectors approximates how similar their meanings are. Embeddings make similarity computable.
Embedding Dimensionality: Embedding dimensionality is the number of values in each vector produced by an embedding model, commonly a few hundred to a few thousand. It is fixed by the model, and every vector in one collection must share it. Dimensionality drives storage size, comparison cost, and, up to a point, how much meaning a vector can carry.
Entity Resolution: Entity resolution is the process of determining that different records or mentions refer to the same real world thing, such as one customer appearing under three spellings across systems. It merges or links those references into a single identity so that facts about the entity accumulate in one place instead of fragmenting.
Episodic Memory: Episodic memory is the record of specific past events an AI system experienced, such as a particular conversation, a task attempt and its outcome, or a tool call and its result. Each entry is tied to a time and a situation, which lets a system recall what happened on a given occasion rather than only what is generally true.
Fact Extraction: Fact extraction is the process of pulling structured statements out of unstructured text, typically as subject, relation, object triples with a source reference. It converts prose into records that can be filtered, aggregated, updated, and traversed. It underpins knowledge graph construction and most durable memory systems that store what was learned rather than what was said.
Faithfulness: Faithfulness measures whether every claim in a generated answer is supported by the retrieved passages given to the model. It asks only about consistency with the supplied evidence, not about real-world truth, so an answer faithfully repeating an incorrect source still scores well. It is the standard way to detect unsupported additions in retrieval augmented systems.
Forgetting: Forgetting is the deliberate removal or downweighting of stored memories so a system does not carry every past detail forever. Mechanisms include hard deletion on request, time-based expiry, relevance decay that lowers a record's retrieval weight as it ages, and supersession when a newer fact contradicts an older one.
Graph RAG: Graph RAG is retrieval augmented generation that retrieves over a knowledge graph built from the source documents, rather than over independent text chunks alone. Entities and their relationships are extracted during ingestion, and queries traverse those connections to gather evidence spread across many documents. It targets questions that no single passage answers.
Grounding: Grounding is the practice of tying a model's output to verifiable source material rather than letting it answer from its parameters alone. A grounded system retrieves evidence, instructs the model to answer only from it, and expects the answer to state when the evidence is insufficient. The goal is answers that can be checked against a source.
HNSW: HNSW, short for Hierarchical Navigable Small World, is a graph based algorithm for approximate nearest neighbor search. It stores vectors as nodes in a layered proximity graph and answers a query by walking from a coarse top layer down to a dense bottom layer, moving toward closer neighbors at each step. It is the default index in many vector databases.
Hybrid Search: Hybrid search combines keyword matching with vector based semantic matching and merges the two result lists into one ranking. Keyword scoring catches exact terms, identifiers, and rare names, while embeddings catch paraphrase and intent. The blended result is usually more reliable than either method alone, particularly on technical corpora full of codes and product names.
Indexing: Indexing is the process of organizing content into a structure that makes search fast, whether an inverted index mapping terms to documents or an approximate nearest neighbor index over embeddings. Without an index, every query would have to scan the entire collection, so the index is what makes retrieval practical at scale.
Keyword Search: Keyword search retrieves documents that literally contain the terms in a query, ranking them by scoring functions that weigh term frequency and rarity. It is the oldest form of text retrieval and still the most predictable, because a user can see exactly why a result matched. Modern retrieval stacks keep it alongside embedding based search rather than replacing it.
Knowledge Base: A knowledge base is a curated collection of documents and structured facts assembled so that a system or a person can look up answers. In AI applications it usually means the indexed corpus a retrieval system searches, covering policies, product documentation, past tickets, and reference material. Its scope, freshness, and accuracy set the ceiling on answer quality.
Knowledge Cutoff: A knowledge cutoff is the date after which a language model's training data ends, so events, releases, and changes occurring later are absent from its parameters. The model may still answer questions about that period, confidently and incorrectly, unless the application supplies current information or the model is instructed to acknowledge the limit.
Knowledge Graph: A knowledge graph stores information as entities connected by typed relationships, such as a person working at a company that acquired another company. Unlike a document index, it makes connections explicit and queryable, which supports multi step questions that require following links between facts rather than matching text.
Late Chunking: Late chunking embeds a long document in one pass with a long-context embedding model, then pools the resulting token representations into per-chunk vectors afterward. Because every token was encoded while the whole document was visible, each chunk vector carries context from the rest of the text, unlike conventional chunking where pieces are embedded independently.
Long Term Memory: Long term memory is information an AI system retains across sessions and recalls much later, such as stable user preferences, learned facts, and records of completed work. It lives in a database, document index, or knowledge graph outside the model, and it is retrieved selectively rather than carried in every prompt.
Mean Reciprocal Rank: Mean reciprocal rank is a retrieval metric that scores each query by one divided by the position of its first relevant result, then averages across all queries. A first-place hit scores 1.0, second place 0.5, third 0.33. It rewards putting a correct result at the very top and ignores everything after the first hit.
Memory Consolidation: Memory consolidation is the background process of turning accumulated raw interaction records into a smaller, more durable store: merging duplicates, summarizing sessions, promoting recurring details into stable facts, and discarding transient noise. It borrows its name from the biological process by which experiences are stabilized into long-term memory.
Metadata Filtering: Metadata filtering restricts a search to records whose stored attributes satisfy given conditions, such as tenant, language, document type, author, or date range. The filter is applied alongside similarity scoring rather than after the fact, so results are both relevant and permitted. It is the mechanism that makes shared vector collections safe for multiple tenants.
Multi-Query Retrieval: Multi-query retrieval generates several variations of a single user question, runs a search for each, then merges the result sets into one ranked list. The variations differ in phrasing, specificity, or angle, so a document missed by one formulation can still be found by another. It trades extra retrieval cost for higher recall.
Ontology: An ontology is a formal specification of the entity types, relationship types, and constraints that a knowledge store is allowed to contain. It defines what a Person, Order, or Incident is, which relations may connect them, and which properties each requires. Without one, a knowledge graph drifts into inconsistent labels that queries cannot rely on.
Prompt Caching: Prompt caching stores the processed form of a repeated prompt prefix so that subsequent requests reusing that prefix skip part of the computation. Because system instructions and retrieved reference material often stay identical across many calls, caching them reduces latency and cost for the unchanged portion while the varying tail is processed normally.
Query Expansion: Query expansion adds extra terms to a search query in order to retrieve documents that express the same idea in different words. The additions can come from synonym lists, from terms found in top ranked initial results, or from a language model asked to generate related vocabulary. It raises recall at some risk to precision.
Query Rewriting: Query rewriting transforms a user's raw input into a better search query before retrieval runs. Typical rewrites resolve pronouns using conversation history, make an implied subject explicit, strip conversational filler, or split a compound question into parts. It exists because what a person types is often a poor query even when their intent is clear.
Recall and Precision: Recall and precision are the two complementary measures of retrieval quality. Recall is the share of all relevant items that a system actually returned, while precision is the share of returned items that were relevant. Improving one often degrades the other, so retrieval systems are tuned toward a target balance rather than a single number.
Reranking: Reranking is a second scoring pass that reorders an initial list of retrieved candidates using a more accurate and more expensive model. A fast retriever fetches perhaps fifty candidates, and the reranker examines each one against the query to produce a better ordering, from which only the top few are kept for reading.
Retrieval Augmented Generation: Retrieval augmented generation is a technique that improves language model output by fetching relevant documents from an external store at query time and placing them in the model's prompt before it answers. The model composes its response from that supplied evidence rather than from its parameters alone, which allows answers to reflect private or recently updated information.
Retrieval Evaluation: Retrieval evaluation measures how well a search system finds the documents needed to answer a set of test queries. It requires a labeled set pairing queries with known relevant passages, and it scores the retriever separately from the model that writes the final answer. Without it, pipeline changes are guesses.
Retrieval Pipeline: A retrieval pipeline is the ordered sequence of steps that turns a user question into the passages a model reads, typically query processing, retrieval, merging, filtering, reranking, and context assembly. Treating it as a pipeline makes each stage measurable and replaceable, which is how retrieval quality is diagnosed and improved.
Semantic Memory: Semantic memory is an AI system's store of general facts and concepts detached from the occasion on which they were learned, such as a customer's billing cycle or a company's approval threshold. It answers what is true rather than what happened, and it is typically held as structured statements, graph edges, or indexed documents.
Semantic Search: Semantic search retrieves results by meaning rather than by literal word overlap. Queries and documents are converted into embeddings, and the system returns the items whose vectors sit closest to the query vector. This lets a search for ending my plan surface a document titled cancellation policy, even when the two share no terms.
Short Term Memory: Short term memory is the recent context an AI system keeps available within a conversation or session, typically the last several exchanges plus any state built during the current task. It lives in the prompt or in a session store, and it is discarded or summarized when the session ends.
Similarity Score: A similarity score is the number a retrieval system assigns to a candidate result indicating how closely it matches the query, used to rank results and sometimes to filter weak ones. Scores are relative to the scoring method and the corpus, so a value that looks high in one system may be unremarkable in another.
Sparse Retrieval: Sparse retrieval scores documents using term based representations in which most entries are zero, matching on the words a query and document actually share. Classical keyword ranking functions are the best known form, and learned sparse models extend the idea by predicting term weights with a neural network. It complements embedding based search rather than competing with it.
Text Extraction: Text extraction is the step that recovers readable text and structure from source formats such as PDFs, word processor files, spreadsheets, slides, web pages, images, and audio. Its output quality sets an upper bound on everything downstream, since a passage never extracted correctly cannot be chunked, embedded, or retrieved correctly.
User Profile Memory: User profile memory is a persistent, structured record of durable facts about an individual user, such as their role, preferences, working context, and constraints. Unlike conversation history it is intentionally small, curated, and injected into most sessions. It exists so an assistant does not have to relearn stable details in every new conversation.
Vector Database: A vector database is a data store built to hold high dimensional numeric vectors and to find the ones most similar to a query vector quickly. It indexes embeddings produced by machine learning models and supports nearest neighbor search, usually with metadata filters. Such systems underpin semantic search and retrieval for language model applications.
Vector Index: A vector index is the data structure that makes similarity search over stored embeddings fast, by organizing vectors so a query can skip most of the collection. Without one, a search must compare the query against every vector. The index type chosen sets the achievable balance between query speed, memory footprint, accuracy, and update cost.
Working Memory: Working memory is the information an AI system actively holds while performing a task, including the current goal, intermediate results, tool outputs, and constraints being tracked. It is scoped to the task rather than to the session, and it is assembled fresh for each model call from the pieces the next step requires.

Back to the AI Glossary

Memory, Knowledge and Retrieval

Start here

How the pieces fit together

Where to start

Memory is a policy, not a feature

What people get wrong

Commonly confused

Semantic Memory vs Episodic Memory

Short Term Memory vs Working Memory

Semantic Search vs Hybrid Search

Cosine Similarity vs Similarity Score

Every term in Memory, Knowledge and Retrieval