What is Document Ingestion?
Also called Ingestion Pipeline, Data Ingestion.
Document ingestion is the pipeline that takes source files and records, extracts their text and metadata, splits them into passages, generates embeddings, and writes everything into a searchable index. It is the process by which raw material becomes retrievable, and it runs repeatedly as sources are added, changed, or removed.
A typical pipeline has stages: connect to the source, fetch new or changed items, extract text and structure, normalize and clean, chunk, embed, and write to the index with metadata. Each stage can fail independently, so pipelines record per document status rather than treating a whole run as a single success or failure.
Change detection determines freshness. Full reingestion is simple but expensive and briefly leaves the index inconsistent, so most systems track content hashes or modification timestamps and process only what changed. Deletions require explicit handling, because a source removed at origin remains retrievable until its vectors and metadata are purged.
Metadata captured at ingestion is what makes filtering and permission enforcement possible later. Source identifier, owner, timestamps, document type, language, and access scope should be recorded at this point, since reconstructing them from chunk text afterward is unreliable. The same applies to structural context such as heading paths and page numbers.
Ingestion is also where content quality problems must be caught. Scanned images without a text layer, boilerplate headers repeated on every page, navigation menus in saved web pages, and near duplicate files all degrade retrieval, and each is cheaper to fix once during ingestion than to work around on every query.
Key points
- Fetch, extract, clean, chunk, embed, index, repeat.
- Track changes to avoid full reingestion each run.
- Deletions must purge vectors and metadata.
- Capture source, owner, dates, and access scope at ingest.
- Fix boilerplate and duplicates here, not at query time.
In practice
A nightly job checks a document folder, finds four files changed and one removed. It extracts text from the four, strips repeated page footers, splits them into ninety passages, embeds each one, and writes them with owner and modification date. The removed file's passages are deleted from the index, so it stops appearing in answers the next morning.