Case study · Leav Aviation · In production
DocsAI
A retrieval assistant over thousands of pages of aviation manuals and regulations, and an agent over the airline's live operations data. Staff ask in plain language; the answer streams back with the exact source page cited inline, beside that page in a PDF viewer. The pipeline is built from scratch, with no LangChain.
Two kinds of question, one interface. A question about a procedure is answered from the manuals, with the page cited and open beside the answer. A question about operations, how many flights ran last week, who is available, whether an aircraft is legal to dispatch under the minimum equipment list, is answered by calling tools against live systems and rendering the result as a table or chart. The assistant chooses the path; either way the answer says where it came from.
The problem
Thousands of pages, and the answer has to show its source
An airline runs on manuals: operations, maintenance, training, and the regulations behind them, thousands of pages that change with every revision. The people who need an answer work in operations, crew planning and maintenance, often mid-task, sometimes on the hangar floor with their hands busy. Finding the right paragraph means knowing which manual, then which chapter, then reading.
In an EASA-regulated operation an answer without its source is not an answer. Whoever acts on it has to be able to open the manual at the page and check, and an auditor later has to be able to see that they could. A general chat model that paraphrases from memory fails that test even when it happens to be right.
The second half of the problem is that not every question is in a manual. Flight history, crew availability and dispatch legality live in operational systems, not documents, and a person assembling one answer from both had to visit both.
The approach
Four decisions
Cite the page, or do not answer
Every answer carries its source page inline and renders that page beside the reply. When the retrieved context is too weak to support an answer, the pipeline re-queries with sub-questions rather than answering from bad ground.
Built from scratch
No LangChain. Routing, rewriting, expansion, hybrid search, fusion and reranking are code we own, so every stage can be traced, tested and costed on its own.
Documents and live data in one interface
The same assistant chooses between document search and typed tools over the operations API, crew availability and minimum-equipment-list legality. Regulatory claims trace to the manual page; operational claims to the system of record.
Measure every query
An admin console traces each interaction to its tool calls, tokens, latency and cost. A 100-case golden eval set scored by a model as judge guards releases. The corpus is an explicit, versioned ingestion set, not a black-box index.
What shipped
Four screens from production
The questions and answers on these screens are real. User identifiers are redacted.
The answer and its page. A question in plain language, an answer streamed back with the source cited inline, and the manual open at that page in the integrated viewer. The reader checks the source without leaving the conversation. Voice input through Whisper transcription covers the hands-busy case.
“How many flights did we do last week?” Answered from live operations data, not documents: the agent calls the operations API and renders the result as tables and charts with route and crew breakdowns. Document retrieval and tool calls run through the same interface, so the person asking never chooses a mode.
Every interaction, costed. The admin console traces each query: confidence, latency, token count and dollar cost, down to the individual tool calls. This is where “what does a query cost?” is answered with data rather than an estimate: about $0.006 to $0.02 each.
Managed, not indexed. The ingested manuals and their catalog state. The set is explicit and versioned: which documents are in, which revision, when they were indexed. A revision is re-indexed deliberately, never silently.
Retrieval
From question to cited answer
Every document question passes through the same stages. Each is ordinary code with its own tests, which is what makes the pipeline traceable end to end and lets a weak answer be opened and read stage by stage.
Blue marks where the model touches the request path after retrieval: the reranker, the sufficiency judgement and the answer. The dashed loop is the self-correction: weak context goes back through routing as sub-questions instead of becoming a confident wrong answer.
Route and rewrite
The question is classified and rewritten for retrieval, then expanded into variants, so a procedure the manual names differently from the way a person asks about it is still found.
Search two ways
Dense retrieval over pgvector (text-embedding-3-large at 2,000 dimensions, HNSW index) runs beside Postgres full-text search. Neither alone is enough: embeddings miss exact part numbers and clause references; keywords miss paraphrase.
Fuse
Reciprocal rank fusion merges the two ranked lists into one, so a passage found by both methods rises above one found by either.
Rerank
An LLM reranker orders the fused candidates by how well each one answers the question, rather than by how similar it looks to it.
Check sufficiency
Before answering, the pipeline judges whether the retrieved context can support an answer. If not, it decomposes the question into sub-questions and retrieves again, instead of answering from bad ground.
Answer and cite
The answer streams back with the source page cited inline, and the viewer opens the manual at that page.
Parent–child chunking
Documents are chunked twice. Children of 200 tokens are what the search matches: small enough that a hit means the passage is about the question. Each child points to a 1,500-token parent, which is what the model reads: large enough to carry the surrounding procedure. Recall precision and answer context stop competing for one chunk size.
Voice input
Whisper transcription lets a question be spoken, for the hangar floor and anywhere hands are busy. The transcript goes through the same pipeline as typed text.
Live data
Agentic answers over operations
Not every question is in a manual. Flight history, crew availability and minimum-equipment-list legality live in the airline's operational systems, and the assistant reaches them through tools.
One agent, typed tools
Per question, a tool-calling agent selects between document search, the operations API, crew availability and legality tools. Each tool has a typed contract, so what the agent is able to do is a finite list that can be read and audited.
Structured rendering
Results come back as tables, charts and ranked lists rather than prose, because a week of flights or a crew roster is data, and reading it as a paragraph is worse than reading it as a table.
Grounded on both paths
Operational claims trace back to the system of record they were read from; regulatory claims to the manual page. Each answer says which kind of source it rests on.
Observability
Quality, latency and cost, per query
Traces. The admin console records every interaction down to its tool calls, token usage, latency and dollar cost. A slow or expensive answer can be opened and read stage by stage. Across production traffic a query costs about $0.006 to $0.02.
Evals. A golden set of 100 questions with reference answers guards releases, scored by a model acting as judge. A change to chunking, prompts or the reranker is measured before it reaches staff, not discovered afterwards.
Corpus. The manual set is an explicit, versioned ingestion set rather than a black-box index: which documents are in, at which revision, indexed when. A revision is re-ingested on purpose.
Tests. 630 backend tests run in CI, alongside the eval set.
Stack
What it is made of
| Layer | Choice |
|---|---|
| Embeddings | text-embedding-3-large at 2,000 dimensions |
| Vector index | pgvector HNSW, inside Postgres |
| Hybrid search | Dense retrieval and tsvector full-text, fused by reciprocal rank fusion, then an LLM reranker and a retrieval-sufficiency check |
| Chunking | Parent–child: 200-token children for matching, 1,500-token parents for context |
| Agent | Tool-calling over document search, the operations API, crew availability and MEL legality; results rendered as tables and charts |
| Backend | FastAPI, async SQLAlchemy, Alembic migrations |
| Frontend | React 19, TypeScript, Tailwind 4, integrated PDF viewer, Whisper voice input |
| Infrastructure | AWS ECS, Cognito, Terraform, CI |
| Tests | 630 backend tests; 100-case golden eval set with LLM-as-judge scoring |
| Cost | About $0.006 to $0.02 per query, read from per-query traces |