Ask your company docs — get answers you can trust
79
Documents
408
Chunks
Hybrid
BM25 + dense
20
Eval questions
What this example is for
This is a smart search over a real knowledge base — guides about reliable software, past outage reports, and team decision notes. You can ask things like “What is an error budget?”, “Why did GitLab’s database go down in 2017?”, or “When should we use a canary release?” It finds the right pages first, then answers from those pages and shows you the sources.
That is better than asking a normal chat model alone. A normal model can guess or make things up. This one sticks to your docs, so you can check where each answer came from — useful for handbooks, support help, and policies where being right matters.
Ask the SRE / DevOps knowledge base. Answers cite retrieved chunks. Toggle Naive vs Improved to compare pipelines.
Architecture Notes
Design decisions for this demo. The goal is inspectability and a fair Naive vs Improved comparison — not a black-box chatbot.
Documents are split on markdown headings so each chunk stays a coherent topic. Target size is ~2.2k characters (hard cap ~3.2k) with 12% overlap so definitions that straddle a boundary are still recoverable. Every chunk is prefixed with title + heading path so both BM25 and embeddings see document context, not orphaned body text.
BM25 catches exact terms (incident names, SLOs, ADR IDs). Dense vectors catch paraphrases. The two ranked lists are fused with Reciprocal Rank Fusion (k=60), which is stable when score scales differ and does not require training a fusion model.
text-embedding-3-small at 256 dimensions (Matryoshka). For a ~400-chunk demo corpus this is enough recall at much lower cost and latency than text-embedding-3-large or full 1536-d vectors. Production corpora with noisier queries may want higher dims or a stronger model — that is a measured trade-off, not a default.
Multi-query rewrite (2 alternate angles) → BM25 + dense per query → RRF merge → lexical/coverage rerank → expand adjacent chunks for section context. The Naive path is intentionally weaker: single query, dense-only, top-5, no rewrite or rerank — so the before/after is honest.
20 gold questions (including multi-hop) scored with Recall@K and MRR. Known failure modes are listed on the Evaluation tab. Groundedness is sampled, not claimed as perfect.
Architecture Notes
Design decisions for this demo. The goal is inspectability and a fair Naive vs Improved comparison — not a black-box chatbot.
Documents are split on markdown headings so each chunk stays a coherent topic. Target size is ~2.2k characters (hard cap ~3.2k) with 12% overlap so definitions that straddle a boundary are still recoverable. Every chunk is prefixed with title + heading path so both BM25 and embeddings see document context, not orphaned body text.
BM25 catches exact terms (incident names, SLOs, ADR IDs). Dense vectors catch paraphrases. The two ranked lists are fused with Reciprocal Rank Fusion (k=60), which is stable when score scales differ and does not require training a fusion model.
text-embedding-3-small at 256 dimensions (Matryoshka). For a ~400-chunk demo corpus this is enough recall at much lower cost and latency than text-embedding-3-large or full 1536-d vectors. Production corpora with noisier queries may want higher dims or a stronger model — that is a measured trade-off, not a default.
Multi-query rewrite (2 alternate angles) → BM25 + dense per query → RRF merge → lexical/coverage rerank → expand adjacent chunks for section context. The Naive path is intentionally weaker: single query, dense-only, top-5, no rewrite or rerank — so the before/after is honest.
20 gold questions (including multi-hop) scored with Recall@K and MRR. Known failure modes are listed on the Evaluation tab. Groundedness is sampled, not claimed as perfect.