AI RAG Chatbot

Problem

General-purpose LLMs lack access to private or mission-specific data, leading to inaccurate or generic information.

Constraints

• • Latency requirements for real-time chat
• • Token window limitations
• • Managing diverse document formats (PDF, Web, Docs)

Architecture

Ingestion pipeline for embeddings, vector storage for semantic retrieval, and a RAG-enabled prompt injection layer.

Key Decisions

• • Use hybrid search (vector + keyword) for better relevance
• • Implement recursive chunking for better context preservation

Tradeoffs

• • Slightly higher latency for significantly higher accuracy
• • Preprocessing overhead for document embedding

Challenges

• • Optimizing chunk size for specific domain knowledge
• • Filtering irrelevant retrieval results to prevent context pollution

Results

• • Highly accurate domain-specific responses
• • Zero-hallucination performance within known data

What I Would Improve

• • Implementing a re-ranking stage for superior retrieval
• • Adding multi-query expansion for complex user intents