Why Your Corporate Chatbot Doesn't Work — And What Does

I built the same corporate chatbot three times. First with naive keyword search. Then with hybrid RAG. Then with a full agentic loop with tool calling.

The first two felt like demos. The third one felt like a colleague who actually read the documentation.

The Setup

The task was simple: answer questions about internal company processes — onboarding, IT requests, travel policies. The kind of stuff that lives in 47 different PDFs on a SharePoint nobody can navigate.

Attempt 1: Keyword Search

Classic full-text search with some ranking. Works fine if you know the exact terminology the document uses. Falls apart the moment someone asks a natural question like “what do I do when my laptop breaks?”

Attempt 2: Hybrid RAG

Better. Vector embeddings plus keyword search. The answers improved, but the system still struggled with multi-step questions or anything that required combining information from multiple documents.

Attempt 3: Agentic Loop

This is where it got interesting. Instead of retrieving and summarizing, the agent reasons about which documents to consult, pulls relevant sections, cross-references them, and constructs an answer. It can handle “I’m traveling to the US next month, what do I need to organize?” — which touches travel policy, visa requirements, expense guidelines, and IT equipment rules.

The Takeaway

Most off-the-shelf solutions I’ve evaluated are stuck at Attempt 2. They’re good enough for a demo, not good enough for daily use. The gap between “impressive in a meeting” and “actually useful on Monday morning” is exactly where the hard engineering work lives.