What makes a knowledge assistant useful to a small business

Every few weeks someone shows me a chatbot they built over their company’s documents, and it works — in the demo. Ask it a question from the onboarding PDF and it answers in a confident paragraph. Then two months later I ask how it’s going and the answer is “people stopped using it”.

I have built a few of these now, for businesses with a dozen to a couple of hundred staff, and the ones that stuck had almost nothing to do with which model was underneath. They came down to three product decisions that are easy to skip because the demo works without them.

1. Every answer shows where it came from

A retrieval-based assistant (the “RAG” pattern: find the relevant passages, hand them to the model, have it answer from those) is only as trustworthy as its worst answer. The first time it tells someone the wrong holiday policy with total confidence, that person is done. And they will tell everyone else.

The fix is not a better model. It is showing the source, every time, right under the answer: the document name, the section, a link that opens it at the right place. Two things happen when you do this.

First, people check. In the beginning they check everything, which is exactly what you want — it is how trust gets built, one verified answer at a time. Later they check only the ones that surprise them.

Second, the assistant becomes a search engine that can talk. Half the value of the tool in the businesses I worked with was not the generated paragraph; it was that the paragraph pointed at the right document, which nobody could find before. A hint of what the model thinks plus a reliable link beats a beautiful answer with no provenance.

Practically this means the retrieval layer has to keep metadata — file, heading, page — alongside every chunk, and the prompt has to require the model to cite chunk identifiers that you then turn into links. It is not much code. It is the difference between a toy and a tool.

2. It can do something, not just say something

The second decision is whether the assistant can act. A knowledge assistant that only answers questions is a nicer manual. One that can also update a record is a colleague.

For one services business, the most-used feature was not question answering at all. It was “mark job 4821 as complete and note that the customer asked for a Tuesday follow-up” — typed in plain language, turned into two writes against the tools staff already used. The assistant read from the same database it answered questions from, so it could confirm what it was about to change before changing it.

This is where the “agent” word earns its keep, and where the engineering gets real:

• Every action goes through a function the assistant is allowed to call, with typed arguments, so it cannot invent a new capability.
• Anything destructive asks first and shows exactly what will change.
• Every action is logged with who asked and what the model saw, so a mistake can be traced and reversed.
• The model never holds credentials. It asks the backend to do things on behalf of a logged-in person, with that person’s permissions.

Do that and staff stop thinking of it as “the chatbot” and start thinking of it as the fastest way to get things done. Skip it and the assistant stays in the “nice, I guess” category forever.

3. Log the questions — that is the real product

The third decision costs almost nothing and turns out to be the most valuable: keep a log of every question, whether it was answered well, and which sources were used. Then put a dashboard on it.

For the business owner, that dashboard is a map of confusion. The questions people ask most are the procedures nobody can find. The questions the assistant cannot answer are the procedures that were never written down. The questions where people click through to the source and then keep reading are the documents that are out of date — the answer was technically right and practically useless.

In one company, the top question for a month was a variation of “how do I get the van key on a weekend”. There was no document. There was a WhatsApp group. Writing a one-paragraph procedure removed the question entirely, and the owner realised that the assistant’s most useful output was not answers but a prioritised list of what to write next.

That is the part I would build first if I had to choose. A retrieval assistant with no question log is a black box that is either working or not, and you will only find out when usage drops. With the log, the assistant tells you how to improve the assistant.

What about the model?

It matters less than you’d think, as long as it is decent at following instructions and staying inside the sources you give it. I have shipped these on hosted frontier models and on smaller open models running cheaply, and users noticed the three decisions above far more than the difference between models.

Where the model choice does matter is cost, and the answer there is boring: count tokens, set a budget per user or team, and show people their usage. A business that can see the meter trusts the tool. A surprise invoice ends the project.

The short version

If you are thinking about building one of these for your own business:

1. Show sources under every answer, with links.
2. Let it take a handful of well-guarded actions in the tools you already use.
3. Log every question and look at the log weekly.

Get those three right and a modest model on a modest budget will be something your team uses every day. Get them wrong and the best model in the world is a demo that dies in two months.

I build these for businesses that want their own data to work harder. If that sounds like yours, tell me what you have and I’ll tell you honestly whether an assistant is the right first step.