In a knowledge assistant demo, someone asks a question whose answer is in the documents. The assistant answers, with a reference, and everyone nods. That is the easiest test there is. It says nothing about what happens when the answer is not in the documents, and that is the question a colleague asks in the first week.
For knowledge work, that's the question that counts. An assistant that invents a plausible answer is more dangerous than no assistant at all. You only find out when someone builds on it under their own name.
It's a known problem
Researchers have known this for a while. For SQuAD, a widely used test set for systems that answer questions from a text, Rajpurkar, Jia and Liang published a second version in 2018 with over 50,000 unanswerable questions, deliberately written to look like answerable ones. A strong system scored 86% F1 on the first version but only 66% on the second. It could answer the questions. It just couldn't tell which ones had no answer.
That study is about an older generation of systems and a standard benchmark, not about your assistant. What it mainly shows is how to build a test set. If you want to know whether a system can say "I don't know", you have to ask it questions that call for it.
Which layer this is
TRACES are the six layers on which you assess an AI system: Trail, Retrieval, Approval, Constraints, Expertise, Storage. Where a layer is missing, there is a risk.
This is Retrieval: the system answers only from a defined collection of sources, with a reference, and says the answer is missing instead of inventing one.
That sentence contains three things you can test. Does the answer come from the collection? Is there a reference? And what does the system do when the answer isn't in there? A demo shows only the first two. For the third, you need questions you devise yourself.
The test set: twenty questions, four types
Make a list of twenty questions about the collection the assistant works with. Spread them over four types.
1. The answer is in there (five questions). The control group. Pick questions where you know which page holds the answer. If the assistant gets these wrong, there's no point testing the rest.
2. The topic is in there, the detail isn't (five questions). The source describes the leave policy but says nothing about leave for interns. The system shouldn't work out what it "probably" is.
3. The topic isn't in there (five questions). A question about something your organisation has never written down. Here you should see "I don't know" more or less literally, or a pointer to who does.
4. The question doesn't match the source (five questions). A false premise: a question about the 2025 policy when the collection only holds 2023, or one that assumes a department that doesn't exist. A good system corrects the premise or says it has nothing on it. A weak one just goes along.
Ask follow-ups on every answer you get, because the first response says little. Ask the same unanswerable question twice, in different words. A system that says "I don't know" the first time and gives an answer the second has no firm boundary.
How to score the answers
Give each answer one of three verdicts.
- Answer with source. The answer is there, with a passage you can open and that really contains it. You have to check that last part, because a reference that looks tidy doesn't prove the passage says what the system claims.
- I don't know. The system says the answer isn't in the sources. That's the right answer to types 2, 3 and 4.
- Answer without source. The system gives an answer, but the passage is missing or says something else. This is the failure the test is looking for.
Count per type. For type 1 you want five answers with source. For types 2, 3 and 4 you want no answer without source, not once. One is not chance but a pattern: the system invents when it finds nothing.
A made-up example. An assistant with the staff handbook gets the question: "How many days of study leave does an intern get?" The handbook only mentions study leave for permanent staff. The system answers: "Interns are entitled to two days of study leave per year." No source is given. That's an answer without source, and exactly the kind that gets through when you only ask good questions.
Repeat the test
The test set isn't a one-off inspection. Whenever the collection changes, a document is added or the model is replaced, the boundary can shift. Keep the twenty questions, run them again and put the scores side by side. That also starts the record that Trail asks for: what was tested, when, and with what result.
If you'd rather start by going through a vendor's public documentation, you can use the vendor scan on this site. It runs a system past the same six layers. The test set above then checks what happens in practice.
What you can do this week
- Pick the collection the assistant works with, or part of it: a folder, a handbook, a year's volume.
- Write twenty questions in the four types above. For type 1, note the page where the answer is.
- Ask all of them and keep the answers. Give each a verdict: answer with source, I don't know, or answer without source.
- Count the answers without source in types 2, 3 and 4. If there is even one, you know enough not to roll the assistant out widely yet.
Source
Rajpurkar, P., Jia, R. & Liang, P. (2018). Know What You Don't Know: Unanswerable Questions for SQuAD. ACL 2018. arxiv.org/abs/1806.03822