A demo is the worst moment to judge an AI system. You see what the system can do, on an example the vendor picked. In knowledge work, what counts is mostly what a system does not do: invent a source, change a text without anyone noticing, take your data somewhere you don't know about. A demo won't show you that. It's in the documentation and the contract, or it's nowhere.
So send your questions before the demo, in writing. A written answer can be set next to a second vendor's, attached to the contract and reread a year later. A vague answer also stands out far more on paper than in a meeting where someone is clicking through screens with enthusiasm.
Six layers, six questions
The questions follow TRACES, the model VergeLabs uses to assess any system.
TRACES are the six layers on which you assess an AI system: Trail, Retrieval, Approval, Constraints, Expertise, Storage. Where a layer is missing, there is a risk.
For each layer below you'll find the question to copy as is, what a usable answer contains, and what an evasive answer sounds like. That last part is just as useful, because evasive answers sound alike whichever vendor gives them.
T · Trail
The question. What does the system record for every answer and every change: the source, the claim, who approved it and when? And can we export that?
Usable. A list of the fields, a sample export, and a commitment that the trail is created as people work. If someone has to keep it up by hand, it won't get done in a busy week.
Evasive. "Everything is logged." Ask again: a log of who signed in when is not the same as a record of which claim rests on which source.
R · Retrieval
The question. Which sources does the system answer from, and what happens when the answer isn't in them?
Usable. Only from a collection you define yourselves, with a reference to the passage for every answer, and an "I don't know" when that passage is missing. Better still, the system can show which claims in a text have no source. Searching alone can never prove that something is absent.
Evasive. "Hallucinations are kept to a minimum." Or: "The model is trained on reliable data." Both answers are about the model, not about your sources.
A · Approval
The question. Where in the work does a person confirm the content, does that differ by type of change, and is the confirmation recorded?
Usable. A description of the approval step, per passage or per type of change, with the source alongside. Fixing a typo doesn't need an expert's judgement; rewriting a figure does.
Evasive. "The user always remains responsible." That's a disclaimer, not a feature.
C · Constraints
The question. Which actions is the system technically unable to perform, even if a user asks?
Usable. A short list, such as sending, publishing, making a commitment to a client or moving data outside your own environment. Plus an explanation that the block sits in the software, outside the model.
Evasive. "The model has been instructed not to do that." An instruction is a request. A model that isn't allowed to do something can in principle still do it; a system that can't, can't.
E · Expertise
The question. Which decision stays with our specialists, and how does the system make that decision easier?
Usable. A named decision, such as the conclusion, the review or the sign-off, and a concrete aid for it. For instance, the source passage shown next to the claim, so nobody has to read the whole source.
Evasive. "The system takes the thinking off your hands." For work that goes out under your name, that is exactly the wrong argument.
S · Storage
The question. Where is our data stored, is it used to train models, and which clause of which contract says so?
Usable. A region, a list of sub-processors and a reference to a clause in the data processing agreement.
Evasive. "We take privacy very seriously." Or "GDPR-compliant", with no document attached.
How to read the answers
Give each answer one of three ratings: present, partial or absent. Present means the answer is specific and you can find it in a document. Partial means the feature exists but you need to ask more, or it was only promised verbally. Absent is an evasive answer, or no answer at all.
Not every layer carries the same weight. Which one weighs most depends on where your own work is vulnerable, and that's what the second model shows. SEAL describes the four phases knowledge work goes through: Source, Edit, Approve and Launch.
SEAL shows which phase leaks. TRACES is how you test whether a system can close that leak.
If your work leaks at Source, because nobody records where a claim comes from, Retrieval is the layer you can't do without. If it leaks at Edit, because a text is no longer checked against its source after rewriting, it's Trail and Approval. A system that scores absent on that one layer is out, however good the rest is.
What you can do this week
- Pick the system you're considering. One you already use works just as well.
- Put the six questions in a single email to the vendor. Ask for written answers, with a reference to documentation or the contract where possible.
- Put the answers in a table with three columns: layer, answer, rating.
- Count the layers rated absent. If one of them is the layer that weighs most for you, you know enough.
If you'd rather go through the public documentation yourself first, the vendor scan on this site takes a system through the same six layers.