With an AI summary, invention isn't the biggest risk. Omission is. The model drops exactly the detail that limited the claim, and the summary ends up saying more than the source. Ordinary subediting won't catch it, because every name, figure and date is still correct.
This isn't a hunch. Uwe Peters (Utrecht University) and Benjamin Chin-Yee (Western University) compared 4,900 summaries of scientific texts, produced by ten widely used language models, with the originals. Most models stated conclusions more broadly than the source, even when explicitly asked to be accurate. For DeepSeek, ChatGPT-4o and LLaMA 3.3 70B this happened in 26 to 73 percent of cases. Compared with summaries written by people, the AI summaries were nearly five times as likely to contain an overly broad generalisation. And newer models tended to do worse than older ones.
The study looked at scientific texts. For a newsroom that has reports, studies and press releases summarised, that's close to home.
Which phase this is
SEAL describes the four phases knowledge work goes through: Source, Edit, Approve, Launch. In every phase the traceability of a source can be lost, and the weakest phase determines what the work is worth.
An overly broad summary is an Edit problem. The question for that phase is: does the source still cover the text after editing?
At Source, something is missing, such as a source that doesn't exist. At Edit, nothing is missing. The report exists, the link works and the figure is there. The text just claims something the report doesn't. That's why this error barely shows up in a final check: everything looks fine.
Three places where meaning shifts
Put the summary next to the source and, for every sentence that makes a claim, look at three things.
1. Scope. Who, where and how many? A study of vocational students in one region becomes "students". A trial at three schools becomes "schools". Find the group, the place and the size in the source, and check they're still in the summary.
2. Certainty. How sure is the source? "Is associated with" becomes "leads to", and "may" disappears. Sometimes a finding even turns into advice: the source describes what happens, the summary says what should happen. Count the caveats in the source passage and look for them again.
3. Time. When did it apply? A measurement from 2019 becomes a statement in the present tense: "had" becomes "has". Check the period of the study and the tense of the summary.
A made-up example, to see the three points side by side:
- Source: "In our sample of 312 secondary school teachers, a minority reported in 2024 that they used AI weekly for marking."
- Summary: "Teachers are increasingly using AI for marking."
The scope is gone: from 312 secondary teachers to all teachers. The certainty has shifted: "a minority" has disappeared, and "increasingly" is a trend the source doesn't measure. And the time: 2024 has become the present. Every word still there is correct. There just aren't enough of them.
Why subediting doesn't find it
Subediting looks at what's on the page: is the name right, is the figure right, does the sentence read well. An overly broad summary fails on what is no longer on the page. You only find that with the source alongside, sentence by sentence. Without a fixed routine that simply doesn't happen when the deadline is close, and an extra read-through changes little: anyone reading without the source is reading a fluent text.
A routine for the newsroom
- An AI summary doesn't go through without the source alongside. Not the link, the passage.
- Whoever checks it goes through scope, certainty and time for each claim, and ticks off the passage.
- If the sentence is rewritten later, the tick lapses. That's the core of Edit: an approval belongs to the passage it refers to.
- Before the summary, have the model list the source's limits: group, period, caveats. That speeds up the check. It doesn't replace it, because in the study the models overgeneralised even when told to be accurate.
The researchers suggest two more measures themselves: a lower temperature setting, and testing models for exactly this error before use. By hand, it stays work. A system that shows the source passage next to every sentence turns it into a matter of reading two texts side by side. In TRACES, that layer is called Retrieval.
What you can do this week
- Take five AI summaries that were used or published in the past month, with their sources.
- In each summary, mark the sentences that make a claim.
- For each sentence, check scope, certainty and time against the source passage.
- Count the sentences that are broader than the source, and see how many of them subediting would have caught.
The difference between those two numbers is your Edit leak.
Source
Peters, U. & Chin-Yee, B. (2025). Generalization bias in large language model summarization of scientific research. Royal Society Open Science. arxiv.org/abs/2504.00025