Sorting a workflow: which steps AI may do

"Can we use AI?" has no good answer. Yes, for what? For the work as a whole the answer is no, and for part of it the answer is yes. As long as the team doesn't pull those two apart, one of two things happens: nobody dares, or everybody uses it everywhere.

Both cost something. In the first case, work sits there that a model could perfectly well prepare. In the second, work goes out the door that nobody has checked against a source.

Why you can't tell from the task

The awkward part is that you can't tell from a task whether AI can handle it. In 2023, researchers from Harvard Business School, MIT and Boston Consulting Group ran an experiment with 758 BCG consultants. They gave the consultants realistic tasks, with and without AI. On tasks the model could handle, the work went faster and came out better. But the researchers also set the consultants a task they had designed so that the model would fail at it. There, the group using AI was 19 percentage points less likely to produce a correct answer than the group without.

The researchers call this the jagged frontier: a boundary that runs unevenly. Two tasks that look equally hard can sit on opposite sides of it. The model writes about both with the same confidence, so reading the result won't tell you which side you were on.

That's why a general rule ("AI for drafts, not for conclusions") only half helps. What does help is breaking the work into steps and sorting each step on its own.

Which phase a step belongs to

First you need to know what kind of step it is. That's what SEAL is for.

SEAL describes the four phases knowledge work goes through: Source, Edit, Approve, Launch. In every phase the traceability of a source can be lost, and the weakest phase determines what the work is worth.

Every step in a workflow falls into one of those four. The question for each phase:

• Source: where does the information come from, and does that source really exist?
• Edit: does the source still cover the text after editing?
• Approve: who has reviewed the content, with what expertise, and against what?
• Launch: who releases the work, and with what visibility?

Searching, organising, summarising and rewriting are Source and Edit steps. A model can be good at that work, and you can set the result next to something to check it. Approve and Launch are different. A system can establish that a source supports a claim, but not that the claim still misleads the reader. That judgement stays with a person, which is exactly why those phases exist.

Three questions, three bins

For each step, ask three questions.

1. Can you set the result next to something you already have? A source, the original, a list. If so, you can find an error. If not, you're just reading fluent text.
2. What does it cost if it's wrong and nobody notices? A wrongly sorted list of sources costs a quarter of an hour. A wrong figure in a report under your name costs a client.
3. Does any data go in that the organisation isn't allowed to let out? Personal data, confidential documents, a client's data under non-disclosure.

Three bins follow:

• Yes. The result can be set next to something and an error is cheap. AI does it, and you spot-check.
• Yes, with a check. The result can be set next to something, but an error is expensive. AI produces it, a person sets every claim next to the source and records the tick.
• No. The result can't be set next to anything, data goes in that mustn't leave, or the step is itself a judgement. A person does it.

One rule of thumb: if you're torn between two bins, take the one with more checking. Moving a step to a lighter bin is always possible later.

An example

A made-up example, to see how it plays out. An advisory firm writes a report for a municipality. The process has five steps:

1. Sorting sources (Source). The result can be checked against the sources themselves, and a mistake costs a quarter of an hour. Bin: yes.
2. Writing a first draft (Edit). It's a draft that will be rewritten anyway. Bin: yes.
3. Rewriting figures into running text (Edit). The result can be checked, but a shifted figure goes out under the firm's name. Bin: yes, with a check: every figure next to the source passage, with a tick.
4. Conclusion and advice (Approve). This is the judgement the client pays for. Bin: a person.
5. Sending it to the municipality (Launch). Bin: a person, on the basis of the three steps before.

These bins aren't necessarily right for your work. What matters is the way of sorting: step by step, with a reason for each.

Try the yes steps first

Because the frontier runs unevenly, sorting on paper is a first pass. Then you test. For every step in the yes or yes-with-a-check bin, take five real examples from the past month, have the model do them, and set the result next to the source. If all five hold up, the step stays in its bin. If one doesn't, you move the step down a bin.

That gets you a sorting you've measured yourself, rather than a guess that sounded good.

What you can do this week

1. Pick a workflow that comes round often: a report, a quote, a newsletter, a test.
2. Write down the steps, six to ten. One verb per step.
3. Put the phase next to each step: Source, Edit, Approve or Launch.
4. Answer the three questions for each step and put it in a bin.
5. Test the steps in the yes and yes-with-a-check bins with five real examples, and adjust the bins to what you see.

The result fits on one side of a sheet. That's the basis for a work policy.

Source

Dell'Acqua, F., McFowland III, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F. & Lakhani, K. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper 24-013. papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321