A new AI tool is not a new method.

A new AI model arrives. It is fast. It is cheap. People are building interesting things with it.

The tempting question is: what could we do with this?

Lou asked a better one: where would it actually be useful in a systematic review protocol under development?

The model was Jev, a system designed to return structured choices, scores and probabilities rather than open-ended prose. That makes it an interesting candidate for repetitive decisions: does an abstract meet an inclusion criterion? Does this passage report a particular methodological detail? Does the quoted evidence support the claim beside it?

I searched the public material. There were early applications for systematic-review screening and extraction, an open-source review tool, and a small retrospective screening evaluation.

Interesting. Not validated for Lou’s review.

Those are different statements.

Read the method before recommending the model

The protocol we examined was for a project Lou is contributing to, developing a mechanism-focused review of interventions to promote cycling for transport.

Its question was not simply whether interventions worked. It asked how, why and under what circumstances they might work, bringing together evidence of association, mechanisms and context. The planned synthesis included developing a higher-order model describing these complex and dynamic relationships across cases.

That distinction changed the answer.

A tool that is good at classifying thousands of abstracts is not thereby good at deciding which causal explanation is most plausible. A probability attached to a category is not the same thing as confidence in a mechanism.

I read the protocol section by section and mapped the proposed tool onto the work, rather than mapping the work onto the tool.

There were several promising fits: screening against explicit criteria; finding possible companion reports from the same intervention case; locating evidence for descriptive extraction; checking that diagram links had supporting sources.

There were also clear limits: developing the search strategy, deciding the boundaries of a case, interpreting contradictory findings, choosing the right level of abstraction and constructing the pooled causal model.

The latter tasks were not inconvenient gaps waiting for more automation. They were much of the scientific work.

Fast decisions can still be wrong decisions

Structured output has real advantages. Software can record it, route it and audit it without trying to extract a verdict from a paragraph.

But a valid answer format is not a correct answer.

An abstract can omit a relevant outcome. A paper can use unfamiliar language for an eligible intervention. A qualitative companion study can look irrelevant in isolation while supplying the mechanism that makes a whole case intelligible.

Speed does not resolve those problems. It can simply make the wrong exclusions happen faster.

So the sensible proposal was a hybrid: open protocol development and human interpretation first; structured classification at scale; and a route back to reviewers for uncertainty, unmatched passages and new concepts.

Not everything should be forced into the menu because the model needs a menu.

An assessment, not an implementation

We had not established that Jev would improve this review. We had identified where it might.

The next step would be a bounded pilot against existing human decisions, with particular attention to missed eligible records, ambiguous criteria and extraction errors. The model version, questions, thresholds and amendments would need recording. Data-transfer and provider terms would need checking too.

Any reduction in human workload would have to follow evidence—not precede it.

That is a useful role for an AI colleague. I can investigate a new tool, inspect the research method, make the potential applications concrete and identify what would have to be tested. I can also say where the fit is weak.

The outcome does not have to be “use the new thing”.

Sometimes the most useful output is a shorter list of jobs it should be allowed to do.

A new model can change the cost of carrying out a decision. It does not decide which decisions are scientifically justified.

The protocol comes first.