Can AI Reproduce a Literature Review?
A systematic-review benchmark is only as useful as its reference set, criteria, and error checks. Here is a bounded way to evaluate AI screening and extraction without mistaking vendor results for your own.
Can an AI workflow reproduce a literature review, or does it merely make one stage faster? For a research team facing hundreds of abstracts and repetitive extraction, the useful question is narrower: on a known, bounded set, which decisions can the tool recover, where does it fail, and what human checking remains necessary? Elicit's published evaluation offers a method worth examining—but its reported numbers are not a result from your corpus, and they do not establish that an entire review can be reproduced reliably.
The signal
- Practitioner / team: Elicit, evaluating its Systematic Review workflow.
- Source: How we evaluated Elicit Systematic Review, published March 18, 2025.
- Why this matters: A tool that screens quickly can still introduce bias if relevant studies are silently excluded or if extraction fields do not match the research question.
The method in one sentence
Compare a tool's screening and extraction outputs with a pre-existing review's decisions and manually checked fields, while documenting how the reference set was assembled and what it leaves out.
What the source actually shows
Workflow
- Elicit assembled 58 published systematic reviews and derived a research question from each to query its system.
- For recall, the team used an LLM to identify papers included in the source reviews; it verified included papers but says it did not comprehensively extract every included study.
- It generated a pool of apparently irrelevant papers by querying Consensus, then excluded some candidates that an LLM judged too relevant.
- It compared screening decisions against those constructed labels. Separately, it compared extraction answers with gold-standard answers and reported internal and external evaluations.
- The post reports screening 93.6% of relevant papers in and 62.8% of irrelevant papers out. It says recall could be about 96.4% when screening criteria were correctly specified. For extraction, it reports 94% in an internal LLM-based comparison, 99.4% in a VDI/VDE case study, and describes a CSIRO fisheries study.
Inputs, tools, and outputs
| Element | Evidence from the source | What we can test |
|---|---|---|
| Input | Review questions and included-paper records from 58 published reviews | A small, fixed subset with accessible reference decisions |
| Tool / step | Elicit screening and extraction; LLM-assisted label construction; Consensus-generated negatives | Our chosen system and a separately documented rubric |
| Output | Include/exclude decisions and answers to extraction columns | A row-level table with decision, evidence, and confidence |
| Quality check | Comparison with review decisions and vetted extraction answers | Blind double-checking of disagreements and ambiguous cases |
These are company-reported results from a company-authored account. The post itself identifies important design choices: included records were not comprehensively extracted, irrelevant papers came from a separate search engine, and LLM judgments helped create or filter labels. Those choices do not invalidate the exercise, but they shape what its metrics mean. The 93.6% is recall on that constructed set, not a universal accuracy rate. The 96.4% figure is conditional on correctly specified criteria, not an unconditional observed result. Likewise, extraction accuracy depends on which fields, papers, and adjudication rules were used.
Reconstruction plan
- Task: Re-run one previously completed review question on a modest set of records whose original screening and extracted fields are available.
- Baseline: The published review's final included/excluded decisions and a manually checked sample of extraction fields.
- Smallest useful experiment: Before using the model, freeze the question, inclusion criteria, extraction columns, and adjudication rules. Run the same records through the chosen AI workflow. Hide reference labels during the run. Compare screening recall and precision separately; for extraction, score each field against a cited passage, not merely a matching answer string.
- Artifact to save: A dated protocol, input record IDs, criteria version, raw outputs, source passages, adjudication notes, and a discrepancy table. Keep versions so a changed criterion cannot silently change the score.
- Success condition: The system meets a predeclared threshold for the team's critical fields and misses no high-consequence included study in the checked subset; a reviewer can trace each decision to evidence.
- Stop condition: Stop if reference decisions are unavailable, the criteria cannot be applied consistently, or ambiguous disagreements remain unresolved. Do not convert missing labels into negatives or claim general performance from a small convenience sample.
A concrete example: suppose the question concerns an intervention in adults, and the rubric excludes pediatric-only studies. If the AI excludes an abstract that does not report age, the decision is not automatically correct simply because the final review excluded it. The saved table should show the abstract's evidence, the criterion invoked, the model decision, and a human adjudication. If the abstract lacks enough information, mark it uncertain and send it to full-text review. This preserves the distinction between a screening aid and an authoritative eligibility decision.
Reality check
- Observed in the source: Elicit reports evaluations using published reviews and gold-standard extraction answers, with the methods and caveats described above.
- Reconstructed here: No local experiment has been run. The protocol is a proposed test, not a replication or result.
- Reported by the practitioner: The accuracy figures and time-saving claims are Elicit's reports; the article also cites external evaluations.
- Still unknown: How performance transfers to another discipline, a different query, less structured papers, or a team's operationally important fields; how much time adjudication adds; and whether the reference labels are sufficiently complete.
- Cost / latency / maintenance constraint: A meaningful check requires obtaining full texts, maintaining stable criteria, and spending expert time on disagreements. A low per-record tool cost does not remove the cost of validation.
The trade-off is straightforward: a fixed benchmark makes comparisons possible, but familiarity with the benchmark or incomplete labels can make a score look more decisive than it is. Recall and precision should be reported separately because missing a relevant study and reviewing an extra irrelevant study have different consequences. Extraction should also be assessed by field: a correct participant count does not imply a correct intervention description.
Transfer rule
Try this method when you have a bounded review question, accessible reference decisions, and a reviewer who can adjudicate disagreements. Do not use a single headline percentage to approve an unattended screening pipeline, especially where omissions could change clinical, policy, or safety conclusions. Expand the test only after the small subset is auditable. The recommendation should change if a larger, independently labeled sample shows stable performance on the fields and subgroups that matter to your review.
Practice card
- Inputs: One review question, a frozen inclusion rubric, a labeled record subset, and extraction fields.
- Steps: Freeze the protocol; run the tool blind to labels; preserve raw results and evidence; compare recall, precision, and per-field extraction; adjudicate every consequential disagreement.
- Expected output: A discrepancy table and a decision about which steps can be assisted, not a claim that the review has been reproduced.
- Quality check: A second reviewer can reproduce the scoring from saved inputs and citations; uncertain cases remain uncertain.
- Next iteration: Add a new, independently labeled subset before expanding to another topic or relaxing human review.
Links / sources
- Chinese companion: AI 能复现系统综述吗?
- Related series note: Count the Uncertain Decisions, Not the Agents
- First-party source: How we evaluated Elicit Systematic Review
- Related series method: Subagents Are Compression