AI Can Reproduce Economics Research—Investment Teams Still Need an Audit Trail
LLMs can compress replication work, but the investment workflow improves only when every data choice, code change, and human judgment remains inspectable.
The investment-research workflow is changing at the point where a paper becomes a testable result. An analyst used to read the paper, locate the data and code, adapt the specification, and decide whether a result deserved another hour. An LLM can now help search the repository, explain unfamiliar code, translate a table into a runnable plan, and suggest robustness checks.
That is a meaningful change—but not evidence that the result is right. A September 2026 NBER working paper by Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro describes an LLM workflow that reproduces, improves, and extends published economics research. The practical lesson for investment teams is narrower: AI can reduce the friction of replication, while the decision record must preserve enough receipts for a human to distinguish a faithful reproduction from a plausible-looking rewrite.
What changed in the investment workflow
The old sequence was linear: read, code, run, interpret. The new sequence is a supervised research loop. The model can map a paper’s claims to data files and code modules, identify missing dependencies, propose an implementation, and generate a list of discrepancies. A researcher then verifies the data vintage, sample window, variable definitions, transformations, and output against the paper.
That changes the scarce resource. It is less often the first draft of code and more often the audit of what the draft actually did. The research object should therefore be a versioned trail: source paper and retrieval date, data and code hashes where available, prompt or instruction version, model output, edits made by the analyst, execution logs, and a table of differences from the published result.
BCG’s October 2026 Global Asset Management Report argues that agentic systems are moving asset managers beyond isolated experiments and into redesigned workflows. That is a useful industry signal, not a causal study of investment performance. It supports a workflow hypothesis: when AI takes on more linked tasks, controls must move from a final memo review to checkpoints throughout the chain.
How an AI-assisted replication desk should work
Start with a frozen research packet. Record the paper URL, version, data-release date, code repository commit, and the exact question being tested. Give the model read access to that packet and a disposable execution environment. Ask it to produce a claim-to-artifact map before asking for code changes.
Next, separate three outputs. “Reproduced” means the stated result was recreated under the stated inputs. “Changed” means a data, code, sample, or specification choice differs. “Proposed” means a new analysis that was not in the paper. Those labels prevent an attractive extension from being mistaken for evidence that the original result survived.
Require a discrepancy ledger. For each mismatch, record the expected value, observed value, likely cause, test performed, and disposition. Human review should cover look-ahead or leakage risk, sample selection, missing-data handling, transformations, benchmark construction, and whether the model silently substituted a convenient proxy. A model can explain code; it cannot certify that an economic interpretation is causal.
The output for an investment committee is not a confident summary. It is a reproducibility packet: the claim, the exact artifact supporting it, the deviations, unresolved questions, and a cautious implication for research—not a security recommendation.
Practice lab
Exercise: In 60–90 minutes, choose one public economics or finance paper with an accessible code repository. Freeze the paper, repository commit, data links, and retrieval time. Use an LLM only for code navigation, documentation, and a clearly logged replication plan.
Baseline and metrics: First spend 20 minutes making a manual claim-to-artifact table. Then run the AI-assisted workflow for the remaining time. Compare the two on correct artifact links, unsupported assumptions, unresolved discrepancies, execution success, and minutes to a reviewable packet. The baseline is the manual table; the observation window is the session plus a next-day check that links and inputs remain available.
Stop condition: Stop if the model invents a file or citation, changes the sample without labeling it, cannot reproduce a stated result after two documented checks, or proposes an investment conclusion from an unverified extension. Keep the exercise research-only: no orders, allocations, or live-account connections.
Reality check
Replication is not validation. A paper may contain an error, the data may be revised, and a reproduced correlation may still have no decision value. A model that selects the easiest papers or repositories creates selection bias; a team that reports only successful runs creates survivorship and reporting bias. Prompt and model changes can make the same workflow non-stationary. Human reviewers may also rubber-stamp fluent explanations, creating measurement bias in the very audit designed to control it.
The NBER paper is a working paper, not proof that LLM-assisted research generates alpha. BCG’s report is an industry perspective, not an independent deployment experiment. Incentives matter: vendors benefit from emphasizing throughput, while research teams may benefit from presenting a clean replication queue. Those gaps are reasons to measure provenance and exceptions, not reasons to dismiss the capability.
For discretionary investors, inspect the discrepancy ledger before the narrative. Systematic teams should version prompts, models, data, and code together. Builders should make “changed” and “proposed” first-class states. The useful frontier is not autonomous investment judgment; it is faster, more inspectable research whose authority remains bounded by evidence.
Links / sources
- NBER Working Paper 35782: An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research
- BCG Global Asset Management Report 2026: Rebuilding Asset Management for an AI-First World
- AI Investment Workflows Need an Exception-Rate Test
- AI Can Reproduce Investment Research—If the Workflow Keeps Its Receipts