Junming Zhao

Working papers

The time machine problem: how much AI forecasters remember

Junming Zhao · Duke University · Version 1, October 2026

AbstractEvery AI investing tool is tested by asking a model about the past, and the model may simply remember the answer. I build a point-in-time database of 4.9 million SEC numbers for 315 companies, each stamped with when it became public, and audit it against six accounting identities: 7.6% of numbers are later re-reported differently, and quarters fail to add up to the year 6.6% of the time. A memory test on eight models with known training cutoffs finds that small date-limited models do not memorize reported earnings at all, so the question has to be put to large commercial models. That is the next experiment.

Keywordslook-ahead bias; point-in-time data; XBRL; language models; backtesting

JEL G14 · G17 · C55 · M41

One-page brief (PDF)Code and full results on request

The question

If a model read the answer during training, a backtest flatters it and live results disappoint. Measuring that takes data that knows exactly what was public and when, and models whose training cutoffs are known.This builds on Glasserman and Lin (2023), Sarkar and Vafa (2024) and Kim, Muhn and Nikolaev (2024), and on model families trained only on text up to fixed dates.

A database that only knows what was known

The warehouse holds every SEC filing and reported XBRL number for 315 US companies, sampled from 2019 filings so that firms which later delisted stay in. A number counts as public from the moment its filing was accepted, with the 4:00 pm rule; quarterly earnings count from the 8-K press release, found for 83% of 6,868 company-quarters. Fourth quarters are never filed on their own, so they are derived as the full year minus nine months, and match Apple's and Microsoft's press releases exactly.Tickers move between companies: XOM now points to a newer ExxonMobil holding company whose filings start in 2026, so histories are pinned to the right filer.

How clean is the SEC's data?

Share of checks that fail, from 6.6 percent for quarters adding up to the year down to 2.7 percent for assets equaling liabilities plus equity.
How often filings break basic accounting identities, checked filing by filing.

Mostly consistent, not clean: 2.7% of balance sheets don't balance and 4.4% of cash-flow statements don't add up. Separately, 7.6% of numbers are later re-reported with a different value, so a backtest on today's data uses information nobody had at the time.EPS looked like 26.6% failures → it was cent rounding. 5.2% now

A memory test for models with known cutoffs

For each company-quarter the test takes the true reported EPS and nine plausible alternatives, 5% to 20% off, and asks whether a model gives the true value more probability. The rank score is the share of alternatives the true value beats, so 0.5 is chance. If a model had memorized earnings, it would score higher on quarters reported before its training cutoff than after.

For eight models, the difference in rank score before and after the training cutoff, with 95 percent intervals that all include zero; mean scores between 0.505 and 0.511.
Rank score on quarters reported before each model's training cutoff minus after, with 95% intervals clustered by company; mean score on the right. 6,868 company-quarters per model.

All eight models sit at chance, and no interval excludes zero, while a control shows the test does catch well-known facts such as 1776. Small date-limited models don't memorize reported earnings.

Can you recognize a company from its numbers?

Before trusting a "masked" backtest, it's worth checking whether the mask hides anything.

Guess the company. Five years of real annual numbers as first reported to the SEC, with the name and years hidden and revenue shown as an index: the re-identification test, played by a person.

The next experiment

A commercial model will forecast earnings twice for each company-quarter, once with the company named and once masked. After the model's cutoff, any gap between the two is distraction, the effect of seeing a name; before the cutoff, any extra gap is memory. The estimator is already tested on simulated data with a planted answer.

Limits

Firms listed after 2019 are not in the sample. 952 earnings-per-share checks still fail after the rounding allowance, with a median gap of about 100% of net income, which suggests a units problem rather than bad filings.

References

  1. Glasserman, P. and C. Lin (2023). Assessing look-ahead bias in stock return predictions generated by GPT sentiment analysis.
  2. Sarkar, S. K. and K. Vafa (2024). Lookahead bias in pretrained language models.
  3. Kim, A., M. Muhn and V. Nikolaev (2024). Financial statement analysis with large language models.
  4. Benhenda, M. (2026). Look-Ahead-Bench.
  5. Drinkall, F. and others (2024). Time Machine GPT; He, Lv, Manela and Wu (2025). ChronoGPT.
  6. Data: SEC EDGAR submissions and XBRL company-facts APIs.

Project log

  • October 2026
    The time machine problem

    Made the earnings-per-share check allow for cent rounding: failures fell from 26.6% to 5.2%. The rest look like a units problem. More.

  • October 2026
    The time machine problem

    First results: 4.9 million point-in-time SEC numbers for 315 companies, an identity audit, and a memory test on eight dated models.

Next

  • The time machine problem. Ask a commercial model to forecast earnings with the company named and masked, before and after its training cutoff.
  • The time machine problem. Trace the 952 earnings-per-share failures that survive the rounding allowance back to the filings.

← The $150,000 cliff