Published 2026-10-14 · Updated 2026-10-25

Test a Trading Rule Without Fooling Yourself

A backtest is the standard way to evaluate a rule — and the standard way to fool yourself. Three checks that separate a result from a screenshot: look-ahead protection, costs, and survivorship.

On this page

What this is about

The first guide closed the loop: an agent pulls a number, code computes, the model explains. This one is about the step after that — you have a rule, now how do you know if it works?

Most backtest problems are not statistical. They are three specific mistakes, and all three are checkable in minutes.

A backtest that fails any of these three is not a result. It is a screenshot.

Check 1 — Look-ahead protection

Look-ahead bias is when the rule uses information that would not have been available at the time of the trade. It is the single most common reason a backtest looks excellent and then fails.

AI makes this worse, not better. A model trained on historical data has, in effect, already seen the future.

The common forms:

Form Looks like
Training-data leakage Asking an LLM to “analyse” a period it was trained on
Restated fundamentals Using a figure as revised later, not as first published
Index membership Testing on today’s constituents, which already exclude the failures
Same-bar execution Deciding on a close and executing at that same close

The check: for every input your rule uses, ask at the moment of this decision, was this number actually available? If you cannot answer with a source, the result is not a result.

Check 2 — Costs

A rule that trades often looks free when it is not.

The failure mode is almost always the same: a backtest assumes thin costs — say 0.1% slippage — and the live market gives you 3% in the moments you need liquidity most.

The check is to state the assumption out loud:

# Every result must declare its cost model.
COST = {
    "commission": 0.0003,   # per side
    "slippage": 0.0010,     # assumed, per side
    "note": "slippage is an assumption — state it, do not hide it",
}

If costs are not in the backtest, the backtest is not measuring the rule. It is measuring the rule as if trading were free.

One more thing the model above hides: in A-shares, stamp duty is charged on the sell side only. A single symmetric commission number is therefore already wrong — every exit costs more than every entry.

Check 3 — Survivorship

Survivorship bias is the distortion that appears when a dataset only contains the things that survived.

In markets: delisted companies vanish from history. A backtest over “all stocks” that quietly excludes the ones that went to zero is testing a market that never existed.

The check: ask whether the dataset includes delisted instruments. If the answer is no, or if nobody has asked, the result carries an unquantified upward bias — and the failures are exactly the point.

Putting it together

A rule is worth taking seriously when you can say, about all three:

Question Passes when
Look-ahead Every input was available at decision time, and you can name the source
Costs The cost model is stated, including slippage as an assumption
Survivorship The universe includes the instruments that failed

Failing all three does not mean the rule is wrong. It means you do not yet know whether it is right — and those are very different states.

Where the data comes from

All three checks require data you can re-query with the parameters stated: the window, the universe, the adjustment convention.

If you cannot re-run the data pull and get the same series, you cannot run any of these checks. That is the practical reason to use a source with a stable, documented shape rather than a scraped page.

# Reproducible pull: same request, same series, any time.
curl "https://api.ashareapi.com/v1/kline?code=sh600519&period=day&count=250"

Next

Sources