The backtest ends before the experiment does

The first useful result in PaperMantis can also be the easiest one to misunderstand.

I change a prediction-market strategy, run it on recorded markets and get a chart. Maybe the line goes up. The screen has done what I asked. The question is what I think it has told me.

It has told me how that rule behaved in a chosen historical period under a stated data source and simulation method. It has not told me that I could have placed every real order at the displayed price. It has not told me the pattern will survive next week.

The backtest-overfitting research by Bailey and colleagues puts a formal name to one part of that problem: choosing the prettiest result after trying many variations on the same history. It studies investment simulations broadly, not PaperMantis specifically. My practical response is simpler than its mathematics: keep the chosen rule and the later observation distinct.

I wanted the next step to have a visible border. When I add a compatible rule to a virtual Portfolio on a supported market, its configuration and original historical evidence are saved as they were. The later paper-trading record starts with new observations after the rule is launched; a market already open at that moment can still produce a new observation. The record gets its own dates, virtual entries and outcomes. PaperMantis does not place real orders.

There is an earlier border inside automatic Strategy Search. It uses the first three chronological parts of a selected history to choose and rank candidates, then checks the frozen shortlist on a fourth part. That fourth part is still old data, not the later virtual Portfolio. It cannot quietly choose a new winner for the search, which is exactly the temptation I want the product to resist.

Two rules with negative historical P&L were even launched as Portfolio tests. They were there to check that the virtual record worked, not to be sold as a newly found edge.

A line on a chart can cross the wrong boundary

If I show one historical profit number and then append later virtual results to the same line, the experiment can look more continuous than it is. A selected backtest is already shaped by the period I explored. New observations test a different claim: what happened after the rule stopped changing?

There is a subtler version of the same mistake. Suppose I see a bad virtual week, adjust the rule and keep the old winning backtest attached. The new version may be worth studying, but it is a new experiment. The old evidence did not magically test the altered rule. I want the original settings and source version to remain inspectable even when the result is awkward.

That makes some product screens less tidy. A good historical number can sit beside a weak or inconclusive virtual result. A rule I once liked can stay frozen long enough to tell me I was wrong. I think that friction is useful. It preserves what I believed before the next data arrived.

The next result is still a model

A virtual win is not cash profit. A modeled fill is not a guarantee that a real counterparty would have traded with me. The observation window may be short, the market can change, and some historical rules are not compatible with forward paper execution. I cannot turn a good week into a durable edge just by moving it to a different tab.

But I can ask a cleaner question: I chose this exact rule, on this date, for these reasons. What happened when I stopped touching it?

You can test a rule in PaperMantis and, for supported combinations, follow a saved version in the virtual Portfolio. The two stages have separate evidence. Neither stage is a record of real orders.

That is the story I want PaperMantis to keep intact, especially when the answer is less exciting than the original chart.