Forward Testing a Trading Bot: What It Shows That No Backtest Can

After the backtest comes the moment a strategy first sees data that didn't exist when it was designed. Forward testing a trading bot - on a demo account or in simulation mode with live data - is the first check without hindsight. Its value lies less in the result than in the deviations it makes visible.

Forward testing is not walk-forward analysis

The two terms get mixed up a lot. Walk-forward analysis works exclusively with historical data: parameters are determined on one time segment and checked on the next, window by window. A forward test runs in real time, on data that didn't yet exist when the strategy was developed. Walk-forward checks the stability of the parameters, the forward test checks the implementation under real conditions. Neither replaces the other.

For how walk-forward analysis and out-of-sample testing help against overfitting, see the article on overfitting in trading strategies.

What a forward test shows that no backtest can

• Execution: whether orders fill at the expected price rather than at the bar's signal price
• Real-time signal behaviour: whether signals form live the way they do in history, or whether the indicator looks different in hindsight
• Time logic: whether trading windows, session anchors and flattening kick in at the right time
• Operations: dropped connections, restarts and missing data - everything that simply doesn't happen in a simulation over history
• Data feed: whether the live data delivers the same values as the historical data the strategy was tested on

The most revealing step: compare trade by trade

At the end of the forward test, the exact same period is recalculated in the backtest with identical parameters. Then both trade lists are laid side by side. Comparing totals isn't enough - two similar end results can come from completely different trades. Only the one-to-one comparison shows where the two worlds diverge, and almost every deviation traces back to one of a few causes:

• Trade only in the backtest: the signal exists only in history - typically evaluation of the open bar or values altered after the fact
• Trade only live: the signal fired mid-bar but wasn't confirmed on the closed bar
• Same trade, different time: time zone, daylight saving, or a diverging session definition
• Same trade, different price: execution, slippage, or the chosen order type
• Same entry, different exit: stop and target within the same bar, which the backtest resolves from bar data differently than the live market did

How long a forward test should run

The answer depends on what's being checked. For the technical check - do signals, times and execution match the backtest? - a few weeks are often enough, because it's about agreement, not statistics. Saying anything about the result, on the other hand, takes considerably more trades and ideally different market phases. The yardstick is the number of trades, not calendar time: a strategy with only a few signals a month needs correspondingly longer.

The limits of a demo account

• Simulated fills usually model queue position and market depth only in simplified form, which tends to make them friendlier than real ones
• Changing parameters during the test makes the comparison worthless, because the backtest of the period then runs with different values than live operation did
• A passed forward test shows that the implementation does what it should - it says nothing about future results

For why time deviations in particular are so easy to miss, see the article on trading hours and time zones in a bot.

A forward test is less a check of whether the strategy makes money than of whether the bot does what the backtest claims. That question can be answered within a few weeks - and it should be answered before anyone thinks about real capital.