The purpose of optimization is not to find the optimal parameters. Yes, that’s not a typo. The main purpose of optimization is to validate backtest results. You’ll remember it was the backtest’s job to validate a trading idea. Well, now backtesting is on the hot seat and optimization is the judge and jury. Optimization will validate that backtest results are not simply the product of chance, and that the system stands a chance of making money in the future. It will do this by creating a profile sketch of the system. I know, sketches are not the first thing you’d associate with spreadsheets. But in our world, it’s what we have to work with.

We are going to go over the basic setup of the optimization run in just a moment, but let’s take a simple statistic from the optimization run and see if it helps illuminate our objective. The statistic will be ‘profitable parameter sets.’ These are permutations that made money on historical data. If we randomly picked a parameter set, what would be our chances that it would have made money? If 50% of our optimized parameter sets are profitable, that means you can basically throw a dart and your chances of making money is, well, 50/50. Of course we not going to throw darts, but you get the idea.

If only 10% of optimized parameter sets are profitable, then you’d better have the aim of a trained assassin. If 80% are profitable, call your half-blind neighbor over and have him fire away. If you absolutely have to be dead-nuts on in your choice of parameter sets, you may consider whether the enterprise is worth your efforts.

The space we’d like to paint (don’t worry, I’ve got some cool looking graphs below) should be basically square in two dimensions. A scan that is too fine will lead to over-fitting. That’s bad. We cannot avoid curve fitting as that’s basically what we’re doing, but we can avoid getting carried away.

Besides providing us with information about how lucky you need to be to make money with your system, optimization will provide an insight into other features of your system, such as what the winning percent is, what sort of drawdowns you should prepare for and how frequent are trades for winning parameters versus the frequency of trades for losers.

Bumblebee was initially backtested with a slow moving average of 30, a fast moving average of 10 and a Bolllinger band around the slow MA of 0.5 standard deviations. We are going to fiddle with the slow MA period and the fast MA period and leave the Bolllinger band width alone. The reason is simple. Moving average length will have the greater impact on the system than tweaking the standard deviation length, and we don’t have time to calculate too many permutations with the method of choice. That method is called brute force. Yes, you need to be brutish and forceful to do this, but it won’t be too terribly distressing, I promise.

Permutations are statistical combinations. If we ran a test with a slow MA of 30 and fast MA of 10, we’d like to ask what would happen if we used a slow MA of 30 and a fast MA of 11. How about 12, 13. 14 and so on. You get the idea. Let me derive a simple method for calculating this monster in the following statistical proof. Or, on second thought, why not you look it up in your statistic book and take my word for it. If we run a range of 20 periods for the fast and 20 periods for the slow, we will have run 400 permutations. Mathematically savvy readers will see that 20 x 20 is 400, and that is the secret shortcut you forgot you once knew.

In keeping with the square space ideal, we are running the fast MA from 3 to 24 in steps of one (that means, 3, 4, 5, etc.) and the slow MA in steps of 2 from 11 to 49 (11, 13, 15, etc.) This yields 410 permutations. After we take out pairs that don’t make sense (where the slow is faster than the fast), we end up with 371 parameter sets. And with no further ado, below are the results of this optimization sketch. First let’s observe the percent of parameter sets that are profitable. If it’s under 30%, I highlighted it in red to indicate potential problems. If it’s over 80%, I highlighted it in green, to indicate the blind squirrel principle applies (ie, a random dart thrower can make money using the system).

I also put a color code to the left of the market to indicate whether or not the system is useful in a particular market. The obvious losers with the Bumblebee include the S&P 500 and the Coffee markets. The markets that show the most promise include the Crude Oil, Gold and Ten-Year markets. The other markets show potential, but also show some warnings signs, so they’ve been designated as cautionary.

Let’s look at the optimization space for a obvious wanker and a potential best-friend. First the wanker. Below is the optimization space painted for the S&P 500 from 1991-92. Only six percent of potential parameter sets showed a net profit. Yes, that ugly.

You can see that overall, you needed to pick just the right parameter set to make money with the system during this two-year period.

Let’s now look at what a more lovely optimization looks like. Below is the optimization space for the Cotton during 1987-88.

You can clearly see that if you avoided a certain area of tiny slow and fast MAs, you’d at least make money. And if you shot for the area in the upper right corner, you’d make some nice returns.

Here is a sample of some other data gleaned from the optimization profile.

This is the spreadsheets results for the Ten-Year between 1995-96. You can see that 92% of parameter sets were profitable. And, 347 parameter sets yielded trades during the period out of a potential 371 sets. That’s impressive. But looking deeper, you can see that the winning parameter sets had a winning percentage of just 51%. The losing parameter sets had a winning percentage of around 35%.

Also, the winning parameter sets (the good guys on our team) traded an average of 10 trades during the two-year period while the losing parameter sets (the bad guys on the other team), traded much more heavily with an average of 35 trades during the two-year period. These are just a few of the revealing characteristics of the winning parameter sets versus the losing parameter sets that can alert you to when things are going wrong. In other words, when you have given the ball to the other team.

September 9, 2009 at 9:11 am |

Very nice. I like to see many winning values, and often gravitate to median values among the best performers, if I'm not wanting to nudge the system toward more or less trades. Another metric along these lines is drawdown, of course, since faith in a system is so easily lost when a position is on (and for a good reason if overoptimized).

September 9, 2009 at 6:34 pm |

Drawdown is an important metric, and also helps sketch the system's personality. The net profit/drawdown ratio is a good basic trait that, like net profit by itself, helps illuminate how a system behaves and if it makes money.

I'll plan a follow-up to the rough net profit sketch that will include drawdown.

September 10, 2009 at 6:38 pm |

Milktrader,

Can you follow me on Stocktwits? I am intriqued by your Black Box and would like to discuss some things with you sometime privately. I am BBtrader1

Thanks.

BBtrader1