Posts Tagged ‘Walk Forward’

Goodbye Bumblebee, Hello Ladybug

September 25, 2009

I had grand schemes and lofty goals for the Bumblebee system. But barring an egregious mistake in my walk-forward of the system, I have to let the critter go. The White Bumblebee (first version) suffered massively in its synthetic real-world experience, like a patient etherised on the table only to suffer reality’s cruel but necessary knife. I had other versions queued up and ready to improve upon the simplest version, but they did no better in their walk-forwards. Not to worry, there are many more insects in the hopper. Next one sprawling on a pin is the Ladybug. It is based on Keltner band breakouts. Keltner bands are like Bollinger bands, but they’re based on ATR instead of standard deviation. The basic idea is that once the band is breached, take a trade in the direction of the excursion and enjoy great inflows of capital into your account. (more…)

Walking forward the Bumblebee – We have issues

September 23, 2009

I was hoping it would be a non-event to blog about how successful the Bumblebee trading system performed under the rigors of walking forward. But alas, the little critter has been tripped up a bit. The initial results I’ve gotten are abysmal. As in I wouldn’t even commit Monopoly money to the enterprise. We started this journey with nine markets. Two of these markets (SP 500 and Coffee) showed early signs that they wanted nothing to do with the system. That’s okay. We’ll deal with them later. But of the seven remaining markets, three showed great promise in the optimization cycle of system development. Clearly, I’m disappointed.

Here is a spreadsheet of results for how White Bumblebee (the system’s first version) performed on fresh data. The basic idea with a walk-forward is to optimize on a set of data and use those optimal parameters in a walk-forward window. The window is forward data, or data that was not subjected to the fondling of the optimization process. It’s fresh, and unbiased. Historical, yes, but the system doesn’t know what year it is so we get away with a little slight-of-hand, to be sure.

The column that is labeled ‘# Periods’ is the number of windows that were traded. The relationship between optimization data (fishing for parameters) and the walk forward window (okay, let’s see what you got) is 200 to 50. Two-hundred days are used for optimization, and the system is traded on 50 days before the process begins anew. It doesn’t take too much mental yoga to see that your optimization data is moved forward after the window results have been recorded. That’s the walking part of walk-forward.

I’ve been giving some limited thought to getting a more clever metaphor for what we’re doing here, but so far walk-forward seems to capture the essence of the process best.

Okay, back to our issues with Bumblebee. I’ve made it easy to spot profitable versus non-profitable cells by painting them green or red. You can see that there is a dearth of green cells. The sigma levels show what happens if the system experiences standard deviations on the downside (where we lose money). Ideally, we’d like to get a little sigma cushion, but none of the walk forwards gave us anything.

So where does this leave us? Well, we can throw the system into the trash can. That would be clean and simple. But sometimes, you just need to play with the pain to win. So, we’re not going to give up just yet.

I’m not sure where things went so terribly wrong. I probably need to check my degrees of freedom and see if I’m doing something stupid there. Or maybe my window is too short and the system is getting re-optimized too much. Or maybe my fitness function is no good.

Uh-oh, I’ve introduced a new concept right when I was wrapping up.

What is a fitness function? Well, it is essentially the arbiter of what are the best parameters. Why give one arbiter dibs on making the decision of what parameters we’ll be using going forward? Essentially, someone’s got to do it. Who do we pick as our front man? We could choose the function that finds the best net profit on a historical basis. But Net Profit is sort of … well, dull and monolithic, unlike the markets we’re trying to squeeze money from. The concept of what is the best fitness function, or by what measure can we determine the parameters that have the best chance of producing profits in the future, is a whole separate topic, worth its own blog at some later time.

I used the pessimistic return on margin (PROM) as my fitness function in the walk forward of Bumblebee. It adds to the losers by its standard error and takes away from the winners by its standard error. I don’t think it’s the problem, quite frankly. But I’ll triple check my programming just to be sure it’s not eating my Bumblebee for no good reason.