Walking forward the Bumblebee – We have issues

I was hoping it would be a non-event to blog about how successful the Bumblebee trading system performed under the rigors of walking forward. But alas, the little critter has been tripped up a bit. The initial results I’ve gotten are abysmal. As in I wouldn’t even commit Monopoly money to the enterprise. We started this journey with nine markets. Two of these markets (SP 500 and Coffee) showed early signs that they wanted nothing to do with the system. That’s okay. We’ll deal with them later. But of the seven remaining markets, three showed great promise in the optimization cycle of system development. Clearly, I’m disappointed.

Here is a spreadsheet of results for how White Bumblebee (the system’s first version) performed on fresh data. The basic idea with a walk-forward is to optimize on a set of data and use those optimal parameters in a walk-forward window. The window is forward data, or data that was not subjected to the fondling of the optimization process. It’s fresh, and unbiased. Historical, yes, but the system doesn’t know what year it is so we get away with a little slight-of-hand, to be sure.

The column that is labeled ‘# Periods’ is the number of windows that were traded. The relationship between optimization data (fishing for parameters) and the walk forward window (okay, let’s see what you got) is 200 to 50. Two-hundred days are used for optimization, and the system is traded on 50 days before the process begins anew. It doesn’t take too much mental yoga to see that your optimization data is moved forward after the window results have been recorded. That’s the walking part of walk-forward.

I’ve been giving some limited thought to getting a more clever metaphor for what we’re doing here, but so far walk-forward seems to capture the essence of the process best.

Okay, back to our issues with Bumblebee. I’ve made it easy to spot profitable versus non-profitable cells by painting them green or red. You can see that there is a dearth of green cells. The sigma levels show what happens if the system experiences standard deviations on the downside (where we lose money). Ideally, we’d like to get a little sigma cushion, but none of the walk forwards gave us anything.

So where does this leave us? Well, we can throw the system into the trash can. That would be clean and simple. But sometimes, you just need to play with the pain to win. So, we’re not going to give up just yet.

I’m not sure where things went so terribly wrong. I probably need to check my degrees of freedom and see if I’m doing something stupid there. Or maybe my window is too short and the system is getting re-optimized too much. Or maybe my fitness function is no good.

Uh-oh, I’ve introduced a new concept right when I was wrapping up.

What is a fitness function? Well, it is essentially the arbiter of what are the best parameters. Why give one arbiter dibs on making the decision of what parameters we’ll be using going forward? Essentially, someone’s got to do it. Who do we pick as our front man? We could choose the function that finds the best net profit on a historical basis. But Net Profit is sort of … well, dull and monolithic, unlike the markets we’re trying to squeeze money from. The concept of what is the best fitness function, or by what measure can we determine the parameters that have the best chance of producing profits in the future, is a whole separate topic, worth its own blog at some later time.

I used the pessimistic return on margin (PROM) as my fitness function in the walk forward of Bumblebee. It adds to the losers by its standard error and takes away from the winners by its standard error. I don’t think it’s the problem, quite frankly. But I’ll triple check my programming just to be sure it’s not eating my Bumblebee for no good reason.

Advertisements

Tags:

8 Responses to “Walking forward the Bumblebee – We have issues”

  1. Woodshedder Says:

    Milk, have it buy when your short MA is stretched X% beneath the long MA fo N days.

    Use your current buy signal as your sell signal.

    I like making the walk forward symettrical. Have it learn 6 months, then test 6 months. You may have skewed performance by getting such a small sample.

  2. Jez Liberty Says:

    How are you finding Traders Studio?

    I find it quite hard to seriously get into it and quite confusing/counter-intuitive (and I am a programmer by trade!). The documentation is pretty poor I find, and I am lacking good code examples to learn more complicated stuff. Such a shame that Trading Box is 6x more expensive!

    Any tips on how you sped up your learning curve (ie website, etc.) as you seem to be managing pretty well on the coding side of things… I am sure the rest will follow
    On my side I did find a website called tradersedgesystems (.com) with some sample code…

  3. Milk Trader Says:

    I'll may use the stretch percent for N days idea for a future version of Bumblebee, but not ready to give up completely on the current 'White' version. There is an issue with using % when dealing with back-adjusted contracts that I need to understand as well.

    Another area that I need to explore is allowing the non-sense trades into the mix. Those are trades where the slow MA is 'faster' than the fast MA. The fitness function was registering these trades until I programmed a filter to get rid of the buggers.

    As to symmetrical optimize:walk-forward periods, I've always seen the ratio as 2, 3 or 4:1, but never 1:1. I'll have to ponder that concept.

    I ran a 3300 permutation overnight with a 300:100 optimize:walk-forward ratio. I also ran the 200:50 ratio with 480 permutations (as documented above) but this time with a different fitness function. The results are here: http://chart.ly/6w8vp4

  4. Milk Trader Says:

    Jez:

    TradersStudio does have a steep learning curve, but I think the process is worth it in the end.

    I've found some help on the Yahoo TradersStudio board for coding issues. I also subscribed to an online Visual Basic learning site (LearnVisual.net) to learn more about programming, as I am a complete neophyte.

    There are various examples of systems that comes with the software, and I slog through them until I get an 'aha' moment.

    I also follow along with Rich Denning's program examples in TASC.

    What is your complex programming issue? I can post it as a Programming Problem of the Month to the Yahoo folks.

  5. Jez Liberty Says:

    Milk:

    Thanks – I did register a while back on the yahoo groups but have not been there much. I do not have a specific problem at hand. I have gone through the docs and sat down to look around the app; and for anything more than "simple" trading systems, it did not appear straight-forward at all and with a doc not so useful. I was planning on compiling a list of questions on either the TradersStudio website forum or the EliteTrader forums where Murray Ruggiero (founder) seems to be pretty responsive to queries – I guess I still havent gotten my aha moment… 😉 I'll keep at it!

    One thing I remember for example is that there did not seem to be an easy way (or not at all) to output the max drawdown as percent of equity as opposed to absolute values.

    Not sure if you looked at trading blox (they have a free trial) but it looked more intuitive (but 3 grand!).

    Regarding back-adjusted contracts (re: your last comment) you should ensure that they are proportionally adjusted if you want to keep percentage ratios intact, as standard back-adjustment will "screw that up" (sorry about the shameless plug but I just posted about it on my "very new" blog: http://www.automated-trading-system.com/what-everybody-ought-to-know-about-continous-futures-contracts/)

  6. Milk Trader Says:

    I'll be sure to look into your post, as I've just started working with back-adjusted futures (Pinnacle Data) and need to understand limitations.

    Murray does frequent the Yahoo board and has actually posed a programming challenge to the group. May be interesting for you to follow.

    And I've heard good things about Trading Blox, but the price tag a little steep.

    About percent of equity drawdown versus an absolute value, do you mean what percent of a 100K account would have suffered an x percent drawdown historically?

  7. Jez Liberty Says:

    Milk-
    Thanks for the tip: I'll be more attentive to the yahoo group.
    For the drawdown percentage: I want to know what percentage of CURRENT equity the drawdown represents. For example if you know that your account grows from 100k to 200k and your max drawdown is 60k – this might be acceptable if the drawdown is at the end (ie from 260 to 200 = 23%) but not right at the beginning (ie 100 to 40 = 60%). Traders Studio seems to only give you absolute max drawdown. There might be ways around it and I need to play with it more seriously again!

  8. Tradingnewscenter Says:

    Excellent post. Just to let you know that we have featured your post on our site today. ( http://www.tradingnewscenter.com/articles/walking-forward-bumblebee-we-have-issues )

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: