Archive for September, 2009

Goodbye Bumblebee, Hello Ladybug

September 25, 2009

I had grand schemes and lofty goals for the Bumblebee system. But barring an egregious mistake in my walk-forward of the system, I have to let the critter go. The White Bumblebee (first version) suffered massively in its synthetic real-world experience, like a patient etherised on the table only to suffer reality’s cruel but necessary knife. I had other versions queued up and ready to improve upon the simplest version, but they did no better in their walk-forwards. Not to worry, there are many more insects in the hopper. Next one sprawling on a pin is the Ladybug. It is based on Keltner band breakouts. Keltner bands are like Bollinger bands, but they’re based on ATR instead of standard deviation. The basic idea is that once the band is breached, take a trade in the direction of the excursion and enjoy great inflows of capital into your account. (more…)

Trade your Equity Curve

September 24, 2009

System traders understand that their approach to trading requires nerves-of-steel patience while in a drawdown, gut-wrenching discipline in taking every signal generated by the system (and I mean every signal – no ifs, ands, buts or maybes) and cat-like reflexes in executing signaled trades before doubt can creep into the kernel regressing, quadratic equation. Whoever said that system trading takes the emotions out of trading doesn’t know the half of it. Doubters and skeptics of system trading recite the ‘past performance doesn’t guarantee blah, blah, blah’ mantra, refer to the perils of overfitting and data mining, and express a general independence from the enslaving nature of system trading. Well, you’ll be happy to know that you can be a system trader and not take every trade. How? Trade your equity curve. (more…)

Walking forward the Bumblebee – We have issues

September 23, 2009

I was hoping it would be a non-event to blog about how successful the Bumblebee trading system performed under the rigors of walking forward. But alas, the little critter has been tripped up a bit. The initial results I’ve gotten are abysmal. As in I wouldn’t even commit Monopoly money to the enterprise. We started this journey with nine markets. Two of these markets (SP 500 and Coffee) showed early signs that they wanted nothing to do with the system. That’s okay. We’ll deal with them later. But of the seven remaining markets, three showed great promise in the optimization cycle of system development. Clearly, I’m disappointed.

Here is a spreadsheet of results for how White Bumblebee (the system’s first version) performed on fresh data. The basic idea with a walk-forward is to optimize on a set of data and use those optimal parameters in a walk-forward window. The window is forward data, or data that was not subjected to the fondling of the optimization process. It’s fresh, and unbiased. Historical, yes, but the system doesn’t know what year it is so we get away with a little slight-of-hand, to be sure.

The column that is labeled ‘# Periods’ is the number of windows that were traded. The relationship between optimization data (fishing for parameters) and the walk forward window (okay, let’s see what you got) is 200 to 50. Two-hundred days are used for optimization, and the system is traded on 50 days before the process begins anew. It doesn’t take too much mental yoga to see that your optimization data is moved forward after the window results have been recorded. That’s the walking part of walk-forward.

I’ve been giving some limited thought to getting a more clever metaphor for what we’re doing here, but so far walk-forward seems to capture the essence of the process best.

Okay, back to our issues with Bumblebee. I’ve made it easy to spot profitable versus non-profitable cells by painting them green or red. You can see that there is a dearth of green cells. The sigma levels show what happens if the system experiences standard deviations on the downside (where we lose money). Ideally, we’d like to get a little sigma cushion, but none of the walk forwards gave us anything.

So where does this leave us? Well, we can throw the system into the trash can. That would be clean and simple. But sometimes, you just need to play with the pain to win. So, we’re not going to give up just yet.

I’m not sure where things went so terribly wrong. I probably need to check my degrees of freedom and see if I’m doing something stupid there. Or maybe my window is too short and the system is getting re-optimized too much. Or maybe my fitness function is no good.

Uh-oh, I’ve introduced a new concept right when I was wrapping up.

What is a fitness function? Well, it is essentially the arbiter of what are the best parameters. Why give one arbiter dibs on making the decision of what parameters we’ll be using going forward? Essentially, someone’s got to do it. Who do we pick as our front man? We could choose the function that finds the best net profit on a historical basis. But Net Profit is sort of … well, dull and monolithic, unlike the markets we’re trying to squeeze money from. The concept of what is the best fitness function, or by what measure can we determine the parameters that have the best chance of producing profits in the future, is a whole separate topic, worth its own blog at some later time.

I used the pessimistic return on margin (PROM) as my fitness function in the walk forward of Bumblebee. It adds to the losers by its standard error and takes away from the winners by its standard error. I don’t think it’s the problem, quite frankly. But I’ll triple check my programming just to be sure it’s not eating my Bumblebee for no good reason.

Vladimir Visotsky redux

September 18, 2009

This is how the market bears feel about the market. Timeless.

Nothing is as it seems.

Upcoming Classes, Objects, Properties and Methods

September 15, 2009

Maybe I’m doing a little too much programming lately, but I’ve decided to lay out my plans for system development into classes and objects. This is more than just creating a super-duper trading system that uses data mining as its justification for massive, unheard-of returns. And it’s more than just creating a ‘system.’ It’s about a style of trading that uses systems, but more in the manner of a symphony and less in the manner of a 3rd grade recorder. I’ll be organizing systems under different classes. An instance of a class of systems will be an object, which itself contains properties that makes it unique from other objects within the class. Getting a little programmy for you? Not to worry. I’ve laid out some of what’s in store below.

So far I’ve shared the development of the Bumblebee trading system. It is a rules-based system, meaning it trades off signals derived from a single market that are clearly defined. The long entry for Bumblebee is when the fast moving average closes above the upper Bollinger band of the slow moving average. That’s an example of a rule-based trigger. Now let’s start organizing.

Bumblebee is an object of the class of systems known as rules-based systems. More specifically, it belongs to the Insect class of trading systems.

Insect Trading Systems (class)

These systems use price action and technical indicators to trigger entries and exits. The source of the price action and the inputs to technical indicators is the market that is traded.

Bird Trading Systems (class)

These systems use intermarket data to trigger entries and exits. Slightly more complicated than the Insect class, Bird class systems include data about what a separate market is doing to trigger trades.

Safari Animal Trading Systems (class)

These systems use neural networks with either basic back-propagation or radial net algorithms, or kernel regression modeling. The inputs into the network can include a wide-range of data. Because of greater jeopardy to overfitting, these systems deploy neural network outputs as enhancements to already viable trading strategies. Too many neural networks have blown up trading accounts, and we’re not going to let that happen here.

Dinosaur Trading Systems (class)

Reserved for more advanced strategies that may use multiple neural networks, for example.

So from that simple outline, one can easily see that the Bumblebee system uses rules to trigger trades, and that its triggering data is all derived from the market to be traded. You likely don’t know about my Pelican trading system, but you can tell from its name that it is an intermarket system that uses data from a separate market to trigger trades.

Now I’d like to designate some properties. We will be using a simple color system that will sound familiar to parents whose children participate in martial arts.

The color property of a system designates the version of the system. The first color is White. White Bumblebee is the first version of the Bumblebee trading system, and the one for which the code, backtesting, optimization profile, etc are publicly available here. Later versions will follow the following order:

Yellow: second version
Purple: third version
Orange: fourth version
Green: fifth version
Blue: sixth version
Brown: seventh version
Red: eighth version
Black: ninth version

There, that’s simple, no?

But we’re just coming to the place where it all comes together. Systems are not going to be traded as stand-alone toys. Instead, they will be grouped together and traded together. They will all play in the same sandbox. And that leads us to a new class known as box. The box is a trading system that incorporates more than one trading system. So the White Box includes the following systems, for example:

White Bumblebee
White Firefly
White Finch
White Zebra

Our White Box trading system incorporates two simple rules-based systems, one inter-market system and one neural network system. The Box’s ‘White’ property signifies that the Box includes only White trading systems and only trades single contracts.

More advanced boxes will follow the same color scheme mentioned above and will get progressively more complex to include more advanced money management schemes, and neural network-based constrictors and expanders, meaning it will have the ability to adapt capital allotments to different systems and markets.

You may be wondering what a particular instance of the White Box system would be. We have designated the Box as a class and White as a property, but there is no Object. Not to worry, each instance of the White Box will be given a name that will differentiate it from other White Boxes. A specific instance of the White Box could include any number of white systems and trade any number of markets, so each time a White Box is created, it will be given a hurricane name.

There, that completes the overview.

This should be fun.

Black Box Chronicles: a Conversation with Optimization

September 14, 2009

I’m having the most pleasant conversation with Optimization over a couple of espressos and biscotti when this hideous creature walks into the room. The she-devil says she has to talk with Optimization in private, so I muddle over to the counter to get two more espressos. But I’m listening carefully. There are threats being volleyed back and forth and some talk about children. The creature leaves and I saunter back to the table with fresh espressos in hand. Curious about the creature, I ask my new friend what that exchange was all about.

‘Who was that?’ I ask.

‘She likes to call herself Optimal, but I call her Overfit.’

Sounds like the perfect description of an ex. I come to find out that Overfit is a mistress of my new friend Optimization. And apparently, they have lots of children together. Optimization is caught in this relationship, and he can’t get out of it. It’s just one of those things.

‘Now, back to my main point,’ Optimization continues.

He gets back into his rant about how it’s not his primary job to find the optimal parameters of a trading system, but rather to test how sturdy a strategy really is.

‘There are really a very wide range of parameters any decent trader can pick from a very wide range of objectives. I offer much more than giving you a dream set. I can tell you if your system has the cajones to trade in future.’

I explain to Optimization that I get it. I also remind him that his job is to find optimal parameters, but also acknowledging that he needs good direction on what to look for from the trader.

‘I can find best parameters based on a variety of measurements,’ he says, ‘I can tell you what the best parameters are based on net profit, net profit/maximum drawdown or some other more clever measurements.’

‘How fond are you of finding best parameters based on net profit,’ I ask. It’s kind of a trick question on my part, really. I realize this isn’t the best way to measure the potential for future profits, but I’d like to hear his take on this.

His eyes start fluttering and he puts down his espresso, almost as a measure to ensure he has nothing to throw at the wall.

‘I don’t like that method, quite frankly.’

I’m about to leave it at that and then he starts in about Overfit.

‘You know that I have certain relationships that I cannot justify to you. Let me be clear, the best way to lead me into my own circle of hell is to ask me to give you the best parameters based on past net profit.’

Okay. Time to change the subject. Clearly not a favorite topic and a little awkward. Just as I try to switch the discussion over to clever fitness functions, in walks a very strange creature named Data Mining.

I wish I could properly describe the look on Optimization’s face when he saw that thing come into the cafe. It was sheer horror that looked comical at the same time.

Data Mining is the albatross around Optimization’s fragile neck. When traders deploy this creature, they are basically torturing Optimization to the core. Data Mining is most annoying even to the uninvolved observer. He mumbles nonsense and has a queer look of promise on his face, not unlike your favorite purveyor of useless kitchen gadgets. As the little troll came into the room, I could hear him repeating a phrase that was barely audible.

‘The chance that the S&P 500 will be up for the year is directly correlated to how many players in the World Series for that year have last names that start with the letter A through M. If most of the players have last names beginning with N through Z, expect a down year.’

‘Is that true?’ I ask Optimization.

‘Probably, but let’s not go there for now.’

I can see how tortured a soul Optimization really is. He is very much a nice fellow should you ever meet him. But he is plagued in the Box. He is an indentured servant to the trader and though he doesn’t always like or agree with what he is made to do, he performs regardless.

I’m starting to see that the Box is not only filled with interesting creatures, but ominous ones as well. I will tread lightly as a precaution.

Optimizing the Bumblebee

September 4, 2009

The purpose of optimization is not to find the optimal parameters. Yes, that’s not a typo. The main purpose of optimization is to validate backtest results. You’ll remember it was the backtest’s job to validate a trading idea. Well, now backtesting is on the hot seat and optimization is the judge and jury. Optimization will validate that backtest results are not simply the product of chance, and that the system stands a chance of making money in the future. It will do this by creating a profile sketch of the system. I know, sketches are not the first thing you’d associate with spreadsheets. But in our world, it’s what we have to work with.

We are going to go over the basic setup of the optimization run in just a moment, but let’s take a simple statistic from the optimization run and see if it helps illuminate our objective. The statistic will be ‘profitable parameter sets.’ These are permutations that made money on historical data. If we randomly picked a parameter set, what would be our chances that it would have made money? If 50% of our optimized parameter sets are profitable, that means you can basically throw a dart and your chances of making money is, well, 50/50. Of course we not going to throw darts, but you get the idea.

If only 10% of optimized parameter sets are profitable, then you’d better have the aim of a trained assassin. If 80% are profitable, call your half-blind neighbor over and have him fire away. If you absolutely have to be dead-nuts on in your choice of parameter sets, you may consider whether the enterprise is worth your efforts.

The space we’d like to paint (don’t worry, I’ve got some cool looking graphs below) should be basically square in two dimensions. A scan that is too fine will lead to over-fitting. That’s bad. We cannot avoid curve fitting as that’s basically what we’re doing, but we can avoid getting carried away.

Besides providing us with information about how lucky you need to be to make money with your system, optimization will provide an insight into other features of your system, such as what the winning percent is, what sort of drawdowns you should prepare for and how frequent are trades for winning parameters versus the frequency of trades for losers.

Bumblebee was initially backtested with a slow moving average of 30, a fast moving average of 10 and a Bolllinger band around the slow MA of 0.5 standard deviations. We are going to fiddle with the slow MA period and the fast MA period and leave the Bolllinger band width alone. The reason is simple. Moving average length will have the greater impact on the system than tweaking the standard deviation length, and we don’t have time to calculate too many permutations with the method of choice. That method is called brute force. Yes, you need to be brutish and forceful to do this, but it won’t be too terribly distressing, I promise.

Permutations are statistical combinations. If we ran a test with a slow MA of 30 and fast MA of 10, we’d like to ask what would happen if we used a slow MA of 30 and a fast MA of 11. How about 12, 13. 14 and so on. You get the idea. Let me derive a simple method for calculating this monster in the following statistical proof. Or, on second thought, why not you look it up in your statistic book and take my word for it. If we run a range of 20 periods for the fast and 20 periods for the slow, we will have run 400 permutations. Mathematically savvy readers will see that 20 x 20 is 400, and that is the secret shortcut you forgot you once knew.

In keeping with the square space ideal, we are running the fast MA from 3 to 24 in steps of one (that means, 3, 4, 5, etc.) and the slow MA in steps of 2 from 11 to 49 (11, 13, 15, etc.) This yields 410 permutations. After we take out pairs that don’t make sense (where the slow is faster than the fast), we end up with 371 parameter sets. And with no further ado, below are the results of this optimization sketch. First let’s observe the percent of parameter sets that are profitable. If it’s under 30%, I highlighted it in red to indicate potential problems. If it’s over 80%, I highlighted it in green, to indicate the blind squirrel principle applies (ie, a random dart thrower can make money using the system).

I also put a color code to the left of the market to indicate whether or not the system is useful in a particular market. The obvious losers with the Bumblebee include the S&P 500 and the Coffee markets. The markets that show the most promise include the Crude Oil, Gold and Ten-Year markets. The other markets show potential, but also show some warnings signs, so they’ve been designated as cautionary.

Let’s look at the optimization space for a obvious wanker and a potential best-friend. First the wanker. Below is the optimization space painted for the S&P 500 from 1991-92. Only six percent of potential parameter sets showed a net profit. Yes, that ugly.

You can see that overall, you needed to pick just the right parameter set to make money with the system during this two-year period.

Let’s now look at what a more lovely optimization looks like. Below is the optimization space for the Cotton during 1987-88.

You can clearly see that if you avoided a certain area of tiny slow and fast MAs, you’d at least make money. And if you shot for the area in the upper right corner, you’d make some nice returns.

Here is a sample of some other data gleaned from the optimization profile.

This is the spreadsheets results for the Ten-Year between 1995-96. You can see that 92% of parameter sets were profitable. And, 347 parameter sets yielded trades during the period out of a potential 371 sets. That’s impressive. But looking deeper, you can see that the winning parameter sets had a winning percentage of just 51%. The losing parameter sets had a winning percentage of around 35%.

Also, the winning parameter sets (the good guys on our team) traded an average of 10 trades during the two-year period while the losing parameter sets (the bad guys on the other team), traded much more heavily with an average of 35 trades during the two-year period. These are just a few of the revealing characteristics of the winning parameter sets versus the losing parameter sets that can alert you to when things are going wrong. In other words, when you have given the ball to the other team.