There’s a commonly bandied about statistic that something like 90% of active retail speculators lose money in a given year.  Incidentally, in this case “retail” means “private trading through a conventional retail broker like E*Trade” and has nothing to do with a retail store.

This statistic is taken from two places.  The first source is a congressional investigation into day trading that resulted in the SEC imposing the pattern day trader rule in 2001.  The second source is records from a number of brokers who have had their accounts exposed by bankruptcy.  The number varies slightly by source, but overall 90% is about right.  The smallest I’ve ever seen it was about 50% losers from a arbitrage-oriented foreign exchange broker.  These are sobering numbers.

Now trading as a whole is a zero sum game – there are going to be a good number of losers regardless of the level of skill of the participants.  It’s like the old quip about Eric Drache – that he was the eighth best stud poker player in the world, but had the poor judgement to only play with the other seven.  As you might guess, Eric was always broke.  But there’s more to the high loss rates among retail speculators than just the difficulty of the game.  Overall, there seems to be a long term net cache flow from retail speculators to institutional speculators (bank and prop desks).  This is harder to prove than the 90% loss rate, because it would require industry-wide statistics to be certain.  But I believe it to be the case, and I don’t think you’ll find many knowledgeable people who disagree.

It stands to reason that if you want to succeed at retail speculation (which is what this blog is trying to teach) that you need to avoid whatever it is that causes retail speculators to fail both individually and in the aggregate.  Somehow you need to be in the 10%, rather than the 90%.  But what’s the difference between the two groups?  I can think of several plausible explanations:

• The 90% retail speculators lack effective methods for predicting market behavior which the 10% and institutional speculators have access to.  As a result the put on worse trades and lose.  For simplicity sake I’ll call this the technical explanation.
• Losing speculators are somehow mentally unsuited for the task at hand, and as a result they do things that sabotage their success.  I’ll call this the psychological explanation.
• The 90% are simply unlucky and the 10% lucky.  The difference in the size of the two populations is due to the negative effects of transaction costs.

Of the three explanations, I think luck can immediately be ruled out.  While I know for a fact that trading has numerous lucky fools and unlucky geniuses, that can only happen in individual cases and can comprise only a small portion of the trading population.  The average trader, almost by definition, is unlikely to have experienced exceptional luck one direction or the other.  I can also rule out the transaction cost explanation – while transaction costs are unquestionably an important factor, in my experience losing traders are losers even ignoring all costs.  Conceptually, this makes sense – if no one’s losing before costs are considered, then it’s a guarantee by conservation of money that no one will be winning after costs are taken into account.  And we all know there are some big winners in trading.  So there must be big pre-costs losers too.  On an individual level, luck can largely be removed from the equation by analyzing trading results for statistically significant groups of trades.  It’s entirely possible to have one lucky trade.  Thirty lucky trades is less likely.  Three hundred is highly unlikely.  I think it’s safe to say that to the extent lucky fools and unlucky geniuses exist in the markets, they mostly have very short track records.

Of the remaining two explanations, it’s not a question of one being right and the other being wrong.  Certainly, both are possible.  A trader could analyze the markets so ineptly that profit was impossible.  Similarly, one could have such severe psychological problems that even perfect analysis could not be turned into profits.  The question is to what degree each explanation impacts the population of losing speculators as a whole.

Clearly, if the goal is to educate winning speculators, then this question is of paramount importance.  It’s akin to, when educating a kid who can’t read English, determining if their problem is a lack of spoken English skills, a lack of practice with phonics, or dyslexia.  If you incorrectly diagnose the problem, all attempts at a remedy are likely to be fruitless.  Similarly, if you want to educate traders, then it’s imperative to understand why traders fail.  What, then, is the balance between the two remaining explanations?

If you had asked me 5 years ago, I would have unquestionably said that the problem is primarily technical.  The shear number of technical trading methods available out there is staggering.  For example, the charting and execution software I use (NinjaTrader) comes stock with about 100 technical indicators.  Each of these could easily be converted into many technical trading systems  – something like buy when the RSI wiggles like this, sell when it wiggles like that.  Each such system could be applied on a wide range of timeframes to a wide range of instruments (either regular ones, or synthetic spreads made up of multiple instruments).  When you look at all the possible alternatives, it becomes clear that the number of possible technical trading system and instruments is essentially unlimited.  Consider that with roughly five thousands NYSE and NASDAQ listed stocks there are about 25 million stock pair spreads one could trade.  That number is so large that one person could not reasonably investigate all the possibilities in a lifetime even with sophisticated computer assistance.  If not infinite, it’s close enough.

So back to 5-years-ago me.  I saw that infinite sea of technical trading methods, and I knew that some of them worked and some (likely, most) of them did not.  Examples like the Turtle method, which clearly used to work, served as proof of the existence of working trading methods.  I saw my task, in learning to trade, as a process of searching through that sea of methods, finding the best one or handful I could, and then committing capital to trade them.  My expectation was that the results would be good, indeed were highly unlikely to be worse than break even.  After all, I would test any method I traded before employing it.  Methods that didn’t test out as profitably wouldn’t get used.  Obviously it was possible that something ineffective would slip in and I would lose a little discovering that, but my expectation was that any such losses would likely be dwarfed by profits from other, successful methods.

If you’ve read the previous articles on this blog, you know it didn’t work out like that.  My first years of live money trading resulted in me losing about twenty thousand dollars.  Even now, it’s difficult and kind of embarrassing to talk about.  I found a couple of methods I thought worked, and started trading them.  But some very bad things happened.  First off, my actual execution of the trades was poor.  If you compared the trades I really took to the ones my method suggested I take you would see only a passing resemblance.  There were numerous reasons for this – accident (buy when I meant to sell etc.), lack of experience, methods that weren’t compatible with my schedule and required trades at times I couldn’t trade, differences between how my paper simulation could push a button vs. how fast I actually pushed it etc.  The whole experience looked like nothing so much as The Three Stooges Trade an Index Future.  It was pathetic, really.

As the losses mounted, it was clear I had to change something.  My first step, and I still think this was conceptually a very good idea, was to invest in trading automation software.  I bought a copy of NinjaTrader and started to use it both to backtest my methods and to automate them so they weren’t dependent on my faulty button pressing.  Some computer problems aside, this was a huge improvement.  Since I’m a pretty good programer, it was fairly easy to specify exactly the behaviors I wanted my trading systems to have.  The automatic computerized execution eliminated the mistakes.  I felt I had solved my problems and was headed for profits.

Sadly, no.  Now that I had eliminated the discrepancies between method and action, I started to notice a disturbing trend.  Trading methods that I had gone to reasonable lengths to convince myself worked would stop working shortly after I started trading them on live money.  I started to get paranoid that I was cursed somehow, that things which ought to work would stop working as soon as it was my money on the line.  I went through about a twelve month period where I tried to power through this problem with more sophisticated technical analysis, more rigorous testing of my methods, etc.  I spent every free hour in front of the computer trying stuff.  The losses continued – in fact they were so consistent it was eerie.  I don’t think I had a winning week for about 3 months, and my trading account shrank bit by bit.  Eventually I quit trading altogether, both for my sanity and to preserve what money I had left.

After a while I felt compelled to give trading another shot, and before I did so I spent a lot of time investigating how successful traders approached the markets.  This wasn’t easy research since there isn’t a lot of detailed information out there on successful traders.  As I previously mentioned it’s more a matter of following a trail of bread crumbs.  But I eventually pieced together several things of interest:

• Most winning traders used a very restricted set of technical tools.  The two most highly correlated with success seemed to be statistical analysis and market profile.  Of the two, I’ve found statistical analysis more useful, but I’m not an expert on market profile, and I do believe it’s useful for many people.  The least correlated  tools were various oscillators and trend following tools based on moving averages – MACD, RSI etc shortly followed by strategies involving drawing lines on charts – trend lines and the like.
• Success was highly correlated with multi-time-frame and multi-instrument strategies.  Even if a trader wasn’t trading spreads, they frequently looked at multiple markets simultaneously.
• The winning traders downplayed the technical aspects of their trading in favor of psychological aspects.

There also seemed to be a shift over time from systematic trading to discretionary trading.  In the 70’s and 80’s, the most successful traders were clearly system traders.  In the 90’s and 00’s, fewer and fewer were.  I think this was caused by the demise of trend following.

Initially I was really disturbed by the psychological focus so many successful traders had.  My reasoning was this: if I have a method I’ve proven works with rigorous statistical techniques, and I trade it in an automated fashion where trading mistakes are basically impossible barring computer failure, my psychology should not matter.  I could be happy, sad, sane, delusional, horny or nearly incapacitated by illness and it should make no difference.  The computer should make money on average regardless.  And I still believe this is a true statement – I’m not mystical about the markets.  I don’t believe anything that happens in my mind moves prices.  They move where they want to go regardless of what’s going on in my head.

That said, it was clear I was losing, and I was willing to change my beliefs to match those of observed winners if it would make me a winner too.  So I started to dig into the question of how my own psychology was causing me to act in the markets.  Once I decided to start, this was a remarkably rapid process.  The first thing I uncovered was a deeply held belief that was clearly irrational: that I had a short period of time to become successful at trading and needed to hurry.  It may seem odd, but I have no idea where this belief came from – I was under no pressure from anywhere to speed up the process of learning to trade.  I had a good job and didn’t need additional income any time soon.  My family was indulgent of the efforts.  Both are still true.  In many ways I had the ideal environment to learn at my own pace.  But despite this, I was creating fantasies in my mind where wild trading success was months, weeks, or even days away – absurd.  Can you imagine someone in the 1st year of medical school with a goal like “by Dec 31st I’ll be one of the best doctors in the world”?  You’d have to think they were delusional.  I was delusional.

The effect this had on my trading I realized was dramatic.  The key to systematic trading is to prove to a very high level of confidence that your system works.  Hurry works against this goal – when you hurry you test less, and test fewer different types of market conditions.  I’m now convinced every single method I traded during this period worked either conditionally, or not at all.  With this realization in hand, I suddenly “got” why psychology is such a big deal – this one absurd belief on my part served to completely overshadow what I knew to be good technical procedures for testing trading methods.

During this process, I uncovered another psychological gremlin which also seriously hurt my trading results.  But this post is already painfully long, and the second problem is deserving of a post all by itself and as such will have to wait.  I don’t really need to go into it to get at why I’m trying to say here.  Initially for a new trader, there’s only one thing worth learning, and that’s technical information.  There are numerous tools for analyzing markets, and some are quite a bit better than others.  I’ll take time in a later post to go into specifics on what I think is worth learning.  But having learned that technical information, it is not enough to succeed.  In fact, it’s frequently just enough rope to hang yourself.  There’s a second phase where you need to “get your head right” for lack of a better term.  And I think those later, psychological issues are the real gate to success.  At least they were for me, and who doesn’t love to generalize as if the rest of the world is like them? 😉  But my belief is now that if you could study the 90% of retail traders who are failing in-depth, you would find that the vast majority of them had been exposed to technical methods sufficiently effective that you could use them to find winning trades.  Therefore I’m forced to conclude the problem is more psychological than technical.  But I also believe that the psychological problems cannot be solved until sound technical methods are learned.