Data - Snooping , Technical Trading Rule Performance , and the Bootstrap

Numerous studies in the finance literature have investigated technical analysis to determine its validity as an investment tool. Several of these studies conclude that technical analysis does have merit, however, it is noted that the effects of data-snooping are not fully accounted for. In this paper we utilize White’s Reality Check bootstrap methodology (White (1997)) to evaluate simple technical trading rules while quantifying the data-snooping bias and fully adjusting for its effect in the context of the full universe from which the trading rules were drawn. Hence, for the first time, the paper presents a means of calculating a comprehensive test of performance across all trading rules. In particular, we consider the study of Brock, Lakonishok, and LeBaron (1992), expand their universe of 26 trading rules, apply the rules to 100 years of daily data on the Dow Jones Industrial Average, and determine the effects of data-snooping. During the sample period inspected by Brock, Lakonishok and LeBaron, we find that the best technical trading rule is capable of generating superior performance even after accounting for datasnooping. However, we also find that the best technical trading rule does not provide superior performance when used to trade in the subsequent 10-year post-sample period. We also perform a similar analysis, applying technical trading rules to the Standard and Poor’s 500 futures contract. Here, too, we find no evidence that the best technical rule outperforms, once account is taken of data-snooping effects. Data-Snooping, Technical Trading Rule Performance, and the Bootstrap 1 Technical trading rules have been used in financial markets for over a century. Numerous studies have been performed to determine whether such rules can be employed to provide superior investing performance. By and large, the recent academic literature suggests that technical trading rules are capable of producing valuable economic signals. In perhaps the most comprehensive recent study of technical trading rules using 90 years of daily stock prices, Brock, Lakonishok, and LeBaron (1992) (BLL, hereafter) found that 26 technical trading rules applied to the Dow Jones Industrial Average significantly outperformed a benchmark of holding cash. Their findings are especially strong since every single one of the trading rules they considered was capable of beating the benchmark. When taken at face value, these results indicate either that the stock market is not efficient even in the weak form – a conclusion which, if found to be robust, would go against most researchers’ prior beliefs – or that risk-premia display considerable variation even over very short periods of time (i.e., at the daily interval). An important issue generally encountered, but rarely directly addressed when evaluating technical trading rules, is data-snooping. Data-snooping occurs when a given set of data is used more than once for purposes of inference or model selection. When such data reuse occurs, there is always the possibility that any satisfactory results obtained may simply be due to chance rather than to any merit inherent in the method yielding the results. With respect to their choice of technical trading rules, BLL state that “... numerous moving average rules can be designed, and some, without a doubt, will work. However, the dangers of data snooping are immense.” Thus, BLL rightfully acknowledge the effects of data-snooping. They go on to evaluate their results by fitting several models to the raw data and resampling the residuals to create numerous bootstrap samples. The goal of this effort is to determine the statistical significance of their findings. However, as acknowledged by BLL, they were not able “to compute a 1 See, for example, Brock, Lakonishok and LeBaron (1992), Fama and Blume (1966), Kaufman (1987), Levich and Thomas (1993), Neftci (1991), Osler and Chang (1995), Sweeney (1988), Taylor (1992), and Taylor (1994). 2 Brock, Lakonishok, and LeBaron (1992), page 1736. Data-Snooping, Technical Trading Rule Performance, and the Bootstrap 2 comprehensive test across all rules. Such a test would have to take into account the dependencies between results for different rules.” This task has thus far eluded researchers. A main purpose of our paper is to extend and enrich the earlier research on technical trading rules by applying a novel procedure that permits computation of precisely such a test. Although the bootstrap approach (introduced by Efron (1979)) is not new to the evaluation of technical analysis, White’s Reality Check bootstrap methodology (introduced by White (1997)) adopted in this paper permits us to correct for the effects of data-snooping in a manner not previously possible. Thus we are able to evaluate the performance of technical trading rules in a way that permits us to ascertain whether superior performance is a result of superior economic content, or simply due to luck. Data-snooping need not be the consequence of a particular researcher’s efforts. It can result from a subtle survivorship bias operating on the entire universe of technical trading rules that have been considered historically. Suppose that, over time, investors have experimented with technical trading rules drawn from a very wide universe – in principle, thousands of parameterizations of a variety of types of rules. As time progresses, the rules that happened to perform well historically receive more attention and are considered ‘serious contenders’ by the investment community, while unsuccessful trading rules are more likely to be forgotten. After a long sample period, only a small set of trading rules may be left for consideration, and these rules’ historical track record will be cited as evidence of their merits. If enough trading rules are considered over time, some rules are bound by pure luck, even in a very large sample, to produce superior performance even if they do not genuinely possess predictive power over asset returns. Of course, inference based solely on the subset of surviving trading rules may be misleading in this context 3 Brock, Lakonishok, and LeBaron (1992), page 1743. 4 Indeed, BLL report that they did not consider a larger set of trading rules than the 26 rules they report results for. 5 See also Lo and MacKinlay (1990) for a similar point. Data-Snooping, Technical Trading Rule Performance, and the Bootstrap 3 since it does not account for the full set of initial trading rules, most of which are likely to have under-performed. The effects of such data-snooping, operating over time and across many investors and researchers, can only be quantified provided that one considers the performance of the best trading rule in the context of the full universe of trading rules from which the best rule conceivably was chosen. A further purpose of our study is to address this issue by constructing a universe of nearly 8,000 parameterizations of trading rules which are applied to the Dow Jones Industrial Average over a 100-year period from 1897 to 1996. We use the same data set as BLL to investigate the potential effects of data-snooping in their experiment. Our results show that, during the sample originally investigated by BLL, 1897–1986, certain trading rules did indeed outperform the benchmark, even after adjustment is made for data-snooping. We base our evaluation both on mean returns and on a version of the Sharpe ratio which adjusts for total risk. Since BLL’s study finished in 1986, we benefit from having access to another 10 years of data on the Dow Jones portfolio. We use this data to test whether their results hold outof-sample. Interestingly, we find that this is not the case: the probability that the bestperforming trading rule did not outperform the benchmark during this period is nearly 12 percent, suggesting that, at conventional levels of significance, there is scant evidence that technical trading rules were of any economic value during the period 1987–1996. To determine whether transaction costs or short-sale constraints could have accounted for the apparent historical success of the trading rules studied by BLL, we also conduct our bootstrap simulation experiment using price data on the Standard and Poor’s 500 (S&P 500) index futures. Transaction costs are easy to control in trading the futures contract and it also would not have been a problem to take a short position in this contract. Over the 13-year period since the futures contract started trading in 1984, we find no evidence that the trading rules outperformed. 6 We thank Blake LeBaron for providing us with the data set used in the BLL study. Data-Snooping, Technical Trading Rule Performance, and the Bootstrap 4 While the current paper adopts a bootstrap methodology to evaluate the performance of technical trading rules, the methodology applied in this paper also has a wide range of other applications. This is important, because the dangers from data-snooping emerge in many areas of finance and economics, such as in the predictability of stock returns (as addressed by, for example, Foster, Smith, and Whaley (1997)), modeling of exchange and interest rates, identification of factors and “anomalies” in cross-sectional tests of asset pricing models (Lo and MacKinlay (1990)), and other exercises where theory does not suggest the exact identity and functional form of the model to be tested. Thus, the chosen model is likely to be data-dependent and a genuinely meaningful out-of-sample experiment is difficult to carry out. The plan of the paper is as follows. Section I introduces the bootstrap data-snooping methodology, section II reviews the existing evidence on technical trading rules, and section III introduces the universe of trading rules that we consider in the empirical analysis. Section IV presents our bootstrap results for the data set studied by BLL, while section V conducts the out-of-sample experiment. Finally, section VI discusses in more detail the economic interpretation of our findings. I. The Bootstrap Snooper Data-snooping biases are widely recognized to be a very significant problem in financial studies. They have been quantified by Lo and MacKinlay (1990), described in mainstream books on investing (O’Shaughnessy (1997), page 24) and forecasting (Diebold (1998), page 87), and have recently been addressed in the popular press (Business Week, Coy (1997)): “For example, [David Leinweber, managing director of First Quadrant Corporation in Pasadena, California] sifted through a United Nations CDROM and discovered that historically, the single best prediction of the Standard & Poor’s 500 stock index was butter production in Bangladesh.” Our purpose in this study is to 7 Lo and MacKinlay (1990) quantify the data-snooping bias in tests of asset pricing models where the firm characteristic used to sort stocks into portfolios is correlated with the estimation error of the performance measure. Data-Snooping, Technical Trading Rule Performance, and the Bootstrap 5 determine whether technical trading rules have genuine predictive ability or fall into the category of “butter production in Bangladesh”. The apparatus used to accomplish this is the Reality Check bootstrap methodology which we briefly describe. White (1997) provides a procedure, building on work of Diebold and Mariano (1995) and West (1996), to test whether a given model has predictive superiority over a benchmark model after accounting for the effects of data-snooping. The idea is to evaluate the distribution of a suitable performance measure giving consideration to the full set of models that led to the best-performing trading rule. The test procedure is based on the l × 1 performance statistic: