본문 바로가기
KERD
논문KCI 등재

Forecasting methods and principles: Evidence-based checklists

1

조회

0

다운로드

1

인용

0

참고문헌

키워드

주제분류경영학

초록

Problem : How to help practitioners, academics, and decision makers use experimental research findings to substantially reduce forecast errors for all types of forecasting problems. Methods : Findings from our review of forecasting experiments were used to identify methods and principles that lead to accurate forecasts. Cited authors were contacted to verify that summaries of their research were correct. Checklists to help forecasters and their clients undertake and commission studies that adhere to principles and use valid methods were developed. Leading researchers were asked to identify errors of omission or commission in the analyses and summaries of research findings. Findings : Forecast accuracy can be improved by using one of 15 relatively simple evidence-based forecasting methods. One of those methods, knowledge models, provides substantial improvements in accuracy when causal knowledge is good. On the other hand, data models – developed using multiple regression, data mining, neural nets, and “big data analytics” – are unsuited for forecasting. Originality : Three new checklists for choosing validated methods, developing knowledge models, and assessing uncertainty are presented. A fourth checklist, based on the Golden Rule of Forecasting, was improved. Usefulness : Combining forecasts within individual methods and across different methods can reduce forecast errors by as much as 50%. Forecasts errors from currently used methods can be reduced by increasing their compliance with the principles of conservatism (Golden Rule of Forecasting) and simplicity (Occam’s Razor). Clients and other interested parties can use the checklists to determine whether forecasts were derived using evidence-based procedures and can, therefore, be trusted for making decisions. Scientists can use the checklists to devise tests of the predictive validity of their findings.

Abstract

Problem : How to help practitioners, academics, and decision makers use experimental research findings to substantially reduce forecast errors for all types of forecasting problems. Methods : Findings from our review of forecasting experiments were used to identify methods and principles that lead to accurate forecasts. Cited authors were contacted to verify that summaries of their research were correct. Checklists to help forecasters and their clients undertake and commission studies that adhere to principles and use valid methods were developed. Leading researchers were asked to identify errors of omission or commission in the analyses and summaries of research findings. Findings : Forecast accuracy can be improved by using one of 15 relatively simple evidence-based forecasting methods. One of those methods, knowledge models, provides substantial improvements in accuracy when causal knowledge is good. On the other hand, data models – developed using multiple regression, data mining, neural nets, and “big data analytics” – are unsuited for forecasting. Originality : Three new checklists for choosing validated methods, developing knowledge models, and assessing uncertainty are presented. A fourth checklist, based on the Golden Rule of Forecasting, was improved. Usefulness : Combining forecasts within individual methods and across different methods can reduce forecast errors by as much as 50%. Forecasts errors from currently used methods can be reduced by increasing their compliance with the principles of conservatism (Golden Rule of Forecasting) and simplicity (Occam’s Razor). Clients and other interested parties can use the checklists to determine whether forecasts were derived using evidence-based procedures and can, therefore, be trusted for making decisions. Scientists can use the checklists to devise tests of the predictive validity of their findings.

참고문헌 (148)

  1. Arkes, H. R., "Principles of Forecasting", pp.495~515 (2001), Kluwer Academic
  2. Armstrong, J. S., "How to avoid exploratory research", Journal of Advertising Research, 10(4), pp.27~30 (1970)
  3. Armstrong, J. S., "Forecasting wiTheconometric methods : Folklore versus fact", The Journal of Business, 51, pp.549~564 (1978)
  4. Armstrong, J. S., "The seer-sucker theory : The value of experts in forecasting", Technology Review, 83, pp.18~24 (1980)
  5. Armstrong, J. S., "Long-Range Forecasting" (1985), Wiley
  6. Armstrong, J. S., "Principles of Forecasting" (2001), Kluwer
  7. Armstrong, J. S., "Principles of Forecasting", pp.171~192 (2001), Kluwer Academic
  8. Armstrong, J. S., "Principles of Forecasting", pp.417~439 (2001), Kluwer Academic
  9. Armstrong, J. S., "Principles of Forecasting", pp.443~472 (2001), Kluwer Academic
  10. Armstrong, J. S., "Findings from evidence-based forecasting : Methods for reducing forecast error", International Journal of Forecasting, 22, pp.583~598 (2006)
  11. Armstrong, J. S., "Significance tests harm progress in forecasting", International Journal of Forecasting, 23, pp.321~327 (2007)
  12. Armstrong, J. S., "Statistical significance tests are unnecessary even when properly done and properly interpreted : Reply to commentaries", International Journal of Forecasting, 23, pp.335~336 (2007)
  13. Armstrong, J. S., "Persuasive advertising: Evidence-based principles" (2010), Palgrave Macmillan
  14. Armstrong, J. S., "Predicting job performance : The moneyball factor", Foresight, 25, pp.31~34 (2012)
  15. Armstrong, J. S., "Illusions in regression analysis", International Journal of Forecasting, 28, pp.689~694 (2012)
  16. Armstrong, J. S., "Exploratory analysis of marketing data : Trees vs. regression", Journal of Marketing Research, 7, pp.487~492 (1970)
  17. Armstrong, J. S., "Error measures for generalizing about forecasting methods : Empirical comparisons", International Journal of Forecasting, 8, pp.69~80 (1992)
  18. Armstrong, J. S., "Causal forces : Structuring knowledge for time series extrapolation", Journal of Forecasting, 12, pp.103~115 (1993)
  19. Armstrong, J. S., "Identifi cation of asymmetric prediction intervals through causal forces", Journal of Forecasting, 20, pp.273~283 (2001)
  20. Armstrong, J. S., "Index methods for forecasting : An application to American presidential elections", Foresight : The International Journal of Applied Forecasting, 3, pp.10~13 (2006)
  21. Armstrong, J. S., "Guidelines for science: Evidence and checklists", ResearchGate, ~ (2018)
  22. Armstrong, J. S., "Brief vs. comprehensive descriptions in measuring intentions to purchase", Journal of Marketing Research, 8, pp.114~117 (1971)
  23. Armstrong, J. S., "Estimating nonresponse bias in mail surveys", Journal of Marketing Research, 14, pp.396~402 (1977)
  24. Armstrong, J. S., "Reaping benefits from management research : Lessons from the forecasting principles project", Interfaces, 33, pp.89~111 (2003)
  25. Armstrong, J. S., "Using quasi-experimental data to develop empirical generalizations for persuasive advertising", Journal of Advertising Research, 49, pp.170~175 (2009)
  26. Armstrong, J. S., "Effectiveness of monetary incentives : Mail surveys of multinational professional groups", Industrial Marketing Management, 23, pp.133~136 (1994)
  27. Armstrong, J. S., "The use of the decomposition principle in making judgments", Organizational Behavior and Human Performance, 14, pp.257~263 (1975)
  28. Armstrong, J. S., "Sales forecasts for existing consumer products and services: Do purchase intentions contribute to accuracy?", International Journal of Forecasting, 16, pp.383~397 (2000)
  29. Armstrong, J. S., "Principles of Forecasting", pp.259~282 (2001), Kluwer Academic
  30. Armstrong, J. S., "Decomposition by causal forces : A procedure for forecasting complex time series", International Journal of Forecasting, 21, pp.25~36 (2005)
  31. Armstrong, J. S., "Golden rule of forecasting : Be conservative", Journal of Business Research, 68, pp.1717~1731 (2015)
  32. Armstrong, J. S., "Predictive validity of evidence-based persuasion principles", European Journal of Marketing, 50, pp.276~293 (2016)
  33. Babst, D. V., "Comparison of multiple regression and configural analysis techniques for developing base expectancy tables", Journal of Research in Crime and Delinquency, 5, pp.72~80 (1968)
  34. Batchelor, R., "Forecaster diversity and the benefi ts of combining forecasts", Management Science, 41, pp.68~75 (1995)
  35. Belvedere, V., "The infl uence of product involvement and emotion on shortterm product demand forecasting", International Journal of Forecasting, 33, pp.652~661 (2017)
  36. Ben-David, I., "Managerial miscalibration", The Quarterly Journal of Economics, 128, pp.1547~1584 (2013)
  37. Boylan, J. E., "Reproducibility in forecasting research", International Journal of Forecasting, 31, pp.79~90 (2015)
  38. Bradburn, N. M., "A sking questions: The defi nitive guide to questionnaire design–For market research, political polls, and social and health questionnaires" (2004), Wiley
  39. Brown, R. G., "Statistical forecasting for inventory control" (1959), McGraw-Hill
  40. Brown, R. G., "Smoothing, forecasting and prediction of discrete time series" (1962), Prentice-Hall
  41. Buchanan, W., "Election predictions : An empirical assessment", Public Opinion Quarterly, 50, pp.222~227 (1986)
  42. Bunn, D. W., "Comparison of seasonal estimation methods in multi-item short-term forecasting", International Journal of Forecasting, 15, pp.431~443 (1999)
  43. Burgess, E. W., "Protecting the public by parole and by parole prediction", Journal of Criminal Law and Criminology, 27, pp.491~502 (1936)
  44. Cerf, C., "The experts speak" (1998), Villard
  45. Chamberlin, T. C., "The method of multiple working hypotheses", Science, 148, pp.754~759 (1965)
  46. Chatfield, C., "Principles of Forecasting", pp.475~494 (2001), Kluwer Academic
  47. Chen, C., "A new accuracy measure based on bounded relative error for time series forecasting", PLoS ONE, 12(3), pp.e0174202~ (2017)[DOI]
  48. Collopy, F., "Rule-based forecasting : Development and validation of an expert systems approach to combining time series extrapolations", Management Science, 38, pp.1394~1414 (1992)
  49. Collopy, F., "Principles of Forecasting", pp.285~300 (2001), Kluwer Academic
  50. Cox, J. E., "Principles of forecasting", pp.633~649 (2001), Kluwer Academic
  51. Czerlinski, J., "Simple heuristics that make us smart", pp.97~118 (1999), Oxford University Press
  52. Dana, J., "The superiority of simple alternatives to regression for social science predictions", Journal of Educational and Behavioral Statistics, 29(3), pp.317~331 (2004)
  53. Dawes, R. M., "A case study of graduate admissions : Application of three principles of human decision making", American Psychologist, 26, pp.180~188 (1971)
  54. Dawes, R. M., "Linear models in decision-making", Psychological Bulletin, 81, pp.95~106 (1974)
  55. Dielman, T. E., "A comparison of forecasts from least absolute value and least squares regression", Journal of Forecasting, 5, pp.189~195 (1986)
  56. Dielman, T. E., "Corrections to a comparison of forecasts from least absolute value and least squares regression", Journal of Forecasting, 8, pp.419~420 (1989)
  57. Dillman, D. A., "Internet, mail, and mixed-mode surveys: The tailored design method" (2014), Wiley
  58. Edmundson, R. H., "Decomposition; A strategy for judgmental forecasting", Journal of Forecasting, 9, pp.305~314 (1990)
  59. Einhorn, H. J., "Alchemy in the behavioral sciences", Public Opinion Quarterly, 36, pp.367~378 (1972)
  60. Einhorn, H. J., "Unit weighting schemes for decision making", Organizational Behavior and Human Performance, 13, pp.171~192 (1975)
  61. Fildes, R., "Against your better judgment? How organizations can improve their use of management judgment in forecasting", Interfaces, 37, pp.570~576 (2007)
  62. Fildes, R., "The organization and improvement of market forecasting", Journal of the Operational Research Society, 45, pp.1~16 (1994)
  63. Fildes, R., "Effective forecasting and judgmental adjustments : An empirical evaluation and strategies for improvement in supply-chain planning", International Journal of Forecasting, 25, pp.3~23 (2009)
  64. Franses, P. H., "Expert adjustments of model forecasts: Theory, practice and strategies for improvement" (2014), Cambridge University Press
  65. Friedman, M., "The philosophy of Economics: An anthology", pp.145~178 (1953), Cambridge University Press
  66. Gardner, E. S. Jr, "Exponential smoothing : The state of the art–Part II", International Journal of Forecasting, 22, pp.637~666 (2006)
  67. Gigerenzer, G., "Simple heuristics that make us smart" (1999), Oxford University Press
  68. Gorr, W., "Short-term forecasting of crime", International Journal of Forecasting, 19, pp.579~594 (2003)
  69. Graefe, A., "Prediction markets", pp.87~95 (2011), Routledge
  70. Graefe, A., "Accuracy of vote expectation surveys in forecasting elections", Public Opinion Quarterly, 78(S1), pp.204~232 (2014)
  71. Graefe, A., "Improving forecasts using equally weighted predictors", Journal of Business Research, 68, pp.1792~1799 (2015)
  72. Graefe, A., "The Sage handbook of electoral behavior", pp.861~882 (2017), Sage
  73. Graefe, A., "Prediction market performance in the 2016 U. S. presidential election", Foresight – The International Journal of Applied Forecasting, 45, pp.38~42 (2017)
  74. Graefe, A., "Predicting elections from the most important issue : A test of the take-the-best heuristic", Journal of Behavioral Decision Making, 25, pp.41~48 (2012)
  75. Graefe, A., "Forecasting elections from voters’ perceptions of candidates’ ability to handle issues", Journal of Behavioral Decision Making, 26, pp.295~303 (2013)[DOI]
  76. Graefe, A., "Improving causal models for election forecasting: Further evidence on the Golden Rule of Forecasting", APSA 2014 (2014)
  77. Graefe, A., "Combining forecasts : An application to elections", International Journal of Forecasting, 30, pp.43~54 (2014)
  78. Graefe, A., "The 2016 Presidential Election: The causes and consequences of an Electoral Earthquake", pp.137~159 (2017), Lexington Books
  79. Green, K. C., "Game theory, simulated interaction, and unaided judgement for forecasting decisions in conflicts : Further evidence", International Journal of Forecasting, 21, pp.463~472 (2005)
  80. Green, K. C., "Structured analogies for forecasting", International Journal of Forecasting, 23, pp.365~376 (2007)
  81. Green, K. C., "Global warming : Forecasts by scientists versus scientific forecasts", Energy and Environment, 18, pp.995~1019 (2007)
  82. Green, K. C., "Role thinking : Standing in other people’s shoes to forecast decisions in conflicts", International Journal of Forecasting, 27, pp.69~80 (2011)
  83. Green, K. C., "Simple versus complex forecasting : The evidence", Journal of Business Research, 68(8), pp.1678~1685 (2015)
  84. Green, K. C., "Methods to elicit forecasts from groups : Delphi and prediction markets compared", Foresight, 8, pp.17~20 (2007)
  85. Green, K. C., "Validity of climate change forecasting for public policy decision making", International Journal of Forecasting, 25, pp.826~832 (2009)
  86. Green, K. C., "Golden rule of forecasting rearticulated: Forecast unto others as you would have them forecast unto you", Journal of Business Research, 68, pp.1768~1771 (2015)
  87. Green, K. C., "Persuasion principles index: Ready for pretesting advertisements", European Journal of Marketing, 50, pp.317~326 (2016)[DOI]
  88. Grove, W. M., "Clinical versus mechanical prediction : A meta-analysis", Psychological Assessment, 12, pp.19~30 (2000)
  89. Hafenbrädl, S., "Applied decision making with fast-and-frugal heuristics", Journal of Applied Research in Memory and Cognition, 5, pp.215~231 (2016)
  90. Hales, B. M., "The checklist–a tool for error management and performance improvement", Journal of Critical Care, 21, pp.231~235 (2006)
  91. Hauer, E., "The harm done by tests of significance", Accident Analysis and Prevention, 36, pp.495~500 (2004)
  92. Hayes, S. P. Jr, "The inter-relations of political attitudes : IV. Political attitudes and party regularity", The Journal of Social Psychology, 10, pp.503~552 (1936)
  93. Haynes, A. B., "A surgical checklist to reduce morbidity and mortality in a global population", New England Journal of Medicine, 360, pp.491~499 (2009)
  94. Hogarth, R. M., "A note on aggregating opinions", Organizational Behavior and Human Performance, 21, pp.40~46 (1978)
  95. Hubbard, R., "Corrupt research: The case for reconceptualizing empirical management and social science" (2016), Sage
  96. Ioannidis, J. P. A., "Why most published findings are false", PLoS Medicine, 2(8), pp.e124~ (2005)[DOI]
  97. Forecasting elections using expert surveys: An application to U. S. presidential elections, "Jones, R. J.", MPRA, ~, pp.MPRA Paper No. 5301 (2007)
  98. Keogh, E., "On the need for time series data mining benchmarks : A survey and empirical demonstration", Data Mining and Knowledge Discovery, 7, pp.349~371 (2003)
  99. Kim, M. S., "Relationships among attitudes, behavioral intentions, and behavior : A meta-analysis of past research", Communication Research, 20, pp.331~364 (1993)
  100. Larrick, R. P., "Intuitions about combining opinions : Misappreciation of the averaging principle", Management Science, 52(1), pp.111~127 (2006)
  101. Lin, V. S., "Accuracy and bias of experts’ adjusted forecasts", Annals of Tourism Research, 48, pp.156~174 (2014)
  102. Locke, E. A., "Generalizing from Laboratory to Field Settings" (1986), Lexington Books
  103. Lott, J. R. Jr, "More guns, less crime" (2010), University of Chicago Press
  104. Lott, J. R. Jr, "The war on guns" (2016), Regnery Publishing
  105. Lovallo, D., "Robust analogizing and the outside view : Two empirical tests of case-based decision making", Strategic Management Journal, 33, pp.496~512 (2012)
  106. MacGregor, D. G., "Principles of forecasting", pp.107~123 (2001), Kluwer Academic
  107. Makridakis, S., "Accuracy of forecasting : An empirical investigation(with discussion)", Journal of the Royal Statistical Society A, 142, pp.97~145 (1979)
  108. Makridakis, S., "The M3-competition : Results, conclusions and implications", International Journal of Forecasting, 16, pp.451~476 (2000)
  109. Makridakis, S. G., "The accuracy of extrapolation(time series)methods : Results of a forecasting competition", Journal of Forecasting, 1, pp.111~153 (1982)
  110. Makridakis, S. G., "The forecasting accuracy of major times series methods" (1984), Wiley
  111. Makridakis, S., "Confidence intervals : An empirical investigation of time series in the M-competition", International Journal of Forecasting, 3, pp.489~508 (1987)
  112. Makridakis, S., "The M2-competition : A real-time judgmentally based forecasting study", International Journal of Forecasting, 9, pp.5~22 (1993)
  113. Mannes, A. E., "The wisdom of select crowds", Journal of Personality and Social Psychology, 107, pp.276~299 (2014)
  114. McNees, S. K., "The uses and abuses of “consensus” forecasts", Journal of Forecasting, 11, pp.703~710 (1992)
  115. McShane, B. B., "Blinding us to the obvious? The effect of statistical training on the evaluation of evidence", Management Science, 62, pp.1707~1718 (2015)
  116. Blakeley B. McShane, "Rejoinder: Statistical Significance and the Dichotomization of Evidence", Journal of the American Statistical Association, 112(519), pp.904~908 (2016), Informa UK Limited[DOI]
  117. Meehl, P. E., "Clinical vs. statistical prediction" (1954), University of Minnesota Press
  118. Miller, D. M., "Shrinkage estimators of time series seasonal factors and their effect on forecasting accuracy", International Journal of Forecasting, 19, pp.669~684 (2003)
  119. Miller, D. M., "Damping seasonal factors : Shrinkage estimators for the X-12-ARIMA program", International Journal of Forecasting, 20, pp.529~549 (2004)
  120. Morwitz, V. G., "Principles of Forecasting", pp.33~56 (2001), Kluwer Academic
  121. Morwitz, V. G., "When do purchase intentions predict sales?", International Journal of Forecasting, 23, pp.347~364 (2007)
  122. Murphy, A. H., "Probability forecasting in meteorology", Journal of the American Statistical Association, 79, pp.489~500 (1984)
  123. Namboodiri, N. K., "The average of several simple regression estimates as an alternative to the multiple regression estimate in postcensal and intercensal population estimates : A case study", Rural Sociology, 36, pp.187~194 (1971)
  124. Nikolopoulos, K., "Relative performance of methods for forecasting special events", Journal of Business Research, 68, pp.1785~1791 (2015)
  125. Nisbett, R. E., "Telling more than we can know : Verbal reports on mental processes", Psychological Review, 84, pp.231~259 (1977)
  126. Perry, M. J., "18 spectacularly wrong predictions made around the time of first Earth Day in 1970, expect more this year", AEIdeas, April 20, ~ (2017)
  127. Pool, I. de S., "Candidates, issues and strategies: A computer simulation of the 1960 and 1974 presidential elections" (1965), MIT Press
  128. Rhode, P. W., "Historical presidential betting markets", Journal of Economic Perspectives, 18, pp.127~142 (2004)
  129. Rowe, G., "Principles of Forecasting", pp.125~144 (2001), Kluwer Academic
  130. Schmidt, F. L., "The relative efficiency of regression and simple unit predictor weights in applied differential psychology", Educational and Psychological Measurement, 31, pp.699~714 (1971)
  131. Schmidt, F. L., "Statistical signifi cance testing and cumulative knowledge in psychology : Implications for training of researchers", Psychological Methods, 1, pp.115~129 (1996)
  132. Schmidt, F. L., "What if there were no significance tests?", pp.35~60 (1997), Lawrence Erlbaum
  133. Shamir, J., "Pre-election polls in Israel : Structural constraints on accuracy", Public Opinion Quarterly, 50, pp.62~75 (1986)
  134. Sheeran, P., "Intention-behavior relations : A conceptual and empirical review", European Review of Social Psychology, 12, pp.1~36 (2002)
  135. Simon, J. L., "The Ultimate Resource II: People, Materials, Environment" (1996), Princeton University Press
  136. Soyer, E., "The illusion of predictability : How regression statistics mislead experts", International Journal of Forecasting, 28, pp.695~711 (2012)
  137. Tashman, L. J., "Out-of-sample tests of forecasting accuracy : An analysis and review", International Journal of Forecasting, 16, pp.437~450 (2000)
  138. Tashman, L. J., "Principles of Forecasting", pp.651~676 (2001), Kluwer Academic
  139. Tessier, T. H., "Decomposition of time-series by level and change", Journal of Business Research, 68, pp.1755~1758 (2015)
  140. Tetlock, P. E., "Expert political judgment: How good is it? How can we know?" (2005), Princeton University Press
  141. Trapero, J. R., "Analysis of judgmental adjustments in the presence of promotions", International Journal of Forecasting, 29, pp.234~243 (2013)
  142. Tukey, J. W., "The future of data analysis", Annals of Mathematical Statistics, 33, pp.1~67 (1962)
  143. Vokurka, R. J., "Automatic feature identifi cation and graphical support in rule-based forecasting: A comparison", International Journal of Forecasting, 12, pp.495~512 (1996)
  144. Winston, C., "Economic deregulation : Days of reckoning for microeconomists", Journal of Economic Literature, 31, pp.1263~1289 (1993)
  145. Wright, M., "The ombudsman : Verifi cation of citations : Fawlty towers of knowledge?", Interfaces, 38, pp.125~139 (2008)
  146. Yokum, J. T., "Beyond accuracy : Comparison of criteria used to select forecasting methods", International Journal of Forecasting, 11, pp.591~597 (1995)
  147. Ziliak, S. T., "Size matters : The standard error of regressions in the American Economic Review", The Journal of Socio-Economics, 33, pp.527~546 (2004)
  148. Ziliak, S. T., "The cult of statistical signifi cance: How the standard error costs us jobs, justice, and lives" (2008), The University of Michigan Press

관련 논문AI추천

이 간행물의 다른 논문 (7건)