The Scientology of Hypothesis Testing in Empirical Research: Emphasizing Economic Significance
DOI:
https://doi.org/10.54536/ajebi.v3i1.2349Keywords:
Scientology, Bayesian, Hypothesis Testing, Economic Significance, Statistical SignificanceAbstract
Over the decades, scientists across various disciplines have cautioned against the practice of over-emphasizing statistical significance at the expense of economic or practical significance. It seems that empirical researchers in the 21st century have not heeded the caution because meeting statistical significance targets continue to take precedence over the wider discovery and publication of scientifically objective findings. Statistically insignificant results are rarely reported, in part due to publication bias towards statistically significant findings. The survey finds that although large sample sizes are widely used in empirical economics and finance research, none of the surveyed papers adopted alternative methods of hypothesis testing, such as Bayesian methods. All of them explicitly or implicitly used the classical Fisher’s hypothesis testing methods. This study finds that discussions on economic significance in nearly all papers (almost 97% of papers) only wrote one sentence or two regarding the magnitude of the effect of the regressors and a declaration that the findings were economically significant. We recommend that to enhance research credibility, other methods of hypothesis testing such as Bayesian methods, should be adopted. Journal article publishers should encourage the publication of statistically insignificant empirical findings. Economic or practical significance should be emphasized and comprehensively discussed.
Downloads
References
Andrew I., and Kasy, M (2019). Identification of and Correction for Publication Bias. American Economic Review 109(8), 2766–2794 https://doi.org/10.1257/aer.20180310
Bakan, D. (1966). The test of significance in Psychological Research. Psychological Bulletin, 66, 423-437
Berchicci, L., and King, A. (2022). Corporate sustainability: A Model uncertainty analysis of materiality. Journal of Financial Reporting, 7, 43–74
Cohen J. (1994). The earth is round (p < .05). American Psychologist, 49, 997- 1003.
Cohen J. (1990). Things I have learned (so far). American Psychologist, 45, 1304- 1312.
Cumming, G. (2013). The New Statistics: Why and How, Psychological Science. http://dx.doi.org/10.1177/0956797613504966.
Ellis, P. D. (2010). Effect sizes and the interpretation of research results in international business. Journal of International Business Studies, 41, 1581–1588
Engsted, T. (2009). Statistical vs. Economic Significance in economics and econometrics: Further comments on McCloskey and Ziliak. Journal of Economic Methodology,16(4), 393–408.
Fisher, R. (1925). Statistical tests of agreement between observation and hypothesis. Economica, (8), 139-147.
Fisher, R. A. (1955). Statistical Methods and Scientific Induction. Journal of the Royal Statistical Society, Series B (Methodological), 17(1), 69-78.
Fisher, R. A. (1959). Statistical Methods and Scientific Inference. New York: Hafner. Second edition.
Graham, J., Campbell, H., and Rajgopal, S. (2005). The economic implications of corporate financial reporting. Journal of Accounting and Economics, 40, 3–73
Gill J. (2002). Bayesian methods: a social and behavioural sciences approach. London: Chapman Hall/CRC.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med. 2, e124.
Ioannidis, J. P. A. and Doucouliagos, C. (2013). What’s to know about the credibility of empirical economics? Journal of Economic. Survey, 27(5), 997–1004.
Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences http://dx.doi.org/10.1073/pnas.1313476110.
Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences http://dx.doi.org/10.1073/pnas.1313476110.
Kim, Jae H., and Philip Inyeob Ji. (2015). Significance Testing in Empirical Finance: A Critical Review and Assessment. Journal of Empirical Finance 34, 1–14
Kewei, H., Chen X., and Lu Z. (2020). Replicating anomalies. The Review of Financial Studies 33(5), 2019–2133.
Keuzenkamp, H. A. and Magnus, J. (1995). On tests and significance in econometrics. Journal of Economics. 67(1), 103–128.
Leamer, E. (1978). Specification Searches: Ad Hoc Inference with Nonexperimental Data. Wiley, New York.
Lindley, D. V. (1957). A Statistical Paradox. Biometrika 44, 187–192.
McShane, B. B., David G., Andrew G., Christian R., and Jennifer L. T. (2019). Abandon Statistical Significance. American Statistician 73(S1), 235–45.
Michaelides, M. (2021). Large sample size bias in empirical finance. Finance Research Letters, 41, 101835.
Mitton, T. (2022). Methodological variation in empirical corporate finance. Review of Financial Studies, 35, 527–575
Mitton, T. (2023). Economic significance in corporate finance. Review of Corporate Finance Studies, forthcoming
Ohlson, J. A. (2022/2023). Empirical accounting seminars: Elephants in the room. Accounting, Economics, and Law: A convivium, (0).
Peden, W., & Sprenger, J. (2021). Statistical Significance Testing in Economics in Conrad Heilmann & Julian Reiss (eds.), The Routledge Handbook of the Philosophy of Economics (2021).
Reichenbach, R. (1938). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641.
Rommel, J., & Weltin, M. (2021). Is There a Cult of Statistical Significance in Agricultural Economics? Applied Economic Perspectives and Policy, 43(3), 1176-1191. https://doi.org/10.1002/aepp.13050
Selleke, T., Bayarri, M. J. and Berger, J. O. (2001). Calibration of p values for testing precise null hypotheses. American Statistician, 55(1), 62–71.
Sneed, D. M. (2016) The Significance of Economic Significance, Oglethorpe Journal of Undergraduate Research, 6(1), Article 2. https://digitalcommons.kennesaw.edu/ojur/vol6/iss1/2.
Terry, S. J., Toni M. W., and Anastasia A. Z. (2022). Information versus investment. Technical Report, National Bureau of Economic Research (NBER).
Wasserstein, R. L., and Nicole A. L. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. American Statistician, 70(2), 129–33.
Zellner, A. and Siow, A. (1979). Posterior Odds Ratio of Selected Regression Hypotheses. http://dmle.cindoc.csic.es/pdf/TESTOP_1980_31_00_38.pdf
Ziliak, S. and McCloskey, D. (1996). The standard error of regressions. Journal of Economic Literature, 34(1), 97-114.
Ziliak, Stephen T., and Deirdre N. McCloskey. (2008a). The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice and Lives: Ann Arbor: University of Michigan Press.
Ziliak, S. T., McCloskey, D. N. (2004). Size Matters the standard error of regressions in the American Economic Review. Journal of Socioeconomics, 33, 527–546.
Ziliak, S. and McCloskey, D. (996). The standard error of regressions. Journal of Economic Literature, 34, 97–114.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Moses Mayondi, Richard Mulenga
This work is licensed under a Creative Commons Attribution 4.0 International License.