Addressing the Challenge of P-Value and Sample Size when the Significance is Borderline: The Test of Random Duplication of Participants as a New Approach

Jose-Gaby Tshikuka; Mgaywa G.M.D. Magafu; Mooketsi Molefi; Tiny Masupe; Reginald B. Matchaba-Hove; Bontle Mbongwe; Roy Tapera

doi:10.6000/1929-6029.2016.05.03.7

Authors

Jose-Gaby Tshikuka Department of Public Health Medicine, Faculty of Medicine, University of Botswana, Private bag 00713, Gaborone, Botswana
Mgaywa G.M.D. Magafu Department of Public Health Medicine, Faculty of Medicine, University of Botswana, Private bag 00713, Gaborone, Botswana
Mooketsi Molefi Department of Public Health Medicine, Faculty of Medicine, University of Botswana, Private bag 00713, Gaborone, Botswana
Tiny Masupe Department of Public Health Medicine, Faculty of Medicine, University of Botswana, Private bag 00713, Gaborone, Botswana
Reginald B. Matchaba-Hove School of Public Health, Faculty of Health Sciences, University of Botswana, Private Bag 0022, Gaborone, Botswana
Bontle Mbongwe School of Public Health, Faculty of Health Sciences, University of Botswana, Private Bag 0022, Gaborone, Botswana
Roy Tapera School of Public Health, Faculty of Health Sciences, University of Botswana, Private Bag 0022, Gaborone, Botswana

DOI:

https://doi.org/10.6000/1929-6029.2016.05.03.7

Keywords:

P-value, Sample Size, Statistical Significance, Borderline Significance, Participant Random Duplication

Abstract

The issue of borderline p-value seems to divide health scientists into two schools of thought. One school of thought argues that when the p-value is greater than or equal to the statistical significance cut-off level of 0.05, it should not be considered statistically significant and the null hypothesis should be accepted no matter how close the p-value is to the 0.05. The other school of thought believes that by doing so one might be committing a Type 2 error and possibly missing valuable information. In this paper, we discuss an approach to address this issue and suggest the test of random duplication of participants as a way to interpret study outcomes when the statistical significance is borderline. This discussion shows the irrefutability of the concept of borderline statistical significance, however, it is important that one demonstrates whether a borderline statistical significance is truly borderline or not. Since the absence of statistical significance is not necessarily evidence of absence of effect, one needs to double check if a borderline statistical significance is indeed borderline or not. The p-value should not be looked at as a rule of thumb for accepting or rejecting the null hypothesis but rather as a guide for further action or analysis that leads to correct conclusions.

References

Morman GR, Streiner DL. PDQ statistics. 2nd ed. St. Louis: Mosby 1997.

Bowers. Medical statistics from scratch. 2nd ed. West Sussex, England: John Wiley & Sons Ltd 2008.

Kirkwood BR, Sterne JAC. Essential medical statistics. 2nd ed. Victoria, Australia: Blackwell Science Ltd 2005.

Peacock JL, Peacock PJ. Oxford handbook of medical statistics. London: Oxford University Press 2011. DOI: https://doi.org/10.1093/med/9780199551286.001.0001

Gordis L. Epidemiology. 3rd ed. Philadelphia: Elsevier Saunders 2004.

Lang TA, Secic M. How to report statistics in medicine: annotated guidelines for authors, editors and reviewers. 2nd ed. Philadelphia: Sheridan Press 2006.

Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence -- imprecision. J Clin Epidemiol 2011; 64(12): 1283-1293. http://dx.doi.org/10.1016/j.jclinepi.2011.01.012 DOI: https://doi.org/10.1016/j.jclinepi.2011.01.012

Alderson P, Chalmers I. Survey of claims of no effect in abstracts of Cochrane reviews. BMJ 2003; 326(7387): 475. http://dx.doi.org/10.1136/bmj.326.7387.475 DOI: https://doi.org/10.1136/bmj.326.7387.475

Khan I, Sarker SJ, Hackshaw A. Smaller sample sizes for phase II trials based on exact tests with actual error rates by trading-off their nominal levels of significance and power. Br J cancer 2012; 107(11): 1801-1809. http://dx.doi.org/10.1038/bjc.2012.444 DOI: https://doi.org/10.1038/bjc.2012.444

Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ 1995; 311(7003): 485. http://dx.doi.org/10.1136/bmj.311.7003.485 DOI: https://doi.org/10.1136/bmj.311.7003.485

Alderson P. Absence of evidence is not evidence of absence: We need to report uncertain results and do it clearly. BMJ 2004; 328(7438): 476-477. http://dx.doi.org/10.1136/bmj.328.7438.476 DOI: https://doi.org/10.1136/bmj.328.7438.476

Graham CD, Rose MR, Hankins M, Chalder T, Weinman J. Separating emotions from consequences in muscle disease: comparing beneficial and unhelpful illness schemata to inform intervention development. J Psychosom Res 2013; 74(4): 320-326. http://dx.doi.org/10.1016/j.jpsychores.2012.09.012 DOI: https://doi.org/10.1016/j.jpsychores.2012.09.012

Schlesselman JJ, Schneiderman MA. Case control studies: design, conduct, and analysis. J Occup Environ Med 1982; 24: 879.

Armitage P, Berry G, Matthews JNS. Statistical methods in medical research, 4th ed. Oxford: Blackwell Science Ltd 1994.

Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall/CRC 1990. DOI: https://doi.org/10.1201/9780429258589

Arya R, Antonisamy B, Kumar S. Sample size estimation in prevalence studies. Indian J Pediatr 2012; 79(11): 1482-1488. http://dx.doi.org/10.1007/s12098-012-0763-3 DOI: https://doi.org/10.1007/s12098-012-0763-3

Fox N, Hunn A, Mathers N. Sampling and sample size calculation. London: The NIHR RDS for the East Midlands/Yorkshire & the Humber 2007.

Tukey JW. Analyzing data: sanctification or detective work? Am Psychol 1969; 24: 83. http://dx.doi.org/10.1037/h0027108 DOI: https://doi.org/10.1037/h0027108

Tukey JW. The philosophy of multiple comparisons. Stat Sci 1991; 100-16. http://dx.doi.org/10.1214/ss/1177011945 DOI: https://doi.org/10.1214/ss/1177011945

Fisher RA. Statistical methods for research workers. 13th ed. New York: Hafner 1958.

Sedgwick P. Clinical significance versus statistical significance. BMJ 2014; 348. http://dx.doi.org/10.1136/bmj.g2130 DOI: https://doi.org/10.1136/bmj.g2130

Mayrent SL, Ed. Epidemiology in medicine. Boston: Little Brown and Company 1987.

Wainer H, Robinson DH. Shaping up the practice of null hypothesis significance testing. Educ Res 2003; 32(7): 22-30. http://dx.doi.org/10.3102/0013189X032007022 DOI: https://doi.org/10.3102/0013189X032007022

Tshikuka JG, Gray‐Donald K, Scott M, Olela KN. Relationship of childhood protein-energy malnutrition and parasite infections in an urban African setting. Trop Med Int Health 1997; 2(4): 374-382. http://dx.doi.org/10.1111/j.1365-3156.1997.tb00154.x DOI: https://doi.org/10.1111/j.1365-3156.1997.tb00154.x

Tshikuka JG, Okenge L, Lukuka A, et al. Severity of outcomes associated to illnesses funded by GFATM initiative and socio demographic and economic factors associated with HIV/AIDS, TB and malaria mortality in Kinshasa Hospitals, DRC. Ethiop J Health Sci 2014; 24(4): 299-306. http://dx.doi.org/10.4314/ejhs.v24i4.4 DOI: https://doi.org/10.4314/ejhs.v24i4.4

Doll R, Hill AB. Mortality in relation to smoking: ten years' observations of British doctors. Br Med J 1964a; 1(5395): 1399-1410. http://dx.doi.org/10.1136/bmj.1.5395.1399 DOI: https://doi.org/10.1136/bmj.1.5395.1399

Doll R, Hill AB. Mortality in relation to smoking: ten years’ observations of British doctors. Br Med J 1964b; 1(5396): 1460-1467. http://dx.doi.org/10.1136/bmj.1.5396.1460 DOI: https://doi.org/10.1136/bmj.1.5396.1460

Frame S, Moore J, Peters A, Hall D. Maternal height and shoe size as predictors of pelvic disproportion: an assessment. Br J Obstet Gynaecol 1985; 92(12): 1239-1245. http://dx.doi.org/10.1111/j.1471-0528.1985.tb04869.x DOI: https://doi.org/10.1111/j.1471-0528.1985.tb04869.x

Mahmood TA, Campbell DM, Wilson AW. Maternal height, shoe size, and outcome of labour in white primigravidas: a prospective anthropometric study. BMJ 1988; 297(6647): 515-517. http://dx.doi.org/10.1136/bmj.297.6647.515 DOI: https://doi.org/10.1136/bmj.297.6647.515

Heal CF, Buettner PG, Cruickshank R, et al. Does single application of topical chloramphenicol to high risk sutured wounds reduce incidence of wound infection after minor surgery? Prospective randomized placebo controlled double blind trial. BMJ 2009; 338: a2812. http://dx.doi.org/10.1136/bmj.a2812 DOI: https://doi.org/10.1136/bmj.a2812