On the Relationship between the Reliability and Accuracy of Bio-Behavioral Diagnoses: Simple Math to the Rescue

Dom Cicchetti

doi:10.6000/1929-6029.2015.04.02.2

Authors

Dom Cicchetti Department of Biometry, Yale University School of Medicine, New Haven, CT 06520, USA

DOI:

https://doi.org/10.6000/1929-6029.2015.04.02.2

Keywords:

Binary Diagnoses, Diagnostic Reliability, Diagnostic Accuracy.

Abstract

An equivalence between the J statistic (Jack Youden, 1950) and the Kappa statistic (K), Cohen (1960), was discovered by Helena Kraemer (1982). J is defined as: [Sensitivity (Se) + Specificity (Sp)] - 1. The author (2011) added the remaining two validity components to the J Index, namely, Predicted Positive Accuracy (PPA) and Predicted Negative Accuracy (PNA). The resulting D Index or D = [(Se + Sp) + (PPA + PNA) - 1] / 2. The purpose of this research is to compare J and D as estimates of K, using both actual and simulated data sets. The actual data consisted of ratings of clinical depression and self-reports of gonorrhea. The simulated data sets represented binary diagnoses when the percentages of Negative and Positive cases were: (Identical; Slightly varying; Mildly varying; Moderately varying; or Markedly varying diagnostic patterns, For both the diagnosis of clinical depression, and the self-reports of gonorrhea, D produced closer approximations to Kappa. For the simulated data, under both identical and slightly different patterns of assigning Negative and Positive binary diagnoses, K, D and J produced identical results. While J produced acceptably close values to K under the condition of Mild discrepancies in the proportions of Negative and Positive cases, D continued to more closely approximate K. While D more closely estimated K under Markedly varying diagnostic patterns, D produced values under this extreme condition that were closer than would have been predicted. The significance of these findings for future research is discussed.

Author Biography

Dom Cicchetti, Department of Biometry, Yale University School of Medicine, New Haven, CT 06520, USA

Biometry

References

Kraemer HC. Estimating false alarms and missed events from interobserver agreement: Comment on Kaye. Psychol Bull 1982; 92: 749-754. http://dx.doi.org/10.1037/0033-2909.92.3.749 DOI: https://doi.org/10.1037/0033-2909.92.3.749

Youden WJ. J Index for rating diagnostic tests. Cancer 1950; 3: 32-35. http://dx.doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3 DOI: https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3

Cicchetti DV. On the reliability and accuracy of the Evaluative Method for identifying evidence-based practices in Autism. In: Reichow B, Doehring P, Cicchetti DV, Volkmar F, Eds. Evidence-based practices and treatments for children with Autism. New York, NY: Springer, 2011; pp. 41-51. http://dx.doi.org/10.1007/978-1-4419-6975-0_3 DOI: https://doi.org/10.1007/978-1-4419-6975-0_3

Schimmel MS, Kaplan M, Soll Rf. Blood transfusion in the neonate- Where are we today? In: Peterson BR, Ed. New developments in blood transfusion research. New York, NY: Nova Science 2006; pp. 1-15.

Feinstein AR. Clinimetrics. New Haven CT: Yale University Press, 1987. DOI: https://doi.org/10.2307/j.ctt1xp3vbc

Fleiss JL, Levin B, Cho Paik M. Statistical methods for rates and proportions. New York, NY: Wiley, 2003. http://dx.doi.org/10.1002/0471445428 DOI: https://doi.org/10.1002/0471445428

Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Coming to terms with the terms of risk. Arch Gen Psychiat 1982; 54: 337-343. http://dx.doi.org/10.1001/archpsyc.1997.01830160065009 DOI: https://doi.org/10.1001/archpsyc.1997.01830160065009

Nelson L, Cicchetti DV. Validity of the MMPI Depression Scale for outpatients. Psychol Assess 1991; 3: 55-59. http://dx.doi.org/10.1037/1040-3590.3.1.55 DOI: https://doi.org/10.1037/1040-3590.3.1.55

Niccolai LM, Kershaw TS, Lewis JB, Cicchetti DV, Ethier KA, Ickovics J. Data collection for sexually transmitted disease diagnoses: A comparison of self-reports, medical record reviews, and state health department reports. Annals Epidemiol 2005; 15: 236-242. http://dx.doi.org/10.1016/j.annepidem.2004.07.093 DOI: https://doi.org/10.1016/j.annepidem.2004.07.093

Cohen J. A coefficient of agreementfor nominal scales. Educ Psychol Meas 1960; 23: 37-46. http://dx.doi.org/10.1177/001316446002000104 DOI: https://doi.org/10.1177/001316446002000104

Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull 1969; 72: 323-327. http://dx.doi.org/10.1037/h0028106 DOI: https://doi.org/10.1037/h0028106

Cicchetti DV, Fleiss JL. Comparison of the null distributions of kappa and the C ordinal statistic. Applied Psychol Meas 1977; 1: 195-201. http://dx.doi.org/10.1177/014662167700100206 DOI: https://doi.org/10.1177/014662167700100206

Cicchetti DV. Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Applied Psychol Meas 1981; 5: 101-104. http://dx.doi.org/10.1177/014662168100500114 DOI: https://doi.org/10.1177/014662168100500114

Cicchetti DV, Volkmar F, Klin A, Showalter D. Diagnosing Autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychol 1995; 1: 26-37. http://dx.doi.org/10.1080/09297049508401340 DOI: https://doi.org/10.1080/09297049508401340

Cicchetti DV, Sparrow SS. Developing criteria for establishing interrater reliability of specific items: Applications to assessments of adaptive behavior. Amer J Mental Deficiency 1981; 86: 127-137.

Landis JR, Koch GG. The measure of observer agreement for categorical data. Biometrics 1977; 33: 159-174. http://dx.doi.org/10.2307/2529310 DOI: https://doi.org/10.2307/2529310

Cicchetti DV, Fontana A, Showalter D. Establishing reliability when multiple examiners evaluate a single case- Part II: Applications to symptoms of Post-Traumatic Stress Disorder (PTSD). Internat J Stat Med Research 2014; 3.