PAC 302 Module 3 Reliability Example

Do Our Counselors Agree? Internal Consistency, Kappa and Intraclass Correlation in an Agency Reliability Study

Student Name

Psychology and Addiction Studies Program, Aspen University

PAC 302: Assessment Procedures in Addiction Studies

Instructor Name

Month Day, Year

Do Our Counselors Agree? Internal Consistency, Kappa and Intraclass Correlation in an Agency Reliability Study

After proposing a more structured intake, as described in Module 1, Lakeview Counseling and Recovery, the composite Grand Rapids agency in these papers, wanted to know whether its existing tools and judgments were consistent before changing them. Its clinical director asked three questions. Does the agency's ten-item alcohol screen measure consistently? When two counselors interview the same client, do they reach the same diagnosis? And when counselors rate the severity of a client's substance problem on a scale from one to ten, do their ratings agree? A three-month pilot gathered data to answer them. This paper explains reliability and interprets the pilot's results. The pilot numbers and the staff are made up, though the measurement research is genuine.

What Reliability Means

Reliability is consistency: the degree to which a measurement gives similar results under conditions where the thing being measured has not changed. Every score contains some error, and different kinds of reliability address different sources of error. Internal consistency asks whether the items of a scale agree with one another. Test-retest reliability asks whether scores are stable over time. Interrater reliability asks whether different raters reach the same judgment. Alternate forms reliability asks whether different versions of a test give similar scores. Reliability is necessary for validity, since a measure that gives different answers each time cannot accurately measure anything, but it is not sufficient: a scale can measure the wrong thing very consistently.

Internal Consistency and Coefficient Alpha

Cronbach (1951) introduced coefficient alpha, which estimates internal consistency from the correlations among a test's items. He showed that alpha equals the average of all possible split-half reliability estimates and argued that it is a useful index of how much a test's items share a common core. He also cautioned against overinterpretation. A high alpha does not prove that a test measures a single trait, and alpha rises with the number of items, so long tests can show high alpha even when their items are only moderately related.

Lakeview's ten-item screen, given to one hundred clients at intake, had an alpha of 0.84. For comparison, Reinert and Allen (2002), reviewing research on the AUDIT, the World Health Organization's ten-item alcohol screen, found that its internal consistency was generally high across many studies and populations, typically in the 0.80 range, and that its test-retest reliability was also good. Lakeview's screen performs at a similar level.

Agreement on Diagnoses and Cohen's Kappa

Two counselors independently interviewed and diagnosed fifty clients, classifying each as having no substance use disorder, a mild disorder or a moderate to severe disorder. They agreed on forty of the fifty, or eighty percent. Percent agreement, however, overstates reliability, because some agreement occurs by chance. Cohen (1960) proposed the coefficient kappa, which subtracts the agreement expected by chance from the observed agreement and divides by the maximum possible agreement beyond chance. Kappa equals observed agreement minus chance agreement, divided by one minus chance agreement.

Based on how often each counselor used each category, chance agreement was fifty percent. Kappa is therefore 0.80 minus 0.50, divided by one minus 0.50, which is 0.60. Commonly used benchmarks describe values between about 0.40 and 0.60 as moderate and values between 0.60 and 0.80 as substantial. The counselors' agreement sits at the border.

Agreement on Severity and the Intraclass Correlation

Three counselors watched thirty recorded intakes and rated each client's severity from one to ten. For continuous ratings like these, the appropriate statistic is the intraclass correlation. Shrout and Fleiss (1979) described six forms of the intraclass correlation and explained that the right one depends on the study's design: whether every client is rated by the same raters, whether those raters are a sample of a larger pool to whom results should generalize and whether the reliability of a single rater or of the average of several raters is of interest. Because Lakeview's counselors are a sample of its staff, all three rated every client and in practice a single counselor will rate each new client, the correct form is the two-way random effects intraclass correlation for a single rater. It was 0.58.

The Pilot Results

Interpreting the Results

The pattern is clear. The standardized screen is reliable, both internally and over time. Counselors' judgments, especially their severity ratings, are less consistent. A severity rating that depends substantially on which counselor does the intake is a problem when that rating helps decide level of care. Benchmarks are guides, not rules, and the stakes matter: research on groups can tolerate moderate reliability, but decisions about individual clients require higher consistency.

Recommendations

Three changes follow. First, the agency should replace the open severity rating with a structured set of criteria, such as the number of DSM-5 criteria met, withdrawal history and prior treatment, so that counselors rate the same features. Second, counselors should meet monthly to rate a recorded intake together and discuss disagreements, a calibration practice that improves interrater reliability. Third, the diagnostic interview should follow a structured format covering each criterion. A second pilot after six months would test whether kappa and the intraclass correlation have improved.

Conclusion

Lakeview's three questions required three different reliability statistics. Cronbach's alpha showed that the screen's items hang together, Cohen's kappa showed that diagnostic agreement was moderate once chance was removed and Shrout and Fleiss's framework identified the right intraclass correlation for severity ratings, which proved the weakest link. Reinert and Allen's review showed that the screen performs like established instruments. Structured criteria and calibration offer a path to making counselors' judgments as reliable as the tools they use.

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Reinert, D. F., & Allen, J. P. (2002). The Alcohol Use Disorders Identification Test (AUDIT): A review of recent research. Alcoholism: Clinical and Experimental Research, 26(2), 272-279. https://doi.org/10.1111/j.1530-0277.2002.tb02534.x

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. https://doi.org/10.1037/0033-2909.86.2.420