PAC 302 Module 4 Validity Example

Motivated Enough for Treatment? A Validity Review of an In-House Score Used to Ration Intensive Care

Student Name

Psychology and Addiction Studies Program, Aspen University

PAC 302: Assessment Procedures in Addiction Studies

Instructor Name

Month Day, Year

Motivated Enough for Treatment? A Validity Review of an In-House Score Used to Ration Intensive Care

Five years ago, counselors at Lakeview Counseling and Recovery, the made-up agency used in these papers, wrote a twelve-item questionnaire called the Commitment Score. Clients rate statements such as "I will do whatever it takes to stay sober" and "I am ready to attend treatment four days a week." Lakeview has twelve intensive outpatient slots and a waiting list, and the Commitment Score is now used to allocate them: clients scoring thirty or above are admitted, and those below thirty wait for an opening. A new clinical director asked whether this use is valid. This paper reviews the evidence. The agency, the scale and its data are composites made for teaching, and the research is real.

Validity as a Property of Uses

Validity is often described as whether a test measures what it claims to measure. Modern measurement theory frames it more precisely: validity refers to the degree to which evidence and theory support the interpretations and uses of test scores. A test is not simply valid or invalid. The Commitment Score might support one use, such as starting a conversation about motivation, and fail to support another, such as deciding who receives care. The question is whether the evidence supports using scores below thirty to delay treatment.

Construct Validity

Cronbach and Meehl (1955) introduced construct validity for tests that measure theoretical attributes, such as motivation, which cannot be directly observed. They proposed that a construct is defined by its place in a network of laws, the nomological network, that links it to other constructs and to observable behavior. Validating a test means testing the predictions that the network implies. If the Commitment Score measures motivation for treatment, it should correlate with other measures of readiness to change, predict attendance and engagement and be relatively unaffected by things unrelated to motivation.

Convergent and Discriminant Evidence

Campbell and Fiske (1959) proposed the multitrait-multimethod approach to construct validity. They argued that a valid measure should show convergent validity, correlating substantially with other measures of the same trait, and discriminant validity, correlating less with measures of different traits. They also warned of method variance: measures can correlate because they share a method, such as self-report questionnaires completed at the same sitting, rather than because they measure the same trait.

Lakeview gathered data from eighty clients. The Commitment Score correlated 0.31 with a validated readiness-to-change questionnaire, a modest correlation for two measures intended to capture the same construct. It correlated 0.48 with a brief rating of how much clients liked their intake counselor, completed at the same time. And it correlated negative 0.36 with a depression questionnaire. By Campbell and Fiske's logic, the pattern is a warning: the score relates more strongly to rapport with the counselor than to an established measure of readiness, which suggests that it may partly measure how clients feel about the intake rather than their motivation.

Messick's Unified View

Messick (1995) proposed a unified concept of validity with six aspects. The content aspect concerns whether items represent the construct. The substantive aspect concerns whether the processes test takers use match the theory. The structural aspect concerns whether the internal structure of scores fits the construct. The generalizability aspect concerns whether interpretations hold across groups, settings and tasks. The external aspect concerns relationships with other measures and behaviors. And the consequential aspect concerns the intended and unintended consequences of score use, including whether they are fair. Messick argued that consequences are part of validity because a test used in ways that systematically harm groups for reasons unrelated to the construct fails as a basis for decisions.

The Consequences of Using the Commitment Score

Lakeview's waiting list data show that clients with moderate or severe depression scored on average six points lower on the Commitment Score than other clients and were twice as likely to be placed on the waiting list. Depression often brings hopelessness, which may lower agreement with statements like "I will do whatever it takes," but it does not mean a person is less in need of intensive care; often it means the opposite. The score's negative correlation with depression and its use to allocate care combine to delay treatment for clients who may need it most.

The Evidence Summarized

Why In-House Scales Are Tempting

In-house scales like the Commitment Score are common in addiction programs, and the reasons are understandable. They are free, they reflect what experienced staff believe matters and they feel tailored to the agency's clients. But the qualities that make them attractive also make them risky. Items written from staff experience may capture staff assumptions, such as the belief that a motivated client will agree to attend four days a week, rather than the construct itself. Without the validation work that published instruments undergo, an agency cannot know what its scale measures or for whom it works, and the longer a scale is used, the more its scores come to feel like facts.

Recommendation

The evidence does not support using the Commitment Score to allocate intensive outpatient slots. Lakeview should suspend this use. Slots could instead be allocated by clinical need, using structured criteria such as severity, withdrawal risk and co-occurring conditions, with first-come order among clients of similar need. The Commitment Score could continue as a conversation tool in motivational counseling, where a low score invites exploration rather than exclusion. If Lakeview wishes to keep a motivation measure for any decision, it should conduct a proper validation study, including whether scores predict attendance and whether they function similarly for clients with and without depression.

Conclusion

The Commitment Score was written with good intentions, but its use to ration intensive care is not supported. Cronbach and Meehl explain that a motivation score must fit a network of predictions, Campbell and Fiske show that its stronger link to rapport than to readiness undermines its claim to measure motivation and Messick shows that its consequences for clients with depression are part of its validity. Allocating care by clinical need, and studying the score before using it for decisions, would bring Lakeview's practice in line with measurement theory.

References

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. https://doi.org/10.1037/h0046016

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741