DNP 825 Module 5 Data Quality and Electronic Quality Measures Example

When the Measure Is Wrong, Not the Care: Validating an Electronic Depression Screening Measure in a Primary Care Network

Student Name

Doctor of Nursing Practice Program, Aspen University

DNP 825: Health Information Management and Informatics

Instructor Name

Month Day, Year

When the Measure Is Wrong, Not the Care: Validating an Electronic Depression Screening Measure in a Primary Care Network

Electronic clinical quality measures are calculated automatically from structured data in the electronic health record. They promise quality reporting without manual chart review, and they increasingly determine payments and public ratings. But an electronic measure is only as accurate as the data behind it. This paper describes a composite primary care network whose electronic depression screening measure showed poor performance, explains how validation revealed that most of the problem lay in the data rather than in care, reviews evidence on the accuracy of electronic measures and the dimensions of data quality, and sets out a plan to correct the measure.

The Problem

The network's quality dashboard reported that only 41% of eligible adult patients had been screened for depression with a standardized tool and, if positive, had a documented follow-up plan. The rate placed the network in the lowest quartile of its accountable care organization and put part of a quality bonus at risk. Clinic managers were asked to improve screening, but nurses and medical assistants insisted that they screened nearly everyone at annual visits. A doctoral nurse leading quality work proposed validating the measure before changing practice.

The pressure to act quickly was strong, since the bonus decision was only months away.

What Validation Found

A manual review of 200 randomly selected eligible patients found that 68% had been screened with a standardized tool and, where positive, had a follow-up plan documented somewhere in the record. The difference arose from where and how data were recorded. In many visits, the screening questionnaire was completed on a paper form and scanned, or recorded in a free-text note, neither of which the measure logic could read. Some clinics used a questionnaire version that was not mapped to the value set the measure required. And follow-up plans were often documented in the note rather than in the structured field the measure used. Actual care was better than the reported rate, although at 68% it still had room to improve.

Evidence on Accuracy

The network's experience is common. A study comparing electronically reported clinical quality measures with manual chart review in a community practice found that the sensitivity of electronic reporting ranged from 46% to 98% across measures. For appropriate asthma medication, electronic reporting found recommended care in 38% of patients, compared with 77% on manual review, and for pneumococcal vaccination 27% compared with 48%; for cholesterol control in diabetes, electronic reporting overestimated performance, 57% compared with 37%. The authors warned that such variation threatens the validity of incentive programs (Kern et al., 2013).

The stakes extend beyond payments. A study of one academic health system found that it reported on 162 unique quality metrics, requiring about 108,000 person-hours and more than $5 million in personnel costs in one year, and that electronic metrics consumed far fewer resources per metric than claims-based or chart-abstracted ones (Saraswathula et al., 2023). Electronic measures can therefore save substantial effort, but only if they are accurate.

Dimensions of Data Quality

A review of methods for assessing electronic health record data quality identified five dimensions: completeness (is the value there), correctness (is it true), concordance (do two sources match), plausibility (is the value believable) and currency (was it entered soon enough to be useful). It also noted that data quality depends on the task for which the data are used (Weiskopf & Weng, 2013). In the depression measure, the main failure was completeness of structured data, although the information existed elsewhere in the record, together with concordance problems between the questionnaire version and the measure's value set.

The Correction Plan

The plan addresses data capture, mapping and practice. First, the paper form is retired and the standardized questionnaire is completed electronically at check-in on a tablet, with scores flowing into structured fields. Second, the informatics team maps every questionnaire version in use to the required value set and removes unmapped versions. Third, a structured follow-up plan field is added to the visit template with common options, such as referral to the behavioral health clinician or medication started, so that clinicians can document plans in one step. Fourth, a monthly validation sample of 30 records compares the electronic measure with manual review until agreement exceeds 90% for three consecutive months, then quarterly.

Only after the data are fixed will the network target practice improvement for the 32% of patients who were genuinely not screened or had no follow-up plan, focusing on visits other than annual examinations, where screening was least consistent.

Communicating the Findings

How validation results are shared matters. Clinic staff had been told for months that their screening rate was poor, and some felt accused. The doctoral nurse presented the validation results at each clinic's staff meeting, beginning with the finding that actual screening was far higher than reported and thanking staff whose documentation made the review possible. The message was that the measure had failed them, and that fixing it required their help with the new tablet workflow and structured follow-up field.

The same results went to the accountable care organization with a request to review the network's reported performance, supported by the validation method and sample. Whatever the organization decided about the past year, the discussion established the network as a partner that validates its data, which strengthens its position in future measure disputes. Internally, the quality committee adopted a rule that any electronic measure used for incentives or public reporting will be validated against a manual sample before targets are set.

Conclusion

An electronic quality measure reported 41% performance where manual review found 68%, because screening data were captured in places the measure could not read. Evidence shows that such discrepancies are common and can run in either direction, and that accurate electronic measures could greatly reduce the cost of quality reporting. Assessing data quality across its dimensions, fixing capture and mapping, and validating regularly allow a network to trust its measures and then direct improvement to the care that truly needs it.

References

Kern, L. M., Malhotra, S., Barrón, Y., Quaresimo, J., Dhopeshwarkar, R., Pichardo, M., Edwards, A. M., & Kaushal, R. (2013). Accuracy of electronically reported "meaningful use" clinical quality measures: A cross-sectional study. Annals of Internal Medicine, 158(2), 77-83. https://doi.org/10.7326/0003-4819-158-2-201301150-00001

Saraswathula, A., Merck, S. J., Bai, G., Weston, C. M., Skinner, E. A., Taylor, A., Kachalia, A., Demski, R., Wu, A. W., & Berry, S. A. (2023). The volume and cost of quality metric reporting. JAMA, 329(21), 1840-1847. https://doi.org/10.1001/jama.2023.7271

Weiskopf, N. G., & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: Enabling reuse for clinical research. Journal of the American Medical Informatics Association, 20(1), 144-151. https://doi.org/10.1136/amiajnl-2011-000681