DNP880 Module 6 assignment: evaluation plan: analysis of process, outcome and balancing measures, a full sample

Rules Before Results: How a Food Insecurity Screening Project Will Judge Its Own Data

Student Name

Doctor of Nursing Practice Program, Aspen University

DNP880: DNP Project

Instructor Name

Month Day, Year

Rules Before Results: How a Food Insecurity Screening Project Will Judge Its Own Data

An evaluation plan states in advance how the project will decide whether the change worked. Writing it before implementation protects against the temptation to choose the analysis that makes results look best. This plan describes how the Maple Street project will analyze its process, outcome, and balancing measures, what thresholds will count as success, and what the design can and cannot show.

Aims of the Evaluation

The evaluation has three aims. The first is to determine whether weekly screening rates rose from baseline and stayed high during implementation at each clinic. The second is to determine whether patients who screened positive were connected to food, measured as referral completion. The third, exploratory, is to describe any change in A1c among patients who screened positive, while monitoring visit length to detect harm to clinic flow.

Why Run Charts

Improvement projects collect data over time while changes are being made, so the analysis must show when change happened as well as whether it did. A run chart plots a measure over time with its median and allows a team to detect nonrandom patterns using simple probability-based rules (Perla et al., 2011). Comparing a single before value with a single after value would hide whether improvement came early, faded, or coincided with something other than the intervention. Run charts also suit a team without statistical software, because they can be built in a spreadsheet and read at the weekly meeting. A fuller treatment of charting improvement data, including when to move from run charts to control charts once enough points accumulate, is available to the team if later cycles need it (Provost & Murray, 2011).

Run Chart Rules Fixed in Advance

Each primary measure will be plotted weekly for each clinic against a centerline set at the baseline median. Four patterns will count as evidence that something beyond chance is happening (Perla et al., 2011). A shift is a streak of at least six weekly values all higher, or all lower, than the centerline. A trend is a climb or fall sustained across five or more consecutive values. A runs test flags a chart that crosses the centerline far more or far less often than a published table allows for its number of points. The astronomical-point rule covers a single value plainly out of line with every other, one that needs no counting to see. Values exactly on the median are ignored when counting streaks and climbs. The rules are written down now so that the chart is read the same way whether the news is good or bad.

Annotating the Charts

Each chart will be annotated with the dates of events that could explain changes: the start of each PDSA cycle, training sessions, community health worker absences, the food bank's schedule changes, and holidays that shorten clinic weeks. This lets the team link signals to specific changes and recognize outside events that might mimic or mask an effect. The annotation log will be kept alongside the charts and reproduced in the final paper.

Targets That Define Success

The project will be judged successful on process if both clinics show a shift above the baseline median in screening rate and reach a median of at least 85 percent over the final four weeks. It will be judged successful on referral if at least 60 percent of referred patients receive food by the two-week call, the prediction set in Cycle 3. These targets were chosen with the team before implementation and are recorded in the project charter. If a target is missed, the final paper will report it as missed and examine why, rather than redefine success.

Exploratory A1c Analysis

For patients who screened positive and have A1c values both before and after the project, the change will be summarized with the mean and median difference and analyzed with a paired t test, or a Wilcoxon signed-rank test if the differences are not approximately normal. Because the sample will be small, the follow-up short, and no comparison group exists, any change will be described as an observation for future study, not as an effect of the project. The analysis will also report how many positive-screen patients lacked a follow-up A1c, since patients who return for testing may differ from those who do not.

Balancing and Equity Checks

Median visit length will be charted weekly; a shift above the baseline median of more than two minutes will prompt the team to review rooming workflow. Screening and referral rates will also be compared by preferred language and by age group at the end of the project, so that a high overall rate does not hide a group that is being skipped. Subgroups with fewer than 10 patients will be combined or suppressed, as the ethics submission requires.

Using Patient Comments and Barrier Data

The run charts show how much changed; the follow-up forms help explain why. Barriers recorded at the two-week call, such as transportation, distribution hours, or a patient choosing not to go, will be tallied by category each month and displayed as a simple ordered bar chart so that the team can see which barrier accounts for most missed referrals. Volunteered patient comments will be grouped into themes by the student and reviewed by the site mentor to check that the grouping is reasonable. These summaries will not be tested statistically. Their role is to guide the next PDSA cycle and to give the final paper an account of the change from the patient's side, which the evidence synthesis found largely missing from published work.

If the barrier tally shows that one problem, for example transportation, accounts for most missed referrals, the plan already provides a response: the clinic produce box pickup tested in a later cycle. The evaluation is therefore connected to action, not only to reporting.

What the Design Cannot Show

This is a before-and-after improvement project without a control group. It can show whether screening and referral processes changed in time with the intervention and whether that change held, but it cannot prove that the intervention caused a change in A1c or in food security itself, and secular trends, such as changes in public food benefits during the project, could influence referral completion. The final paper will report results following the SQUIRE 2.0 guidelines, which ask authors to describe the context and the limits of what the study can support (Ogrinc et al., 2016).

Conclusion

The evaluation will read weekly run charts with four signal rules fixed in advance, annotate them with the timing of each change, judge success against targets recorded before implementation, treat A1c as exploratory with a paired analysis, and check for harms and gaps by subgroup. Together, these steps allow the project to report what changed and how confidently, without claiming more than its design supports.

References

Ogrinc, G., Davies, L., Goodman, D., Batalden, P., Davidoff, F., & Stevens, D. (2016). SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): Revised publication guidelines from a detailed consensus process. BMJ Quality & Safety, 25(12), 986-992. https://doi.org/10.1136/bmjqs-2015-004411

Perla, R. J., Provost, L. P., & Murray, S. K. (2011). The run chart: A simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1), 46-51. https://doi.org/10.1136/bmjqs.2009.037895

Provost, L. P., & Murray, S. K. (2011). The health care data guide: Learning from data for improvement. Jossey-Bass.