Did It Change the Bedside? Evaluating High-Flow Oxygen Training by Its Effect on Patient Care
Student Name
Master of Science in Nursing Program, Aspen University
N584: Curriculum Development, Implementation and Evaluation
Instructor Name
Month Day, Year
Did It Change the Bedside? Evaluating High-Flow Oxygen Training by Its Effect on Patient Care
Continuing education is often evaluated with a satisfaction form and a post-test, which show whether learners liked a session and remembered its content but not whether patient care changed. The high-flow nasal oxygen program developed in this course was planned around a performance outcome: that nurses on both participating medical-surgical units would document respiratory rate and a combined oxygenation index at least every two hours for patients on high-flow therapy and escalate according to unit criteria. This paper presents an evaluation plan designed to show whether that outcome is achieved and whether patients benefit, using an outcomes framework built for continuing education.
The Evaluation Framework
Moore et al. (2009) described a framework of expanded outcomes for continuing education that begins with participation and satisfaction, moves through declarative knowledge, procedural knowledge and competence, and ends with performance in practice, patient health and community health. Its value lies in distinguishing what learners know, what they can show they can do in a controlled setting, and what they actually do in practice, levels that often diverge. It also encourages planners to design the evaluation at the same time as the program, so that the data needed at the higher levels are collected from the start. This plan measures six of the seven levels; community health is beyond the scope of a unit-based program.
Measures at Each Level
The table lists the measure, timing and target at each level.
| Level | Measure | Timing | Target |
|---|---|---|---|
| Participation | Percentage of eligible nurses completing both parts | Weekly during rollout | 90% by week ten |
| Satisfaction | Three-item survey on relevance, clarity and intent to change | After each session | 85% agree or strongly agree |
| Declarative and procedural knowledge | Online module post-test, including index calculation and trend interpretation | End of module | Mean at least 85%, compared with the 71% needs assessment baseline |
| Competence | Simulation checklist: recognizes rising work of breathing, calculates index, escalates by criteria, communicates in structured format | During simulation | 90% of nurses complete all critical items |
| Performance | Chart audit of respiratory rate and index documentation every two hours; review of escalations | Monthly for six months | 85% of audited patient-hours documented, from 12% baseline for the index |
| Patient health | Rapid response calls, unplanned ICU transfers and emergency intubations among high-flow patients; delayed escalation safety reports | Quarterly for one year | No delayed escalations; transfers occurring earlier in deterioration |
Analyzing Change in Practice
The performance measure is the most important and the most vulnerable to misinterpretation. A single before-and-after comparison can mislead, because documentation might have been improving anyway or might fluctuate from month to month. The plan therefore uses statistical process control. Benneyan et al. (2003) described control charts and run charts as tools for distinguishing common cause variation, the ordinary month-to-month noise in a process, from special cause variation, which points to a genuine change, and for monitoring whether improvements are sustained. Monthly documentation rates for the six months before the program, drawn from a retrospective chart audit, and for six months afterward will be charted month by month, with the program and the documentation prompt marked as interventions. A sustained shift above the baseline median, meeting standard run chart rules, would indicate that the change is real rather than chance.
Because the electronic documentation prompt goes live during the program, the evaluation cannot fully separate the effect of education from the effect of the prompt. The plan accepts this: the program was designed with the prompt as part of the intervention, and the aim is to improve practice, not to isolate the effect of teaching alone. Comparing documentation by nurses who completed the program with those who had not yet done so during the rollout weeks will give a partial indication of the education's contribution.
Patient Outcomes and Their Limits
Patient outcomes are the ultimate reason for the program but the hardest to attribute. High-flow patients on two units are relatively few, perhaps several hundred a year, so rates of intubation or intensive care transfer will vary considerably by chance. The plan therefore focuses on process measures closely linked to outcomes, such as the time from the first documented index below threshold to escalation and the proportion of intensive care transfers that occur before rather than after an emergency, and reviews every rapid response and transfer among high-flow patients in a monthly case review. Any delayed escalation will be reviewed in depth, both as a safety event and as information for refining the program.
Balancing measures guard against unintended harm. If nurses escalate more readily, rapid response calls may rise, which is acceptable if calls occur earlier and lead to timely care, but it could strain the rapid response team. The evaluation will therefore track the number of rapid response calls for high-flow patients and the team's workload, and ask the team whether calls were appropriate. It will also monitor whether patients are transferred to intensive care unnecessarily. The aim is earlier, appropriate escalation, not more escalation for its own sake.
Reporting and Using Results
Results at each level will be reported to the unit practice councils and nurse managers monthly during the first six months and quarterly afterward. The evaluation is not only a verdict on the program but a guide to improving it: if competence is high in simulation but performance in the chart remains low, the barrier lies in the work environment or the record rather than in knowledge, and the response should target those. If implementation measures from the previous module, such as penetration and fidelity (Proctor et al., 2011), show low participation on nights, weak results on nights should be read in that light rather than as a failure of the content.
Conclusion
An evaluation that stops at satisfaction and a post-test cannot show whether a continuing education program changed the bedside. By following the high-flow program through participation, knowledge, simulated competence, charted performance and patient outcomes, with baselines, targets and run chart analysis of practice data, this plan can show whether the program achieved the outcome it was built for and where to improve it if it did not. Evaluating at the level of performance is harder, but it is the only way to know whether nurses on these units now see and act on failing high-flow therapy in time.
References
Benneyan, J. C., Lloyd, R. C., & Plsek, P. E. (2003). Statistical process control as a tool for research and healthcare improvement. Quality and Safety in Health Care, 12(6), 458-464. https://doi.org/10.1136/qhc.12.6.458
Moore, D. E., Green, J. S., & Gallis, H. A. (2009). Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions, 29(1), 1-15. https://doi.org/10.1002/chp.20001
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7
How this N 584 Module 7 example is structured
Aspen does not publish N584 module prompts, so check your classroom for the exact instructions. This example explains why satisfaction and post-tests are not enough, introduces an outcomes framework for continuing education, sets measures and targets at each level in a table, analyzes practice change with statistical process control, addresses patient outcome limits and explains how results will be used.
N584 Module 7 questions, answered
What does N584 Module 7 usually ask for?
N584 includes the evaluation of the continuing education program being developed, so an evaluation plan with measures at several levels is a typical late-course assignment. Check your classroom for the framework required.
What is the difference between competence and performance?
Competence is what a learner can demonstrate in a controlled setting such as simulation. Performance is what the learner actually does in practice, which is shaped by the work environment as well as knowledge and skill.
Why use a run chart instead of a before-and-after comparison?
A run chart shows the pattern over time, which distinguishes real, sustained change from ordinary month-to-month variation that a single before-and-after comparison can mistake for improvement.
Write yours, or have the desk draft it
This paper is an original model document written by our desk, not a submitted student paper and not an official Aspen University document. Read it for the moves, then write your own to the instructions in your classroom. If you want one built to your exact prompt and rubric, the first custom sample is free and arrives in 24 to 48 hours.