Formative program

A formative evaluation is a method of assessing the value of a program while the program activities are being developed. The main purpose of these evaluations are to catch deficiencies early on in the process so that the proper adjustments can be made before the program is implemented.

These evaluations are conducted when a new program is being developed or when an existing one is being modified. Formative evaluations provide ongoing feedback to the instructional designers to ensure that what is being created meets the learning objectives and the needs of the intended audience. A summative evaluation is a method of assessing the value and effectiveness of a training program at the end of the course or program activities.

The findings are used to help decide whether a program should be continued or modified for improvement. The primary purposes of an evaluation are to assess the processes and outcomes of a specific initiative and to facilitate ongoing program management.

Formative evaluation provides information to guide program improvement, whereas process evaluation determines whether a program is delivered as intended to the targeted recipients Rossi et al. Summative evaluation informs judgments about whether the program worked i.

Outcome evaluation focuses on the observable conditions of a specific population, organizational attribute, or social condition that a program is expected to have changed. Whereas outcome evaluation tends to focus on conditions or behaviors that the program was expected to affect most directly and immediately i.

For example, assessing the strategies used to implement a smoking cessation program and determining the degree to which it reached the target population are process evaluations. Reduction in morbidity and mortality associated with cardiovascular disease may represent an impact goal for a smoking cessation program Rossi et al.

Several institutions have identified guidelines for an effective evaluation. For example, in , CDC published a framework to guide public health professionals in developing and implementing a program evaluation CDC, Although the components are interdependent and might be implemented in a nonlinear order, the earlier domains provide a foundation for subsequent areas. They include:. Five years before CDC issued its framework, the Joint Committee on Standards for Educational Evaluation created an important and practical resource for improving program evaluation.

The Joint Committee, a nonprofit coalition of major professional organizations concerned with the quality of program evaluations, identified four major categories of standards — propriety, utility, feasibility, and accuracy — to consider when conducting a program evaluation. Propriety standards focus on ensuring that an evaluation will be conducted legally, ethically, and with regard for promoting the welfare of those involved in or affected by the program evaluation.

In addition to the rights of human subjects that are the concern of institutional review boards, propriety standards promote a service orientation i. Both approaches can examine how an intervention was implemented, the barriers and facilitators to implementation, and the effects of the intervention on various outcomes.

Although both components can provide feedback on the effectiveness of an intervention and offer ways to improve it, they differ in frequency, aim, and focus.

Formative evaluations stress engagement with stakeholders when the intervention is being developed and as it is being implemented, to identify when it is not being delivered as planned or not having the intended effects, and to modify the intervention accordingly. The stakeholders include payers, clinicians, practice staff, patients and their caregivers, and other decisionmakers. This conceptual distinction was first suggested by Scriven to describe the two main functions of evaluation: 1 to foster development and improvement of a program formative and 2 to assess whether the results of that program met its stated goals summative.

A summative evaluation generally provides feedback to stakeholders at the end of program implementation. In contrast, a formative evaluation focuses attention on ongoing, midstream assessments that feed information back to intervention implementers, allowing them to make real-time adaptations and refinements to ineffective aspects of an intervention.

Formative feedback often leads to decisions about program development such as whether to modify or revise the intervention , whereas summative feedback often leads to decisions about whether to ultimately continue, expand, or adopt the program Worthen, Sanders, and Fitzpatrick, An example can illustrate the difference between the two approaches.

Suppose that a quality improvement intervention allows patients at a primary care practice to obtain laboratory results through a secure Web portal.

Formative evaluations provide findings such as these to practices and program sponsors on an ongoing basis, along with specific recommendations on how to improve patient access. In this example, this information could be used to refine the intervention by sending email alerts to patients when new lab results are entered. In contrast, a summative evaluation might also note the problem, but might wait to do so until the next scheduled summary report.

A summative evaluation would not provide the intervention implementers with timely information or concrete suggestions that they could use to refine the intervention as it unfolds. In practice, most evaluations contain both formative and summative approaches. Implementing complex interventions in complex settings such as the PCMH is a difficult task that requires researchers and program managers to have a clear understanding of what should be implemented, how to best implement a suggested strategy, which elements may hinder or facilitate the implementation process, and why a strategy did or did not work once implemented.

A formative evaluation can provide this information on an ongoing basis as the intervention is being delivered. Stetler, Legro, Smith, et al. Complete a needs assessment. Formative evaluations focus on pre-planning for the intervention design before it is implemented, which Stetler, Legro, Smith, et al. Before the intervention begins, the evaluator conducts a needs assessment about areas where the practice should focus improvements by understanding the context the practice operates in, potential barriers and facilitators to practice change, and the feasibility of implementing the intervention as initially designed.

Below we discuss these three components and the types of information they contribute. Unlike summative evaluations, formative evaluations use these components to deliver information back to intervention implementers, with a certain frequency and intensity, to change the delivery of the intervention itself.

During the intervention, combining implementation and progress analyses can provide a comprehensive assessment of the intervention Stetler, Legro, Smith, et al. Conduct implementation analysis. An implementation-focused analysis assesses discrepancies between the implementation plan and the execution of that plan. This can include assessing fidelity to the implementation strategy and the clinical intervention, understanding the nature and implications of local adaptation, identifying barriers, identifying new intervention components or refining the original strategy to optimize the potential for success, and identifying the critical details necessary to replicate the implementation strategy in other settings.

Data sources might include semistructured interviews with stakeholders, structured surveys, focus groups, direct observations through site visits, document reviews, electronic health records or charts, and management information systems.