Using Program Theory to Guide Art and Science: An Introduction and Overview of Program Evaluation

Why Do Program Evaluation?

Let’s start with why organizations put energy and resources into doing program evaluation. Historically, there has been a tension between doing program evaluation and being a part of a program that is seen as providing a positive good or service. Unlike program evaluation of a clinical practice or a housing policy, for example, there was a belief that doing well – or at the very least doing no harm – is sufficient and that nonprofits didn’t need to embrace program evaluation.

Nevertheless, program evaluation is a tool nonprofits use to better understand themselves, their stakeholders, and, in certain environments, more specifically communicate their impact by:

• Ensuring effective programs are maintained
• Monitoring progress toward programs goals
• Determining if desired outcomes are being met
• Comparing outcomes across stakeholder groups
• Identifying opportunities for continuous improvement
• Measuring effective use of resources
• Advocating for additional support, including financial support.

(List adapted from the CDC and MEERA.)

Let’s go back to the example from the last blog post using the ant emoji. Why might an entomologist want to undertake an evaluation of emoji? In this particular example, it appears that this scientist wants to clearly tell certain designers and developers that there is a right and a wrong way to present an ant as an emoji. And they would like to inform those who work is not up to par how to improve. This emphasizes both why program evaluation is a valuable and why it is cyclical. Those who may want to improve their emoji rating have clear action items. And those who do improve could likely be improve their rating on a future evaluation. They have, in essence, applied what they learned through evaluation to make improvements.

Similarly, organizations undertake evaluation in order to engage in continuous improvement of our programs and services. Doing so requires so to continually ask about experiences – both the highlights and the points where things could go more smoothly. Evaluation is done to ensure that programs live up mission and values, such as creating high-caliber experiences that promote learning, expand community services, or better community health.

Tension in Program Evaluation: Art vs. Science

Even with the best of intentions to learn from program evaluation, there are still tensions in doing this work. One of the major tensions in program evaluation draws on its roots in both science and art: do organizations and stakeholders interpret the concepts being evaluated or central to the evaluation the same way?

For some concepts – those that are routed in science or in a well-known idea – it is much easier for stakeholders and organizations to be on the same page.  Consider, for example, a program designed to help participants become faster and better runners. In evaluating whether or not the program was successful, it would likely not be difficult to know if the runners became faster. With a good evaluation plan, the organization might be able to measure faster each participant does (or does not) become by comparing times for particular distances before and after the program period. And with good program data, it would be hard to argue over “faster” as time is a clearly measurable and discreet unit.

However, we may have trouble agreeing on a definition of better. What does it mean to be a better runner? For some, better and faster may be synonymous while for others being a better runner might entail having better form, developing a more consistent pace, developing a routine or habit for running, or even enjoying running more. As a result, an important part of determine if the program has made runners better is not establish a clear definition of what better means to the organization, and why, in order to engage stakeholders in this aspect of the conversation.

The Tools Program Evaluation Can Use

In order to have clear definitions of what more abstract goals or outcomes mean, organizations can rely on tools such as a program theory or logic model to lay out their goals, communicate them stakeholders, and create consistency about desired outcomes.

A program theory is similar to a roadmap: it lays out what an organization wants to do and how it will get there. Thus a program theory encompasses both a theory of change (usually creating or influencing a specific change or action) and a theory of action (through activities, practices, programs, etc.).  Similarly, it is a program’s theory of action, or how it will do its work, that leads to its theory of change (the desired outcome).

Thinking about our fictional running program, we might imagine that the organization has articulated a program theory that emphasizes the importance of running – perhaps improving personal health – while also addressing how it will improve perceptions of running – perhaps something about running as accessible to everyone, that with good form and/or knowledge about how to run people will find it more enjoyable, or some similar combination. It’s theory of change builds on this idea by committing to work toward a larger community of runners who run not only because of the health benefits but run primarily because they enjoy it.

Components of the theory of action and theory of change can be brought together in a program logic model, where an organization lays out is why and how. Those why elements (the theory of change) include:

• the context in which an organization is operating,
• the underlying issue the organization seeks to address, and
• the desired impact an organization hopes to have.

Similarly, the how elements of a program logic model (or the theory of action) include:

• the organization’s partner and stakeholders
• the inputs, or materials and capital, put into a program or effort
• the environments, actions or processes in which a program or effort occurs.

The theory of action and theory of change meet through a program or organization’s outputs and outcomes. An important distinction here is the difference between an output and an outcome. In program evaluation, we use output to describe those tangible or quantifiable results of a program or effort, such as time it might take a runner to travel one mile. In contrast, we use outcomes to describe changes in knowledge, behavior or attitude as short-, medium-, and long-term participation in a program or effort.

A Fictional Example of Program Theory

Here’s a complete example for our fictional program:

I Can Run Program Theory
Context
The I Can Run Program is an all ages athletic program focused on improving an individual’s ability – their speed, pace, form – and their appreciation for running through group activities and coaching.
Underlying Issue
Attitudes and perceptions about running vary widely. While many individuals find joy and a sense of accomplishment in running, others do not understand how to make running an enjoyable exercise, thus losing out on the tremendous health benefits of this activity. Developing a good running habit can contribute to better individual physical and mental health.
Stakeholders
Community members/athletes/runners, coaches & trainers, health care professionals, physical education teachers.
Inputs	Environments, Actions, Processes	Outputs	Outcomes	Desired Impact
Community members, coaching professionals, expert-planned curriculum and exercises, appropriate gear (sneakers, time pieces, etc.), etc.	Indoor running programs, outdoor trails and routes, routes of increasing distance and duration, group runs and coaching sessions, individual coaching, stretching and cool down programs	Runners will develop better speed. Runners will run more frequently. Runners will develop a distance or route preference for their running. Runners will develop warm-p and cool-down routines.	Runners will have more knowledge of the science of running, how to select appropriate gear, develop good form, and improve their gait and pace. Runners will report that they derive value from the exercise – joy, sense of calm, or sense of relaxation.	TIPP will build a community of runners who run for the love of being active and healthy, and who know how to run with good form, pace, sharing their knowledge and joy with others.

When an organization is able to articulate its program theory, stakeholders and audiences should be able to see a clear progression from the underlying issue to the desired impact. We might infer from this logic model that the coaches hired for this program support the organization’s perception of the underlying issue and its desired impact. As a result, we imagine that the coaches would work with the community members to move toward more desirable output and outcomes through a program that address runners’ concerns and builds their skill levels. Through this intervention, we could imagine that someone interested in running could more toward being a better runner and finding things they enjoy about the activity.

How Our Community Can Use This Information

A well-constructed program theory signals why a certain organization is the best suited to work toward the desired impact. There clear measures of what the organization will consider success. As a result, those who interact with an organization or participate in a program will know how success is defined – where the art and the science will come together.

This allows stakeholders to ask questions about the work of the organization and allows organizations to engage with stakeholders as they evaluate programs. For external stakeholders, such questions can include:

• Do I understand how this organization will work for change? Do I agree that this approach makes sense based on my experiences?
• Do I know how the program defines success? Is it clear to me what a good outcome is for a program participant? Do I agree with the program’s established decisions?
• If I don’t agree, do I have a way to ask questions of the organization to gain clarity about their perspective?
• If I do agree, do I understand how the metrics will be gathered based on the information in program theory?

In an upcoming blog post about program evaluation, we will explore this last question more deeply and will look at how the final these questions address another tension: knowing when to use data to make a change and when data might signal an unusual or exceptional event or experience. We will also talk generally about how organizations might undertake a program evaluation using various tools and instruments, such as surveys, focus groups, or participant interviews.