Category Archives: Uncategorized

Why Do Program Evaluation?

Let’s start with why organizations put energy and resources into doing program evaluation. Historically, there has been a tension between doing program evaluation and being a part of a program that is seen as providing a positive good or service. Unlike program evaluation of a clinical practice or a housing policy, for example, there was a belief that doing well – or at the very least doing no harm – is sufficient and that nonprofits didn’t need to embrace program evaluation.

Nevertheless, program evaluation is a tool nonprofits use to better understand themselves, their stakeholders, and, in certain environments, more specifically communicate their impact by:

• Ensuring effective programs are maintained
• Monitoring progress toward programs goals
• Determining if desired outcomes are being met
• Comparing outcomes across stakeholder groups
• Identifying opportunities for continuous improvement
• Measuring effective use of resources
• Advocating for additional support, including financial support.

(List adapted from the CDC and MEERA.)

Let’s go back to the example from the last blog post using the ant emoji. Why might an entomologist want to undertake an evaluation of emoji? In this particular example, it appears that this scientist wants to clearly tell certain designers and developers that there is a right and a wrong way to present an ant as an emoji. And they would like to inform those who work is not up to par how to improve. This emphasizes both why program evaluation is a valuable and why it is cyclical. Those who may want to improve their emoji rating have clear action items. And those who do improve could likely be improve their rating on a future evaluation. They have, in essence, applied what they learned through evaluation to make improvements.

Similarly, organizations undertake evaluation in order to engage in continuous improvement of our programs and services. Doing so requires so to continually ask about experiences – both the highlights and the points where things could go more smoothly. Evaluation is done to ensure that programs live up mission and values, such as creating high-caliber experiences that promote learning, expand community services, or better community health.

Tension in Program Evaluation: Art vs. Science

Even with the best of intentions to learn from program evaluation, there are still tensions in doing this work. One of the major tensions in program evaluation draws on its roots in both science and art: do organizations and stakeholders interpret the concepts being evaluated or central to the evaluation the same way?

For some concepts – those that are routed in science or in a well-known idea – it is much easier for stakeholders and organizations to be on the same page.  Consider, for example, a program designed to help participants become faster and better runners. In evaluating whether or not the program was successful, it would likely not be difficult to know if the runners became faster. With a good evaluation plan, the organization might be able to measure faster each participant does (or does not) become by comparing times for particular distances before and after the program period. And with good program data, it would be hard to argue over “faster” as time is a clearly measurable and discreet unit.

However, we may have trouble agreeing on a definition of better. What does it mean to be a better runner? For some, better and faster may be synonymous while for others being a better runner might entail having better form, developing a more consistent pace, developing a routine or habit for running, or even enjoying running more. As a result, an important part of determine if the program has made runners better is not establish a clear definition of what better means to the organization, and why, in order to engage stakeholders in this aspect of the conversation.

The Tools Program Evaluation Can Use

In order to have clear definitions of what more abstract goals or outcomes mean, organizations can rely on tools such as a program theory or logic model to lay out their goals, communicate them stakeholders, and create consistency about desired outcomes.

A program theory is similar to a roadmap: it lays out what an organization wants to do and how it will get there. Thus a program theory encompasses both a theory of change (usually creating or influencing a specific change or action) and a theory of action (through activities, practices, programs, etc.).  Similarly, it is a program’s theory of action, or how it will do its work, that leads to its theory of change (the desired outcome).

Thinking about our fictional running program, we might imagine that the organization has articulated a program theory that emphasizes the importance of running – perhaps improving personal health – while also addressing how it will improve perceptions of running – perhaps something about running as accessible to everyone, that with good form and/or knowledge about how to run people will find it more enjoyable, or some similar combination. It’s theory of change builds on this idea by committing to work toward a larger community of runners who run not only because of the health benefits but run primarily because they enjoy it.

Components of the theory of action and theory of change can be brought together in a program logic model, where an organization lays out is why and how. Those why elements (the theory of change) include:

• the context in which an organization is operating,
• the underlying issue the organization seeks to address, and
• the desired impact an organization hopes to have.

Similarly, the how elements of a program logic model (or the theory of action) include:

• the organization’s partner and stakeholders
• the inputs, or materials and capital, put into a program or effort
• the environments, actions or processes in which a program or effort occurs.

The theory of action and theory of change meet through a program or organization’s outputs and outcomes. An important distinction here is the difference between an output and an outcome. In program evaluation, we use output to describe those tangible or quantifiable results of a program or effort, such as time it might take a runner to travel one mile. In contrast, we use outcomes to describe changes in knowledge, behavior or attitude as short-, medium-, and long-term participation in a program or effort.

A Fictional Example of Program Theory

Here’s a complete example for our fictional program:

I Can Run Program Theory
Context
The I Can Run Program is an all ages athletic program focused on improving an individual’s ability – their speed, pace, form – and their appreciation for running through group activities and coaching.
Underlying Issue
Attitudes and perceptions about running vary widely. While many individuals find joy and a sense of accomplishment in running, others do not understand how to make running an enjoyable exercise, thus losing out on the tremendous health benefits of this activity. Developing a good running habit can contribute to better individual physical and mental health.
Stakeholders
Community members/athletes/runners, coaches & trainers, health care professionals, physical education teachers.
Inputs	Environments, Actions, Processes	Outputs	Outcomes	Desired Impact
Community members, coaching professionals, expert-planned curriculum and exercises, appropriate gear (sneakers, time pieces, etc.), etc.	Indoor running programs, outdoor trails and routes, routes of increasing distance and duration, group runs and coaching sessions, individual coaching, stretching and cool down programs	Runners will develop better speed. Runners will run more frequently. Runners will develop a distance or route preference for their running. Runners will develop warm-p and cool-down routines.	Runners will have more knowledge of the science of running, how to select appropriate gear, develop good form, and improve their gait and pace. Runners will report that they derive value from the exercise – joy, sense of calm, or sense of relaxation.	TIPP will build a community of runners who run for the love of being active and healthy, and who know how to run with good form, pace, sharing their knowledge and joy with others.

When an organization is able to articulate its program theory, stakeholders and audiences should be able to see a clear progression from the underlying issue to the desired impact. We might infer from this logic model that the coaches hired for this program support the organization’s perception of the underlying issue and its desired impact. As a result, we imagine that the coaches would work with the community members to move toward more desirable output and outcomes through a program that address runners’ concerns and builds their skill levels. Through this intervention, we could imagine that someone interested in running could more toward being a better runner and finding things they enjoy about the activity.

How Our Community Can Use This Information

A well-constructed program theory signals why a certain organization is the best suited to work toward the desired impact. There clear measures of what the organization will consider success. As a result, those who interact with an organization or participate in a program will know how success is defined – where the art and the science will come together.

This allows stakeholders to ask questions about the work of the organization and allows organizations to engage with stakeholders as they evaluate programs. For external stakeholders, such questions can include:

• Do I understand how this organization will work for change? Do I agree that this approach makes sense based on my experiences?
• Do I know how the program defines success? Is it clear to me what a good outcome is for a program participant? Do I agree with the program’s established decisions?
• If I don’t agree, do I have a way to ask questions of the organization to gain clarity about their perspective?
• If I do agree, do I understand how the metrics will be gathered based on the information in program theory?

In an upcoming blog post about program evaluation, we will explore this last question more deeply and will look at how the final these questions address another tension: knowing when to use data to make a change and when data might signal an unusual or exceptional event or experience. We will also talk generally about how organizations might undertake a program evaluation using various tools and instruments, such as surveys, focus groups, or participant interviews.

November 2020

A Note About Evaluation and Research

Before we continue, one note about how research and evaluation can differ. In general, data we collect for evaluation is used to support operations – efficiency, decision-making, planning, etc. – and is not used for other purposes. As a result, some of the tools used in research are not necessary for use in evaluation, namely using randomization to study the effect of a particular program or intervention, and a control group of students not participating in those programs and interventions.

If the research is our way to understand and communicate best practices, policies, and new knowledge with the community of gifted education parents and professionals, then evaluation is how we turn the mirror to ourselves, and reflect on the work that we do as an organization. Both sets of work involve data, our students, and our stakeholders, but through them we work to accomplish different goals.

What is Program Evaluation?

Program evaluation is both art and science. It’s science as it uses research methods and techniques to gather and analyze data and its an art as it is driven by things like mission, values, and our interpretation of those elements. Program evaluation is part of a family of evaluation strategies that can look at processes, outcomes, and impacts. Broadly defined, an evaluation is

A process of critical examination, involving the collection and analysis of information about activities, characteristics, and/or outcomes, in more to make judgments, improve (effectiveness), and/or inform decision-making.

(Patton 1987)

This definition emphasizes the science of program evaluation. It’s grounded in a cyclical process, which includes periods of analysis and informed decision-making. Evaluation is also a determination of whether or not an initiative has delivered the intended and expected outcomes (ICAP, 2012). This touches on the art. We might look at data have several impact stories we could tell, choosing which to amplify with different audiences. We might have more questions than answers. In some instances, our determination of effectiveness or impact might be challenged.

Here’s a fun example that can show the interplay between science and art when doing an evaluation. In this exercise, an entomologist rates ant emoji used in various apps and platforms. In this example, we can clearly see the interplay between science and art. As an entomologist, curlicuecal has specific standards each emoji must meet. It’s clear from the analysis that realism is an important factor in the final ratings. Ants must have realistic bodies, legs, and antennae. Nevertheless, the feelings that a particular emoji evokes contribute significantly to its final rating; the highest rated ants have a beauty that are absent from those rated lower.

As a result, we as an audience know that curlicuecal rates aesthetic highly, if not as highly as accuracy. And while we might want to argue with them about the beauty of a particular emoji, we (unless we are also entomologists studying ants) are not able to argue with him about the accuracy of the drawings. Or at least, we shouldn’t.

In the next blog post, we’ll look at the tensions that can exist in the work of program evaluation, like how the beauty of an ant emoji is interpreted, and how using tools like program theory can create a positive framework from which to build an evaluation.

November 2020

Welcome to Cardinal Evaluation & Consulting, LLC!

My goal is to help organizations working in the educational and nonprofit space become data-informed decision-makers and strategic planners.

Why?

As financial landscapes become increasing competitive, and there are more organizations and institutions competing for grant money, philanthropic, and donor support, organizations that are successful at accessing those resources are those who can articulate what they are doing, what impact their work has, and where their work will go.

I can help your organization develop a cohesive and comprehensive narrative – your story and your impact.

I do this through a iterative partnership and process. If you don’t have an articulated program theory, I can help you write one. Together, your organization and I can link your vision to the steps you are taking. For there, we can identify how you will learn about your community of clients and partners, what data you need, and how you can collect it it a way that reflects your mission and values.

I can work with you and all of your stakeholders to goal-set, either in a retreat setting for your short-term gain, or through a longer, and more consultative, process of writing a strategic plan.

I want organizations that do good work in their communities to thrive – and to do that, my experience has shown me that blending data with goals, and stories with action is the best way to create the impactful narrative that organizations need to bring in fundraising money and grants.

My role is to be a facilitator and interpreter – to help you gather and understand the data your organization has, needs, and wants to act on. We’re partners, working together to get you where you want to go.

Let’s get started.