November 2020
A Note About Evaluation and Research
Before we continue, one note about how research and evaluation can differ. In general, data we collect for evaluation is used to support operations – efficiency, decision-making, planning, etc. – and is not used for other purposes. As a result, some of the tools used in research are not necessary for use in evaluation, namely using randomization to study the effect of a particular program or intervention, and a control group of students not participating in those programs and interventions.
If the research is our way to understand and communicate best practices, policies, and new knowledge with the community of gifted education parents and professionals, then evaluation is how we turn the mirror to ourselves, and reflect on the work that we do as an organization. Both sets of work involve data, our students, and our stakeholders, but through them we work to accomplish different goals.
What is Program Evaluation?
Program evaluation is both art and science. It’s science as it uses research methods and techniques to gather and analyze data and its an art as it is driven by things like mission, values, and our interpretation of those elements. Program evaluation is part of a family of evaluation strategies that can look at processes, outcomes, and impacts. Broadly defined, an evaluation is
A process of critical examination, involving the collection and analysis of information about activities, characteristics, and/or outcomes, in more to make judgments, improve (effectiveness), and/or inform decision-making.
(Patton 1987)
This definition emphasizes the science of program evaluation. It’s grounded in a cyclical process, which includes periods of analysis and informed decision-making. Evaluation is also a determination of whether or not an initiative has delivered the intended and expected outcomes (ICAP, 2012). This touches on the art. We might look at data have several impact stories we could tell, choosing which to amplify with different audiences. We might have more questions than answers. In some instances, our determination of effectiveness or impact might be challenged.
Here’s a fun example that can show the interplay between science and art when doing an evaluation. In this exercise, an entomologist rates ant emoji used in various apps and platforms. In this example, we can clearly see the interplay between science and art. As an entomologist, curlicuecal has specific standards each emoji must meet. It’s clear from the analysis that realism is an important factor in the final ratings. Ants must have realistic bodies, legs, and antennae. Nevertheless, the feelings that a particular emoji evokes contribute significantly to its final rating; the highest rated ants have a beauty that are absent from those rated lower.
As a result, we as an audience know that curlicuecal rates aesthetic highly, if not as highly as accuracy. And while we might want to argue with them about the beauty of a particular emoji, we (unless we are also entomologists studying ants) are not able to argue with him about the accuracy of the drawings. Or at least, we shouldn’t.
In the next blog post, we’ll look at the tensions that can exist in the work of program evaluation, like how the beauty of an ant emoji is interpreted, and how using tools like program theory can create a positive framework from which to build an evaluation.
Cardinal Evaluation & Consulting, LLC 