Notes on Theory of Change approach to Program Evaluation

Source: Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects, and Problems. by James P. Connell and Anne C. Kubrisch Link

A theory of change approach to program evaluation: as a systematic and cumulative study of the links between activities, outcomes, and contexts of the initiative.

The first step toward evaluating a program is to determine its intended outcomes, the activities it expects to implement to achieve those outcomes, and the contextual factors that may have an effect on implementation of activities and their potential to bring about desired outcomes.

We now identify at least three reasons to begin the design and evaluation of a CCI with a good theory of change:

1. A theory of change approach can sharpen the planning and implementation of an initiative. Used during the design phase, it increases the likelihood that stakeholders will have clearly specified the initiative’s intended outcomes, the activities that need to be implemented in order to achieve those outcomes, and the contextual factors that are likely to influence them.

2. With a theory of change in hand, the measurement and data collection elements of the evaluation process will be facilitated.

3. A theory of change specifies, up front, how activities will lead to interim and longer term outcomes and identifies the contextual conditions that may affect them.

   A theory of change approach would seek agreement from all stakeholders that, for example, activities A1, A2, and A3, if properly implemented (and with the ongoing presence of contextual factors X1, X2, and X3), should lead to outcomes O1, O2 and O3; and, if these activities, contextual supports, and outcomes all occur more or less as expected, the outcomes will be attributable to the intervention. Although this strategy cannot eliminate all alternative explanations for a particular outcome, it aligns the major actors in the initiative with a standard of evidence that will be convincing to them.

What is a good theory of change?

It should be plausible. Do evidence and common sense suggest that the activities, if implemented, will lead to desired outcomes?

It should be doable. Will the economic, technical, political, institutional, and human resources be available to carry out the initiative?

It should be testable. Is the theory of change specific and complete enough for an evaluator to track its progress in credible and useful ways?

Is impact evaluation possible?

What sets the social policy evaluation field apart from most of the rest of the scientific world has been its conclusion that all forms of hypothesis-testing are inadequate relative to experimental approaches that include random assignment of units of observation (individuals, institutions, communities) to treatment and control conditions (Hollister and Hill, 1995). The argument made by those who seek to retain this single standard of evidence for impact is that it is the only way to establish a convincing counterfactual.

But, part of the reason for the dominance of the experimentalists is that the magnitude of the change expected to occur as a function of many social interventions has been very small. The smaller the change expected, the more solid the counterfactual must be in order to attribute cause. However, in most current CCIs, the magnitude of change expected in the long-term outcomes is not small. They seek neighborhood transformation and meaningful improvements in individual and family well-being. Whether or not stakeholders agree that these expectations are realistic-that is, whether the theory of change is plausible and doable-should be determined before they are tested in an evaluation. The more significant the change that occurs, we and others (such as Gueron, 1996) would argue, the less the need for airtight counterfactuals to attribute impact to CCIs.