The word ‘evaluation’ has at its core the concept of ‘value’.
Evaluations involve making an assessment of the value of interventions, initiatives or innovations. They provide information about the worth or merit of that activity, and often underpin decisions to change, or cease to fund, that activity.
Although evaluations are powerful tools, the process of defining and agreeing the values which will be used to arrive at evaluative judgements is one of the less systematic parts of an evaluation.
The role of values in evaluation was a key theme at this year’s International Evaluation Conference held in Launceston in August, of which ACIL Allen was proud to be a gold sponsor. Many speakers stressed the importance of understanding and being explicit about values in all stages of the evaluation process – from the organisational or program ‘values’ as an input to the logic model, to the ‘value’ delivered by the program.
One of the first steps to conducting an evaluation is to determine what ‘success’ looks like. In other words, if the activity is working as intended what will we be able to see? Success is often explicitly stated in program aims or policy objectives, but in reality may be experienced very differently by different stakeholders. Evidence of success for some, may for others indicate a failure.
Values help determine what success looks like and are particularly useful when translated into criteria, or indicators, or standards. Evaluation rubrics are one way of working collaboratively with stakeholders to build a framework that provides a clear, shared understanding of what success looks like from varying points of view. Rubrics help evaluators move beyond linear thinking about the relationship between activities, outcomes and impact, and to capture all the attributes of value as experienced by stakeholders.
Evaluation requires careful thought about measuring not just the tangible but also the intangible and tacit benefits. The latter can often be significant but difficult to quantify, especially in public and social policy settings where there is an increasing need to consider cross-sectoral and transformational factors, and many of the derived benefits or value may have an indirect relationship with the stated outputs of the program.
Multi-criteria analysis is typically used as the 'go-to' for assessing a range of tangible and intangible value measures. But this approach fails to recognise that intangible value by its nature does not usually ‘add-up’ in the same way that conventional financial or economic value does. It is also important to understand the ways in which the different value attributes can interrelate and influence each other.
When it comes to evaluation, it is important to be aware of these competing value attributes and whenever feasible make them transparent. The best program evaluations will explicitly consider and reconcile different values and, in turn, be able to deliver more holistic assessments of the relevance, effectiveness and efficiency of an activity.
ACIL Allen has a well-defined process for first classifying, and then understanding and combining, all the attributes of value being delivered—that is, assessing the total value not just the value that is easy to count.