Please note, this post was originally published on ArtsFwd.org
A strategy for embedding a developmental evaluation approach requires paying attention to relationships, and individual and organizational meaning-making, culture and practice. While the specifics of any evaluation will depend on context, these four principles provide a place to start as you think about integrating a developmental evaluation approach into your organization’s work.
Build trusting relationships
The success of any developmental evaluation depends on trusting relationships between the evaluator and other team members. Team members and stakeholders need to feel comfortable being honest about their experiences (including challenges) and having the evaluator share back trends in that learning. Moreover, being a trusted team member can support being heard and included in strategic decision making. Being transparent about the process, aims, and expectations; developing and maintaining clear, open lines of communication; listening intentionally; supporting participatory and collaborative work; and creating authentic connections with team members are essential to building these relationships. This inevitably requires time and intentional energy.
Make assumptions visible, and continually re-visit them
Learning is an ongoing process. It is important to continuously establish clear snapshots of team members’ assumptions and perceptions about the context and strategy. One way to make this thinking visible is to create a Theory of Change alongside team members (for a detailed resource to create a Theory of Change, see the Center for Evaluation Innovation’s Strategic Learning in Practice toolkit). A Theory of Change lays out the logic of how strategies contribute to desired outcomes. It also highlights the assumptions that underlie this logic. The connections and gaps within the Theory of Change become the basis for an evaluation plan, which is continually adjusted based on learning. Doing this as a team supports a common understanding of the strategy and evaluation. It also helps to tie the evaluation plan more closely to strategic priorities, making it easier to feed learning into strategic decisions and operations.
One assumption that we can forget to question is team members’ vision of evaluation’s role. Having a shared understanding of the role of evaluation is essential to the effectiveness of the learning strategy. This includes clarifying: what they expect to learn; how data will be collected; how, when, why and with whom findings are shared; and the role of evaluation in strategic decision making. Because developmental evaluation is an approach that differs significantly from more conventional approaches, it is possible that the team’s expectations do not align with developmental evaluation principles. In this case, part of the evaluator’s role will become actively creating buy-in around the approach, over time.
One way of surfacing assumptions within a team is through interviews. Interviewing is a strategy to support critical thinking and reflection. Having people articulate themselves allows individuals to become aware of patterns in their thinking, their assumptions, and how they map onto organizational approaches and expectations. Making these patterns explicit creates the foundation for developing realistic, shared expectations.
Pay attention to process
Collecting data on process is essential to tease apart the relationship between strategy and outcomes. This includes observing how strategies are developed and implemented. Moreover, defining quality process will be part of the patterns that the evaluator is paying attention to. This may include paying attention to programmatic elements but also relational aspects of your work that can be measured and tied to quality process. These processes will be continuously adapted as the interactions between the strategy and context become clearer.
Integrating learning into everyday activity
Evaluation is often treated as a parallel process, which happens alongside the implementation of a strategy. Feedback forms are given at the end of a workshop; formal reports are generated at the end of a process; and interviews are scheduled as separate activities. This has the disadvantage of evaluation feeling like an “add-on,” which requires extra time and effort. It can also isolate the responsibility of being aware of what’s being learned and developing learning strategies to the evaluator. There are two simple considerations that support integrating evaluation into a team’s broader approach.
First, sharing learning in real-time can tighten and clarify the connection between what was learned, what needs to be learned, operational realities, and strategic decision making. Bringing tactical discussions back to the strategic level by asking reflective questions, questioning how decisions are being made, and highlighting what has already been learned, is part of the feedback loops the developmental evaluator is working to support.
Second, data collection can be embedded into opportunities where stakeholders are already coming together. After determining what you want to learn and from whom, you can systematically integrate learning questions into every day discussions or debriefs that can be recorded as observation data. One important consideration when people share their reflections in a group setting is to be aware of power dynamics in the room:
- Do people feel comfortable sharing their perspective?
- Is an authority figure influencing what people share?
- Who is choosing to speak up?
As an evaluator, you can mitigate this by being explicit about the context in your records and supplementing this approach with other strategies.