There has been a great discussion on the KM4Dev mailing list the last 10 days or so about evaluation, impact and measurement. In the context of international development, this is critical. Why do something if it doesn’t make a difference. However, often we don’t do a very good job figuring out what does make a difference, let alone know why (causality.) Dave Snowden posted something that just rang the bell for me. I hastily copied it down to share here. The link to the web archive of the email discussion is at the bottom, if you want to mine for the rest of the thread. Emphasis is mine.
The linear concept of input, leading to outputs, leading to outcomes which in turn leads to impact is I think at the heart of the problem, It implies (and I can see why people would want this) a causal chain that can be replicated.
However if the system is complex (in the sense of complex adaptive) then any input is a stimulus or modulator which influences but does not determine impact. That means we need to start measuring the sensitivity of a system to different stimuli, and the way in which some stimuli produce a disproportionate effect in that they catalyze other inputs. This is newly developing area which has not hit the development sector yet, but we are working on it in related fields, loosely termed modulator mapping. It also leads us to evolutionary representations (such as fitness landscapes) and measure based on stability of landscapes. In all those cases mathematics are simplified by representation and linked micro-narratives. There is no point in measuring anything if the results do not convince both donors and recipients alike to take action
All of that moves the “impact” agenda on. I didn’t confuse outputs and outcomes, I conflated them as the model means there is no real difference in what is measured in practice.
via Discussions.
I am now going to start paying attention to this idea of “measuring sensitivity of a system to different stimuli.” This relates closely to two projects I’m working on where I have been sensing this, but hadn’t had the words for it. Now I have a toehold. Onward!
Photo credit:
