BetterEvaluation: 8 Tips for Good Evaluation Questions

BEQuestionsFrom BetterEvaluation.org’s great weekly blog comes a post that has value for facilitators, not just evaluators! Week 28: Framing an evaluation: the importance of asking the right questions.

First let me share the tips and the examples from the article (you’ll need to read the whole article for full context), and then in blue I’ll add my facilitator contextual comments!

Eight tips for good evaluation questions:

  1. Limit the number of main evaluation questions to 3-7. Each main evaluation question can include sub-questions but these should be directly relevant for answering the main question under which they fall. When facilitating, think of each question as a stepping stone along a path that may or may not diverge. Questions in a fluid interaction need to reflect the emerging context. So plan, but plan to improvise the next question.

  2. Prioritize and rank questions in terms of importance. In the GEM example, we realized that relevance, effectiveness, and sustainability were of most importance to the USAID Mission and tried to refine our questions to best get at these elements. Same in facilitation!

  3. Link questions clearly to the evaluation purpose. In the GEM example, the evaluation purpose was to gauge the successes and failures of the program in developing and stabilizing conflict-affected areas of Mindanao. We thus tried to tailor our questions to get more at the program’s contributions to peace and stability compared to longer-term economic development goals. Ditto! I have to be careful not to keep asking questions for my OWN interest!

  4. Make sure questions are realistic in number and kind given time and resources available. In the GEM example, this did not take place. The evaluation questions were too numerous and some were not appropriate to either the evaluation methods proposed or the level of data available (local, regional, and national). YES! I need to learn this one better. I always have too many. 

  5. Make sure questions can be answered definitively. Again, in the GEM example, this did not take place. For example, numerous questions asked about the efficiency/cost-benefit analysis of activity inputs and outputs. Unfortunately, much of the budget data needed to answer these questions was unavailable and some of the costs and benefits (particularly those related to peace and stability) were difficult to quantify. In the end, the evaluation team had to acknowledge that they did not have sufficient data to fully answer certain questions in their report. This is more subtle in facilitation as we have the opportunity to try and surface/tease out answers that may not be clear to anyone at the start. 

  6. Choose questions which reflect real stakeholders’ needs and interests. This issue centers on the question of utility. In the GEM example, the evaluation team discovered that a follow-on activity had already been designed prior to the evaluation and that the evaluation would serve more to validate/tweak this design rather than truly shape it from scratch. The team thus tailored their questions to get more at peace, security, and governance issues given the focus on the follow-on activity. AMEN! YES!

  7. Don’t use questions which contain two or more questions in one. See for example question #6 in the attached—“out of the different types of infrastructure projects supported (solar dyers, box culverts, irrigation canals, boat landings, etc.), were there specific types that were more effective and efficient (from a cost and time perspective) in meeting targets and programmatic objectives?” Setting aside the fact that the evaluators simply did not have access to sufficient data to answer which of the more than 10 different types of infrastructure projects was most efficient (from both a cost and time perspective), the different projects had very different intended uses and number of beneficiaries reached. Thus, while box culverts (small bridge) might have been both efficient (in terms of cost and time) and effective (in terms of allowing people to cross), their overall effectiveness in developing and stabilizing conflict-affected areas of Mindanao were minimal. Same for facilitation. Keep it simple!

  8. Use questions which focus on what was achieved, how and to what extent, and not simple yes/no questions. In the GEM example, simply asking if an activity had or had not met its intended targets was much less informative than asking how those targets were set, whether those targets were appropriate, and how progress towards meeting those targets were tracked. Agree on avoiding simple yes/no unless of course, it is deciding if it is time to go to lunch. 

I’m currently pulling together some materials on evaluating communities of practice, and I think this list will be a useful addition. I hope to be posting more on that soon.

By the way, BetterEvaluation.org is a great resource. Full disclosure, I’ve been providing some advice on the community aspects! But I’m really proud of what Patricia Rogers and her amazing team have done.

Vectors of Learning

SharonI’m just back from a week in Nairobi, Kenya, with a group of amazing practitioners doing a wide variety of community based development work across Africa. They are masters of building value chains, community based learning, rural finance and many other domains. We gathered to spend four days expanding their practice of supporting communities of practice and networks of learning online. For me, these are yet another vector for learning.

Due to the travel, I missed the first week of my Acumen sponsored MOOC (Massively Open Online Course) on Human Centered Design. I have been one of those “enroll but never do anything” people and hoped the F2F gathering group here in Seattle would pull me in. I still have my fingers crossed. It’s about vectors for learning.

So I was delighted today to be pointed to a great post on “Charlie’s Blog: To Notice and to Learn” (Hat tip Stephen Downes). Charlie shares a reflection from a humanities professor on the depth of engagement in the online discussion threads of his MOOC, “The Fiction of Relationship” (Coursera link). This line summing things up from Charlie grabbed me.

Lifelong learning is a bouquet of flowers that we must gather and arrange ourselves, and MOOCs are the stem of new type of flower, on which beautiful new petals might blossom.

via A Heartfelt Note from a Humanities MOOC Professor | Charlie’s Blog – To Notice and to Learn.

The bouquet, if we follow the metaphor, is rich with possibilities. With people’s time more fractured than ever, there is a seemingly growing disbelief that we can meaningfully engage, build trust and relationships, learn, work and play — even in asynchronous discussion threads. That promise is there. It has been there for a long time. What we only need to add is our time, care and attention.  WE have the vectors. Now let’s learn.

 

 

IgniteSeattle21: S#*!, is it time to think about a legacy?

In May I had the great privileged of giving my second Ignite Seattle talk (the first one was at the second ever Ignite). The video hit YouTube today (thanks to Bootstrapper Studios) I’ll let it speak for itself. Slides are also embedded below just for fun.

via Nancy White – S#*!, is it time to think about a legacy? – YouTube.

And the slides…

[slideshare id=21289545&doc=igniteseattle-nancywhite-legacies-130516194147-phpapp01]

Edit July 2: See Rob Cottingham’s talk which has some resonance with mine… and he is a LOT funnier! http://robcottingham.ca/2013/07/if-i-knew-then-what-i-know-now-my-presentation-to-the-iabcbc-event/

Openness is Easy: Recipe Sharing, Knowledge Sharing

yummy - that's my pieceA while back someone wanted a copy of a recipe from me. I wanted to find (re-find!) a recipe. And I decided to create a little email group where I could email myself recipes, and then easily find them later. I shared with my sister and mom. A few others joined when they heard, but it has stayed a fairly small circle.

Today, in my typical Friday morning work avoidance, I was following links from Twitter to cooking blogs, copying recipes into the list and thought, HEY, I SHOULD SHARE THIS! After all, a bit part of my work is about knowledge sharing. So I went in and made the postings public. You can see them here:

99+ Recipe Circle – Google Groups.

The significance of these few, little clicks is that this sort of thing happens all the time. We have our private or small group piles of good stuff, of information, of knowledge, and yet we don’t even THINK of making it open. Hey, it is EASY. So just do it. Find one treasure you can make open today. Open it up. Let your friends, networks, even the world know about it.

I you would like to be added to the list, let me know (and what email address you’d like me to use). And maybe find one of these great recipes and cook up something loving and wonderful for friends and family this weekend!

Community events: dinner parties and rock concerts

A month ago I facilitated the fourth of six workshops I’m doing for a client on (online) communities of practice. The topic was “community heartbeats,” with a focus on the role of events as heartbeats. Of course, I tried to pack too much into an hour and one topic we never got to was thinking about the strategic differences of small, intimate events (a.k.a. the dinner party) and big, elaborate events (a.k.a. the rock concert). I promised to blog something as follow up because I have been thinking about some heuristics around how to decide what kind of event is useful at any particular time in a community’s life, particularly communities of practice. So here are a few thoughts, along with a question for you: what kind of events (online and offline) have been particularly useful and generative for your communities (of practice or whatever!)

IMG_0200

Small Events – “Dinner Parties” – Connection

You know those intimate gatherings, 4, 6 or 8 people, a well planned menu, attention to atmosphere and, most importantly, thinking about the chemistry between people. When friends introduce friends, connections can be activated in a flash. We walk in the room strangers, we walk out as friends, telling stories.

When our community goal is to create critical connections, small events can be the most effective way to “close triangles” between people. The intimacy creates context. Of course, good food and wine help too. But one things of a four way Skype call as a virtual small event, we can still bring that kind of care. How we introduce people, what the invitation looks like … more than “I want you two to meet.” Plant the seeds of possibility in the invitation by sharing what the individuals have in common. Come with a great question to start the conversation. And with online events – more than with F2F events – consider some follow up contact. The “magic” sometimes takes more than one virtual contact.

IMG_4277

Medium Events – “Workshops” – Action

Medium sized events are the workhorses of communities.  This is where the action happens. Groups of 7 – 50 (often broken into subgroups) are the places to get things done: focused learning, team work, thinking together, tackling tough problems or opportunities and planning. Online or F2F, we can think about workshops, planning meetings, consultations and other gatherings, all time delimited and focused. This is a key to the productivity. They can’t be too long or too broad.

A distinction that I think IS worth making here is that it is too easy to make these into content delivery events, moving participants to the role of audience rather than community members. There are some great resources for rethinking HOW we do these, such as Liberating Structures, GroupWorks Deck, World Cafe and Open Space — all of which focus on participant ownership and engagement of the interaction. If I could make one strong recommendation about these “medium” events, it would be to move straight content delivery into videos and save the “face time ” (online or F2F) for designs that fully engage everyone.

IMG_0403Big Events – “Rock Concerts” – Community Identity

There is something about a “happening.” The crush of the crowd. The energy that groups of over 150 generate. It is often far less about the community’s domain or content, and more a statement of identity of the group. In full force, our parties rock our organizations. In full force, we may influence in a way that is actually larger than our numbers when they are dispersed across the organization. We create presence. We celebrate. We see both what we have in common and how we are different — both very powerful community aspects. Our “rock concerts” should not be everyday affairs. Once a year, or even less , can create the feeling both for the community, and it’s visibility to the wider organization or world.

What did I miss? Misinterpret?