Throwing off our results-chains

Here’s something I’ve been meaning to write about: one diagram that’s done more than anything else to stop NGOs truly measuring their performance. You’ll know it if you work in the sector. It’s the results-chain:

The example about teacher training shows why it’s so wrong. Look at it for a second and you can see it’s just not true.

An NGO can train teachers. But anywhere round the world a hundred other factors affect whether students get better education: Do schools have classrooms and textbooks? Can students go to them? Do the teachers go to them? Are teachers well managed and supported?

A thousand other factors affect whether students get better jobs: Do the jobs exist? Do students have the right skills? Do women have to stay at home and look after children? Is economic growth keeping pace with population growth?

And who knows how many factors affect average incomes: employment laws, the number of job-seekers, investment policies, European trade barriers, the international financial crisis – and many others.

Here’s the problem. The diagram shows a set of aspirations. An NGO may hope that training teachers will help improve education and employment. There may sometimes be good grounds to suggest that activities will contribute to specific results.

But when it comes to social change, involving real thinking people living their own complicated lives, it’s by no means certain.

In practice, the results-chain tends to be seen as reality, showing what will definitely happen rather than what we hope might happen. It reinforces the wrong logic, suggesting that social change is predictable, straightforward and mechanical – and that NGOs determine long term changes.

But any field experience shows that it’s not and they don’t.

From the point of view of a results-chain, local people are cogs in the wheels of our projects, expected to behave in certain pre-determined ways – rather than independent people making their own decisions.

The approach pushes NGOs to measure the wrong things, like whether pre-planned activities have all been carried out, what ‘impact’ has occurred and whether there’s an impact for every activity listed in the plan. None of this has any bearing on the quality of the teacher training or whether training teachers is still a good idea – which would probably help the NGO achieve more.

On the other hand, the results-chain has some appealing aspects that have made it popular. It’s easy to understand. It creates a framework for management and action. It sets out a ‘project logic’, apparently linking activities to long term results. It provides a useful summary of goals and intentions. It also suggests that NGOs are important: good news when you’re trying to impress donors.

Our challenge is to come up with alternative frameworks, that are as intuitive and easy to use, meeting the needs of field staff as well as managers, without doing such violence to reality.

Then we can throw off our results-chains and get on with the important business of measuring – and maximising – our performance.

Here’s a people-centred approach I use. You can see how it fits with the analysis on this website. What others are there out there?

8 Responses

  1. Hi,

    At the outset, let me say that I agree completely that the results chain, used as in the example, is simplistic and useless. One of the contributors to this getting adopted has been over-dependence on log frames which are necessarily linear.

    The main problem is not just the complexity that exists out there but the assumptions that the planner makes. When I say assumptions I refer to the underlying rationale for either doing something or an output to translate in to an outcome.

    To take the example you have provided, the reason the planner did a teacher training was the assumption that a better trained teacher would deliver better education. That is a HUGE assumption and almost a leap of faith.

    I have found logic models / theories of change to work better in terms of

    1. determining necessity and sufficiency of interventions and
    2. rationality of assumptions.

    Of course, logic models should also not be static; that would defeat the purpose. The programme needs an in-built process of action-learning-action to increase chances of making a difference.

    Of course for a planner to be able to embed that flexibility needs sound understanding on part of donors, governments and beneficiaries. That could also lead to difficulties but more of that sometime later.

  2. Hi Alex. You are absolutely right. The issue I think you are refering to is called “attribution” as far as I know. How do you “attribute” a change to the efforts of your organisation? Positive (i.e. a successful outcome) change appears to be claimed by many. Negative (i.e. unsuccesful change or unforessen change) appears to be “not our fault”. The answer of course is that the outcome may not be what we all want. The outcome is a product of factors and influences and human decision. We can claim some part of the outcome but not all of it. Interestingly the larger the amount of money spent on a problem the larger the likelihhood “attribution” will be claimed – but this is not necesarily true or fair. Another way of looking at this issue is to do the classic economist analysis of “with and without” rather than the “before and after” which the first model you show is a product of. the “with and without” is difficult to do but likely to produce more interesting results and of course heated debate.

  3. A people-centred approach is good (change happens through people changing in some way) but could we throwing the baby out with the bath water? It seems a reasonable working assumption to make for a NGO that training teachers will improve educational standards. Many things can go wrong with this, of course, but isn’t the alternative to end up not doing anything because we can’t do everything?
    Isn’t the results chain worth continuing to work with but adapting it to include the fuzziness of life, e.g. showing it graphically less as a line with solid arrows and more curves and circles. We need to know how to express uncertainty and (other) external factors in graphical representations of models to avoid them looking mechanistic but do they need to have some degree of specificity to avoid them being too vague to be of use?

  4. Hi Alex. Outcome mapping works right into this space. It is concerned with mapping the contributions of boundary partners that are necessary contributors to envisaged outcomes; and then engaging purposefully with them as a part of one’s programme activities. For those who are unfamiliar with the approach, there is a wealth of reference material and guidance to be found at http://www.outcomemapping.ca

  5. Very thoughtful and challenging. As the “father” so to speak of results frameworks at USAID let me throw out a couple of points which I think supplement yours, but your underlying issue is not wrong – as soon as we become captive of analtyic “rigor” that is too predictive and too rigid, we might as well give up. So the question then is: how exactly DO we develop analytic tools which help you break those chains and yet on the other hand don’t allow you to freefall into the seductive world of “everything is linked to every other thing…”? One advantage of approaches such as OM, I think (still being a novice in that approach) is that it puts the “observer” and his/her “data points” into the game; development after all is not a social science experiment and we are just as involved as anyone else in change (like EH Carr’s point, ages ago, about trying to report on a parade as it winds in and out of city streets, while marching in it..).

    First, a defense or two of what we tried to design into USAID’s systems in 1996, and what we are now trying to bring back, with considerable tweaks:

    RFs were not really designed, at least in my mind, to BE the logic, but just rather a simple way to express it. But as soon as one takes those boxes to BE the reality, the power and utility of the construct is lost. What matters to me are the “lines” that link the boxes NOT the boxes themselves. What we call a development hypothesis. AND like any hypothesis is is what we think might happen, and should be “testable” in some way. In our revision of AID’s programming guidance I think we will not only emphasize the underlying logic of the hypothesis (rather than the graphic) but also its role as a heuristic tool as one implements, not just as a poster board of a project wiring diagram…

    Many if not MOST results in an RF done “well” most likely won’t be your responsibility; I can think of very few, if any, dedvelopment programs where one entity (be it govt, a donor or an NGO) essentially controls all the variables, in fact even the majority of the variables. Likewise, I can think of very few where the “logic” is ironclad and “always this way”. In fact I’d say even the most “obvious” hypothesis (people will want health services if the quality of those services is improved”) should be taken with a grain of salt and open for questioning.

    I personally think it’s nuts for AID to ask contractors and grantees to prepare their own results frameworks, since the more grounded and tactical you go the harder it is to see are the external factors affecting you. I think AID and its partners are shirking their responsibility by dictating to implementing entities to come up with RFs on their own which are supposed to cover work beyond what you yourselves do, or could be responsible for.

    Results frameworks and logframes are not the same thing, at least in my book. logframes are nice tools for deciding how to spend your budget but they tend to trap one in both an iron logic of causality AND put into the “assumptions” column all sort of things which in fact need to be paid attention to. Assuming the sun will come up in the morning is one thing, but assuming that the Govt will cede control over revenue in the forestry sector, as a for instance, is quite another thing. In my mind that’s not an assumption at all but a result; a change in behavior the broader community of folk involved in that objective want to see happen. OM seems to be perfect for thinking that through, but I THINK it’s possible to combine OM with what we intended results frameworks to be.

    Enough for now, but I did want to thank you for raising this; very stimulating perspective. We are starting a blog within USAID on these issues and will open it up to all comers later this year. Can I share your ideas with my colleagues in the meantime?

    Many thanks –

    C.Anthony (Tony) Pryor
    Senior Programming Advisor
    Bureau for Policy Planning and Learning (PPL)
    USAID
    cpryor@usaid.gov

  6. […] led to specific results. There is a fascinating discussion about this question on Alex Jacob’s NGO Performance […]

  7. Thank you for these thoughtful comments. I’m very glad you think I’m on the right track.

    It’s also fascinating to hear Tony’s view from inside USAID: results frames are meant to be ‘development hypotheses’, not project blueprints – and USAID should do more to develop these hypotheses themselves. Good luck with the next iteration. Can you design something that will survive the corrosive force of funding dynamics on the one hand and middle management on the other?

    It reminds me of how GTZ (German aid) has relegated logframes to being one potential approach among many others, while DFID (British aid) has continued to beef them up and make all their grant recipients use them. References at the bottom of https://ngoperformance.org/related-initiatives/logframes/

  8. Hi Alex,
    Good thoughts. As you know, I very much agree, yes we should release the results chains. But as you also know, I still think it matters to us to know if people are healthy, can they read and write and yes, are they having a better income.
    After all, that’s why we started planning interventions.
    Throwing off the chains reminds me of women in the 70-s burning their bras. Did women get more empowered? And if not, did their daughters went back to their moms? No, they kept doing better in education and kept doing better in economy and yes, will have to keep fighting for gender equality and making sure man get on their side.

    So may be – in your line of thinking, can we ask the people what matters to them? And if what matters to them got improved, could they indicate why that happened?
    This implicates a reversed form of monitoring and carried by the target population themselves.
    Probably more anecdotal, but with high numbers, still providing quantitative insights.

    And I am sure tou will agree, this is excatly what we want to happen.
    Cheers,
    Tjipke

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: