Getting beyond impact

Some interesting developments in the debate about measuring impact, in particular around Social Return On Investment (SROI) – and some lessons from microfinance.

A new report by the Third Sector Research Centre, comments that SROI “leaves ample room for judgement”, that it is not a good basis for comparing different organisations and that it “provides a weak basis for understanding how and why impacts occur”. Oh dear.

It’s not all bad news. Properly used, the method can help organisations map and value the effect they have on different people. But it doesn’t provide a reliable way of comparing performance between organisations. It’s also pricey.

Pretty much the same concerns are raised about impact evaluations in international aid. Critics say that randomised control trials don’t explain how change happens; and it happens differently in different contexts; so these trials don’t provide a guide for planning work in the future. What worked in Malawi last year may not work in Madagascar next year.

Critics also point out that both control trials and SROI can easily sideline the views of the most poor and marginalised people.

We shouldn’t be too quick to dismiss SROI. It’s a serious effort to find a better way of measuring results and allocating resources. I hope it continues to develop and evolve. But it runs into some deep seated problems about impact.

Microfinance has faced the same issues. The sector came up with a great way to measure impact, developed by Progress out of Poverty. Their easy-to-use and reliable scorecards help measure whether microfinance clients are poor and if they stay poor. It’s an impressive approach, that deserves to be widely known. But, as Progress out of Poverty comments, “poverty scoring does not indicate what caused change”.

Something more is needed to provide helpful, direct data that managers can use to contribute better to social change – what Lawrence Haddad, Director of the Institute of Development Studies, calls the “post-impact agenda”.

What has microfinance come up with? Feedback. You might have heard it somewhere before!

Social performance management‘ includes structured feedback from clients, among other measures. It’s not the only tool, but it is central. Simple, cheap to collect and powerful, it provides managers with the information they need to improve their services. It also provides summary data that policy makers can use to review performance.

It’s a fascinating example of how sector-wide collaboration can generate practical management tools to strengthen performance across the field. Here’s hoping that similar efforts drive the “post-impact agenda” elsewhere.

2 Responses

  1. I understand that there is a real desire to find a”magic bullet” which gives a difinitive answer as to performance. I am not sure it is out there. If monitoring and evaluating projects and programmes, using DAC criteria for example, is done professionally and independently and is based on many years of experience then this does provide a valid opinion on performance. We cannot get away from subjective view or judgement. By example, the legal world is always doing this as is the medical world which we pay for.

    Important to keep in mind, are the objectives being met and are the problems being solved. if not then something is wrong. This leads into all sorts of issues such as whose objectives and whose problems. As a professional monitor and evalution practioner, I try to be objective and provide an opinion on the evidence in front of me.

    Gathering opinion from stakeholders in the form of structured feedback may not be a magic bullet either given it is only opinion after all. The issue is that even this is not independently arrived at and likely to be biased. True independent monitoring and evalution should accommpany these other instruments for understanding impact and performance.

    Benchmarking performance which takes all these elements (transparency, accountability, responsibility, relevance, efficiency, effectiveness, impact and sustainability) into account must be agood thing. Performance is an all emcompassing term and therefore needs to be all encompassing in the way it is measured.

    I have developed an on-line tool to try and do just this and would welcome exploring how to realise it more widely.

  2. […] This makes it much easier to compare different approaches, for instance to contributing to policy change or providing clean water. NEF’s Sept 2010 position paper on value for money suggests coming up with standard approaches for measuring results that are specific to different sectors. I think there’s real promise down that route. Though I’m unsure about Social Return on Investment. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: