Tools not indicators

How to measure a man?

I’m losing count of the number of times I’ve met people who aim to change how NGOs manage their work by ‘defining the right indicators’.

It’s a powerful line of thought. First, identify indicators that define what you want to achieve – like changes in average incomes or exam results. Then have all your programmes use the same menu of indicators to establish their objectives and measure their performance. Finally, compare performance between programmes to see which ones work best; do more of the good ones and less of the bad ones.

It’s a simple analogy to profit in the private sector: a common way of assessing performance to drive continual improvement. Right?

Wrong. Lists of indicators are already available. But I’ve yet to see the approach work. Some of the problems are:

  • difficulties in finding one set of indicators that apply to different programmes and places,
  • difficulties in finding indicators for personal empowerment and political change,
  • changes in indicators are hard to attribute to a specific NGO’s work,
  • the long time lag between carrying out activities and seeing changes in indicators,
  • the length and complexity of the links between activities and indicators,
  • centrally selected indicators may not reflect local peoples’ concerns, and
  • the expense of measuring indicators separately from on-going work.

The approach is attractive at a senior level. It promises a clear way to tackle the issues and generate information about how well overall goals are being achieved. But it seldom works because of difficulties at the local level.

(BTW, I’m a fan of impact evaluations which do need well defined indicators. They just don’t help day-to-day managers run their programmes better.)

Luckily, there are alternatives that generate useful information for local managers. They help local people tackle their own problems and also generate summary high level data. They start from a different point: developing the right tools, based on assessing performance from the point of view of intended beneficiaries.

For instance, the Outcomes Star is used in the UK homeless sector. It defines ten dimensions of homelessness, from homeless people’s point of view. The tool structures how staff work on the front line and also generates management data. It could never have been created by ‘defining the right indicators’.

The Keystone Partner Survey provides detailed information on how well international NGOs support their southern partners. It is based on how the relationships work in practice, combining a range of indicators into a powerful aggregate picture. It also generates summary quantitative data that can be benchmarked.

The Coping Strategies Index generates numerical data on people’s level of food security. It is based on asking local people what they do when they start to run out of food – seeing the problem from their point of view, rather than trying to measure around them.

CAFOD’s Voice and Accountability tool provides a way of measuring organisations’ ability to carry out advocacy. Listen First provides a practical way of measuring how well an NGO involves local people in their work.

The NGO sector doesn’t need more lists of indicators. We urgently need creatively designed tools that allow us to measure performance from local people’s perspectives and help front line staff do their jobs better. Once created, different NGOs should use the same tools so we can compare performance and drive learning and improvement across the field.

One Response

  1. Sometimes I read a blog post and wish I’d written the whole thing from beginning to end, and this is one of those times. Thank you. The only thing I would add would be, for heaven’s sake, let’s not repeat this entire learning process in the social enterprise/entrepreneurship/public-private-partnerships sphere, either! It will not be any different just because we are funding it differently. Social change is just hard to measure like that.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: