Impact report

If it was straightforward to understand the impact of the sorts of work that foundations support and do, it would have been done by now. This year we began a new programme of work to create a fuller and more accurate analysis of the impact of our funding. This understanding, though difficult to obtain, is critical for organisations like ours – and our grantees.

Our mission is to maximise opportunities for individuals and communities to realise their potential and to experience and enjoy a better quality of life. This can involve addressing unpopular or challenging issues. We are interested in finding better ways to do things and in helping organisations to sustain and develop their work. With these values, it is important that we know whether we are helping to create impact in the areas we have chosen to target through our work and funding.

Further, we need to be able to make some informed judgements about whether this impact is good enough and whether doing things differently – or doing different things – would improve the contribution we make to the quality of life and opportunities for the people our grantees work with. This means not only assessing impact but also bringing together our understanding of how and why participants have been able to change things for the better, to help others to learn from our experience.

We believe that if we can shed some light on these questions, learn from what we find and report publicly on the impact of what we do, we will also be putting into practice the sort of accountability, and the focus on learning and outcomes, that we look for in our grantees.

Evaluating the impact of PHF’s funding brings considerable challenges. As is obvious from this Yearbook, the work we fund takes many different approaches to meeting a wide range of needs. With as many as 400 Open Grants ‘live’ at any one time, and 11 Special Initiatives under way across the three UK programmes, there is a considerable volume and rich diversity of work and outcomes to map and understand.

At PHF we try to pay particular attention to the types of change that can take a long time to bring about and to doing what we can to ensure that changes are more than short- lived. This means identifying change over quite long periods, which increases the difficulty of knowing whether change can be attributed to any particular activity and source of funding.

Such challenges are familiar to all with an interest in evaluation but they do not lessen the need to understand impact. Following the commitment in our current strategic plan to ‘develop a systematic approach to understanding, assessing, monitoring and evaluating the outputs and outcomes of the activities we support’, we decided to develop and test a new approach to understanding the impact of the work we fund. This analysis will be particularly important as we look ahead to our next strategic planning period.

Evidence sources

At PHF we have a wealth of information about the outcomes of funded activities, through grantees’ reports to us and commissioned evaluations of our Special Initiatives. Our approach involves extracting evidence of change from these reports, coding and then grouping it to develop an ‘impact framework’ that classifies the outcomes of the work we have funded over the past few years.

There are three levels of change in the framework – change for individuals and communities, change in organisations, and change in policy and practice – each of which contains a number of broad impacts, giving 14 in total. Within each of these, there are several more specific types of change. So, for example, one of our 14 impacts is on marginalised young people, who improve their wellbeing and/or skills for a more successful future. This impact comes from the range of outcomes that our grantees reported to us. The children and young people they worked with:

• Developed skills to support a more successful and happy future, such as resilience and team working
• Improved attendance at school, when at risk of drop out
• Progressed in educational attainment
• (Re-)engaged in education, training, jobs or volunteering
• Improved their speaking and listening skills

And those who had been involved in the criminal justice scheme:

• Reduced their incidence of re-offending

By identifying types of change and grouping them together to form our impact framework, we are able to put together the ‘big picture’ of the impact of PHF funding. We have trialled this approach with a sample of 40 per cent of completed Open Grants, from our three UK programmes, and can already see impacts that are contributing to our key strategic aims (enabling people to experience and enjoy the arts; developing people’s education and learning; and integrating marginalised young people who are at times of transition).

Some of our impact comes through organisations we fund developing new business models, services and partnerships or through training and professional development of staff and volunteers. Grantees’ evidence that they become more effective at responding to new needs or different groups helps us to understand more fully our impact on organisations.

We also hope that the work we fund can contribute to wider changes in the practice and policy of other organisations – changes that will have a positive effect on the young and disadvantaged people with whom we are especially concerned. This type of change is particularly difficult to attribute and can take longer to happen. While we see some signs of it happening, we may need to think about further follow-up work to investigate – and perhaps bring about – the sorts of change that evidence shows to be helpful.

Next steps

As we bring together evidence from more grantees’ reports and extend the same approach to the evaluation of our Special Initiatives, we will continue to build the picture of the types of impact that PHF-funded work has been able to achieve. As we do so, it is particularly important that we and others are able to learn from the wealth of experience and expertise that has gone into bringing about these changes.

To help us with that objective we have also been assessing the strengths and weaknesses of the outcomes evidence that grantees include in their reports to us. The picture is very varied. Some of the evidence is based on a robust but straightforward approach to monitoring and evaluation, with the analysis clearly presented, helping us to understand the work and its outcomes. This is important to us because we know that organisations that have good information about their own work are better placed to improve their outcomes and to make the case for further funding.

However, not all the evidence was of such a high standard and a small proportion was quite poor. The next phase of our work will include talking to grantees about how we might be able to help them improve it, always recognising that the organisations we fund need a pragmatic approach to evaluation that will provide them with the information they need, while not costing too much in time or money. Already we have revised our reporting guidelines to grantees; we hope that reflecting and reporting on their work will help both PHF and our grantees to learn from the work they are doing.