Assessing the impact of PHF funding: the approach
Our aim was to develop and test an approach to understanding the impact of our whole portfolio of funding. We wanted to build up a picture from existing evidence rather than commission new research or evaluation.
Effective evaluation strategies have to fit their organisation’s overall strategic approach. Our funding strategy means that assessing the overall impact of PHF funding involves particular challenges. With as many as 400 Open Grants ‘live’ at any one time (lasting up to three years each) and ten Special Initiatives under way, there is a considerable volume and rich diversity of work and outcomes to map and understand. Further, with our commitment to disadvantaged young people cutting across all our programmes and our interest in fostering the interaction between the arts and learning, none of our strategic aims are the exclusive territory of any one programme.
If PHF had fewer grantees, a narrow set of objectives and a prescriptive approach to how grantees should achieve them, we could ask those we fund to sign up to a limited number of intended outcomes and some standard ways of measuring their results. As the funder, we could then compile a useful guide to the overall impact of our funding by adding up the results from our different grantees. But that would not fit PHF’s approach to grant-making and social change.
Like many grant-makers, we work with each grantee to agree what they intend to achieve as a result of the work we are funding. Our intention is that self-evaluation will help the grantee manage their work, understand how to improve their impact and share what they learn with others. We agree up to five outcome targets with each grantee for each year of their funding. These outcome targets are not standardised or shared between different grantees. Rather, they are very specific to each organisation’s objectives and activities, context, client groups and other factors. Results are measured and reported in different ways, depending on the particular outcome target.
Grantees report to us regularly (usually on an annual basis for grants lasting more than one year) on the actual outcomes of the work we have funded. These reports are the basis for discussions between PHF staff and the grantee about how well the work is achieving their objectives and what the challenges are. In longer grants, the report informs our discussions and agreements with the grantee about the next year’s objectives. Of course, intentions and results are often different and we are keen to understand what the grantee is learning from their work and what additional or unexpected outcomes have resulted.
We find that, far from lacking information about the outcomes of the work we fund, PHF has a wealth of information – from grantees’ reports and the reports on Special Initiatives that we commission from external evaluators. Our aim was to develop a way of grouping this large collection of pieces of very specific evidence into categories, so that patterns of impact were revealed. In other words, we needed a map to enable us to see the ‘wood for the trees’. Our map should be a map of actual outcomes on the ground, whether intended at the outset or not.
The categories were defined through a process of sorting the evidence of actual outcomes from grantees’ and Special Initiatives’ reports into groups of related outcomes. We found that we needed a classification with two levels in order to organise the evidence into a manageable framework, which would provide both an immediately accessible overview and allow a finer grained understanding of the types of changes being made within each category.
The result was, at the upper level, a framework comprising 14 actual outcomes of PHF-funded work, which we could group into the three forms of impact to which PHF aspires: on individuals and communities, organisations, and policy and practice. Six of the 14 are outcomes experienced by individuals and communities; three are outcomes of organisational change; five are changes in wider practice and policy, based on the results of PHF-funded work. These 14 give the overview of the impact of PHF funding.
But for some purposes we need a map on a larger scale – one that shows more detail and names the different neighbourhoods and districts within the city, as well as the city itself. So, at the next level in the framework are 37 more specific types of change, which we also call ‘sub-outcomes’, each of them a sub-category of one of the 14 main outcomes.
The framework is essentially a classification of the outcomes achieved through PHF funding. The development of the framework enabled us, for the first time, to know, from a systematic approach, what these outcomes are – across the whole of the Foundation’s funding and at two different levels of detail. At the upper level there are 14 broad outcomes, each of which can be more deeply understood through its sub-outcomes (the lower level).