5 Outcomes, impact and effectiveness

There is no story without numbers; but numbers don’t mean much without a story.

Alliance Magazine

5.1 Developing an impact reporting framework

As a strategic investor, the Foundation seeks to make an impact at three levels: individual and community, organisation, and policy and practice.

One of the objectives of the Strategic Plan, set out under the aim of becoming an exemplar foundation, is to ‘Develop a systematic approach to understanding, assessing, monitoring and evaluating the outputs and outcomes of the activities we support.’

Evidence of impact of philanthropic investment is notoriously difficult to collect and furnish. All of the grant-giving friends of the Foundation consulted for this review – from a large Lottery Distributor to a small foundation – confessed to struggling to find ways of assessing in ‘return on investment’. There is an ongoing debate within the Foundation world on effective practice.

It is, however, possible to tell a partial story about the Foundation‘s impact through evaluations and reviews commissioned and through the data collected through outcomes-based system of reporting introduced to the Open Grants scheme in 2007.

These sources tell us that desired outcomes are being achieved at individual and community level and on the level of organisations. There is less evidence of impact on policy and practice. With exception of Musical Futures on music teaching, we have yet to see any systemic radical changes from Special Initiatives because all of the others are still at an early stage. But there are possibilities here.

Stakeholders see PHF as an effective grant-maker and catalyst for change. Advisors report that the Foundation’s investments are starting to make changes in their sectors, but this is still largely anecdotal. The Foundation’s records – going back to 2007 when the outcome-based system was first introduced –tell us that 60% of Open Grants are exceeding or meeting their outcome objectives, with a further 17% partly meeting them (see below for reasons for failure and Appendix 3). This is likely to be higher than the figure for the period as a whole, since the quality of applications has increased and the new two stage system is much more rigorous in selecting projects and defining objectives. A 23% failure rate might be seen as relatively low for a risk-taking grant-maker.

The real challenge, though, is finding a simple way to tell an overall story about impact, which brings data on Special Initiatives together and which aggregates the outcome data from individual grants. Using the outcome-based system as a start, the consultants have worked with senior management to design an overall impact and outcomes reporting framework.

The prototype below attempts to distil at a top/generic level the outcomes and impact achieved or being pursued at each of the three levels. It has been built up by analysing the range of outcomes negotiated as part of Open Grants, analysing the objectives of Special Initiatives and referring back to the overall strategic aims of the Foundation expressed in the Strategic Plan. There is not enough data, categorised in the right way at present to make full use of this framework at present.

Top-level outcomes and impact reporting framework (work in progress)

This reporting prototype framework has a number of potential applications going forward: helping in grant assessment, developing Special Initiatives and helping the organisation to focus more on desired long-term impact. In future it might also be used to develop ways of assessing value for money. The collection and coding of grants data will need to be modified to fit into the generic outcomes categories and indicators. It will also require the Foundation to collect data from organisations once the grant period is complete.

5.2 The Three Levels of Change and Impact

In the meantime, for the purposes of this review we illustrate the actual and potential impact at the individual and community and organisational levels, by setting out grant expenditure (using categories currently used in the PHF Yearbook), supplemented by short project examples – one from each UK programme area. For the level of policy and practice, we provide project examples and also summarise the main objectives of the Special Initiatives.

Individual and Community Level Outcomes

These are the specific changes brought about in the lives of individuals and the communities and institutions they are part of as a result of participation in a funded project. They mostly involve giving people, who would not otherwise have the chance, the skills and confidence to progress to a new level in their lives.

Table 3 shows that 80% of the Foundation’s grant expenditure is primarily directed towards achieving individual and community impact.

Table 3: Grant Expenditure by Funding Focus 2006-11: Individual and Community Impact

Project Examples

Organisational Level

This level consists of better equipped organisations, producing more and better work, in new and more effective ways and reaching an even larger and or more diverse audience.

Table 4 shows that about 20% of the Foundation’s grant expenditure is primarily focused on impact at the organisational level.

Table 4: Grant Expenditure by Funding Focus 2006-10: Organisational Impact

Individual Project Examples

 

Policy and Practice Level

These are the effects of a project or programme at a higher and broader level. In most cases this type of impact will be felt in the medium to long term – often beyond the life of a particular project or programme. Some of these impacts may be unforeseen or unintended – but never the less positive.

Project Examples

The table below sets out the actual and potential impact of the three pre-existing and six new Special Initiatives.

Special Initiatives: Top-Level Impact (Actual and Potential)

5.3 Reporting Failure

In spite of all the talk about risk, we are not comfortable with failure and have no way of reporting it.

Trustee

The new outcome-based reporting system allows the Foundation to record instances of failure, where grants do not achieve their desired outcomes. There seem to be two main sets of reasons for failure: unrealistic expectation at the outset about the time and resources required to develop and deliver the project and the challenges of engaging excluded groups. Unexpected changes within the grantee organisation itself , especially of personnel, is sometimes a factor – though surprisingly rare considering the fragility of most small voluntary organisations. Below are three examples of failed grants, as recorded by the new reporting system.

Failed grants

 

5.4 Impact by age group

Table 3 shows that PHF is achieving its aims of supporting people under the age of 30 (Strategic Aims 2 and 3). 40% of expenditure over the period had been targeted specifically on this age group. The same age group is also well represented in the ‘multi-age groups’ category, which makes up over 50% of expenditure.

Table 3: Grant Expenditure by Age Group 2006-11