Framework (methodology)

Coding and counting

Knowing what these outcomes are is a step forward. A further step would be to know how many Open Grants and Special Initiatives produced each of the types of outcome and sub-outcome in the framework, including which were the most and least numerous results to flow from PHF funding.

To answer these questions, we reviewed, grant by grant, grantees’ reports and extracted a summary of each of the outcomes reported by each grantee. We then assigned to each grant codes for the appropriate outcome and sub-outcome categories from the framework i.e. the categories into which the specific outcomes of the funded work fell. The same process followed for Special Initiatives, using evaluators’ reports. Most Open Grants and all the Special Initiatives were assigned a number of different codes, since most of the activities we support are multi-faceted and many spread over several years.

We incorporated the coding system into GIFTS, the off-the-shelf grants management system used by PHF and many other funders, so that the data could be analysed with
management information about the coded grants.

Sampling

To test the approach, and because the reviewing and coding of evidence is quite a slow process when done thoroughly, we sampled grants rather than looking at the totality of grants made during the strategic plan period in which we were interested.  Between October 2007 and March 2012, 236 Open Grants were approved and completed within PHF’s three UK programmes. We analysed the outcomes of 51% (120) of these, a sample which is proportionate to the different numbers of grants made by each of our three UK programmes during this period. The variables used for selecting the sample were: the financial value of the grant, length of time covered by the grant, and the start date. We used GIFTS to select the sample.

Our Special Initiatives run for longer than most Open Grants. Most are ongoing and a number have been running long enough to have produced interim reports. So we included all the eight initiatives that were started and/or active during the strategic plan period and are either complete or have been running long enough to have evidence of outcomes to date.

This process of summarising and coding outcomes provides what we describe as the ‘impact map’ of PHF funding.