The Heritage Lottery Fund (HLF) commissioned an independent review of the self-evaluation process and outcomes achieved for 326 completed Heritage Grants projects funded under HLF’s Third Strategic Plan (SP3), which ran between 2008 and 2013.
This review includes a comparative appraisal of the quality, scope and methodology of the self-evaluated reports against six criteria and the type, range and quality of activities and outcomes achieved by completed projects, including a mapping of the impact of their work onto HLF’s current framework of 14 outcomes for heritage, people and communities.
Quality of the Self-evaluations
The quality of the submitted self-evaluation reports was assessed on a four-point scale (excellent, good, adequate or poor) using six criteria. These six criteria focused on the extent to which the evaluation:
- provided a logical framework
- included appropriate and methodological ways of providing robust evidence
- demonstrated that data was subject to robust analysis and provided evidence on outcomes
- was objective and free from bias
- presented the results clearly
- included sufficiently clear conclusions and recommendations to enable stakeholders to apply any lessons learned
Overall just over a third, 37%, of reports were graded as good or excellent with just under two thirds, 63%, falling within the adequate or poor categories.
The aggregated findings in this report show that the quality criteria scores have significant dependence on several characteristics, confirmed by a range of statistical tests:
- Evaluation was external: external consultants/organisations tended to write better quality reports.
- Planned share of grant initially allocated to evaluation: where higher amounts of expenditure had been originally allocated for evaluation, the reports tended to be of better quality.
- Report length: longer reports which contained more explanation and more data, tended to be of better quality than the shorter reports.
- The number of HLF outcomes assessed as being met: better quality reports tended to record a greater number of outcomes being met.
The most commonly mapped outcomes were all outcomes for people and for heritage. There was evidence in over three-quarters of reports for:
- People will have volunteered time (87%)
- Heritage will be better interpreted and explained (79%)
- Heritage will be in better condition (79%)
- People will have learnt about heritage (78%)
- People will have developed skills (77%)
- People will have had an enjoyable experience (76%)
The least commonly reported framework outcome was that a project had helped to boost the local economy, with only 15% of reports containing evidence of this.