Last week we hosted an informative live chat about evaluation. We were joined by a panel of expert guests and also received questions and comments from external contributors.
"Tell your project story and share your successes, but also explore what worked and what didn't work, as this will assist future projects (your own and other grantees'), and feed into HLF's strategies and processes."
There was so many good tips - as well as a lot of recurring themes - that we've summarised the key points for you here.
What are the main activities to take into account when planning your evaluation?
- A good and straight-forward starting point is to use a logic model: think about what inputs (resources) your project needs, what activities are required to create your project outputs, and what outcomes and impacts you hope will be achieved by your project.
- Budget and timing – are you giving adequate time and resource to the evaluation being undertaken?
- It's not just about showing that you have done what you said you were going to do (prove), but understanding the impact of the activities you've undertaken and the challenges you have encountered along the way (improve).
- Tell your project story and share your successes, but also exploring what worked and what didn't work will assist future projects (your own and other grantees'), and feed into HLF's strategies and processes.
- Never start with the method. Start with what you want to find out and why you want to find it out, as this will determine the type of evaluation you need to do (ie: front end, formative and summative), and the most appropriate methods to use.
How should grantees use the HLF outcomes framework to evaluate their project?
- You will have identified aims using the framework in your application, so include the outcomes as headings in your report, and ‘tell the story’ of your project in a range of ways, including both the ‘hard’ evidence of statistics, alongside quotes.
- The outcomes are necessarily quite broad, so start by making them more specific to your project and then think about what the indicators for them having been met might be. You then design your evaluation to capture those.
- The 'more and wider' outcome highlights the need for capturing baseline data so you can compare the ‘before’ and ‘after’ results.
- Do make sure you leave some scope to capture unintended outcomes, too. Using broad, open-ended questions can be a good way of doing this.
What do you think is realistically feasible for small-scale projects when conducting an evaluation?
- Some evaluation awareness training during project start-up is a great thing to factor in costs for, because it helps everyone to understand the benefits of evaluation and be on the same page for the rest of the project. It can also be a super activity to galvanise the team.
- Hiring a consultant to help create a framework and some tools/templates can be a good use of limited budget.
- Threading evaluation throughout everything you do as part of the project will help to make it less of an effort at interim or summative stage to pull together findings and a report.
- On a small scale project, you run the risk of the evaluation dominating or overshadowing the main activities if you use too many methods. Start with what do you want to know and why do you want to know it, then stick to one or two appropriate methods.
- If there is an experimental dimension to the project, you may want to pay more attention to assessing whether the new approach/idea actually achieved its aim effectively.
- Top tips: do what's manageable within your resources and capacity - better to do what you can well, than try to do everything and struggle; be honest and relfective - what advice would you give yourself in hindsight; use stories and stats - clear and succint descriptions can bring the numbers and percentages to life.
- Is it possible to collect data from other readily-available sources, such as Trip Advisor?
What are the advantages and disadvantages of self-evaluating your project?
- Experiencing visitors’ reactions first-hand makes the findings more memorable and helps build a body of knowledge about your visitors that is held within an institution.
- However, in particularly sensitive situations, it may be easier for an external evaluator to feedback findings that are negative or in any way hard to hear. If you need to lend weight to your arguments, an external voice can also be helpful.
- People often employ an external evaluator when they are short of time, but don't underestimate the time still required to recruit, brief, and manage the contractor, and comment on data collection tools and final reports.
- Its important to demonstrate that you are approaching things from as unbiased a point of view as possible, with a real desire to know what worked and what didn't work. Grantees shouldn't be afraid to be honest - it shows they're a learning organisation that is progressive and working towards being more resilient.
- A hybrid model could be most effective: work with a specialist consultant who can help to train your staff and volunteers, mentor and act as a critical friend, but also bring independence and capacity to your report writing.
What makes for a good evaluation report, ie: structure, content, etc?
There is no one size fits all, but some principles that should be adhered to include -
- An executive summary that gives a quick snapshot of the project and what it acheived.
- A detailed background and context which sets out the original intent of the project.
- Explain the methods used to evaluate the project and the rationale for choosing those methods, plus detail any limitation to the research undertaken.
- Use the main activities as described in your application, alongside the HLF outcomes, as headings to explore in further detail.
- Include baseline figures and demopgraphic analysis.
- Explore the lessons learned, and future improvements.
- A detailed conclusion of what difference the project has made.
- Data collection tools used, such as questionnaires or observation guides, should be included in an appendix.
- Use charts, infographics, and images to make the report more engaging.
- It is also vitally important to think of how you make your evaluation accessible to others so information can be shared. Here's a Sharing Heritage report for the project Joined by the River: 50 years of the London Borough of Richmond upon Thames: http://www.richmond.gov.uk/home/services/arts/orleans_house_gallery/education_at_orleans_house_gallery/our_education_reports.htm
What do you think - what advice can you share about conducting heritage project evaluations? Or what further questions do you have about conducting your own evaluation? Please share below.
As requested during the live chat, please find attached the Heritage Lottery Fund 2017/18 evalaution strategy, outlining our objectives and how we intend to meet them.
Our organisation is relatively experienced in evaluation techniques. However, Amy's post has prompted some new ideas of how to keep evaluation 'fit for purpose' in relation to our proposed project.
Glad to hear that, Glyn