Blog: Closing the circle - Including co-production in evaluation and reviews

Daniel Turner (Quirkos) and Lucy Pickering (University of Glasgow) explores co-production, evaluation and research.

What’s the difference between co-production and consultation? Lots of people talk about co-production and participatory approaches, but read between the lines and many are still only scratching the surface of what’s possible.

Including service users and community representatives in the design and implementation of service provision can not only greatly improve user satisfaction and use of resources, but also create new services to meet previously unidentified needs. While many projects can demonstrate excellent co-production in design, the assumption is often that the ‘professionals’ analyse the data, including participants again only when writing summaries of the already interpreted data. But if we are including service users and carers for their ‘expert’ personal knowledge of systems, services and need, aren’t they also qualified to contribute to understanding the data?

This is actually central to co-production, indeed one of the SCIE’s key points for reviews and evaluations is that evaluation of co-production should itself be co-produced, and the Co-production Star also stresses the importance of ‘co-assessing’. However, very few of the case studies on co-production include much detail on the evaluation stage of the project, and some claiming involvement happened only credit academic researchers. We realise that a lot of professionals and researchers are wary of giving up control of the analysis and interpretation process, but we know from our experience that it can be done, and have practical advice for those willing to close the circle, and try co-production in every stage of a project.

First, resist the temptation to have project leads do the evaluation, and just present their interpretations to service users for feedback. While this can help challenge researchers’ assumptions and create outputs that resonate better with users and the community, when not well managed it has the potential to be little more than ‘sense checking’. Denying participants a chance to interpret data for themselves means that participants’ insights are not being fully used. If the evaluation focuses on a metric like ‘satisfaction’, when for service users the most important metric is ‘accessibility’, for example, it risks having success being defined by the wrong measures.

Secondly, to ensure co-production in evaluation, accept that service users and carers have the ability to contribute meaningfully to interpreting data. After all, interpreting data is something we all do every day: whether listening to others, reading the paper or watching TV, we are always evaluating – we are asking who and what can be trusted, are there underlying messages or patterns present? Those are the same basic skills required to evaluate and analyse qualitative data such as interviews, focus groups and feedback surveys typically used in a co-produced project.

However, systematic interpretation is skilled work, so there is a need for training and familiarisation, as in any of the other stages of co-production. It requires planning to make best use of service users time and existing and developing skills. For example, giving one participant dozens of interviews or hundreds of surveys to analyse is likely too big a draw on their time. But creating short workshop exercises where participants work in groups to summarise a single text, or key quotes or selected sources are used to promote discussion can work well. Or rather than ask participants to code an entire transcript, can they co-develop a framework for analysis? Can the framework they develop ensure the right topics and terminology is used. Quirkos was designed to make qualitative analysis software simple enough for participatory approaches, but basic tools like flipchart paper and pens work well too.

The resources listed below give some more practical (and theoretical) advice on coming full circle with including participants in all stages of co-production projects. So for Scottish Co-production Week this year, why not make the resolution to include service users in evaluating your next project? You might be surprised at how easy it is, and at how much their full contribution can add.

Further Reading

Participatory data analysis: a step too far? Nind, 2011

A Participatory Group Process to Analyze Qualitative Data
Jackson, 2008

Workshop exercises for participatory qualitative analysis, Turner, 2016