Man walking tightrope above a ravine at sunset

Walking the Tightrope: Quality of Evidence v Quality of Experience

Published: 12/11/20 | Categories: Author: Mike Corcoran

How can we balance the quality of evidence our evaluation requires with the quality of experience it delivers to all those we work with? The Co-production Network for Wales’ Mike Corcoran investigates.

Back in February (though it feels as if it was hundreds of years ago now!) I was l fortunate enough to take part in Inspiring Impact’s ‘Evaluating Front Line Services – What Works’ event, in collaboration with WCVA, Data Cymru, The Wales School for Social Care Research and the Co-production Network for Wales.

We spent the day with an amazing group of passionate and dedicated people from all over the country – getting to know each other, and sharing our stories of the trials and tribulations of getting evaluation right.

One of those stories really stuck with me. The story of a care home where passionate and diligent staff duly recorded their activities throughout the day, using electronic devices into which they could input information about every meal, every check-in, every toilet trip, every visitor … each and every aspect of daily life at the home. On the face of it, a compliant, simple, efficient, easy to use system, producing reliable evidence of the highest standard – but something wasn’t quite right. From the perspective of the residents of the home, what they saw was staff more interested in an electronic device than on them, opportunities missed to create those ‘magical moments’ that can happen when we shift our focus from processes to people, and systems which didn’t have the space to capture them even if they did.

Making evaluation work in practice is, like many things, about striking a balance. A balance between the conflicting pressures on limited time, money and resources. A balance between the expectations of service users, managers, funders and regulators. And a balance between the quality of evidence our evaluations must produce, and the quality of experience we owe to all they involve.

Producing data of good quality, which is robust, reliable and respected is central to any evaluation – but this should not come at the expense of the experience for all those our evaluation involves. Get it right, and evaluation can empower people and consolidate and strengthen relationships. Get it wrong, and (as in the case of our care home) evaluation can provide perfectly reasonable evidence whilst at the same time inadvertently disempowering, disengaging, and undoing all the hard work which came before it.

The solution – keep your eyes on the prize – at the heart of your evaluation are your people, put their best interests first, and allow everything else to follow. Through our Measuring What Matters Evaluation Tool, we make the central question of any evaluation, ‘what matters most to the people my evaluation involves?’ and through the new ‘Simple Questions Framework‘ we have developed with WCVA, we hope to foster evaluation cultures where processes fit around people, not the other way around. There is abundance of fantastic and hugely diverse evaluation tools and approaches out there, and huge scope for creativity and compassion in their application. Putting people first does not need to mean more time, more money or even more expertise. It just requires a different way of thinking about what evaluation is really all about – not just measing impact, but an opportunity to positively enhance it.

Mike Corcoran works with organisations across Wales and around the world as an advisor on engagement and impact. He is a long-term associate of the Co-production Network for Wales and led the development of the Network’s ‘Measuring What Matters’ evaluation tool and WCVA’s ‘Simple Questions’ evaluation framework. 

INSPIRING IMPACT EVENTS

Inspiring Impact’s events programme is running with monthly events through to February 2021.

  • 21st October – 7 Simple Questions: An exploration of the relationship between routinely asking simple questions and developing a rigorous, holistic approach to evaluation
  • 10th November – Quality of Evidence vs Quality of Experience: An exploration of standards of evidence, how they are used and how to balance the quality of evidence required with the preferences and experiences of those evaluation is required from.
  • 1st December – Evaluation, Funding and Procurement: An exploration of the relationship between funding and procurement requirements and evaluation methods and strategies.
  • 19th January – Evaluation and Complexity: An exploration of development and participatory evaluation approaches and techniques and their relevance to evaluation in rapidly changing complex environments
  • 9th February – Reflective Practice: An exploration of the relationship between reflective practice, evaluation and organisational culture

Booking is free, and for more information visit https://www.eventbrite.co.uk/o/wcva-cggc-12161492678