Tag Archive for: Independent Evaluation UK

Capturing impact in people’s own words

At The Evaluator we recognise the value of collecting qualitative information for our evaluations. We take pride in capturing the genuine impact our projects or organisations have on individuals through hearing about their experiences in their own words.

The questions we ask are often simple and open to encourage individuals to speak honestly and let their stories unfold naturally. We feel privileged to get to hear about the amazing feedback and stories from people, and we love to share these stories with our clients.

We would also like to share some genuine quotes with you, spoken by real people who have experienced the impact from some of our clients:

We have heard from volunteers

“I truly appreciate everything the team does throughout the year to keep things moving forward. Thank you for helping to make so many people feel empowered, and seeing all of the work being done leaves me feeling so much hope for the future. Thank you for shining a light on all that can be done to make live music more accessible and equitable!” – Volunteer

“That was the biggest opportunity I’ve been given and that changed my life” – Volunteer

“It is such a joy and has made a really positive impact on my life. It has even changed the direction of my career” – Volunteer

 

We have also heard from participants

 

“For my general health I consider [client] more important than the NHS” – Participant of a wellbeing service

“[Client] has just been a godsend for me. Absolute Godsend” – Participant of a wellbeing service

“I really enjoyed the course as I love art and it also gave me a push to get out of the home and socialise more thus contributing to my wellbeing. Thank you”  – Participant of an Arts project

Thank you for the opportunity. Since starting with [project] I’m growing in self confidence and feel like I’m getting my life in order again. I’m putting plans in place and I’m more optimistic for my future.” – Participant of an Arts project

 

If you’re interested in learning more about our evaluation processes or exploring how we can help your organisation measure its impact, don’t hesitate to reach out. We’d love to chat!

What are the Indices of Multiple Deprivation?

What are the Indices of Multiple Deprivation?

If you have ever read one of our reports, you’ll likely see reference to something called the Indices of Multiple Deprivation, or ‘IMD’.

We use the IMD as a powerful tool to analyse and understand the barriers and socio-economic status of audiences, participants, and volunteers. We use postcodes to measure the IMD status, and this means that people find it easy to answer. This does mean that we are looking at the household status too.
The results give our clients an idea of which people they are attracting to their projects and what barriers they may be facing in life.

Statistically, people who live within areas of greater deprivation have more barriers in their lives and are less likely to engage with the arts and nature and are more likely to have greater issues with their wellbeing. That’s one of the reasons funders do like to understand if projects and organisations are reaching people from the most deprived areas.
How are the indices of multiple deprivation worked out?
The IMD is worked out using a range of factors which are assessed across the UK and given a score. The factors are listed below:

  • Income deprivation: how many people in the area are experiencing low-income levels.
  • Employment deprivation: what is the rate of unemployment in the area?
  • Education deprivation: what are the average qualifications within the area, and how easy is it to access quality education?
  • Health deprivation and disability: what is the average life expectancy and disability prevalence in the area?
  • Crime: how much crime occurs within the area
  • Housing deprivation: how affordable is housing within the area, and are there other barriers to people accessing housing
  • Living environment: what is the air quality like in the area, and are there green spaces?

These scores are compiled to give a ranking, which are then categorised into ten deciles. The entirety of England falls equally into one of ten deciles, with decile 1 indicating the most deprivation, and area 10 having the least deprivation.
If you were targeting deprived areas, you might want to reach 50% of people living in IMD1 and 2 for example. Or if you wanted a perfect section of the UK represented in your project, you might want to see 10% of your audience from each decile.

We think the IMD is a useful tool and aim to use it in every evaluation where relevant.

Cheers to a Festive Break: Wrapping Up a Year of Amazing Projects!

As the year draws to a close, we reflect on how grateful we are to work with such outstanding clients, providing evaluation and data analysis services throughout the year. It has been our pleasure to have been involved in some of the world-changing positive projects our clients have been working on.

In the spirit of the festive season, the Evaluator team will be taking a break to refresh and recharge during Christmas. Therefore, we would like to inform you that our office will be closed for the holiday period starting from the 22nd of December and will resume operations on the 8th of January.

We appreciate your understanding, and we assure you that we will return in the new year fully revitalised and ready to continue delivering our exceptional service.

 

Below are some of the clients we’ve had the pleasure of working with this year:

 

Wishing you a Merry Christmas and a Happy New Year!

New client: Community Arts by ZK

The Evaluator is delighted to be working on a new project, Healing Arts for All (HAFA) from Community Arts by ZK.

This project is place focused and based in Pendle, on our doorstep!

HAFA describes itself as a “3-year people powered art and heritage project”, funded jointly by the National Lottery Heritage Fund and the Paul Hamlyn Foundation.  The project will include participants working with professional artists as well as gaining an understanding of working within heritage.  Heritage will focus on understanding the history and impact of Pendle’s Brierfield Mill, an iconic grade II listed former cotton mill built in 1931.  This mill originally attracted the South Asian community to move to the area to work in the textile industry.

You can see more about  Community Arts by ZK on their website here. 

We are particularly looking forward to evaluating the personal impact that this project, blending creativity and heritage, has on the participants from the South Asian community.

 

Learning about data; Working with quantitative data

In a previous blog, we described quantitative data as data that can be counted or measured in numerical values. A spreadsheet is a good basis for organising data ready for analysis. Now let us imagine you have some data to hand, this is how you might want to try and use it.

Ways to analyse quantitative data.

Example 1: Looking at what the data tells you itself.

We generally start with calculating averages to consider the range of data (the highest score and the lowest score collected).  If we consider 2 sets of data (Data set A and Data set B):

From Data Set A, it is apparent that a similar experience was shared by the large majority of the audience because the average score is 40, but the range is from 36-44. That means the lowest answer was 36, which is quite close to 40 and the highest was 44 which is still quite close to 40.

However, in Data Set B, there was a much wider range of experience.  This should be questioned as to why experiences are so varied, illustrated by the wide range in the data.  Why did some individuals have such a higher score than others? Did some people have a different tutor, or venue, or have more time? There are many reasons why the range could be so high.

Example 2: Comparing data to other national data available.

Another possible way to compare data could be as in the worked example below where the Warwick Edinburgh Scale of Wellbeing was used to measure participant wellbeing.

Participants scores were collected over a period of time, firstly at the start of their engagement and then, at the end.  This graph represents a comparison in:

  • Each individual’s wellbeing from the start versus at the end. This is often a reflection of the impact of their engagement.
  • The individuals’ wellbeing who took part versus the national average
  • The individuals’ wellbeing score versus the NHS score of 40 which is believed to be indicative of poor wellbeing.
  • The individuals’ wellbeing versus one another.

Using quantitative data to help with forecasting & future decision making

Using percentages makes making comparisons easier to relate to and understand.

We use an example of a cinema and record audience attendance numbers of time.  Attendance could be affected by the popularity of the content of the film, seasonal trends or weather.  The trend line shows the overall audience is growing and then this line can be used to forecast on at the same trajectory.  This will help you to identify if you are likely to achieve your audience targets.

This information can also help you to make decisions about your capacity too and streamline your resources. For example, if you were thinking about moving a cinema location to a larger space, it would be worth looking at the trend line to think about venue size. If you were looking at two venues and one had a capacity of 100, and another 200, you might want to look at what your predicted audience size would be in six months’ time and negotiate the lease or hire accordingly. This is an example of making a data-driven-decision, something we are passionate about at The Evaluator.

Taking notice of negative space in data

It’s important to take note of the 70% who are agreeing but also to take note of the 30% who are disagreeing and find out why this is the case.  It is worth delving a bit deeper into on the minority and finding out what was the cause of their response.

At The Evaluator, we tend to represent 3 answers ‘yes’, ‘no’ and ‘prefer not to say’ in our reporting. If there’s a high number who indicate that they would ‘prefer not to say’ then it would be suggested they might insecure about completing the survey.  Often their indecision is explained in their qualitative answers and this is worth taking note of when creating future surveys.

Dealing with ‘satisficing’ survey responses.

This is the term we use when people respond with the answers that they think you are looking for and may not read all the questions.  An indication of satisficing is when respondents repeatedly choose 3 when presented with a 1-5 scale. We don’t come across these very often, as we spend a lot of time making sure our surveys are easy to complete, and varied, but if we do spot them, we will try and remove these answers from our data analysis.  It’s important to encourage honesty in answering survey questions.

A final tip

All data can be segmented but you do need to think about the time you have to spend on this as it is time consuming.  In segmentation ideally you are looking for what is more than 10% different to the average. In the example of the graph below, it is worth looking at segmenting to see the results demographics of the areas that are 10% above and 10% below the average line.

 

 

New Client: Love Withington Baths

The Evaluator is delighted to be working with a new client, Love Withington Baths.

The Evaluator is working with the community leisure centre to carry out a Social Return On Investment (SROI). You can read our blog post all about SROI here, but it’s basically a formal process of identifying what matters to people taking part, starting with the changes they identify. We are looking forward to chatting away to people using the centre, and figuring out what changes it has led to in people’s lives. It’s a really well used centre, so we should get to talk to lots of people!

You can see more about Withington Baths here. 

New client: The Bureau for the Arts, Blackburn

The Evaluator is delighted to be starting a new evaluation; Chip In. Isn’t that a great name? Northern, does what it says, and everyone loves chips!

Helping people, building community, changing lives!

Chip In is a place- based project which aims to engage and mentor young people aged 16-30 in volunteer opportunities across Blackburn with Darwen’s arts, culture, sport, heritage and environment sectors over the next two years. The Project aims to break down barriers which young people face in volunteering, bring together opportunities and build skills, confidence and community.

The Bureau Centre for the Arts is among over 160 organisations across arts, culture, sport, civil society, youth, and heritage sectors to benefit from the £4.6 million Volunteering Futures Fund.

Chip In (formerly Volunteering Now) is our two year volunteer partnership programme for young people aged 16-30. We have teamed up with a range of local partners in the borough to provide opportunities for volunteering in a range of areas such as events, music, project development, admin, exhibition invigilation, workshop support, stewarding, curation, collection handling, supporting youth sessions, social media and blogging, food growing and much more!

You can see more about the project here or read more about Volunteering Futures here. 

Learning about data; Working with qualitative data

From time to time, we share a director’s blog post, where we share some learning about how to use your data (information) better. This is a post all about how to use qualitative data, that’s the information that is made up of words rather than numbers.

The chances are that your organisation already has a bank of this data but it may not be well organised and easily accessible. Below is a list of some of the many ways you might have collected qualitative data, possibly without realising it.

How credible is your qualitative data?

There is a distinction between primary and secondary data and it’s important to understand what you have collected as the credibility differs between the two.  Primary data is more credible and robust.  It is the data that people told you or have written themselves.  Secondary data is that which is overheard in discussion or something that someone has heard and told you about.

To check the credibility of your secondary data, you can apply the Rule of 3. If you’ve heard it from 3 different places or if it’s come from 3 different people independently then it can be considered as credible and representative of a thought or opinion.

How to make the qualitative data more usable? 

We use word or phrase frequency analysis to evaluate our qualitative data to look for patterns of frequency of words to identify common themes.  This can be done online using free tools, like this one,  and they just count the words for you.  We often use the top 10 words or phrases.  You can also use the ratio of positive to negative words that are used or how far down the feedback the first negative word appears.  Using these methods helps to quantify data and make it more digestible and can be used in marketing or to track changes over time.

Mind maps can be useful to illustrate and develop on the themes identified. You can just draw these freehand to have a look at what the main themes are.

Word clouds provide a visual representation at a glance of the qualitative data and this is also a resource that can be sourced freely online.  The most common words appear the biggest in the cloud, making this data easier for the reader to understand visually.

Top tip – try to be objective, it can be hard to hear negative comments but it is how we improve and know what to fix, and they’re often in the minority compared to positive comments.

How to use and share qualitative data?

This data can be used in many different ways:

We also use qualitative data to create case studies which illustrate people’s journeys and direct engagement.  Case studies can be shared at board level and to show case your project’s work for marketing purposes.  Case studies appeal to a wide audience and are particularly useful in attracting funders as voices are recorded and reflected in these studies.

In our experience as evaluators, a case study from a project that we evaluated got shared with funders (Green Recovery) who sent it on to The National Lottery Heritage Fund who then sent it on to the Department of Work and Pensions.  Case studies are impactful in that they record real voices and can attract publicity and raise awareness of change.

 

If all of this sounds like too much work and you don’t have time, get in touch. We are happy to have a chat about your individual requirements and to see how we could help.

Recent Client Feedback

At The Evaluator, we are committed to learning and improving our work over time. To make sure we achieve this we do evaluate ourselves and ask our clients to give us feedback. Here are some recent results.

 

And let’s end on a detailed client testimonial, from when The Evaluator worked in partnership with Simon Lees from Countryside Training Partnership to create an evaluation framework.

“Kirsty from The Evaluator and Simon from Countryside Training Partnership have challenged and encouraged the Climate FORTH project team and Board to think about monitoring and evaluation in a holistic way, right from this project development stage. The Monitoring and Evaluation Framework created will support our application to NLHF for delivery phase funding and show how we wish to go beyond monitoring outputs to considering the wider impact of National Lottery and other investment through our project. Communication was excellent throughout this contract and an extension was kindly accommodated when our timelines were delayed. The final report is clear, concise and well considered – spot on – we hope that we get an opportunity to enact it! The Evaluator & Countryside Training Partnership have proposed workable and proportional monitoring and evaluation methods to follow in a project delivery phase, identified suitable baseline data and provided friendly advice as we move through our project development phase. I would be pleased to recommend both parties to other organisations and partnerships looking for a personalised professional approach to this type of work.” Kate Fuller, Climate FORTH Project Manager, Inner Forth Futures.

Evaluation: To Have and To Heal

When you hear Ancient Egypt for Modern Times, what do you think of? We thought of pyramids, and mummies and actually the whole of Egyptian Culture. We have loved evaluating this project around connecting modern people to ancient themes. This project aims to take the whole of that culture and make it relevant now. Of particular poignancy is how people have coped with tough times, and bereavement throughout the generations, which is timely given the last few years.

The Evaluator is delighted to be working the Manchester Museum team to understand the impact of their project, which includes an inter-generational project working across schools and care homes and an online project, of which more is below.

The official project description is:

To Have and to Heal is a unique new arts and wellbeing programme, supporting Covid recovery and resilience, which explores Manchester Museum’s World Class Egyptology collection and popular fascination with ancient Egypt. Sessions provide a safe space to share responses to and personal reflections of experiences of the pandemic.

Participating groups are invited to explore a chosen theme, through which discussion and reflections are shared in relation to the objects and the personal experiences they may connect with. Themes range from Magical Healing and Touch, to Loss and The Meaning of Art.

The interactive online programme includes real time engagement and Q & A with the Museum’s Curator of Egypt and Sudan, showcases high quality digital images of rarely seen objects from the collection, together with a behind the scenes virtual visit to the Museum’s storerooms.

The online programme uses Manchester Museum’s collections as a catalyst for providing inspiration, and meaningful support to groups and organisations that have been heavily affected by the coronavirus pandemic, and is open to:

*​Third sector organisations offering vital support to people at risk of social isolation, with health conditions, or caring responsibilities.

*Schools and home-schoolers seeking inspirational new resources to support ‘catching-up’ on disrupted education

*Care homes and living independent living facilities

You can find out more about the project by clicking here. There are a number of short films which can be freely viewed to explore some of the project themes and objects from Ancient Egypt on the same link.

This project is funded by The Esmée Fairbairn Collections Fund – delivered by the Museums Association.

Manchester Museum reopens soon at 10am on the 18th February and is opening with a free exhibition, Golden Mummies. You can find out more here.