Thursday, 8 August 2019

Evaluation Tips Series; What questions do you need answering?

This is part two in our Evaluation Tips Series, you can see part one here

Once you have planned out who the evaluation is for, it's time to think about what questions you need answering.



Key Questions to think about:

  • What are / were the project / activity aims?
    • If a project is trying to improve confidence or health, make sure you ask about changes in confidence or health
  • What is the impact you expect to happen?
    • For example, do you think people will be more physically active as a result of the work your charity does? Then ask them how many days they were active in the last two weeks at the start of your work with them and repeat this and see what has changed
  • What is happening during the project? 
    • Do people take part for years or on just one day? If it's just one day, you will need to collect all your information then, if it is years you might want to have an annual collection month and talk to all your participants every February for example
  • What do we want to know the answers to?
    • Sometimes it can be as simple as are the sessions on the right day and time, how did people hear about the programme and would they tell a friend to come along? Sometimes it's more nuanced and you want to know if they have learnt about a particular topic or that they get on better with their family because of the work you have done
  • How much time is there to collect?
    • If you only have a little bit of time, only ask a few questions
  • How much time is there to analyse?
    • If you only have an hour to analyse the information, just ask a few questions. It's better to look at what you learn rather than collect lots of information and have no time to do anything with it! 
  • What information already exists?
    • Often projects overlook the information they already collect, if there are registers - do people come regularly? If you have an enrolment form, you may already collect all the demographics you need. 



Demographics

Demographics simply relate to the kind of people you work with; young or old, male or female, whether from an ethnic minority or not. Funders almost always want to know if you have reached the people you thought you would so if your evaluation is designed for funders, you will almost certainly want to collect this information.

Collecting Age

Here you can choose whether to ask people their date of birth, or get them to tick age brackets. Some people do not like giving this information so always make sure there is a 'prefer not to say' option.

The reason for collecting age is simply to find out how old the people are who are taking part. Once you know the age you can answer the following questions; Is this project aimed at a particular age group? Did you reach them? Who is missing and what does that mean for planning in the future?

Collecting Gender

This is usually best kept simple and you can ask people to tick

  • Male
  • Female
  • Non Binary
  • Prefer not to say


Unless there is a reason otherwise, you would normally like to try and hit 50% male and 50% female. If you didn't make a 50/50 split, ask yourself why not? What does that mean?

Collecting Ethnicity

If you do only see people very briefly you could ask people if they consider themselves from a white background or from a black and minority ethnic background (often referred to as BME) or give people the option to choose 'prefer not to say'. Sometimes people refer to BAMER now which stands for black and minority and refugee backgrounds. On the whole, though, it's probably best to use the standard ethnicity question as recommended by the census, office for national statistics and almost all funders. The standard question is below, in italics, so you can just copy it from here if you want.



What is your ethnic group?
White
·         English/ Welsh/Scottish/ Northern Irish
·         Irish
·         Gypsy or Irish Traveller
·         Other White background
Mixed
·         White and Black Caribbean
·         White and Black African
·         White and Asian
·         Other Mixed/ multiple ethnic background
Asian or Asian British
·         Indian
·         Pakistani
·         Bangladesh
·         Chinese
·         Other Asian background
Black or Black British
·         African
·         Caribbean
·         Other Black/African/Caribbean background
Other
·         Arab
·         Other

·         Prefer not to say

Collecting Postcodes

Here at The Evaluator we always try and collect postcodes. Postcodes are very helpful if you want to show the location of people on a map – which can be done through Google Maps for free.

Collecting Disability

There are a few ways of collecting this but we tend to go for 'Do you consider yourself disabled?' and give four options;

  1. Yes, very
  2. Yes, a little
  3. No 
  4. Prefer not to say
That's because we believe in simple questions. It is really important to collect information on disability, we often find that people with disabilities don't have the same outcomes as people who are not disabled and this means projects need to think carefully about the work they are doing and how they can make it better, so everyone can achieve. 

Finally, some funding bodies want information collected on Socio-Economic Status – Employment Status or Household income, and Sexual Orientation and Gender Reassignment. If your project is funded, check your funding body requirements first – it will make your life a lot easier at the end.