As a mystery shopping provider we frequently get involved helping our clients with survey design because a well thought out survey is critical to the success of the programme. Essentially a good survey will be short and punchy and contain a series of logical, objective questions. My five golden rules are as follows;
- Keep the question set short and punchy
- Don’t ask questions about things the shopper is unlikely to remember
- Ensure your questions are objective
- Each question must only address one issue
- Ensure the shopper explains their scoring decisions
To provide a bit more understanding, I’ll work through each of the above.
1. Keep the question set short and punchy
Maybe the first question you will ask yourself is ‘how many questions should my survey contain’. The short answer is, ‘as few as possible’.
Let’s start with the ‘one question’ principle, if you asked the mystery shopper to answer just one question, you would find that the information provided was 100% accurate and reliable. The further you move away from this principle, the less reliable the information.
Of course, a lot depends on the type of visit and the method of reporting. If the visit is video and/or audio recorded then you can ask a lot more questions because the person reviewing the media can answer the questions. However, let’s assume this is a plain vanilla written programme. On this basis, we would recommend between 20 and 40 questions.
With written reporting, the shopper is working from memory so you have to be realistic in terms of what the average person can remember. The survey will demand an answer so if the mystery shopper can’t remember, then they will simply guess/falsify the answer. In this case, they will always answer in the positive.
So, in essence, try to limit the number of questions in the survey and don’t allow other departments to load up your survey with questions of dubious relevance. Remember, there are too primary issues with overly long surveys;
- The mystery shopper will not remember all the facts and will probably end up guessing the answers.
- When you feedback the results to the people who matter, the results will lack focus and the key messages in your results may be lost amongst all the fluff.
2. Don’t ask questions about things the shopper is unlikely to remember
I apologise if this sounds obvious but it is a serious issue. Mystery shoppers will remember aspects of the visit that they interacted with directly. Imagine you go to a function and you have a conversation with a stranger. You are subsequently asked about the conversation you had, what they said, what you said and how the conversation ended. It is likely that you will remember the conversation and be able to answer reliably. Now imagine you are asked what the colour of the carpet was. Most people wouldn’t remember but if an answer was demanded, you’d resort to guessing.
Mystery shopping is there as a service to answer questions about the customer experience which you have no other way of answering. If you want to know if the opening hours are displayed or if there is a hole in the carpet, these things are better addressed by an area manager or at least someone who is not concealing their identity and can take notes as they move around.
3. Ensure your questions are objective
Now that we have the focus, the next thing to consider is question configuration. In customer satisfaction surveys, it’s good to present the respondent with multiple choice options. A scale of 1 to 10 is great for customer sat. purposes because it allows the subtlety of responses to come through. In mystery shop, the opposite is true. The perfect mystery shopping question has a defined reference point and can be answered Yes or No.
Let’s take customer acknowledgement as an example. A client may suggest a question such as ‘Were you acknowledged within a reasonable time’. Maybe the client likes this question because it takes into account the level of trade and the circumstances of the visit. We would advise against this and suggest including a time measurement or floor space measurement. Our preferred approach would be to ask for example ‘Were you acknowledged within 2 minutes’ or ‘Were you acknowledged within 5 metres of the entrance’.
When you ask a question that includes a subjective element like ‘reasonable’ then the answer is open to interpretation. What is reasonable for one person may not be reasonable for another. Equally, how do you push through this principle in your team training sessions? How do you get your teams of staff to buy into acknowledging customers within a ‘reasonable’ amount of time?
4. Each question must only address one issue
In addition to removing subjectivity from the questions, a good question should only have one variable. Over the years, I’ve seen some crazy questions which actually attempt to answer five or six different things.
Earlier, I’ve encouraged you to limit the number of questions but that does not mean questions should be amalgamated. If you ask a question like, ‘Was the staff member in uniform and wearing a name badge’ what does the answer tell you when the shopper replies No? Do you have an issue with uniform supply or just the fact that the assistant hasn’t put their name badge on?
If you look over your questions and you see lots of commas or the word ‘and’ just ensure that each question is only attempting to answer one thing.
5. Ensure the shopper explains their scoring decisions
If you are reading this, you are probably in a management position and therefore you will want your project to deliver great management information. However, most programmes are not short of graphs and tables. What many programmes are short of is buy-in from the people at grass roots level. This can be because they do not trust the findings of the mystery shopper.
Many surveys demand an explanation from the shopper when they answer a question No. There are two main problems with this;
- It encourages the shopper to answer Yes because if they do this, they won’t have to write anything.
- Often the comments are little more that the question being regurgitated. For example, the survey asks ‘Was the assistant wearing a name badge?’ and the shopper replies No. Their comment may well be ‘The assistant was not wearing a name badge’.
We have found that the most productive method is to demand a narrative response from the shopper for each section within the survey and generate consistency by applying a minimum character count for the shopper response. This must be completed however the survey has been scored. Many mystery shopping providers don’t like this approach because the shopper will expect to be paid a little more and it takes more effort to quality check the work.
However, when you feedback the findings at store level, you will get much more buy-in from the staff if there is a short paragraph explaining who said what and giving a general feel for the flow of the experience. As you will find, buy-in at grass roots level is critical when using the results to facilitate change.
Hopefully that has been useful in helping you design your mystery shopping survey. There are many other aspects to a programme to consider and we are always happy to assist our clients with overall design to ensure they get the possible return on their mystery shopping investment.