Research Methods: Pre-Testing A Survey With A Focus Group

Today, I tried out a new survey pre-test technique: a focus group. In the past, I’ve relied solely on individual feedback and pilot studies. It was a fantastic experience, and the feedback was much more detailed than what I have been able to get from one-on-one interviews and pilot studies.

Recruitment
I placed advertisements on Craigslist soliciting participants for the discussion. I offered $30 per person for an hour. About thirty people responded within a few hours, and I selected nine people who represented a diverse age range.

Format
After they signed consent forms, participants took the survey on their laptops. I gave each participant a notepad and asked them to take notes. Next, I kicked off the discussion with a round-robin ice breaker activity. The ice-breaker was effective at getting people comfortable. We then discussed each of the survey questions in order, with the survey projected at the front of the room. For each question, I asked: “Do you understand the terms used in the question?”, “Were you able to answer the question?”, “Are the options sufficient to express your opinion?”, “Was there anything else that you disliked about the question?” As necessary, I tailored each of these discussion points to the survey question that we were discussing.

Benefits of the focus group

• The major benefit of a group discussion is that some people will bring up points that resonate with others, triggering additional comments that would not have come about in a one-on-one conversation. For example, one person said, “I don’t like X because of Y.” Another person replied, “It seemed strange to me, too, but I didn’t know why. You’re right. I really wish that it said Z instead.”
• There were also several occasions where the participants disagreed. It was valuable for me to see both perspectives at once.
• In one-on-one interviews, shy interviewees will answer questions monosyllabically or not at all. This is awkward in a one-on-one setting. In the focus group, it’s easy to ask, “Jane, I saw you shaking your head a little. What do you think of what Bob said?”

Challenges

• People were hesitant for the first 2-3 questions. If your first few questions are important to your study, I recommend inserting some fake questions to start with.
• Some people will dominate the discussion. This is challenging because you need to ensure that you hear everyone’s opinion. On the other hand, having opinionated people helps start and stir a lively conversation. For the next focus group I run, I will aim for one dominant personality instead of two. I was able to (correctly) identify the most opinionated people from their e-mails because they self-identified as “experts” on the subject without prompting.
• Participants have strong opinions on survey design based on past surveys that they have seen. These past surveys may not have been very good. For example, two participants said that we should move our demographics questions to the beginning because they “usually” see them at the beginning. This is contrary to best practices; demographics questions should be at the end to avoid stereotype threat.
• Our attendance rate was low (5 of 9 showed up). This is much lower than what I’ve experienced with individual interviews. I suspect that there is less social pressure to attend a group discussion. I recommend stressing the importance of each individual’s opinion and attendance in the confirmation and/or reminder e-mails.

Textbook reading
I heavily relied on Focus Groups: A Practical Guide for Applied Research, Second Edition, by Richard A. Kreuger (1994). You can find the third edition on Amazon. The book is a fast read and contains a lot of practical directions on how to run a focus group. It is targeted at running a focus group to collect general qualitative information about a topic, but most of the lessons are applicable for pre-testing a survey.