Thursday, July 05, 2007

The Naked Eight - 8 rules Producers MUST know to get the most out of qualitative research

I’ve done quite a bit of research over the course of my career, both as a facilitator and/or ‘manager’ of the research and also as a Product or Marketing Manager looking to find answers. I’m not saying I’m an ‘expert’ – I know way too many talented professionals to put myself in their class – but from a creative point of view, let’s just say that I’ve put enough time in the field to have some lessons to share.

So if you plan to be behind the mirrored glass anytime soon, here are the 8 key rules you MUST follow if you want to get to the naked truth.

#1 Hire the best
Most of us don’t have unlimited funds for research. You might WANT to do five geographically-desirable cities with full rotations of multi-day sessions, but you’ll be lucky if your budget covers a single day in a city you can drive to. So hire the absolute BEST. Schedule around them, accommodate their needs, do whatever you have to do – but make sure you don’t skimp on the talent that will help you get the information you need and work around your limitations.

#2 Talk to the right people
Product Managers generally work with research experts to set up their research and therefore ‘trust the expert’ to complete the screening process on their behalf. The “screener” is the script that the research expert will use to choose who will be in the room to tell you what they think in regards to your product or service. Inevitably the ‘expert’ doesn’t have anywhere near the level of understanding of the target audience as the product guru, which can result in full groups or sessions that are a complete and total waste of time.

A colleague of mine had a great example of rule #2 with a session this week featuring an online product that will use tons of video and 3D animation. Because the screener didn’t weed out dial-up users (are those really still out there?), one entire hour was wasted on a Mom who not only wouldn’t be able to experience the product – but who was seriously disgruntled and used the time to vent her frustrations about high-bandwidth content. In a 12 hour/10 group schedule, that was 10% of his budget out the window.

You know your audience and what you need to learn – make sure that EVERY cent counts. Now if you follow #1 and hire the best, then you should also TRUST the best and use their expertise. But don’t do it blindly – you need to contribute too. And the screener is the critical step that ensures that when the day comes, you’ve got the right people in the room to give you the answers you need.

#3 Deliver your materials as early as possible, and then iterate
It is inevitable that Product Managers and Creative Directors will be tweaking concepts or prototypes up to the absolute last second to be able to test as ‘real’ a product as possible. This irrefutable fact often results in the facilitator not seeing the product until the day before or the day of the research.

Why is this bad? Because you want your facilitator to be totally, completely comfortable with your product so that they are focused on the people in the room. If you are sharing your demos iteratively leading up to the sessions, they will be much more at ease and can focus totally on asking the RIGHT questions at the RIGHT time – and will be much more likely to get to the core of why people are responding to (or more importantly, NOT responding to) your materials.

#4 Show everything in context
This is not rocket science, but if you are testing a website, then make sure to show your pages within a web browser. If it is an email, set up a fake email account for pick up or at the very least show screenshots or boards that place your email in the most popular email hosts. If your concept is related to a movie, a TV or an established product – then start off by showing a clip or demo of the current product. This quickly gets the respondent thinking in terms of how they ‘feel’ when interacting in a certain way or with a particular property and therefore helps them to provide much truer answers to your questions. Provide the expected context so that the focus is on the message, design or feeling you are trying to convey.

#5 Preparation is everything
One of the biggest mistakes that I see Product Managers make is in not putting together the materials in a way that the facilitator can manage them. This is especially true with anything that has to do with new media. Keep in mind that most facilitators are NOT web experts or email marketers or film producers or product designers. Their core strength is generally in making strangers feel comfortable, connecting with them, and asking great questions that get to the heart of a matter. Let them do that – by making sure that the flow of materials and what they have to do to keep it moving is 100% idiot proof.

#6 Shut up and listen
The best groups are those that have great big servings of positive AND negative discussions. In fact, the things that people DON’T like about your creative are generally much more useful than the things they do like. After all – wouldn’t you rather know before you put your product out in the marketplace? Listen to absolutely everything your respondents have to say, even if they don’t give the answers you were hoping for.

#7 Schedule breaks
Another key to success is to schedule a good break or two throughout the day. A lot of the time we are so eager to get as many sessions in as possible that we put no more than :15 min breaks in between each session in a schedule that rolls straight through from beginning to end. This is a mistake!

Not only is it highly likely that you will run behind your planned schedule due to forces beyond your control (late respondees, technical issues, whatever), but some groups might just take longer. If you don’t have a good medium or long break, your later sessions will suffer as your facilitator gets tired. The reason a tired facilitator is so bad is not that you’ll miss out on the obvious questions from your script, if you hired the best this won’t be an issue. It’s the opportunity cost of missing that critical follow-up question that digs deeper or keeps people on track and off of extended tangents.

#8 Be a positive force
There is no rule more important than this one – if you want peak performance from your facilitator, then be a positive force. If you feel the facilitator is leading the respondents, missing key questions or misrepresenting the product – DON’T tell them everything they are doing wrong in between sessions or when they come out for notes, even if it's a disaster. Beating up your facilitator typically results in just getting them flustered – or worse – focusing only on exactly what you told them and then messing up the things they were already doing right.

Focus on the positive, tell them what a great job they are doing, and give a maximum of two notes at a time. If you did a good job of preparing for the big day you won't need more than small nudges to provide direction. Treat your facilitators not only with respect, but also provide support and encouragement throughout and they will work harder to do the best job they can to get you results.




No comments: