Think twice before listening to your users

Back in the old days when I was learning how to develop behavior change interventions, everyone started the same way: with the focus group.

However, of late, I’ve been seeing more and more scientists applying the same rules to their digital health interventions. When designing a digital health intervention, I think you have to be very judicious about when to use focus groups and how much weight to give the findings.


Two reasons:
1) Users don’t know what they want.
2) Users can’t comprehend the tradeoffs.

John Gruber just posted an anecdote that nicely captures the challenge (BTW — yes, it’s that Marissa. From way back in 2006.

Marissa started [her talk] with a story about a user test they did. They asked a group of Google searchers how many search results they wanted to see. Users asked for more, more than the ten results Google normally shows. More is more, they said. So, Marissa ran an experiment where Google increased the number of search results to thirty. Traffic and revenue from Google searchers in the experimental group dropped by 20%. Ouch. Why? Why, when users had asked for this, did they seem to hate it? After a bit of looking, Marissa explained that they found an uncontrolled variable. The page with 10 results took .4 seconds to generate. The page with 30 results took .9 seconds. Half a second delay caused a 20% drop in traffic. Half a second delay killed user satisfaction…

The lesson, Marissa said, is that speed matters. People do not like to wait.

Now, I suspect that focus group users could’ve articulated that they like speedy web pages. No one likes watching webpages load (even if we were more tolerant of slower speeds back in 2006). But, what if we asked those users how they prioritized speed? I think most would’ve said, “I’ll happily wait a bit longer for more/better results.”

The lesson?

Base your designs on what users do, not what they say.