Watch What I Do, Don’t Listen to What I Say.
Let me preface this whole thing by saying: I love in-person user tests. In fact, we do them about once every-other-week at Rival IQ, just to watch a new user interact with our app. It’s always eye-opening and definitely helps you reset your perspective on your latest work(s). We’ll always do them.
With all that said: it’s almost total bullshit.
You have to take everything you see with a grain of salt. User tests are nigh impossible to run well enough to get concrete direction from. So, this blog post is about not doing that (maybe I’ll write one about users tests another time).
My biggest beef is that the user is in a totally foreign context. “Come to our office, sit in this unfamiliar chair, use this Mac, talk out loud about everything you’re doing, feel awkwardly watched and, oh, can you be here at exactly 2:30PM? And, one more thing, we’re recording this and simulcasting it to the other room where eight more people are watching.”
Here is how someone normally uses your product for the first time: “Oh shit! I’ve got a boat load of reporting to do for my next meeting. What was that website that Jamie told me about? I should use that next time.” Repeat that 5 times, and then at some point you’ll finally visit our site. You’ll probably be sitting at your desk, using your normal computer, drinking coffee, get interrupted by email, coworkers, lunch, coffee, and you won’t feel like you’re being watched.
So, what to do? Your first step is to watch people use your product in their normal context. Use tools like Inspectlet and Mouseflow. These are great. Use them.
But, this is the real thing I want you to do: instrument the ever-loving-hell out of your app, and look at how people use your site in bulk. Some really smart and cheap people (like myself) tend to favor Google Analytics, some favor Kiss Metrics, and even some favor homegrown logging with Tableau and/or Splunk (those people tend to work at really well funded companies, because the price tag isn’t small).
If you don’t suck at building things on the internet, your site will eventually have more traffic than you can evaluate by watching everyone on the site. Inspectlet, Mouseflow, and user tests become gut checks… but reality doesn’t come until your metrics are measuring a meaningful difference.
Here are my two insanely actionable pieces of advice (sorry, I’ve been reading too much buzzfeed and inbound.org recently)
How should you instrument? Here is a piece of advice from my dad about interacting with air traffic controllers: “Tell ‘em who you are, where you are, and what you are doing.” Turns out, it applies to remote chunks of code reporting to a centralized service too.
What should you instrument? Everything the user does on the site and everything the site does for the user. So, you’ll track: page views, button clicks, etc. And, you’ll track emails sent, remote data collected, batch processes running on their behalf, etc.
Here is a bonus piece of insanely actionable advice: Make sure your user data is joinable to your metrics data. I guarantee when you get to that “I don’t suck at the internet” level of users, you’ll want to know things like: “how do users with gmail addresses behave?” Or “How likely is someone who analyzes a big/medium/small company to buy our product?”
Adam, out.