Data Addict

Startups, Self Quantification, and Internet Culture

Flower

Archive for August, 2010

3 Things Steve Krug Didn’t Tell You About Usability Testing

I first read Steve Krug’s book “Don’t Make Me Think” about a year ago. I have never finished it. The reason is that I got to the chapter about how to do one-on-one usability testing and was so fired up about it, I couldn’t stand the thought of continuing to read until I’d at least tried to put his ideas into practice.

Since that time my two cofounders and I (we run Skritter)  have run quite a few usability tests.  My closest guess at this point is that I’ve run usability sessions with more than 50 people. We have benefited enormously from having one of us sit down face-to-face with someone who isn’t an internet addict and have them use our creation.

Over the last year we have become much better at usability testing, and I thought I would share 3 of the lessons we learned that Krug doesn’t mention in his book. The following tips were gleaned from several group and many one-on-one sessions, and I hope that prove useful as you start your own usability testing:

1) Never present a page as “redesigned” or “new.”

Krug mentions that testers are often polite. This is an understatement.

Case in point: we redesigned a page where people were stumbling on the UI. In the next batch of tests I asked users to tell me whether they liked the interface on the old page or the new one. The results were astounding: my new page was preferred 95% of the time! Wow, I’d massively improved the main vocabulary page! But wait, why was the number so high? Just for the heck of it, I started switching the pages that were “new” and “old.” Whereas before I’d give the testers variant A and then said “here’s a redesigned version,” and show them B, now I lied and did the opposite. You know what? Suddenly people loved the old page overwhelmingly.

Never underestimate the social graces of a tester, especially when you are paying them for their time (more about that later). I have tried and tried to get across to testers that I want them to criticize and be rude, but I’ve given up. If you absolutely need to test two pages against each other and you can’t use a multivariate or A/B testing framework (as was necessary in this case unfortunately), never tell a tester which version you just spent the last week building.

2) Pointedly ignore social cues for help.

At the beginning of a testing session, I always tell testers that I’m going to give them a task to complete, shut up, and watch them get stuck. I tell them I won’t offer help. If I’m really on task that day, I’ll repeat this point twice. So far, most people I’ve tested either don’t hear or don’t fully realize what I’m saying. They get stuck on our (inevitably bad) interfaces and turn to me and say: “Uhhhh …. am I doing this right? …. do I click here? ….”

In situations like this, you have to ignore any desire you have to smooth over the awkward moment by helping, and remain intentionally quiet. To help the person is to destroy the value of watching them muddle through it. In our case, I’ve learned incredible things from being quiet and letting people get by on their own. To give a few examples, I learned that people were trying to log in to the site by using the “create a new account” fields (which required them to do a double-take and re-enter their info), that the link to our press page was all but invisible, that after creating accounts only about 1/5 could figure out how to study a list, and that there is something irresistible about clicking this one out-of-the-way menu. We were able to fix a lot of things because I didn’t talk.

So halt thy tongue and let your tester’s actions do the talking.

3) People are often willing to do a usability test for a lot less than $50.

In his book, Krug mentions that he generally pays $50 for people without expertise to test web pages. When we first decided to start testing, we batted around various per-test figures. I wanted to do $30/session. My two cofounders thought I was throwing money away and that we should start low and gauge the response. Thank goodness I listened to them. We put up posters advertising $15/session for testers and we were inundated. If you choose the right demographic (we chose college students), testing can be done for much less than even Krug suggests.

We are still testing and refining our technique, and I’d be interested to hear if anyone else has tricks that help them wring more from their usability testing sessions!

Be Happier: Rent Everything

The Backstory

I recently read a short book entitled “Life Nomadic.” It was written by a programmer who one day he realized that living in Austin Texas wasn’t that cool. He put up a Craigslist ad telling people to come to his house and take all his belongings. Shortly thereafter he flew out of the country and began a life hopping from continent to continent working from his laptop.

This story fascinated me and my cofounders. The idea of living entirely out of a small backpack and sculpting the very texture of one’s life struck us as equal parts satisfying and empowering. This got me thinking: the biggest hurdle between me and a life like that was the fact that I owned too much stuff.

America Wants You to Own, not Rent

In his 2007 essay “Stuff,” Paul Graham explains that too many people in America gain too much from having you buy stuff. The result is that you will be pressured to buy things all the time. It takes a particularly strong individual to both notice and then resist the call of companies, some of which have spent hundreds of years perfecting their sales pitch.

These companies, and the people in them, make more money from you if you rent than if you own, which I believe is why ownership is prized: to own is to obtain something of value for less than you would pay with the corresponding financing plan. American middle class life is largely based on imitation of the rich, and one way to imitate the rich is to get good stuff for less than advertised. This is why I suspect that ownership is a quintessential aspect of American middle class life.

However, I’d like to challenge the conventional wisdom that it’s somehow better to own. I believe that for young, mobile, educated people, it makes more sense to avoid ownership wherever possible. I was taught to believe that a mortgage payment is superior to a rent check and that a monthly car payment is more noble than a similar-sized lease. In a strict economic sense my parents and grandparents didn’t lie to me: I will end up paying more to rent than own. But they overlooked the high cost of actually owning things.

Ownership Makes You Less Happy

There are substantial non-monetary advantages to not owning stuff that directly impact happiness. There are obvious situations in which owning belongings becomes a hindrance. Moving, for example, is harder if you own 2 tons of stuff and easier if you were just renting the same quantity. But it’s the more insidious costs of ownership that creep up on otherwise rational people and decrease their happiness.

Owning a pet can be a delight, but it is also a burden. When my girlfriend visits she has to get a cat sitter and she worries when she can’t get a hold of them.

Owning a large house gives you more personal space and makes day-to-day living more pleasurable. At the same time, bigger houses cost more to upkeep, they take more time to keep clean, and perhaps most dangerously: they allow you to accumulate stuff without noticing. My Uncle has a huge suburban home that is so full of stuff he can’t park his cars in the garage.

Speaking of which, having a car is convenient (and sometimes necessary), but is a monetary black hole and a source of stress if it’s having problems. My old Subaru got me from A to B but probably raised my blood pressure by more than the mileage as I worried about what would go wrong next.

Finally, owning very nice things makes using them less pleasurable because you have to be extra careful not to damage them (as Paul Graham says in his essay “Nothing owns you like fragile stuff”).

In short, ownership dampens your happiness and burdens you psychologically. Being able to leave your home and not worry about a pet adds to your happiness. Keeping your housing costs low and not accidentally accumulating stuff allows you to spend time and money on things that matter more. Riding the bus frees you from mechanic’s fees and being worried about “that squeaking sound the car makes when starting up.” And avoiding very nice things allows you to avoid the “good china” problem where you don’t want to use something for fear of damaging it.

So Question Ownership: Rent

Owning lots of things makes you less free and less happy, but renting is more expensive, so why do I advocate it? Simple: if you are young, frugal, and employed (something I’m well aware doesn’t describe everyone), you can almost certainly afford to pay a premium to avoid the stress of owning stuff. And at no other time in your life will renting improve your lifestyle more. So comparatively, the renting premium is a pittance. When you are young and don’t have a family, the extra freedom you can derive from not owning tons of stuff enables you to go to Europe for 3 weeks on a whim, quit your job and pursue your passion, invest all your energy into learning a skill, or build/strengthen relationships. Surely these things are worth an extra few hundred a month for a few years.

There is definitely a time and place for ownership. I just don’t believe it’s anywhere near as early in life as we Americans introduce it.

So I challenge you: do a cost-benefit analysis and see if you absolutely need that extra vehicle. Run the numbers to see if that smaller apartment in downtown wouldn’t help you reduce your pile of stuff. Look for a furnished apartment instead of hauling everything with you. Take small leaps at first, and maybe someday you too can be a happy globetrotter with only a backpack and an online freelance job of your own making.