Rockstar

Like most jobs, mine has its share of drudgery. But one thing I do (that I have to do) I truly enjoy.

I get to test software on people.

Yeah, I know - that doesn't sound like much, but in my line of work - it’s the Superbowl. Testing a design with typical users and seeing how the software can be made better.

There is considerable debate as to whether or not this is science, art, or outright chicanery - but I think anyone who has participated in a lab test would agree - it gets your attention.

Having spent months in design, hashing out what an application needs to do - then spending more months designing a good way to do it - you finally put a prototype in front of someone and see what they think of it.

I may be mentally ill, but to me, it's a hell of a good time.

The doomsday project has stabilized to the point that it almost doesn't deserve the name, and I've spent the past month or so putting together testing scenarios and trying to find the best way to get a user to interact with the interface in ways that will give us useful feedback.

This is not easy. The first rule of user testing is that the participants will always surprise you. A task you are sure will be simple will prove maddeningly difficult for them. Participants will fly through your thorniest scenario with ease, only to fail on task because you made a bad choice in labeling the final "commit" button.

The ideal is to give users tasks that they would perform in real life. Then, maybe they'll act like it's real - and then maybe they'll even tell you what they really think. My scenarios are (to my mind) real-world, hands-off - but touch on all the design elements we care about. Trying to find the right mix of organic observation and manipulation can drive you nuts.

That said, testing week is like a carnival. AV setups - recording software - network connections, scheduling people, blah blah blah. I have to run a meeting tomorrow, I owe project work for other things - everything gets benched. People want the test results, they'll just have to deal.

Our prototype is (at least to my mind) a miracle of JavaScript posing as a live system. We have a design that would probably be easier to code as a working system than to simulate with script - but it's ready.

I start cataloging a list of all the things that could go wrong.

One by one, they don't.

The participants show up - despite subzero temperatures and crappy roads.
Inevitable shortcomings of a prototype do not break our test.
People tell us what they think.
The videoconference works.
IMing remote questions proves useful.
Our new screen capture software not only works, but is pretty damn cool.

Me and my team are loving it. We toast our prototyper.

Rockstar!

This becomes our chosen word of congratulations.

When the facilitator deftly handles a test snag, or when the notetaker pulls off the always-elusive "How do I ask this without telling them the answer?" question, over and over.

Rockstar!

We're getting really great comments - people are tanking on one task, but we know why - so we can fix it. All sorts of side comments are pointing out the vast difference between how our users actually work - and how our developers think they do. I'm drafting a list of follow up issues for other projects that are just starting up - so we can head off some really bad choices.

Living the life.

My boss comes in. They love the prototype, shower the team with praise. Rockstar!

The Client walks in. Wow.

This is a good thing. Most times, we can't get dev types or clients in the room to see what we see. The mere possibility that a client could actually watch a participant is great for us.

Assumptions are made in corp-world that have no basis in reality. Seeing a participant struggle with someone's pet design will usually trump business politics and get needed improvements a shot in the arm.

Participants aren't shy. They loudly announce that the corporate types have no idea what their needs are, point out workflow shortcomings and highlight obvious improvements that have somehow remained undone for years.

Our team has fought tooth and nail for credibility on this project. Lord knows, the H situation cost us heavily. We can really gain some corp-cred - if nothing goes wrong for this session.

I know what you're thinking, but nothing does go wrong. Aside from being overly talkative, the session goes much like previous sessions. Problems on the known issue, A/B testing results about the same. A few choice comments about the interface, and fade to black.

Me and my team are smug - cracking wise and cutting up on the IM.

I'm jazzed. I turn to the Client. "I can see some obvious improvements to this area of the design, but once people understand what it is trying to do - they do it easily."

The Client is un-moved. "This new design is replacing a prior one. There is nothing in this test that will allow me to compare the new design to the old. I still don't know which design is more successful."

I start with my list of proposed improvements to the new design. Our prototyper starts to chime in.

Client addresses the prototyper like I'm not even there. "We started with two options, we still have them - and we get nothing definitive about them from this test. I saw that as soon as the test started."

All the air goes out of the room. Client's right. I've put together this whole thing and I've missed one of the main goals. Client could care less about my list of findings, our list of issues, much less our ideas for future projects.

Three weeks delay, two huge projects waiting for the results of this test - and here we are, right back at a guessing game because I didn't force my participants to make a choice.

Client spells out what they would have done to get this answer - and I'm just sitting there taking it.

About the time they're done, my laptop fires off a warning shot - my boss has invited me to a meeting about some other project on the other side of campus in about three minutes.

I'm ten minutes late if I hurry.

I grab my things and bail - I've lost all cred with the Client, and now they get to watch me scramble to get my things and dash off to a meeting I've forgotten.

The whole way there, I'm just projecting the fallout. "Hey, nice test! Looked like fun! Complete waste of time, though." I'm too spent to even start on damage control. The benched work is rushing up in my mind - that roomful of people in tomorrow's meeting (that I haven't prepped for), I'm envisioning a really fun Q & A session...

And just like that, it's doomsday again.