*ahem*
Just needed to be said.
Move along, now. Nothing to see here.
Saturday, February 24, 2007
Tuesday, February 13, 2007
Fratricide
Test over. It’s time for the report. We have a protocol that you cannot test a design that you helped develop. This means that my test’s final report will be written up and presented by the people who conducted it.
They’re a good crew. The facilitator, SR, appears a bit casual - but they are a whiz at avoiding leading questions and practicing self-restraint. A person who knows the answer has to work hard not to hint to a participant who is struggling – SR is an ace at just responding minimally and getting the participant to explain their thought process.
SR is paired up with a serious usability pro, KR, so the report is sure to be airtight and professional. They are offsite, so we’ve scheduled a videoconference so the clients and developers can hear the report and ask questions.
Given the fact that my test plan has missed a key objective – I’m expecting to get tagged on the report. SR and KR won’t ding me for my plan – but a public reading of their findings can only highlight the gap in what I’d asked them to find out. I’m genuinely sweating it.
It’s the last item of the day, and all my other meetings cancel – so I’ve got all day to imagine the myriad ways SR will say “we don’t have much data on X.”
I ping SR twice about the report. “Make sure you hit the main test objectives first, the clients won’t want to hear about ancillary stuff until they have the big stuff.” I ask about a finding they mentioned in passing. I’m wondering if this finding will re-open old arguments with the Client. I get some assurances and go to lunch.
At work, I’ve been ever the optimist – but I just can’t relax. I hit one of the few decent restaurants near work and tell myself I might as well eat first.
Last time we used KR and SR, there were some communication errors that resulted in findings that were beyond the scope of the project, or made it look like my department didn’t understand the project they were working on. I’ve tried to minimize this risk as much as I can – but there’s so much information to get across. I’m afraid that SR will say something that will make us lose cred.
My department, we’re a latecomer to most projects, and we are generally regarded as just another mouth to feed. I’m convinced half the time that the Client would happily be rid of us – but somebody above them is forcing them to work with us.
The history of this project has been tumultuous – I smoke-jumped in just as people were getting canned. The timeline and scope keep getting changed in threatening ways – and we’ve been at loggerheads in design because the culture is “don’t change anything.” It’s the classic usability problem. Developers want the UI to look like the back-end code, the SME’s assume that everyone using the system eats and sleeps the operations manual, just like they do – it’s a mess.
I’ve gone out on a limb countless times, just to see if I can provoke discussions that lead us away from “do it the way we do now.” I’ve been shot down so many times, I’ve stopped noticing - but I’m determined to at least try to make things better.
Very rarely, one of our ideas gets approved, and my team celebrates. Sure, the whole design is a Frankenstein’s monster – but there’s this little part (call it function X) that we got into the project. We think it’s neat – we hope it lives until go-live.
It’s been a real struggle – and patience on some issues has worn so thin that we’ve abandoned all discussion of them. We’re just hoping that once things have cooled off, we can revisit them in better times.
So, here we are – having amassed a modest amount of cred – and I’m waiting for this presentation to highlight my test’s poor design.
I’ve told SR that he should plow through the biggies and expect interruptions. My group prefers the piranha method of discussion, so any suggestion that we hold questions is just a waste.
And we start.
---
SR starts on the part of the UI that I failed to evaluate against a previous design. Participants encountering the new design failed to see and interact with it 100% of the time.
No biggie, we just prompt them to look at it and observe their reactions to it with a prompt. Having seen most of the sessions – I know that most people were able to use the UI with a prompt. This is not ideal, but it means that our problem is likely one of visibility and labeling rather than actual function.
Except SR didn’t see it that way. SR details failure after failure in this UI even after prompts. This I do not recall, but I wasn’t at all the sessions. I’m a bit surprised, because this is a key piece of UI that two projects were waiting for. I’d thought (and unfortunately, said) that the issues with it were cosmetic and we could probably fix it easily. SR details a series of convincing issues that say otherwise. One of the project managers, short on time, expresses alarm and then leaves.
SR continues into another section I thought went well and proceeds to document why it is unadulterated crap. I’m really not liking where this is going.
Questions? The room is dead quiet. Everybody’s here, project managers, the Client, my boss – nobody says a thing.
It’s my show on this end, so I just tell SR to plow onward and they do. Right into a pet issue of mine. One of those little victories that I’d endured weeks of abuse to get into the design is now being held up as a cautionary tale. The Client and I were on opposite sides of this one and I’d carried the day mainly on persistence. Now, SR is basically listing all the same complaints the Client gave me when I first came up with the design.
The Client and the rest of the dev team go quiet – they remember the same discussions. They thought I was nuts – and SR is telling them all they were right to say so.
Now, normally, stuff like this – I don’t care. A design dies, but you make a better one out of the ashes. Everybody’s design takes a pounding, everyone gets a turn. But with this crowd, there will be people who will use this moment to discount any future input from me.
It’s dead quiet. SR’s clearly a bit surprised by this, and tries to liven things up. He addresses me directly “You look like you have something intelligent to say.”
I’m about to reply when I’m cut off –
“No, he doesn’t” says the Client, scoffing.
We have a few pro forma questions and then it’s time to go.
---
SR’s report was honest and accurate – yet it managed to hit every issue where the Client had doubts about us and strengthen them. I’m absolutely stunned.
My boss is gone. No chance to debrief. One of the more sympathetic types comes up and says “Back to the drawing board, eh?” I’m looking over at the Client, who is in a huddle with the people they still have respect for.
I can’t even get my head around this. Every discussion after this will be uphill times ten.
Next meeting, all my suggestions will be stillborn – the Client may want me off their project and my boss, too for all I know.
I don’t blame SR at all – these were honest conclusions, but what are the odds that all the stuff singled out for abuse would be mine? Yeah, I know law of averages, but did it have to frickin' happen now??!!
All this fighting for a place at the table, some respect and now –
Shit. Square one if I’m lucky.
They’re a good crew. The facilitator, SR, appears a bit casual - but they are a whiz at avoiding leading questions and practicing self-restraint. A person who knows the answer has to work hard not to hint to a participant who is struggling – SR is an ace at just responding minimally and getting the participant to explain their thought process.
SR is paired up with a serious usability pro, KR, so the report is sure to be airtight and professional. They are offsite, so we’ve scheduled a videoconference so the clients and developers can hear the report and ask questions.
Given the fact that my test plan has missed a key objective – I’m expecting to get tagged on the report. SR and KR won’t ding me for my plan – but a public reading of their findings can only highlight the gap in what I’d asked them to find out. I’m genuinely sweating it.
It’s the last item of the day, and all my other meetings cancel – so I’ve got all day to imagine the myriad ways SR will say “we don’t have much data on X.”
I ping SR twice about the report. “Make sure you hit the main test objectives first, the clients won’t want to hear about ancillary stuff until they have the big stuff.” I ask about a finding they mentioned in passing. I’m wondering if this finding will re-open old arguments with the Client. I get some assurances and go to lunch.
At work, I’ve been ever the optimist – but I just can’t relax. I hit one of the few decent restaurants near work and tell myself I might as well eat first.
Last time we used KR and SR, there were some communication errors that resulted in findings that were beyond the scope of the project, or made it look like my department didn’t understand the project they were working on. I’ve tried to minimize this risk as much as I can – but there’s so much information to get across. I’m afraid that SR will say something that will make us lose cred.
My department, we’re a latecomer to most projects, and we are generally regarded as just another mouth to feed. I’m convinced half the time that the Client would happily be rid of us – but somebody above them is forcing them to work with us.
The history of this project has been tumultuous – I smoke-jumped in just as people were getting canned. The timeline and scope keep getting changed in threatening ways – and we’ve been at loggerheads in design because the culture is “don’t change anything.” It’s the classic usability problem. Developers want the UI to look like the back-end code, the SME’s assume that everyone using the system eats and sleeps the operations manual, just like they do – it’s a mess.
I’ve gone out on a limb countless times, just to see if I can provoke discussions that lead us away from “do it the way we do now.” I’ve been shot down so many times, I’ve stopped noticing - but I’m determined to at least try to make things better.
Very rarely, one of our ideas gets approved, and my team celebrates. Sure, the whole design is a Frankenstein’s monster – but there’s this little part (call it function X) that we got into the project. We think it’s neat – we hope it lives until go-live.
It’s been a real struggle – and patience on some issues has worn so thin that we’ve abandoned all discussion of them. We’re just hoping that once things have cooled off, we can revisit them in better times.
So, here we are – having amassed a modest amount of cred – and I’m waiting for this presentation to highlight my test’s poor design.
I’ve told SR that he should plow through the biggies and expect interruptions. My group prefers the piranha method of discussion, so any suggestion that we hold questions is just a waste.
And we start.
---
SR starts on the part of the UI that I failed to evaluate against a previous design. Participants encountering the new design failed to see and interact with it 100% of the time.
No biggie, we just prompt them to look at it and observe their reactions to it with a prompt. Having seen most of the sessions – I know that most people were able to use the UI with a prompt. This is not ideal, but it means that our problem is likely one of visibility and labeling rather than actual function.
Except SR didn’t see it that way. SR details failure after failure in this UI even after prompts. This I do not recall, but I wasn’t at all the sessions. I’m a bit surprised, because this is a key piece of UI that two projects were waiting for. I’d thought (and unfortunately, said) that the issues with it were cosmetic and we could probably fix it easily. SR details a series of convincing issues that say otherwise. One of the project managers, short on time, expresses alarm and then leaves.
SR continues into another section I thought went well and proceeds to document why it is unadulterated crap. I’m really not liking where this is going.
Questions? The room is dead quiet. Everybody’s here, project managers, the Client, my boss – nobody says a thing.
It’s my show on this end, so I just tell SR to plow onward and they do. Right into a pet issue of mine. One of those little victories that I’d endured weeks of abuse to get into the design is now being held up as a cautionary tale. The Client and I were on opposite sides of this one and I’d carried the day mainly on persistence. Now, SR is basically listing all the same complaints the Client gave me when I first came up with the design.
The Client and the rest of the dev team go quiet – they remember the same discussions. They thought I was nuts – and SR is telling them all they were right to say so.
Now, normally, stuff like this – I don’t care. A design dies, but you make a better one out of the ashes. Everybody’s design takes a pounding, everyone gets a turn. But with this crowd, there will be people who will use this moment to discount any future input from me.
Remember that one thing? How you insisted it was better? Well, you were wrong, so shaddup!SR’s report plows onward, ripping on two other items that I’ll take the fall for, before finally winding down on issues that were too minor for the team to care much about.
It’s dead quiet. SR’s clearly a bit surprised by this, and tries to liven things up. He addresses me directly “You look like you have something intelligent to say.”
I’m about to reply when I’m cut off –
“No, he doesn’t” says the Client, scoffing.
We have a few pro forma questions and then it’s time to go.
---
SR’s report was honest and accurate – yet it managed to hit every issue where the Client had doubts about us and strengthen them. I’m absolutely stunned.
My boss is gone. No chance to debrief. One of the more sympathetic types comes up and says “Back to the drawing board, eh?” I’m looking over at the Client, who is in a huddle with the people they still have respect for.
I can’t even get my head around this. Every discussion after this will be uphill times ten.
Next meeting, all my suggestions will be stillborn – the Client may want me off their project and my boss, too for all I know.
I don’t blame SR at all – these were honest conclusions, but what are the odds that all the stuff singled out for abuse would be mine? Yeah, I know law of averages, but did it have to frickin' happen now??!!
All this fighting for a place at the table, some respect and now –
Shit. Square one if I’m lucky.
Thursday, February 08, 2007
Rockstar
Like most jobs, mine has its share of drudgery. But one thing I do (that I have to do) I truly enjoy.
I get to test software on people.
Yeah, I know - that doesn't sound like much, but in my line of work - it’s the Superbowl. Testing a design with typical users and seeing how the software can be made better.
There is considerable debate as to whether or not this is science, art, or outright chicanery - but I think anyone who has participated in a lab test would agree - it gets your attention.
Having spent months in design, hashing out what an application needs to do - then spending more months designing a good way to do it - you finally put a prototype in front of someone and see what they think of it.
I may be mentally ill, but to me, it's a hell of a good time.
The doomsday project has stabilized to the point that it almost doesn't deserve the name, and I've spent the past month or so putting together testing scenarios and trying to find the best way to get a user to interact with the interface in ways that will give us useful feedback.
This is not easy. The first rule of user testing is that the participants will always surprise you. A task you are sure will be simple will prove maddeningly difficult for them. Participants will fly through your thorniest scenario with ease, only to fail on task because you made a bad choice in labeling the final "commit" button.
The ideal is to give users tasks that they would perform in real life. Then, maybe they'll act like it's real - and then maybe they'll even tell you what they really think. My scenarios are (to my mind) real-world, hands-off - but touch on all the design elements we care about. Trying to find the right mix of organic observation and manipulation can drive you nuts.
That said, testing week is like a carnival. AV setups - recording software - network connections, scheduling people, blah blah blah. I have to run a meeting tomorrow, I owe project work for other things - everything gets benched. People want the test results, they'll just have to deal.
Our prototype is (at least to my mind) a miracle of JavaScript posing as a live system. We have a design that would probably be easier to code as a working system than to simulate with script - but it's ready.
I start cataloging a list of all the things that could go wrong.
One by one, they don't.
The participants show up - despite subzero temperatures and crappy roads.
Inevitable shortcomings of a prototype do not break our test.
People tell us what they think.
The videoconference works.
IMing remote questions proves useful.
Our new screen capture software not only works, but is pretty damn cool.
Me and my team are loving it. We toast our prototyper.
Rockstar!
This becomes our chosen word of congratulations.
When the facilitator deftly handles a test snag, or when the notetaker pulls off the always-elusive "How do I ask this without telling them the answer?" question, over and over.
Rockstar!
We're getting really great comments - people are tanking on one task, but we know why - so we can fix it. All sorts of side comments are pointing out the vast difference between how our users actually work - and how our developers think they do. I'm drafting a list of follow up issues for other projects that are just starting up - so we can head off some really bad choices.
Living the life.
My boss comes in. They love the prototype, shower the team with praise. Rockstar!
The Client walks in. Wow.
This is a good thing. Most times, we can't get dev types or clients in the room to see what we see. The mere possibility that a client could actually watch a participant is great for us.
Assumptions are made in corp-world that have no basis in reality. Seeing a participant struggle with someone's pet design will usually trump business politics and get needed improvements a shot in the arm.
Participants aren't shy. They loudly announce that the corporate types have no idea what their needs are, point out workflow shortcomings and highlight obvious improvements that have somehow remained undone for years.
Our team has fought tooth and nail for credibility on this project. Lord knows, the H situation cost us heavily. We can really gain some corp-cred - if nothing goes wrong for this session.
I know what you're thinking, but nothing does go wrong. Aside from being overly talkative, the session goes much like previous sessions. Problems on the known issue, A/B testing results about the same. A few choice comments about the interface, and fade to black.
Me and my team are smug - cracking wise and cutting up on the IM.
I'm jazzed. I turn to the Client. "I can see some obvious improvements to this area of the design, but once people understand what it is trying to do - they do it easily."
The Client is un-moved. "This new design is replacing a prior one. There is nothing in this test that will allow me to compare the new design to the old. I still don't know which design is more successful."
I start with my list of proposed improvements to the new design. Our prototyper starts to chime in.
Client addresses the prototyper like I'm not even there. "We started with two options, we still have them - and we get nothing definitive about them from this test. I saw that as soon as the test started."
All the air goes out of the room. Client's right. I've put together this whole thing and I've missed one of the main goals. Client could care less about my list of findings, our list of issues, much less our ideas for future projects.
Three weeks delay, two huge projects waiting for the results of this test - and here we are, right back at a guessing game because I didn't force my participants to make a choice.
Client spells out what they would have done to get this answer - and I'm just sitting there taking it.
About the time they're done, my laptop fires off a warning shot - my boss has invited me to a meeting about some other project on the other side of campus in about three minutes.
I'm ten minutes late if I hurry.
I grab my things and bail - I've lost all cred with the Client, and now they get to watch me scramble to get my things and dash off to a meeting I've forgotten.
The whole way there, I'm just projecting the fallout. "Hey, nice test! Looked like fun! Complete waste of time, though." I'm too spent to even start on damage control. The benched work is rushing up in my mind - that roomful of people in tomorrow's meeting (that I haven't prepped for), I'm envisioning a really fun Q & A session...
And just like that, it's doomsday again.
I get to test software on people.
Yeah, I know - that doesn't sound like much, but in my line of work - it’s the Superbowl. Testing a design with typical users and seeing how the software can be made better.
There is considerable debate as to whether or not this is science, art, or outright chicanery - but I think anyone who has participated in a lab test would agree - it gets your attention.
Having spent months in design, hashing out what an application needs to do - then spending more months designing a good way to do it - you finally put a prototype in front of someone and see what they think of it.
I may be mentally ill, but to me, it's a hell of a good time.
The doomsday project has stabilized to the point that it almost doesn't deserve the name, and I've spent the past month or so putting together testing scenarios and trying to find the best way to get a user to interact with the interface in ways that will give us useful feedback.
This is not easy. The first rule of user testing is that the participants will always surprise you. A task you are sure will be simple will prove maddeningly difficult for them. Participants will fly through your thorniest scenario with ease, only to fail on task because you made a bad choice in labeling the final "commit" button.
The ideal is to give users tasks that they would perform in real life. Then, maybe they'll act like it's real - and then maybe they'll even tell you what they really think. My scenarios are (to my mind) real-world, hands-off - but touch on all the design elements we care about. Trying to find the right mix of organic observation and manipulation can drive you nuts.
That said, testing week is like a carnival. AV setups - recording software - network connections, scheduling people, blah blah blah. I have to run a meeting tomorrow, I owe project work for other things - everything gets benched. People want the test results, they'll just have to deal.
Our prototype is (at least to my mind) a miracle of JavaScript posing as a live system. We have a design that would probably be easier to code as a working system than to simulate with script - but it's ready.
I start cataloging a list of all the things that could go wrong.
One by one, they don't.
The participants show up - despite subzero temperatures and crappy roads.
Inevitable shortcomings of a prototype do not break our test.
People tell us what they think.
The videoconference works.
IMing remote questions proves useful.
Our new screen capture software not only works, but is pretty damn cool.
Me and my team are loving it. We toast our prototyper.
Rockstar!
This becomes our chosen word of congratulations.
When the facilitator deftly handles a test snag, or when the notetaker pulls off the always-elusive "How do I ask this without telling them the answer?" question, over and over.
Rockstar!
We're getting really great comments - people are tanking on one task, but we know why - so we can fix it. All sorts of side comments are pointing out the vast difference between how our users actually work - and how our developers think they do. I'm drafting a list of follow up issues for other projects that are just starting up - so we can head off some really bad choices.
Living the life.
My boss comes in. They love the prototype, shower the team with praise. Rockstar!
The Client walks in. Wow.
This is a good thing. Most times, we can't get dev types or clients in the room to see what we see. The mere possibility that a client could actually watch a participant is great for us.
Assumptions are made in corp-world that have no basis in reality. Seeing a participant struggle with someone's pet design will usually trump business politics and get needed improvements a shot in the arm.
Participants aren't shy. They loudly announce that the corporate types have no idea what their needs are, point out workflow shortcomings and highlight obvious improvements that have somehow remained undone for years.
Our team has fought tooth and nail for credibility on this project. Lord knows, the H situation cost us heavily. We can really gain some corp-cred - if nothing goes wrong for this session.
I know what you're thinking, but nothing does go wrong. Aside from being overly talkative, the session goes much like previous sessions. Problems on the known issue, A/B testing results about the same. A few choice comments about the interface, and fade to black.
Me and my team are smug - cracking wise and cutting up on the IM.
I'm jazzed. I turn to the Client. "I can see some obvious improvements to this area of the design, but once people understand what it is trying to do - they do it easily."
The Client is un-moved. "This new design is replacing a prior one. There is nothing in this test that will allow me to compare the new design to the old. I still don't know which design is more successful."
I start with my list of proposed improvements to the new design. Our prototyper starts to chime in.
Client addresses the prototyper like I'm not even there. "We started with two options, we still have them - and we get nothing definitive about them from this test. I saw that as soon as the test started."
All the air goes out of the room. Client's right. I've put together this whole thing and I've missed one of the main goals. Client could care less about my list of findings, our list of issues, much less our ideas for future projects.
Three weeks delay, two huge projects waiting for the results of this test - and here we are, right back at a guessing game because I didn't force my participants to make a choice.
Client spells out what they would have done to get this answer - and I'm just sitting there taking it.
About the time they're done, my laptop fires off a warning shot - my boss has invited me to a meeting about some other project on the other side of campus in about three minutes.
I'm ten minutes late if I hurry.
I grab my things and bail - I've lost all cred with the Client, and now they get to watch me scramble to get my things and dash off to a meeting I've forgotten.
The whole way there, I'm just projecting the fallout. "Hey, nice test! Looked like fun! Complete waste of time, though." I'm too spent to even start on damage control. The benched work is rushing up in my mind - that roomful of people in tomorrow's meeting (that I haven't prepped for), I'm envisioning a really fun Q & A session...
And just like that, it's doomsday again.
Subscribe to:
Posts (Atom)