Alright, you're gonna think I'm looking for something new to slag on - but this really happened.
And
I
get
to
b!tch
about
it.
* * *
Rewind to a several months back. Noddy's webform is only three months late, and it's barely a form. Scratch that - it's barely the
idea of a form. Noddy's written up a story for the devs that says basically, "put some text boxes and dropdowns on a web form. Oh, and give them labels, too." No validation, no function - a dead end of a feature that mimics the same old schlock in the rest of our app.
Which is sad, because Noddy has been given a gift - and was blowing it. All the webforms in our app were built with a common engine. A common engine that was built before we thought about UX at all. As a result, all our forms are uniformly terrible. Improving one form meant doing the heavy lifting of overhauling the entire engine.
But not Noddy's form. Noddy got the greenlight to be a standalone new form. Which means Noddy is free to do all the modern UI they want.
And they're replicating the same old schlock.
Runner has noticed this and is trying to browbeat some quality into Noddy's up-till-now lame-@ss feature. Runner's been appointed the guardian of the style guide and there are so many things that need fixing in our forms - they cannot bear seeing the same crap happen all over again.
Our forms have many problems but two of the newly fixable problems are our multiple required indicators and lack of conditional display.
Our forms have two kinds of required indicators (field is required to save, field is required if you want to mark the document as 'finished'). This is bad enough, but is compounded by the fact that we
don't have an indicator for fields that are required to submit the form to Nachen.
Mind you, the whole reason you fill out our forms, is to send them to Nachen. And we
don't have required indicators on fields that Nachen requires.
Beyond insane.
If we just add more required indicators, we would end up with THREE different required indicators.
Which
simply cannot happen. Since whatever solution we come up with could influence other forms, we call a UX huddle to agree on a first step. Runner, Noddy, myself and Ahhh end up in a room and spell out the problem.
Required to save seems like the easiest indicator to lose. It turns out only two fields of data are required to save a form. One is prefilled, and the other is just a date field.
I've seen other forms with this problem and rather than make you read little symbols on the page to see what needs to be done, they frontload some prerequisite fields and don't let you see the rest of the form until you fill them out.
So, let's put the two required to save fields on a mini form before the main form. Fill out the date field and hit "continue" and we'll show you the rest of the form. Then, once you get to the actual form, you can save at any time. *Poof* 1 less required indicator.
We come up with a conditional display of the Nachen indicators that seems good.
The team seems okay with this as a first shot, and Ahhh (true to form) wants to have a wide ranging discussion of how we will come up with a design language for validation going forward. Ahhh is just compelled to make discussions larger and less fruitful - we call a halt for now and I send out the meeting minutes.
From a possible 3 indicators, we're down to 1, with an option to turn on a second. Not bad for an hour meeting.
* * *
Let's skip to a few weeks later. Noddy's demoing the form with the mini form up front and Ihaq is laying in like a buzzsaw. "I've never seen that before. I'd like to have a discussion about this."
When Ihaq says things like this, you can tell they are not pleased. True, this is a new UI element, but Ihaq is the same SME who thought our previous required indicators were for Nachen fields. This is our Subject Matter Expert - who as worked here for years, misinterpreting the standard interface of our App.
There is no better proof we have a UI problem with required fields.
But back to the UI - the SMEs get to weigh in on their field of expertise, but they don't get to design the interface. That's our job. If the SMEs have a specialist objection to the workflow, like they know the user won't have the required data up front, then sure - we'll listen to them and adjust. But if they just don't like the button, tough bounce.
Post showcase, Gumby asks after the UI, "did we test this?"
No, I say,
this was a decision that came out of UX. I can tell Gumby is still smarting from being called out in the stakeholder meeting, but there's not much to do about it.
Later, Runner, Noddy, Gumby and Ahhh meet with the SMEs about Nachen work - and Ahhh pulls a reversal. They denying playing any part in making the mini form, despite being in our meeting and agreeing to the approach. Despite the fact that progressive disclosure was something Ahhh had suggested in prior meetings.
I'm irritated and dig up the meeting minutes that show our decision to use the form. I don't send it to Gumby, but I show Runner so they know that Ahhh is full of it. Standing up to the SMEs can be hard, and I can see the pressure to please them - but you can't just surrender the role of UI design. SMEs inform the design, they don't draft it. Ihaq doesn't like the progressive disclosure of the mini form - because it is new. Not because it is a real problem, they just were unprepared for it.
Gumby wants us to vet the entire form in a user test - and our new guy, Dash is supposed to put that together. Heck, at some point I was going to be doing it - but Arwafn blew up and put that to rest. Dash would be great - they are a veteran user tester and completely outside of the design.
A designer should
not test their own design. Bad things happen if you do.
Somehow, the test doesn't manage to happen. I'm eye-deep in the Arwafn hairball and forget we were even thinking of testing the form.
* * *
So, a few days ago at standup, Ahhh mentions that they are working on a user test. I'm like
great! I love user tests - they are awesome. Sure,
sometimes they kick your ass, but they are so worth it. Normal run of things back at CorpWorld was invites would go out to the full team to observe - so everyone can learn from the test.
A test has a primary and a second. Primary runs the show and writes up the findings, with final edit. The second is pretty much there to help with the metric ton of work that a good test requires.
A good test should look effortless to the participant, but getting credible task scenarios is not easy. Getting a working prototype to test those scenarios is pretty frickin hard. And notetaking, and analysis, and on and on....
I don't know who is leaping on all of that, but I'm glad that we'll finally get some user testing done at Nerdhaven.
Whoo! Heck, if things calm down in a week or so, I might have time to help out.
The next day, I get an email from Ahhh "I can't get the form to load and I've got a user session in a few hours."
I'm headed out the door to pick up the kids (it's my early day) so after I figure out what they are asking I get QA to help Ahhh out.
User session? WTH?
But I'm out the door.
* * *
Day after that, Ahh is talking about the next user test they're setting up, and there's still problems with the form loading. I can't help myself, I fire up IM:
User test? Can you forward the webmeeting link? I'd love to observe? I back off a bit.
Or are you recording?
Ahh takes awhile to respond. All the while I'm thinking:
How is it that they are holding tests and haven't emailed the team about it. No one else on the team even knew this was happening.
This is weird. Somebody should be helping Ahhh with the test. Not only is this the first user test that Ahhh is doing for NerdHaven - this is the first user test that's been done at NerdHaven. Period.
How is this being done by one person? And our newest UXer, to boot?
Ahh responds with "I'm recording. I'll get you the link." The implication is, I'll get back to you. Now, I don't need to observe in realtime - observers don't get to ask questions normally anyway - too much pressure on the participant. But the tone was bothersome.
Ahhh was at the point of having participants in sessions and nobody had heard a thing. Asked for access, Ahhh has basically said I'll tell you when it's over.
Later that day, Dash sits back from their desk - and snorts.
"I tried to get a hold of Product and they say they are busy because - get this - they are observing Ahh's user test."
WHAT? Okay, people on your team - whose interface you are testing (Dash's prototype is also being tested) have not been allowed to observe - but Product is sitting in.
Nice.
I have a quick client call and end up debriefing with the account Rep on how it went. The Rep's like "No problem, it went fine. I was on that user study with Ahhh earlier and that was a little rough, but it all works out."
I can't help myself - "Can you send me the link to the recording?"
That's sh!tty of me, but Ahhh said they'd send the recording, and they haven't. And really, the recording is made by the rep, so I'm jumping past the middle man.
The rep - in an act of karmic payback - totally forgets to send me the link. They are good people, and I believe this is so - but it is right that I should not get this from some back ally maneuver.
I should get it from Ahhh - directly. And Ahhh is pointedly not giving me, or any other UXer access to their tests.
* * *
I cannot stress enough how bizarre this is. Designers can get pretty prickly about their pet designs, but any designer worth even half a hoot knows you are going to get better design with collaboration.
More to the point, any designer worth considerably less than that knows their own bias is going to hang them if they don't have regular contact with opinions outside their personal bubble.
Ahhh makes much of their history in UX. They have an MBA, they've been doing user research for almost twenty years. I have heard this ad nauseam pretty much every time our team is asked to introduce themselves to some guest.
I get it, Ahhh. I do. You are not wet behind the ears when it comes to UX. So why the F*&k are you doing user research in secret?
The ENTIRE POINT of research is to share findings - and while convincing people with research is nigh impossible - it is
absolutely impossible if your audience thinks your research came from some black box process.
Erika Hall gave
a talk on design research that just blew my mind - not because it's some previously unthought hunk of wisdom, but because it perfectly reveals wisdom that is hiding in plain sight. Research that comes from the mountaintop in the hands of some sage is trivial to dismiss.
Sages...amIright? Research that tells a story can resonate. Stories you participate in are far more compelling than stories that are told to you.
If you give a crap about research, watch Erika make it make sense - or buy her book - or both - because I'm going to get back to griping about work.
* * *
So...
Whisky Tango Foxtrot, right? I do not go to the boss, because this is not a discussion that needs management. I haven't confronted Ahhh about the situation, I'm merely asked and not gotten much of a response. I've vowed to be direct about things at this job, and I'm going to do this if it burns every bridge around - because dammit, direct communication should not be a last resort.
I'm neck deep in some Arwafn stuff, when Ahhh gives her standup update as, "I have user testing session today."
What?? Still no upfront notice? Okay, we'll talk, you and I.
I call Ahhh out at standup and ask to be able to observe the test. Ahhh's on the video conference, in front of the entire team. Rather than look petty in front of all of us, they hedge, "If you
want to... I guess..."
I do. And I make it known again, that I do. I corner Ahhh on IM and ask them for the web conference link, and book a room on our office to invite the entire UX team.
Ahhh becomes very prickly over IM when I mention that I'm grabbing a room.
"WHO is going to be attending?"
I am upfront to the max.
I'll send you a list of the attendees and everyone in the room will understand that we are *observing* not questioning or interfering. We know the drill.
I'm starting to get the sense that Ahhh is afraid they will have some higher up observing them, and they are all kinds of on guard. Which I would understand if we were back in CorpWorld, but this is NerdHaven - we don't do that kind of bullsh!t here.
I'm not ambushing you by videoconference, jeezus.
Most of our client contacts are managed by Reps, who maintain client relationships far better than anyone on our team could - so I ask who Ahhh's Rep is on the test.
"Does it matter?" is Ahhs response.
Well, yes - actually. I'd like to invite them to the room I'm booking for the other observers - in case they want to be there.
A lot of Reps do all their conferences from their desk, but some find value in being in a room of people who are also on the call. I'm amazed this has not occurred to Ahhh, but I spell this out.
They icily give me the name of their Rep and sign off.
Well...
So, now I've got a conference room with Runner, Heater, Dash, and the Rep, and we're finally going to observe the test. Irritation aside, I'm actually excited, because tests are exciting. Real users, doing stuff.
w00t!
The session starts up, and Ahhh - twenty years of UX/research etc - is H-O-R-R-I-B-L-E. I don't know any other way to say it. I mean, typically if you have a test, you have scenarios with test data that the participant uses to attempt each task. There are success criteria/ fail criteria. There's a notetaker, there's...
Ahh has nothing. They have the live system and a participant, sure, but "Hey, try to fill out this form" is not a valid set of instructions. Our users - like most users - are truly bad at making up a set of data to fill an ill defined scenario. I'd actually say our users are worse than most, because our users are highly trained, detail oriented, and used to working in incredibly complex environments.
They sweat every detail and anything out of place will have them derailed and talking about bogus data for twenty minutes. They - as a rule - do not make up data. They want to know details before they proceed.
Ahhh has started a 'test' by loading up the mini form and saying 'have at it.'
'Oh, and tell me what you think of what you're seeing....'
The participant is looking at the mini form which is asking for a specific date. They are like 'you want me to make up a date?'
Ahhh is like 'oh sure'
So the participant types in the current date and hits the button to go to the rest of the form.
Ahhh does not ask them what information they would start the task with, nor have they supplied any. They don't ask why they chose today's date, if that would be representative, or where they would get a real date if this were an actual scenario.
What Ahhh
does ask about - is the mini form. 'What did you think of that - not seeing the rest of the form until after you put in the date?'
The participant is confused 'uh, it was fine.'
Ahhh asks about this a few more times, phrasing the question differently each time. I'm looking over at Dash like
what the...? until I get it.
Ahhh doesn't like the mini form - and they are really hoping that the participant will find fault with it. When that doesn't happen, Ahhh gives them a few more chances, in hopes of hearing what they want to hear.
Which is RULE NUMBER ONE as to why you don't test designs you've had a hand in making if you have any way of avoiding it.
I'm shaking my head when Ahhh asks about a piece of form UI that they designed - and believe is crucial to the workflow. "What do you think of that?'
The participant is confused. They hadn't noticed Ahhh's pet piece of UI until now. 'uh, I dunno.'
Ahhh repeats the question, gets a similar response - then tells the participant what the UI element is and what it is trying to do.
At this point, I'm about to roll my eyes when I catch Dash shaking their head. This is another thing you just DO NOT DO in a user test - if you have any way at all of avoiding it. You do not tell the participant what the interface is, or what it is doing. You let them tell you what they think it is doing. This way you can learn if if it's intuitive or not. As soon as you tell them the answer, you've lost the chance of learning more about the participant's point of view.
Now, there are lots of research techniques for lots of different situations, but this is a user test. The participant's unadulterated behavior is the entire point of the exercise. They may falter, but you do not give them assistance until you've learned what their perspective was
without assistance.
Ahhh wants their pet UI validated by the test, and they cannot stop themselves from interfering. They do this repeatedly until Dash and I are absolutely disgusted.
There are no success or fail criteria, no test scenario, so the participant gets to the full form and basically stops doing anything and waits for more information. Ahh pokes them a bit by asking for their thoughts about the form, what they are thinking - typical UX probing techniques. But there is no task, so this is just an airy, off the cuff discussion.
This is not a test. This is a walk in the woods.
"What do you see?"
I see trees. I like trees...
The entire session is just a colossal waste of resources and time. I've worked hard to get access to see this so I can learn more about our users - and all I've learned is that Ahhh is
really, seriously bad at testing.
A lot of what Ahhh did in the session would have been fine if it was just an informative interview - "hey, tell us about this" but this was set up to be a test - more to the point - this is being *represented to the organization as a test* and it was nothing of the sort. I'm not confident that the organization is ripe to receive test findings yet, but if they were to get a report based on this load of... crap - I'm not sure what would be worse: accepting it at face value, or tearing into the method.
* * *
And I want to be clear - the mini form is not a design I'm terribly fond of. This is not about pride of ownership - if the participant had trouble with the mini form, I'd cheerfully bin it and move on. Never mind the fact that Ahhh had suggested progressive disclosure like the mini form to solve our validation issues to begin with - then dropped it like it was leprous as soon as Ihaq came out against it.
Ahhh's behavior has me seriously questioning Ahhh's professional integrity. For all appearances, it looks from the outside like Ahhh has kept the UX team outside of their research, while they have been - evidently - seeing what they want to see.
There is no doubt in my mind what Ahhh's conclusions will be at the end of their 'tests.' They will reject the mini-form and completely validate their pet UI element.
And that is just sad. Because that means NerdHaven's first round of user testing has been performed by someone who cannot be trusted to do user research.
And now, having discovered this problem. We are left with the problem of what we are going to do about it.