Monday, November 02, 2015

Action Needed

It's the end of July. We've just finished up showcase, where one of my stories got demo-ed - the first for BigDog's team in quite a while.

I'm sitting next to Devil, who is proxy for Product Actual. Devil is awesome. Today, everything went more or less to plan, so what happens next is mostly formality - but the forms must be obeyed.

Was everything accepted?

Agile requires that our team demonstrate what we've done to Product, and they get final sign off on whether or not it is done to their satisfaction. Devil nods, and I look in the queue and click the 'accept' button three times:

One.
Two.
Three.

And immediately forget about it. There's an iteration planning meeting coming up. I need to be ready. I think some of my stuff might get picked up.

I'm focused on that as I walk out, unaware of how three clicks will turn into a sh!t show.

*   *   *

From: Ahhh 
To: murph
Cc: Nugget
Subject: thanks, but please don't move the tickets to accepted

Hello!  There’s work I need to do on the tickets and having them in the showcase queue, checking them, and then moving them is my process.  I don’t think there’s any immediate hurry outside of the showcase day.  If you’re being asked to move them please let me know so I can address it with whoever is asking you.  Thanks
Brief aside, here: Ahhh refers to stories as 'tickets' and refuses to call them anything else. Whatever. Ahhh has cc'ed Nugget, who is our Scrum Master, for reasons that are clear to pretty much no one.

From: murph
To: Ahhh
Cc: Nugget
Subject: RE: thanks, but please don't move the tickets to accepted

What additional process is involved in accepting stories?

My understanding was that you were minding the RFD queue with regards to priority. If this has changed, that’s news to me. I moved the cards to accepted because I was sitting next to Product when they said they were accepted.


From: Ahhh
To: murph
Cc: Nugget
Subject: RE: thanks, but please don't move the tickets to accepted

If there’s any of the field entries out of whack/missing/etc. I get a talking to.  So, no.  Not just priority.
At this point, Nugget comes over to my desk to ask me if there is some irrevocable aspect to marking a story 'accepted.' There isn't, I say, and since Ahhh is an admin on the site, they are able to edit any aspect of a card. Nugget seems confused by why Ahhh is making a deal out of this. I share their confusion. Nugget walks off.

From: Nugget
To: Ahhh, murph
Subject: RE: thanks, but please don't move the tickets to accepted

I like the process of updating right away in the showcases as the Dragon team does it that way too.  I also try to update the release page shortly after (don’t always get to it) and look for Accepted before marking them off.

Are there some fields that are not updateable after the status is changed to Accepted?


From: Ahhh
To: Nugget, murph
Subject: RE: thanks, but please don't move the tickets to accepted

I sure can change my process to accommodate your process.  Who will be the responsible party for moving the tickets to Accepted?  Thanks



From: Nugget
To: Ahhh; murph
Subject: RE: thanks, but please don't move the tickets to accepted

Ok, thanks.  I think you should own all the activities with the board which includes marking as Accepted.
Translation: Nugget is washing their hands of this. 

From: Ahhh
To: Nugget, murph
Subject: action needed RE: thanks, but please don't move the tickets to accepted

Thanks!  murph, Do you agree? 
(Subject change and highlighted text in original)

From: murph
To: Ahhh, Nugget
Subject: action needed RE: thanks, but please don't move the tickets to accepted
I Do Not 8-) 

Partly because I think the point of a tool like Mixer is to share a workflow, so that tasks have more coverage and people can help each other out.


But also because I really don’t see limiting use in Mixer as solving this issue. If you’re getting grief because there are missing fields, we can add validation to the transitions to enforce those fields (and share the issue with the larger team, so that new cards are filled out correctly, rather than have the same process lead to more cards to clean up).


From: Nugget
To: murph, Ahhh
cc: Gumby
Subject: action needed RE: thanks, but please don't move the tickets to accepted

Copying [your boss] as well for their thoughts.  I agree the team should all move their cards through the workflow.  Am suggesting that Ahhh owns the “bookends” to that flow meaning she will get it moved through the analysis phase and get it RFD and then be the one to close it out by Accepting it.  If the Accepting can be done during the showcase, then this should work (at least for me).

If we still disagree, let’s set up some time to discuss further.



From: Ahhh
To: Nugget, murph
cc: Gumby
Subject: action needed RE: thanks, but please don't move the tickets to accepted

Nugget, Could you describe what moving it through the analysis phase looks like for you?  We have various UXers assigned to the analysis work on the different cards – and typically the assigned UXer is the one to present it for estimation (in pre-ipm) and for RFD.  It would be helpful to understand you thoughts in more detail.  Thanks



From: Gumby
To: Ahhh, Nugget, murph
cc: Gumby
Subject: action needed RE: thanks, but please don't move the tickets to accepted

First and foremost, my expectation is that the team works together. Attached are the expectations based on what is needed for the team…regardless of who is doing the work.  Ahhh is the single point of contact and we look to them for the work to be completed (like before a Pre-IPM). However, if murph is moving stories to accepted during the showcase, I would see this as a nice gesture.  

I’m not sure who is providing the “talking to” as referred to below. The bottom line is BigDog’s team is relying on us to manage the queue. If managing the queue is creating such issues that it creates strings of emails on a regular basis, then we need to rethink the process.

When I read discussions like this, I feel like a lot of energy is being focused on the Mixer board that would be more productive if focused elsewhere.  
And when I read discussions like this, I feel like hurling my laptop out of window.

Sunday, October 25, 2015

Progressive Disclosure

Alright, you're gonna think I'm looking for something new to slag on - but this really happened.

And
I

get

to
b!tch

about
it.

*     *     *

Rewind to a several months back. Noddy's webform is only three months late, and it's barely a form. Scratch that - it's barely the idea of a form. Noddy's written up a story for the devs that says basically, "put some text boxes and dropdowns on a web form. Oh, and give them labels, too." No validation, no function - a dead end of a feature that mimics the same old schlock in the rest of our app.

Which is sad, because Noddy has been given a gift - and was blowing it. All the webforms in our app were built with a common engine. A common engine that was built before we thought about UX at all. As a result, all our forms are uniformly terrible. Improving one form meant doing the heavy lifting of overhauling the entire engine.

But not Noddy's form. Noddy got the greenlight to be a standalone new form. Which means Noddy is free to do all the modern UI they want.

And they're replicating the same old schlock.

Runner has noticed this and is trying to browbeat some quality into Noddy's up-till-now lame-@ss feature. Runner's been appointed the guardian of the style guide and there are so many things that need fixing in our forms - they cannot bear seeing the same crap happen all over again.

Our forms have many problems but two of the newly fixable problems are our multiple required indicators and lack of conditional display.

Our forms have two kinds of required indicators (field is required to save, field is required if you want to mark the document as 'finished'). This is bad enough, but is compounded by the fact that we don't have an indicator for fields that are required to submit the form to Nachen.

Mind you, the whole reason you fill out our forms, is to send them to Nachen. And we don't have required indicators on fields that Nachen requires.

Beyond insane.

If we just add more required indicators, we would end up with THREE different required indicators.

Which simply cannot happen. Since whatever solution we come up with could influence other forms, we call a UX huddle to agree on a first step. Runner, Noddy, myself and Ahhh end up in a room and spell out the problem.

Required to save seems like the easiest indicator to lose. It turns out only two fields of data are required to save a form. One is prefilled, and the other is just a date field.

I've seen other forms with this problem and rather than make you read little symbols on the page to see what needs to be done, they frontload some prerequisite fields and don't let you see the rest of the form until you fill them out.

So, let's put the two required to save fields on a mini form before the main form. Fill out the date field and hit "continue" and we'll show you the rest of the form. Then, once you get to the actual form, you can save at any time. *Poof* 1 less required indicator.

We come up with a conditional display of the Nachen indicators that seems good.

The team seems okay with this as a first shot, and Ahhh (true to form) wants to have a wide ranging discussion of how we will come up with a design language for validation going forward. Ahhh is just compelled to make discussions larger and less fruitful - we call a halt for now and I send out the meeting minutes.

From a possible 3 indicators, we're down to 1, with an option to turn on a second. Not bad for an hour meeting.

*     *     *

Let's skip to a few weeks later. Noddy's demoing the form with the mini form up front and Ihaq is laying in like a buzzsaw. "I've never seen that before. I'd like to have a discussion about this."

When Ihaq says things like this, you can tell they are not pleased. True, this is a new UI element, but Ihaq is the same SME who thought our previous required indicators were for Nachen fields. This is our Subject Matter Expert - who as worked here for years, misinterpreting the standard interface of our App.

There is no better proof we have a UI problem with required fields.

But back to the UI - the SMEs get to weigh in on their field of expertise, but they don't get to design the interface. That's our job. If the SMEs have a specialist objection to the workflow, like they know the user won't have the required data up front, then sure - we'll listen to them and adjust. But if they just don't like the button, tough bounce.

Post showcase, Gumby asks after the UI, "did we test this?" No, I say, this was a decision that came out of UX. I can tell Gumby is still smarting from being called out in the stakeholder meeting, but there's not much to do about it.

Later, Runner, Noddy, Gumby and Ahhh meet with the SMEs about Nachen work - and Ahhh pulls a reversal. They denying playing any part in making the mini form, despite being in our meeting and agreeing to the approach. Despite the fact that progressive disclosure was something Ahhh had suggested in prior meetings.

I'm irritated and dig up the meeting minutes that show our decision to use the form. I don't send it to Gumby, but I show Runner so they know that Ahhh is full of it.  Standing up to the SMEs can be hard, and I can see the pressure to please them - but you can't just surrender the role of UI design. SMEs inform the design, they don't draft it. Ihaq doesn't like the progressive disclosure of the mini form - because it is new. Not because it is a real problem, they just were unprepared for it.

Gumby wants us to vet the entire form in a user test - and our new guy, Dash is supposed to put that together. Heck, at some point I was going to be doing it - but Arwafn blew up and put that to rest. Dash would be great - they are a veteran user tester and completely outside of the design.

A designer should not test their own design. Bad things happen if you do.

Somehow, the test doesn't manage to happen. I'm eye-deep in the Arwafn hairball and forget we were even thinking of testing the form.

*     *     *

So, a few days ago at standup, Ahhh mentions that they are working on a user test. I'm like great! I love user tests - they are awesome. Sure, sometimes they kick your ass, but they are so worth it. Normal run of things back at CorpWorld was invites would go out to the full team to observe - so everyone can learn from the test.

A test has a primary and a second. Primary runs the show and writes up the findings, with final edit. The second is pretty much there to help with the metric ton of work that a good test requires.

A good test should look effortless to the participant, but getting credible task scenarios is not easy. Getting a working prototype to test those scenarios is pretty frickin hard. And notetaking, and analysis, and on and on....

I don't know who is leaping on all of that, but I'm glad that we'll finally get some user testing done at Nerdhaven. Whoo! Heck, if things calm down in a week or so, I might have time to help out.

The next day, I get an email from Ahhh "I can't get the form to load and I've got a user session in a few hours."

I'm headed out the door to pick up the kids (it's my early day) so after I figure out what they are asking I get QA to help Ahhh out.

User session? WTH?

But I'm out the door.

*     *     *

Day after that, Ahh is talking about the next user test they're setting up, and there's still problems with the form loading. I can't help myself, I fire up IM:  User test? Can you forward the webmeeting link? I'd love to observe? I back off a bit. Or are you recording?

Ahh takes awhile to respond. All the while I'm thinking: How is it that they are holding tests and haven't emailed the team about it. No one else on the team even knew this was happening.

This is weird. Somebody should be helping Ahhh with the test. Not only is this the first user test that Ahhh is doing for NerdHaven - this is the first user test that's been done at NerdHaven. Period.

How is this being done by one person? And our newest UXer, to boot?

Ahh responds with "I'm recording. I'll get you the link." The implication is, I'll get back to you. Now, I don't need to observe in realtime - observers don't get to ask questions normally anyway - too much pressure on the participant. But the tone was bothersome.

Ahhh was at the point of having participants in sessions and nobody had heard a thing. Asked for access, Ahhh has basically said I'll tell you when it's over.

Later that day, Dash sits back from their desk - and snorts.

"I tried to get a hold of Product and they say they are busy because - get this - they are observing Ahh's user test."

WHAT? Okay, people on your team - whose interface you are testing (Dash's prototype is also being tested) have not been allowed to observe - but Product is sitting in.

Nice.

I have a quick client call and end up debriefing with the account Rep on how it went. The Rep's like "No problem, it went fine. I was on that user study with Ahhh earlier and that was a little rough, but it all works out."

I can't help myself - "Can you send me the link to the recording?"

That's sh!tty of me, but Ahhh said they'd send the recording, and they haven't. And really, the recording is made by the rep, so I'm jumping past the middle man.

The rep - in an act of karmic payback - totally forgets to send me the link. They are good people, and I believe this is so - but it is right that I should not get this from some back ally maneuver.

I should get it from Ahhh - directly. And Ahhh is pointedly not giving me, or any other UXer access to their tests.

*   *   *

I cannot stress enough how bizarre this is. Designers can get pretty prickly about their pet designs, but any designer worth even half a hoot knows you are going to get better design with collaboration.

More to the point, any designer worth considerably less than that knows their own bias is going to hang them if they don't have regular contact with opinions outside their personal bubble.

Ahhh makes much of their history in UX. They have an MBA, they've been doing user research for almost twenty years. I have heard this ad nauseam pretty much every time our team is asked to introduce themselves to some guest. I get it, Ahhh. I do. You are not wet behind the ears when it comes to UX. So why the F*&k are you doing user research in secret?

The ENTIRE POINT of research is to share findings - and while convincing people with research is nigh impossible - it is absolutely impossible if your audience thinks your research came from some black box process.

Erika Hall gave a talk on design research that just blew my mind - not because it's some previously unthought hunk of wisdom, but because it perfectly reveals wisdom that is hiding in plain sight. Research that comes from the mountaintop in the hands of some sage is trivial to dismiss. Sages...amIright? Research that tells a story can resonate. Stories you participate in are far more compelling than stories that are told to you.

If you give a crap about research, watch Erika make it make sense - or buy her book - or both - because I'm going to get back to griping about work.

*   *   *

So...
Whisky Tango Foxtrot, right?  I do not go to the boss, because this is not a discussion that needs management. I haven't confronted Ahhh about the situation, I'm merely asked and not gotten much of a response. I've vowed to be direct about things at this job, and I'm going to do this if it burns every bridge around - because dammit, direct communication should not be a last resort.

I'm neck deep in some Arwafn stuff, when Ahhh gives her standup update as, "I have user testing session today."

What?? Still no upfront notice? Okay, we'll talk, you and I.

I call Ahhh out at standup and ask to be able to observe the test. Ahhh's on the video conference, in front of the entire team. Rather than look petty in front of all of us, they hedge, "If you want to... I guess..."

I do. And I make it known again, that I do. I corner Ahhh on IM and ask them for the web conference link, and book a room on our office to invite the entire UX team.

Ahhh becomes very prickly over IM when I mention that I'm grabbing a room.

"WHO is going to be attending?"

I am upfront to the max. I'll send you a list of the attendees and everyone in the room will understand that we are *observing* not questioning or interfering. We know the drill.

I'm starting to get the sense that Ahhh is afraid they will have some higher up observing them, and they are all kinds of on guard. Which I would understand if we were back in CorpWorld, but this is NerdHaven - we don't do that kind of bullsh!t here. I'm not ambushing you by videoconference, jeezus.

Most of our client contacts are managed by Reps, who maintain client relationships far better than anyone on our team could - so I ask who Ahhh's Rep is on the test.

"Does it matter?" is Ahhs response.

Well, yes - actually. I'd like to invite them to the room I'm booking for the other observers - in case they want to be there.
A lot of Reps do all their conferences from their desk, but some find value in being in a room of people who are also on the call. I'm amazed this has not occurred to Ahhh, but I spell this out.

They icily give me the name of their Rep and sign off.

Well...

So, now I've got a conference room with Runner, Heater, Dash, and the Rep, and we're finally going to observe the test. Irritation aside, I'm actually excited, because tests are exciting. Real users, doing stuff.

w00t!

The session starts up, and Ahhh - twenty years of UX/research etc - is H-O-R-R-I-B-L-E. I don't know any other way to say it. I mean, typically if you have a test, you have scenarios with test data that the participant uses to attempt each task. There are success criteria/ fail criteria. There's a notetaker, there's...

Ahh has nothing. They have the live system and a participant, sure, but "Hey, try to fill out this form" is not a valid set of instructions. Our users - like most users - are truly bad at making up a set of data to fill an ill defined scenario. I'd actually say our users are worse than most, because our users are highly trained, detail oriented, and used to working in incredibly complex environments.

They sweat every detail and anything out of place will have them derailed and talking about bogus data for twenty minutes. They - as a rule - do not make up data. They want to know details before they proceed.

Ahhh has started a 'test' by loading up the mini form and saying 'have at it.'

'Oh, and tell me what you think of what you're seeing....'

The participant is looking at the mini form which is asking for a specific date. They are like 'you want me to make up a date?'

Ahhh is like 'oh sure'

So the participant types in the current date and hits the button to go to the rest of the form.

Ahhh does not ask them what information they would start the task with, nor have they supplied any. They don't ask why they chose today's date, if that would be representative, or where they would get a real date if this were an actual scenario.

What Ahhh does ask about - is the mini form. 'What did you think of that - not seeing the rest of the form until after you put in the date?'

The participant is confused 'uh, it was fine.'

Ahhh asks about this a few more times, phrasing the question differently each time. I'm looking over at Dash like what the...? until I get it.

Ahhh doesn't like the mini form - and they are really hoping that the participant will find fault with it. When that doesn't happen, Ahhh gives them a few more chances, in hopes of hearing what they want to hear.

Which is RULE NUMBER ONE as to why you don't test designs you've had a hand in making if you have any way of avoiding it.

I'm shaking my head when Ahhh asks about a piece of form UI that they designed - and believe is crucial to the workflow. "What do you think of that?'

The participant is confused. They hadn't noticed Ahhh's pet piece of UI until now. 'uh, I dunno.'

Ahhh repeats the question, gets a similar response - then tells the participant what the UI element is and what it is trying to do.

At this point, I'm about to roll my eyes when I catch Dash shaking their head. This is another thing you just DO NOT DO in a user test - if you have any way at all of avoiding it. You do not tell the participant what the interface is, or what it is doing. You let them tell you what they think it is doing. This way you can learn if if it's intuitive or not. As soon as you tell them the answer, you've lost the chance of learning more about the participant's point of view.

Now, there are lots of research techniques for lots of different situations, but this is a user test. The participant's unadulterated behavior is the entire point of the exercise. They may falter, but you do not give them assistance until you've learned what their perspective was without assistance.

Ahhh wants their pet UI validated by the test, and they cannot stop themselves from interfering. They do this repeatedly until Dash and I are absolutely disgusted.

There are no success or fail criteria, no test scenario, so the participant gets to the full form and basically stops doing anything and waits for more information. Ahh pokes them a bit by asking for their thoughts about the form, what they are thinking - typical UX probing techniques. But there is no task, so this is just an airy, off the cuff discussion.

This is not a test. This is a walk in the woods.

"What do you see?"
I see trees. I like trees...

The entire session is just a colossal waste of resources and time. I've worked hard to get access to see this so I can learn more about our users - and all I've learned is that Ahhh is really, seriously bad at testing.

A lot of what Ahhh did in the session would have been fine if it was just an informative interview - "hey, tell us about this" but this was set up to be a test - more to the point - this is being *represented to the organization as a test* and it was nothing of the sort. I'm not confident that the organization is ripe to receive test findings yet, but if they were to get a report based on this load of... crap - I'm not sure what would be worse: accepting it at face value, or tearing into the method.

*   *   *

And I want to be clear - the mini form is not a design I'm terribly fond of. This is not about pride of ownership - if the participant had trouble with the mini form, I'd cheerfully bin it and move on. Never mind the fact that Ahhh had suggested progressive disclosure like the mini form to solve our validation issues to begin with - then dropped it like it was leprous as soon as Ihaq came out against it.

Ahhh's behavior has me seriously questioning Ahhh's professional integrity. For all appearances, it looks from the outside like Ahhh has kept the UX team outside of their research, while they have been - evidently - seeing what they want to see.

There is no doubt in my mind what Ahhh's conclusions will be at the end of their 'tests.' They will reject the mini-form and completely validate their pet UI element.

And that is just sad. Because that means NerdHaven's first round of user testing has been performed by someone who cannot be trusted to do user research.

And now, having discovered this problem. We are left with the problem of what we are going to do about it.

Monday, July 06, 2015

Carli Lloyd Willed It - Thus, It Is So

I'm just gonna put this here for now... because, DAMN:




Carli Lloyd can do whatever the hell she wants:

Tuesday, June 30, 2015

Götterdämmerung

Scoreline - June 30, 2015
USA: 2
Lloyd 69' PK; O'Hara 84'
Germany: 0

Biggest match up in women's soccer history.

Everything to prove - against Germany.

And we OWNED them.

Dream Team? Yeah, kinda.


Everybody was playing out of their minds: Holiday, Brian, Sauerbrunn, Krieger, Lloyd, O'Hara, Heath, Rapinoe, Klingenberg, Morgan, Solo....

Oh, and Johnston! Hell, yeah, Johnston - I'd say the player of the tournament - but Lalas said that, and I swore a blood oath to disagree with everything that man says.

Still - Johnston - what a game - offense, defense - and YES - she pulled down an attacker in the box, but the Soccer Gods decreed that this would not stop us.
Johnston: Clearly better than you.

And the crowd... Just magic - Morgan's first big shot wide had the place jumping outta their seats.

Lloyd? Flat out aces. Old Guard? I'm fricking DRIVING this bus.

Swarming midfield, Solo psyching out Sasic...

Yes, Germany can pout about the PK, but they had their chance and blew it. Crappy calls are part of Soccer - and lord knows, the US has had their share. (You think I forget? I FORGET NOTHING).

This was the culmination of the USWNT's campaign. They were on their game against the world's best and beat the crap out of them.

I feared for this team in the group play, but they are clearly on afterburners now.

Go USA.

Monday, June 08, 2015

Rapinoe to Rapinoe

Rapinoe: American badass.

Not our best effort by a long shot. We looked really ragged at the back for most of the first half.

And I don't know if Wambach is putting too much pressure on herself, but two flat our misses by our used-to-be guided missile forward just boggle.

And after a fortuitous bounce goal, Rapinoe looked like she was trying too hard as well. All kinds of whiffed crosses, flubbed passes - even a missed throw in.

And then she does this sh!t:



And this was after getting the ball at the half-way line. She ran half the length of the field, looked up and thought

Well, everyone thinks I'm gonna cross - I'll just shoot this F*&-er.

And boom! Low to the far post, for the icer.

Take NOTHING away from Press & Laroux's efforts to get the go-ahead goal. Leroux is all kinds of hustle and guts, and Press threw down ironclad evidence (if any was needed) that-



COACH? 

I AM 

A FORWARD

Hell's yeah, she's a forward -  and make sure she gets more playing time in the middle, eh coach?


Sad for the Matildas. De Vanna was her usual full-throttle self, she was criminally unmarked in the box and instantly reminded the US why that is a bad thing.

I hope the Aussies get some success, but looking at the rest of their schedule - it's not like it gets easier from here.

USA: 3
Rapinoe 12', 78'; Press 61'
Australia: 1
De Vanna 27' (who else, seriously?)

Sunday, June 07, 2015

Feels like 2011

Hell yeah, I remember.

122nd minute.

Ian Darke, Julie Foudy and the rest of all right thinking people losing their minds as one.

I just shortened the lives of countless Brazilians.

A fan could live a lifetime for this sport and never get a moment like that.

The Cup doesn't care if you are worthy and the Soccer Gods have always been @ssholes

Japan was a fairy tale worth telling, but Germany 2011 should have been ours.

And here we are, back for another helping of whatever the Soccer Gods choose to serve us.

Defying reason and the odds, we still have Rapinoe, Rampone - even Wambach.
Germany is - once again - terrorizing the minnows, and the Samba Queens are the same as always.
This time, the French are looking solid and the Dutch look to be the newcomers who could mess things up.

We have injuries, we have scandal, and everyone would love to be the team that sent us home.

Still, I can't wait.

Go USA.

Rapinoe to Wambach for the win.
Always and forever.

Friday, June 05, 2015

Farewell to Megatron

I'm in our stakeholder meeting, a few weeks ago - presenting one of our new features done by the west coast devs and things are not going according to plan.

Being me, I blurt out "That's not right."

In front of a crowd is never the right time to encounter a new bug. Yet it happens.

I try a few more times, before realizing that the app is behaving fine, I've just completely misread what was happening.

I reset, and get the demo done and hand things over to Noddy.

Noddy launches into their part of the demo - a feature that was supposed to be RFD in January that is only now taking tangible shape in code. Noddy's dev's have endured missing requirements, late requirements, incomplete requirements - and late edits when they were still coding.

Noddy has created features with a single requirement - that is wrong. Noddy's feature work has spawned endless "fix it" stories, to correct some omission or other mistake. Each correction will require changes to automated testing, more time and additional expense.

But now, Noddy has landed the first wedge of their feature - and it's...

...it's a mess. The overall problem that this feature is supposed to solve - CANNOT be solved directly by our product. This feature's value rests on the dubious assumption that our users (who hate data entry) will enter data into our application, print out that data, and then re-enter the same data into the Nachen website.

This makes sense to absolutely no one.

Because Noddy has been on this feature, it is being built in incredibly small slices of functionality. Which is agile, but the fact is this feature is a webform. Something as basic as a webform does not need to have a lot of slices to be built by an agile shop.

Sure, you can say, build me the basic form without the ability to validate it. Then validate it. Then make the printout. Then show me how to find all the forms I've filled out. And so on.

Noddy has turned this into story pong, where features are added - then moved, then shifted to other stories, and finally landing in a heap right before a looming deadline.

Noddy's built their form, and it's maybe a third of what it needs to be in order to be a basic, dumb form. The stakeholders are not mean, but they ask pointed questions until Noddy starts deflecting in their usual way. Talking technical and promising future work until people lose interest.

Pinning Noddy down is just too much work sometimes. Even I've given up most of the time.

The meeting ends and I'm off to get a beverage. I meet HockeyTwo in the breakroom. They laugh at my struggles in the demo, but as they put it "at least you fess up to problems. If that had been Noddy? They'd have pressed on, never said a word, and hoped nobody noticed."

It's a remarkable how often a co-worker can (out-of-the-blue) break into a critical observation about Noddy.

*     *     *

The Noddy effect. It's amazing.

I've been in the hallway and been accosted by co-workers, "GOD, Noddy is just so good at pissing people off, huh?"

I've been at team building exercises that have turned into full blown Noddy venting sessions. One of my co-workers related the story where once, at a work sponsored family fishing outing, their six year old child had caught a fish and wanted to catch-and-release like everyone around them.

Noddy had intervened and told this child (mind you, someone else's six year old child) that "No, that fish is invasive, you need to kill it."

Now, I can imagine saying something along those lines, in the absence of thought. I might do that. But faced (as Noddy was) with a weeping six year old asking Why? Why does my fish have to die? 

-most of us would reconsider. Not Noddy. They insisted - to a six year old - that the fish must die.

And on and on. Ad nauseum to anyone unfortunate enough to be near an employee of NerdHaven. Everyone has a Noddy story.

*     *     *

I leave the breakroom and head back towards my desk, then I remember I have something to check on with Dragonman.

On the way there, I see Atlas talking to Noddy. Atlas wants Noddy to meet with them in their office. Noddy is begging off, they have to do something, but Atlas is insistent, "Right after."

Now, I know Noddy's been on the hot seat for a while now. In fact, most people on UX know that as well. Noddy gets regular meetings with the boss, and a host of performance monitoring activities have become the norm for our entire team.

Hearing Atlas wants to meet with Noddy is filed away.

*     *     *

The day goes on. I'm back to my desk. Runner, Dash and I have our own little UX Shangri-La, separate from our team, and a genuinely nice setup. Windows, natural light and plant life.

Plus? No Noddy.

Our team hired two more devs and there weren't enough chairs for everyone, so Runner and I volunteered to move to the next room.

Noddy offered to move as well - and BigDog was about to go along with it, logic being move all of the UX crew at once - but Runner shot BigDog a look that could maim.

And BigDog pivoted on a dime. "murph and Runner will move over to the other room." Noddy would end up staying in the team room. In a way, this made the most sense. Noddy would be creating work for the team, while Runner and I were doing work exclusively for the west coast devs.

But mostly, we were sick to the brim of Noddy.

I've thought long (way too long) about what I think of Noddy as a co-worker.

They are not:
  1. Honest
  2. Hardworking
  3. Diligent; or
  4. Professional
Above that, they have a disastrous personality and virtually everyone hates their guts. 

Runner deserved a break, and I was happy to get one as well. With the new UXer, Dash - we've got a great thing going. Dash is cool - and a pro.

I miss our devs, but I love our new digs.

I ground out some work for the rest of the day, and was on the way out when I run into OneTwentyEight. 

"What's up with Noddy?"

Huh?

"After showcase, they came back, groused something about 'might as well hand in my resignation' and left in a huff."

!!!

This news is positively electric. Nobody wants to call it out as fact, but everyone is thinking it.

Noddy is gone??  

*     *     *

The next day, Noddy is nowhere, and their laptop is missing. Questions are asked at standup, but no one has solid info. Someone thinks that Noddy's child was in the hospital. This is greeted with a certain amount of disappointment. 

Later that morning, we get an email from the boss: Noddy has a family emergency and will be out until the next week.

So that's it. 

Runner is jubilant. Noddy is gone! But family emergency has me wondering - if they are gone, the boss would just say so. I remember all the crazy with H, that dragged on and they ended up coming back. I don't want to get happy too soon.

But days roll on and Noddy does not come back. Frankly, with Arwafn blowing up and damage control in full swing - I don't notice their absence. 

The next week, Gumby tells everyone that Noddy will be gone "indefinitely." This gets all the rumor mills flying, but there's still precious little to go on. Noddy's not here, and we don't know when or if they will be back.

*     *     *

I speculate openly with HockeyOne & HockeyTwo - towards the end of the week, I'm feeling flip and launch into "I think this may be it."  HockeyTwo is skeptical, but clearly thinks this would be great news. Noddy being gone has created a news item, and more than a few water cooler discussions stray into what if?

*     *     *

And then its Thursday, and I walk into standup late. I've been in some catchup session with Gumby or something or other - I've been running late for everything lately - and I blow into the team room right behind Runner.  They look pissed.

The team goes through the standup routine and BigDog is going through their list -

- And I see Noddy back into view. They'd been standing right behind Runner such that I couldn't see them at all. 

And there's the answer. When's Noddy coming back? Right now.

And they are back with a vengeance. We get the story on their child being hospitalized - they are okay now, but it sounded like quite an ordeal - and then Noddy is back onto muddling their way through their allotted work. They have meetings with Ihaq, and are working with Ahhh, the new UX in the west coast office.

Ahhh is a special case, and I'm starting to think that Noddy and Ahhh are on the same page. Noddy can wield a firehose of bullsh!t and I'm afraid Ahhh is starting to think Noddy is credible.

For that matter, I'm worried that Ahhh & Dash are going to need the talk about Noddy in the near future. Runner and I have studiously avoided talking to Dash or Ahhh about the Noddy situation. We'd hoped that the situation would be sorted out before it became necessary, but that hasn't happened.

Dash and Ahhh deserve to know that Noddy should not be relied on. That their pronouncements on Nachen work should be double-checked - always. 

They need to know that Noddy is borderline useless in a great many ways - but...

How do you start that conversation? I want Dash and Ahhh to make their own conclusions. I don't want to just start in with "Noddy's a useless hump," but I don't know a middle ground. I've punted and avoided the issue.  When Dash needs to work with Noddy, Runner and I try to assist.

Dash is no fool, and they have to know that there is a schism between Noddy and the rest of UX - but watching them struggle to collaborate with Noddy is hard. We want to jump in and say "Forget what Noddy is saying, do what you think is right."

The devs hold their fire as well. They are pros, and they are focused on getting work done, not smoothing things out for the new UXers. Sooner or later, they will figure it out on their own.

*    *     *

I'm running late on a Wed and want to check in with the boss before standup. Gumby's in their office meeting with Portal and Noddy on something, so I head to the desk and chat up Runner.

BigDog strolls into UX Shangri-La.

"One at a time, I need all of you to come to BigSad." BigSad is the main conference room.

Runner & Dash are confused - thinking we are each to go to BigSad by ourselves and then come back before the next person goes.

I took it to mean we were all to go to BigSad, but one by one so as not to make a big deal out of things.

I have a momentary flash of panic. OMG BigDog is leaving! Which would be biblically bad. BigDog is awesome and they are about to have another child. Why would they change jobs? They can't do that to us!

*     *     *

We make BigSad together, contrary to orders. Most of the team is there. BigDog tells us to close the door and starts in.

"No other way to say this, but to say it..."

Please, no.

Just no.

I love working with BigDog, they are awesome. They are the most supportive dev I have every worked with - and they kill it in the dev space. Their status as team lead has taken them away from the coding they love and they've been clearly frustrated by it... And they've been fed a steady diet of crap work by Noddy.

OMG, Noddy - if you've cost us BigDog, I will personally-

BigDog finishes in a rush.

"Noddy is being let go today. Portal asked me to get the team out of the room so it won't get awkward."

*     *     *

When the moment finally arrived - it was a total ambush. I was exhausted from the Arwafn grind, terrified that BigDog was leaving, and frankly - numb.

Of course. Portal. Duh. They always do the reaper work.

And today they showed Noddy the door.

Noddy. The person who has caused untold misery and wasted hours - will burden us no longer.

I made an excuse to go to the breakroom and get a beverage. Before coming back, I swung by HockeyTwo.

On the down low, deliverance is at hand.

HockeyTwo nodded somberly. They were totally on the same page.

Back in BigSad, we are told the information is close hold until we hear specifically that it is not.

Oops, already blew that. Odds are good HockeyTwo has at least told HockeyOne.

After a brief stay in BigSad, we are allowed to go back to our desks. Noddy has already been shown the door.

*     *     *

Official word comes during yet another Arwafn meeting. Gumby sends out an email to our team announcing Noddy no longer works at Nerdhaven. 

I'm sitting across from Teacher, who utterly detested Noddy - and have to share. With Gumby's email maximized on screen, I spin my laptop around and offer it to Teacher.

We're on a phone conference, so Teacher keeps composure, but leans over to Juice and relays the news in a whisper. Juice looks at me, eyes wide and I make the touchdown sign.

Somehow, spreading the news makes it real.

This has really happened.

*     *     *


The team went out for a celebratory lunch, and we read Dash into the situation. Dash took it well. "I guess I don't have to feel bad about wanting to redesign all of Noddy's work, then."

Nope!


Later, I'm in the hall and I run into SnT, a wonderful human being who does great, high stress work and yet still finds time to be amazingly nice. 

Did you hear?? I want to say, but SnT's clearly clued in. This news has whipped through NerdHaven like a shockwave.

SnT and I swap Noddy stories for a bit and they're all like "I wanted to give them the benefit of the doubt, but after that whole egg-kicking thing, I just knew they were a total asshole."

To hear someone as generous as SnT launch into profanity is no small thing.

I laugh and share our nickname for him "We called them Noddy."

SnT blinks.

"Because they were always falling asleep."

SnT busts out laughing. "We called them Megatron."
(apparently, Noddy was fond of using a particular buzzword that ended in '-tron' - thus, Megatron).

I bust out laughing, and we stand there laughing in the hall until we have to explain ourselves to a passing co-worker. And then they start laughing.

*     *     *

And before folks start accusing me and mine of being mean spirited, I would like to offer the following justification.

I tried for over a year to be sympathetic to Noddy. I gather their home life is not easy. But I have watched this person bring misery and additional work to a team I am very fond of for a very long time. 

An untold amount of work was wasted cleaning up after Noddy and stepping up to do stuff they should have done. 

Noddy added stress and uncertainty to the team - and very little else.

Everyone at NerdHaven gets to be happy that those burdens have been lifted from them.

I can separate any feelings of empathy for Noddy and their family from that. Yes, I hope things don't go too badly for them. I genuinely hope they land on their feet. That they find work that inspires them to actually do the work and not just phone it in.

And I sincerely feel sympathy for Noddy's family.
A family whose chosen breadwinner decided to surf Ebay for collectables rather than do their effing job. 

Who, when faced with the prospect of imminent professional death, chose to continue the same half-assed everything that landed them in hot water in the first place.

That family deserves sympathy.

But mostly, I'm happy for my team. The UX lost a staff member and productivity will increase. Morale will improve.

And we will do better work for our customers. 

We get to be happy about that. 

Free and clear.

Thursday, June 04, 2015

Derailment, part IV

That's not Agile!

I cannot tell you how often these words are uttered these days. Arwafn has gone over the cliff, and Nerdhaven has had to publicly walk back from its launch posture of "Wow, look at this!" to the less inspiring message of "Watch this space."

Upper management is involved to an alarming degree.

Faced with the possibility of completely shutting down Arwafn, dev effort was flung at the TURN issue to determine which customers could retain the feature and which could not. The SMEs drew a line in the sand and in the end, about a third of our customers could keep the feature.

Which would be encouraging, if it weren't for the filters. Arwafn has more than a few, and two of them in particular are near total flops. Both are based on fields that customers have never been required to supply. We'd assumed they were, and our limited beta tests looked promising. Data flowed in, and the filters acted on that data.

Trouble was, that data was crap. For some customers, Filter A's key field was being populated only 20% of the time - or was co-mingled with another field, rendering it unusable. In beta, these filters appeared to work, but what they would show was wildly inaccurate.

In beta - our users were not doing a rigorous comparison between their system's data, and what Arwafn was showing. Our users viewed the beta sessions as a preview - play time. And while they surely care about the accuracy of the report - they evaluated Arwafn based on what they imagined it was doing.

Fellow UXers of the world: learn from me. Spell out exactly what you want your beta testers to do. If you need them to do system testing, you need to tell them that. Regardless, you should ensure that your feature has system testing with live data before you green light it for wider release.

But some customers had good data. And there were options for us to explore - an alternate key field might get us there. It would mean an immense amount of re-tooling, though. The thought of it made my mind hurt.

Filter A was the good one.

Filter B was based on a field that not only had issues with being populated only some of the time, when it was populated it was almost always the right kind of data - at the wrong point in time.  Filter B is supposed to present its value as of a specific moment - and it was showing a value that was a few days later. And those few days were killing us.

We had a few customers who were happy to get the data at all - even asked for Filter B to be left on despite the time shift - but Arwafn is supposed to follow a standard. A few days later was not the standard. With over 100 clients active on the report, we only found 1 who had a working Filter B.

Worse, when I spoke with Data about our prospects for getting Filter B working, I read out the point in time requirement and Data just groaned. "If we ask the customers to send us an entirely new set of data, we could get closer to what is needed. But the literal criteria for Filter B does not exist in any data element that I know of."

For the millionth time, I ask myself - How is it I am learning this now? How did we get to the point where we have code in production, and I literally just found out that the data point I need does not exist?

And I don't have a good answer for that.

UX was primarily focused on the front end stuff - what the user sees and touches - and the back end stuff (where the digital bodies are buried) was supposedly the realm of other smart people. Our former UXDir encouraged us to provoke the change by asking for things without looking under the hood. "We'll figure it out," they'd say, "we can't design based on just what we have now, or we'll never make new things."

And there is a point there. If every design is vetted against the current codebase, legacy code limits what your customers get.

Fellow UXers of the world: learn from me. If a feature is dependent on a lot of user supplied data, design it to fail gracefully when some or all of those elements don't show up.

That's pretty basic, but its one thing to nod at a good idea in principle, it's quite another to realize that your feature is built on a house of cards while you are working on it. When all of this was happening, it all seemed so...right. And now Arwafn was a pile of roadkill in the middle of the road.

*     *     *

And everyone was rushing in to help. Which is great, but with upper management calling folks out, the agile culture in NerdHaven was beginning to crack.

Upper management said: Turn off Filters A and B. NOW.

And NerdHaven jumped to obey. Devs created branch code, stopped all other feature work and slammed two shutoff switches for Filters A and B into production. Devs were up in arms, because upper management never tells NerdHaven to work on this feature or that feature - and now they were.

But a fire was burning and this would put it out, so everyone got it done.

Next up was the TURN problem - and dear God what a mess. Telling upper management that the key value for Arwafn is being adulterated by our product was the easy part. Management realized there was a problem and TURN fit that bill. Resources were allocated to fix this issue and commandments were given.

FIX THIS. NOW.

The problem was, as we fixed TURN for Arwafn, we realized that TURN was used all over our product - and fixing it for just Arwafn would mean that other features would be out of synch with Arwafn.

Mind you, I'd always thought of Arwafn as a modest feature - and it is. Just a little report in one corner of the product. But TURN is a value that is used in the NumeroUno feature, and Nachen work, and more besides.  I spoke with Sprint when we started fixing TURN and they gave me the full list of what they knew would be affected and it was alarming.

Then, in the middle of one of our situation room meetings for Arwafn, I started spelling that out to the group.

This launched another round of anxiety telephone and soon fixing the TURN issue for Arwafn, became fix TURN everyGoddamnWhere.

Which meant the scope of work just ballooned - which naturally had to be reported up the food chain.

Upper management's response was predictable:

FIX EVERYTHING NOW. AND SWEAR IN BLOOD THAT THIS WILL BE THE LAST OF THE TURN PROBLEMS.

Now, you're an agile dev shop and you get this kind of directive. What do you say to that? There is absolutely no guarantee that we will get everything in one go.

The SMEs are being asked to sign in blood that everything is fine from their perspective - and they are already anxious about the current state of things. They don't see everything and now they want to see EVERYTHING.

We've got SMEs crawling down our necks asking for more access and more analysis. And then Gumby's asked to provide proof that UX has discussed our designs with SMEs (in a meeting, with no warning - so they don't have that information, which itself becomes a thing)...

...and now we're careening towards Waterfall.

Waterfall: an epithet in dev shops. Shorthand for let's add more analysis and more documentation and more signoffs and when that doesn't result in a perfect product - slow down and add still more analysis/docs/signoffs. Lather, rinse, repeat.

All the UX folks hate this idea with a passion, and I pointedly ask if this change is the new way of things going forward, or just until we get Arwafn/TURN fixed.

Gumby is direct, "this is going forward."

I feel like throwing up. Yes, we steered the company in the ditch, but we learned a boatload and absolutely nobody wants to go through that again. We will be more vigilant, and we don't need more goddamn documentation.

I point out (for the umpteenth time) that more SME contact in our process would not have caught our data issues. The SMEs don't go under the hood any more than UX does. But we have experts in house who do, and we will be joined at the hip with them the next time we roll out features driven by client data.

Let's do that, I say, before we get all process heavy. We are already getting berated for being slow, all this bloat is not going to make software happen faster.

Wasn't this why Agile was invented? to prevent this kind of stuff?

But the powers that be are unassailable - we will add process and hope this will get better  - or blow over.

Or something.

*     *     *

Today was a day that threatened to be the end of all things. I was roped into three meetings in a row, one about some future data issue I knew absolutely nothing about, one with the SMEs to discuss their (well aged) laundry list of desired features (that have absolutely nothing to do with Arwafn/TURN), and one that I was dreading most of all: a review of all our Arwafn/TURN work with the SMEs.

The laundry list meeting was a total bust - the SMEs want us to build what they want (as opposed to what Product has prioritized, or what the customers want). Their list of wants will solve issues that the SMEs have experienced, but may not do much to address root causes of user pain. And it is very unclear if all the items on the list should be solved with more computer code. Some might literally be user error. And all of the items are broadly worded concerns that cannot be vetted without specific examples of the behavior in question.

Sprint is in the meeting with Gumby and I and (heroically) volunteers to vet the list for the SMEs. I say (as diplomatically as I can) that this list should be given to our support staff, who can explain the inner workings of the application and perhaps get to the root of the matter without us trying to build a solution based on a speculative issue. I'm not sure the message took, but Gumby and Sprint are clearly way ahead of me. They want to do right by the SMEs, but route support work to support resources.

And then the review meeting. I wanted to keep my hackles in check, but this had all the hallmarks of being a "why did you put the button there?" kind of discussion. One of the SMEs, Ihaq, had ridiculed our design of a new form and I was dreading a new session of "Move that button, because I want you to."

The lead SME was Voldemort: brilliant, hardworking and overworked beyond all reason. They started out with a few probing questions which Gumby fielded for the first few minutes. Ihaq was there and clearly confused about what we would be running through. They had a few questions they wanted to run through before we got started. Ihaq's questions are never brief, and usually involve follow ups.

It was getting awkward.

When we got to the list of feature work, Gumby gave me the podium.

At the risk of appearing arrogant - I am pretty good at explaining things. So long as I understand something, I can transfer that information to pretty much anyone.

And the work we've been neck deep in for the past two months is stuff I can't help but understand. I wake up in the middle of the night thinking about this stuff.

Grok it? Hell, I wrote it.

And I go to town. You have a question, Ihaq? I got you. Voldemort wants clarity on the hideously technical matter of historical TURN values?  I got you, too.

And as soon as I realize that the SMEs are (rather predictably, I might add) solid professionals trying to do their jobs - the tension goes out of the meeting. The SMEs are not there to grill us, or re-design our work - they are literally trying to understand where the work is now and where it will be when we are all done.

Gumby asked our new UXer, Dash, to study our stakeholder demos and they pointed out a host of issues with how we present our work and how it is consumed and retained.

Predictably, shortcomings in our presentations leave lasting misunderstandings in the SMEs as well as others in our organization.

Having this Q and A session allowed us to close that gap for Voldemort and Ihaq at least, and at the end of it - it was a really good feeling.

Voldemort was exultant "That was awesome."

And I cannot tell you how much better that made damn near everything.

Yes, we have a metric sh!t-ton of work ahead of us, but we are beginning to round the corner and we've started to mend things with the SMEs.

The technical work is in flight, the release date is set. Our culture has taken a massive hit to the jaw.

But if today was any indication-

We can do this.

Friday, May 22, 2015

Derailment, part III

Continued from Part II

I literally go from the airport, to Gumby's office, and thence to G's house to watch basketball. I'm absolutely shredded and a long weekend away from it all could not be better timed.

Hoops with the gang is sacred. I know next to nothing about basketball, but the gang puts up with me anyway. We eat a lot, drink a lot and loaf a lot - which is good for your mental state.

And my mental state needed a helping hand. I'd booked my PTO back when the West Virginia trip was not a threat - then the trip was moved to be right on top of Hoops.

Not cool. I ride out the weekend and soak up the camaraderie. Then I slouch back home to deal with Arwafn.

First there's the anxiety tag: Nerdhaven, like any business, has factions - and the loudest faction at present is the SMEs. The SMEs are frontline staff, up to their eyes in subject matter expertise, but also expected to perform a slew of customer contact, sales & support.

Since they have real jobs, they're not deeply plugged into what goes on back at Nerdhaven, and a lot of the work that our UX team does seems (to them, at least) to come from outer space. There are a lot of reasons for this: distance, iterative design, and a long history of unsatisfactory dealings with UX.

We don't build exactly what they ask for. Largely because every request out of the SMEs starts with a solution. They are smart people, and they know what they want.

Being in UX, we know that any solution causes more problems - and the trick is to look for root causes before you "just build what they ask for." It doesn't mean that UX folks have the answers (lord knows we don't) but our approach is to get more data to see if we can gain enough insight to make a 'big win' kind of solution.

The SMEs don't have time to wait for that kind of thing. They need answers yesterday - and are confident that their chosen approach will begin paying off immediately.  They may be right - but one of the things we hear from our end users is classic alert fatigue: the users have more warnings then they have time to respond to. And a lot of the requests from SMEs amount to 'gimme a new alert option.'

Its difficult, and frequently frustrating for both UX and the SMEs and I don't see a good way out of it.

*     *     *

With that bit of background, you can appreciate the amount of anxiety tag that is ripping through the SMEs after the launch of Arwafn. They've done demos of the report in front of clients. They've described its many features and clients have oooohed and aaaaaaahed. And now the product has been flipped on for their clients and the result is a big, wet, thud.

And it's the SME on the horn or in the room with the client having to take the abuse, not me. Nobody in UX is on the phone with a baffled customer trying to explain why their numbers don't add up.  Historically, the SMEs have to run with whatever features make it out of UX and they quickly share the skinny on what they've seen so they can go into client contact forewarned and forearmed.

I'm eventually cc'ed on a series of messages between the SMEs "Did you hear? This filter is a failure across the board!!" and that SME tells the next one, and the next - and soon you have a whirlwind of anxiety making its way to the higher ups.

The higher ups have seen this movie - and they rapidly direct their power on various minions who are ordered to bring them clarity about what the hell is going on. One of the higher ups is Gumby, new to Nerdhaven and that means I'm getting pulled into their office and other meetings to explain everything I know in detail.

I spell it out over and over again. I state my mistakes and assessment of where the product is at the moment. I spell out the issue with TURN codes and list out a half dozen other things that could be made to work better.

Then I do it again in front of the the execs. I have to go before the Wizard, who is mighty, but never really says much. I live in mortal terror of having to go before MGM, who is mightier still, but those fears are thankfully not realized.

Airing the TURN issue in front of this crowd means we are going to fix the issue. HockeyOne will be happy, Ajax will be relieved, and our client's data will at last be made whole.

And the exec level folks will make our lives an unending hell until all of this is done.

Product is leaving the company - completely unrelated to this situation, they just got a VP gig elsewhere, so this 'fix the Arwafn' effort is going to need a point person.

Absolutely no one volunteers. So, after a loooooong pause, Gumby raises a hand.

As the newest manager at Nerdhaven, Gumby has a free pass in the situation. They had nothing to do with what has happened up until now. Their role is enthusiastically endorsed by a roomful of cowards.

*     *     *

The boss is in this up to their neck, and UX is now on the hook in a larger sense. Gumby has to prevail.

And Nerdhaven has to fix Arwafn. All of it.

Thursday, May 21, 2015

Derailment, part II

Continuing from Part I

Arwafn is coming undone. And for a reason I first started hearing about a month after I came to Nerdhaven.

One of the base data elements of Arwafn is something I'll call TURN. TURN is a series of codes that factor into the key datapoint of the Arwafn report. Calling it the lynchpin of the report would not be overstating it. The customer sends Nerdhaven TURN codes and we group them, count them, and make them look beautiful.

Classically, a TURN code is a single character value. Like T, or U, or R, or N. And Nerdhaven's process is built on this standard.

Single character come in, single character go into database. Arwafn is happy.

But what if TURN comes in as multiple characters?

Instead of T, let's say it comes in as T<*

Then what?

This question had been posed and answered for me by Noddy. Noddy had raised an issue about multichar TURN codes in their usual manner: Airily, arrogantly "raising a concern" via email to a slew of senior staff. Runner and I were cc'ed so we would be aware of his brilliance.

Look at me!

That's Noddy, and the truth is - its hurts their message, since the resulting eyeroll drowns out the content. But the issue sounded serious.

According to Noddy, when a multi-character TURN code came in to a system I'll call Dragon, it would get truncated. Dragon would take just the first character and bin everything else.

T<* would become simply, T.

And that didn't sound so bad, and really wasn't bad, until new values were added to the TURN code like NO and TDD. These were values that were distinct from T, U, R, and N.

But if Noddy was right, Dragon would take an incoming code like TDD and convert it to T. NO would become N.

And that really would be a problem. Because now our system would be adulterating client data - and that is not cool. Arwafn would be inaccurate, and other features in our product would be wrong as well. TURN is deep, deep in the works. And Noddy wants us to know he's found this problem.

I remember Heater responding to Noddy's initial email. Heater had a few stories written up to deal with this issue - they'd never been given priority before, but maybe now the time was ri-

Dragonman kills that noise almost immediately. Dragonman is lead dev on the Dragon system, and his response took the wind out of Noddy & Heater.

Here's a count of all the incoming TURNs we've received in the past two months: out of millions of messages, we're only gettting a few hundred N values. Even if all of them were really 'NO' values, its a negligible amount of data.

In other words, Dragonman was saying this is not a problem.

Heater lives in fear of Dragonman, and Noddy never follows up on anything. So that was enough to make them drop the issue.

I'd remember talking with UXDir and they were frustrated that we weren't fixing this issue. "We shouldn't treat this like a tech support call! Who cares if there isn't a lot of volume now? We should make this right, so we future proof the system!"

I also remember talking to Product & suggesting that supporting longer TURN values might be a selling point for our product. Product said we didn't have any data to support that it would be worth it. Between that and the technical pushback, the idea was buried.

Yeah, we truncated the TURN codes, but it didn't matter enough to warrant allocating dev resources to fix it.

*     *     *

That answer became the party line: We truncate, but so what?

Months later, even I used that line to shoot down Suit when they tried to grandstand on the discovery that OMG, we're truncating TURN values!

Suit was engaging in that same chicken little exercise where "lookee everybody! I found a problem (and gosh, aren't I smart?)" and I was in no mood. I forwarded Dragonman's original response on TURN to Suit and cc'ed about everyone. This is a crap issue, leave it alone.

I felt proud of myself, too. I knew what was going on under the hood with TURN. It's no big deal.

*     *     *

Rewind to about five months ago. I'm on the phone with Goldeagle, one of our beta testers, and they are looking at their version of Arwafn and they are confused.

The numbers for this item just look wrong. It's way too high, and I think the count is off.

I'd seen this before, but it had always been a setting problem. The user had misconfigured things and broke the output. But a quick check showed that Goldeagle had the settings right. The numbers were just wrong. I told them I would figure out what the problem was.

This was weeks after Arwafn went live to over 100 customers. And this was definitely not good.

I asked BigDog to give me the under the hood data for Goldeagle's report. I do the math and everything adds up to the same crappy numbers Goldeagle was wondering about. And I notice a pattern in the data: the calculation is based on the occurrence of T's out of the total of T's, U's, and R's.

And there are no R's at all.

Which is odd, because Arwafn is report that spans a year or more. In the course of a year, for this particular datapoint - there should be at least an R here and there.

I check again. None.

As all the data I'm looking at is in the Guardian database, I wonder if we're screwing things up somewhere upstream: in Dragon, maybe.

But Dragon doesn't hold data, they take incoming values and slot them into our Guardian db. True, Dragon would be the place where we are truncating TURN codes, but truncating still passes something and I was seeing no Rs at all. Was there a process that was eating the Rs?

And why was it only for this one record? Arwafn has dozens of record lines, each crossing columns with other values. And only this one combination was showing this total lack of Rs.

What the hell?

At this point, I trot over to the other side of the building and visit HockeyOne. They are a code slinger, an expert on incoming client data, and they're one of the folks that are always asking me painful questions when I'm showcasing new features.

HockeyOne lives and breathes in another system that is upstream of Dragon. It's called Lizard. And Lizard is very low level stuff, grabbing up massive volumes of text files sent by our clients and flinging them into the jaws of Dragon.

HockeyOne is a guru of all things Lizard and I start asking them if they have any idea what could be happening to my missing Rs.  HockeyOne looks at me with a sort of amused smile and says "have you looked at the KIM fields?"

No. KIM fields are not TURN fields, why the hell would I look there?

"Because if the TURN is too long to go into Dragon, we'll append it to the end of a KIM field and pass a blank TURN value."

Shut
the
front
door.

HockeyOne goes on to explain the problem. Lizard wants to pass all TURN values to Dragon, but it will get an error if it tries to pass a TURN that is longer than 1 character, since Dragon will only take a 1 char TURN code.

Rather than give up, Lizard will look for a neighboring field, KIM, and append the TURN value to it so the data has somewhere to go in Dragon. Lizard doesn't truncate the TURN, it puts it elswhere and passes nothing at all to the TURN code in Dragon.

I start to sweat. "Can you run a query to show me what kinds of extra noise is getting tacked on to Goldeagle's KIM field?"

HockeyOne says "sure!" and writes up byzantine query in a few minutes. They show me a list of values that are in the KIM field, but supposedly came in as values in TURN:

  • BLES
  • UB
  • CLAB

Whiskey Tango Foxtrot? I've never heard of these values coming across in TURN. I ask HockeyOne to make sure - absolutely sure - these values exist in the client's messages to our system. HockeyOne goes deep into the weeds to show me - that these are real values Goldeagle is sending to us AND that they are sending to as as TURN codes.

I ask for samples that prove this, so I can show them to others. HockeyOne has just demonstrated that we are NOT truncating TURN codes, we are moving them elsewhere. Which to downstream features like Arwafn, is essentially the same thing as deleting them, since Arwafn is looking at the TURN field, which ends up blank every time one of these multicharacter values bounces off Dragon's front door.

HockeyOne is excited. They've always hated the fact that Lizard was shoving data in weird places, and they've asked repeatedly for Dragon to take longer values and store them. They give me all kinds of examples of what is going on in Goldeagle's incoming messages - and I find front end examples of this data mangling so I can sell this as a real issue.

Then I get back with Goldeagle, and ask them what is going on with these other codes. What do they mean? and why are they being sent in a field that should only have 6 discrete values?

Goldeagle tells me that their system uses those other values to track TURN as well as an entirely different dimension called BLES. In their system, the code BLES = R and also has some additional meaning as well. UB is also the equivalent of R.

Which means I've found my missing R values, they are being sent as multichar values in TURN, which means Lizard isn't putting them into our db as TURN values, because Dragon won't let them.

Which means Arwafn is pretty much screwed at this point, because Arwafn only sees TURN values in the Guardian database. If multichar TURN values aren't landing in Guardian's TURN field, they might as well not exist - because to Arwafn - they don't.

I ask HockeyOne how often Lizard would move TURN values into KIM and send a blank value in TURN to Dragon. I mean, I'd always been told we truncate.

HockeyOne is pretty sure that we do this for a majority of our sites. Which means that Goldeagle has a lot of company. And a lot of our Arwafn customers are going to be missing data.

I flip out and go to Data and Heater and declare that the sky is falling. Data tells me that what I am saying is inaccurate. "We truncate."

I show them the messages. "No, we don't. At least not for this site."

My equivocation doesn't help my case. I may have just found an outlier. I email a wider group and provoke an exasperated response from Ajax, who is the sheriff of these kinds of data issues. Ajax is a supremely competent person and they tell me that I'm describing the situation wrong. That any moving of data into different fields would require changes that they would be aware of. And they have seen no such changes.

I take this as "murph, you are full of sh!t" and as I have only one site as a datapoint, I can't press much further.  I get more data by hitting up OneTwentyEight to run HockeyOne's byzantine query against all of my beta sites.

More long TURN values show up, noisier ones like "NEG" and "POS" and a slew of things that look like equations. I run this past Data and Sprint and start to get some traction.

Sprint (who is awesome) runs ahead and sorts out a list of what chunks of work would need to be done to get these longer values into our system correctly.

We get Dragonman into the fold and a reasonable solution is proposed. Dragon will create a path for longer values to make it into Guardian. We draft up a list of work to present to Product.

We're on the verge of pitching this plan when all hell breaks lose on Arwafn.

I'm in West Virginia getting an alarmed email message from Product asking what the hell is going on with Arwafn. I'm wrung out from a long day of client visits yet completely unable to sleep - and so I lie in bed and type an epic mea culpa with my thumbs. We didn't go under the hood enough when we were in beta. This is over and above the TURN problem, but this starts the crazy train rolling.

I fly back home and drive straight to the office from the airport and see Gumby and Atlas clearly having a strategy meeting. I make myself available and soon we are in stage one of responding to an epic managerial sh!tshow.

The executives have heard that Arwafn isn't all it was cracked up to be and they are demanding answers.

The execs are flipping out about filters on the report not working - and they are pissed.

And at this point, they have no idea about the issues with TURN.

Its about to get a whole lot worse.