Thursday, September 25, 2014

Two Tables and a Dropped Ball

It is one thing not to know, it is quite another to not even suspect.


About a month ago, I'm getting pinged by the Bass - a really excellent customer relations rep. The Bass has been aces at getting Runner and I access to clients interested in the Arwafn.

In one of our sessions, I'd griped to the universe at large about a particular feature request. Clients almost universally wanted Arwafn data separated into two groups. 

Conceptually, this is a dead simple UX request. Write it up, ship it!

But y'know: devil, details...

In order to actually split the data, I have to associate a classification of group A or group B to all of the possible data elements in report.

No problem. This data exists in the world. Somebody has compiled a list of which elements are in A or B.

Right?

I'd thought so, months ago... until I asked our SMEs who were like "Gosh, I dunno... I think it's just something you kind of know."

Point of order: Arwafn has over 2,000 data elements.

Nobody just knows that many classifications. They might know their favorites, and might be able to spot the ones that just seem wrong. But no way - there has to be a list somewhere.

So I search for such a thing and find partial lists and dodgy compilations - but nothing I could hang a hat on and present to customers. "Trust me, I found it on the interwebs...!"

Frustrating. A known set of data that is unavailable in a reliable, complete form.

So, I'm griping about this to the Bass - and they, being cool, file that away.

Days later, they IM me to say that their clients rely on a published text that gets them what they need on Arwafn grouping.  I'm like: Score! Where do I get such magic?

An hour later, I'm walking into my local megaBookstore and pulling a copy of this magical tome off the shelf.

-and trying not to let it get blown away by a stiff breeze. MegaBookstore only had the pocket edition of this reference - about 3 inches on a side - in 7 point type.

But I plow in and rapidly determine that PocketGuide will get me about 10% of what I need. But it is a published authority. A start.

Plus, a certain number of the values can be derived when applying them to similar data elements. 10% might turn into 20% or so.

At this point, I'm tipping down the rabbit hole and start doing more research. I blunder across a government database that is intended to allow for publication searches, but their header system nicely classifies the data it finds... under group A or group B.

Double Score!

Details beget details and soon I learn that the Arwafn data element list will have Groups A, B, C, D, E, F and possibly others. I only care about A & B, but with 2,000 plus elements, classifying each one will take a long time, regardless of what each value turns out to be.

I find another database in the UK that fills in more gaps - and then the big score: I find a government listing of data elements that is pretty much my Arwafn list - and it has database codes.

Winner! Better still, it has aliases for a huge chunk of my data elements. A ton of things I could not find are definitively associated with a radically different name. Searches on those names get me a classification for everything. It takes three days, plus another three to proof it and double check. 

It is not 100% but this is AGILE. We'll make this happen and improve it as we go.

But during the proofing, I notice something odd. I'd rigged some conditional formatting in a spreadsheet so I could compare my Arwafn element list vs the government element list.

And I see a bunch of things in our list that look like typos. Element names that are close to what I think they would be, but are clearly either alternate spellings or just plain mistakes.

And there are additional elements on the govt list. Things that we might want to have on our list. Some are clearly duplicates - but perhaps its better to list the same thing on the list multiple times so people can find it under the name they prefer.

It's not a huge list of discrepancies - about 50 - but my Arwafn list is currently being used by our customers in other features. If we're wrong, we are visibly wrong in front of clients.

This is bad. I think.

I mention this to Data in one of our collaboration sessions. Data is very skeptical that this is possible. And Data is a very precise kind of person. I respect their opinion, and let's face it - I'm literally cobbling together some sources into a Franken-list. 

Also, the government list I found was dated January of this year. We may already have a process in place to address this and we just haven't caught up with the new list. I really don't know, but that's why I'm checking with Data.

Meeting over - I go back and check my tables in excel. The Arwafn table vs the Govt table.

And we really do look like we have some mistakes. The Govt table looks like it has the odd typo here and there, but most of the oddness is on our end. 

And I know I pulled our list from production code.

I go do it again, and do a compare to make sure I'm not seeing things. 

I'm not.  So I copy just the 50 odd records into a file and send them to Data. Just a "Here's what I found" kinda message. Data will at least be able to tell me if I'm nuts.

A few days pass and Data pings me back "I've had a look at the file you sent. You need to set up a meeting with you, me and Noddy."

*     *     *     *

Okay, so the first thing you need to know is that part of the Reindeer Games that have been going on the past few months have involved Data avoiding Noddy almost entirely. A meeting with Data in it would typically begin with the question "are we alone?" Meaning: is everyone I can see on the video monitor everyone there is? or are there other people in room but off-camera?

Subtext: Is Noddy in the room?

The answer was generally, "Yes, who you see is all there is," but if Noddy was on the meeting invite, Data would skip the meeting so long as they had any professional cover.

Data's a pro - and they don't shirk work - but they dread working with Noddy more that most of us. Probably because they've put up with them longer  - seen more of their bullsh!t, watched too many good features die under their care.  

But this time, Data wants to meet with Noddy.

I'm wondering whether I should hide or get popcorn.

*     *     *     *

When the appointed day arrives, Runner, Noddy and I file into our conference room and dial up the west coast. Data's there and we get down to biz.

I start by stepping through what I've done to find these discrepancies and Data (sharp as ever) identifies a flaw in the way I've associated the two tables with each other. I just eyeballed them, because they line up almost perfectly. Govt table has a few more values, but inserting a few blank records into the Arwafn table makes the two virtually identical. They are clearly originating from a common list.

To Data, this is not much to count on - they want ID numbers, standardized codes, then you know that one element is truly the same thing as another. Because the codes match. 

Display text? Please, you're embarrassing yourself.

But even with this ding, Data can see we have an issue. Our list should not have typos in it - and it should match the government list.

And it doesn't.

I'm thinking this is Data's bailiwick - they are Data, after all. So I start in like an innocent, "so is there a regular process in place to check our table with the government table?"

Data is blunt. "Clearly there isn't, but there should be."

Now, I'm getting nervous. I really like Data, but my imprecision in earlier dealings with them have left me feeling like I'm always annoying them with dumb questions. Now I've come to them with a question that sounds like I'm saying their house is not in order.

I.... uh... what do we do now?  

All I wanted was to give Arwafn its grouping feature. It's just a bunch of typos, is this really a big-

"So, Noddy..." Data is staring daggers through the vidcom, "How do you usually handle this sort of thing?"

Standing neck hair. 

Lightbulbs.

Data's mad. Just not at me.

Data's grokked the issue right off the bat. Me, I'm just catching up. This data element table I'm using for Arwafn originates from a module (that I'll call Nachen) that Noddy is the go-to for. 

Anyone wants to know about the Nachen module, they always go to Noddy. They're a dork in lots of ways, but Noddy knows all about Nachen.

Except, this table and its list of data clearly has a good chunk of fail in it and Noddy clearly has no idea why that would be so.

Noddy is even slower than I to realize that this meeting is aimed entirely at them.

"Uh..."

Data is merciless. "What is your procedure for checking up on this data and ensuring that it is up to date?"

Noddy, as ever, pivots: "Well you know this government agency, they are always making errors that I have to remind them about."

"No. Most of these issues are on our side. How do you go about making sure we are current? This list murph was drawing on was from January, eight months ago. How do you go about getting us current?"

Noddy squirms a bit, but Data has set the hook. We write up action items and all of them belong to Noddy.
  • Noddy will follow up with Pooka, one of our DB gurus and see about what is involved in getting our table updated.
  • Noddy will also look into making sure this is a full and correct list of corrections.
Noddy slinks out and Runner and I exchange looks of 

OOOoooooooohhhh

and

DaaaaAAAAAAaaaammnn!

*     *     *     *

Two days ago - I get an email right after I leave work for the day. Pooka has been fired.

I've seen Pooka get into some regular disagreements with our Devs about coding approaches. Pooka's skillset is databases and not much else. Our Devs do it all. 

Apparently, Portal had put Pooka on notice that their productivity was not what it could be. Pooka had been volunteering for extra work here and there, but I didn't pick up on their anxiety.

The day of our team photo - including Pooka - was the day stuff came to a head.

Pooka was gone.

Sad for them, but I'd trust Portal's call. They are fiercely intelligent and don't put up with slackers.

*     *     *     *

So it's today. Our regular UX team meeting is actually occurring, and that means everyone is there - (well, except Heater). Which means Data and Noddy are in the same room for the first time in about three weeks.

We talk UX turkey - map out how impossible it will be to get all our work done by the end of the quarter and I'm gearing up to ask Sprint to look at my belated prototype-

-And Data speaks up.

"So, Noddy... Do you have anything to report on updating the Arwafn table?"

Noddy turns to the screen and it is instantly clear to everyone that they have completely forgotten the Arwafn table, their action items, the meeting from three weeks ago, hell - their own name.

Noddy is a deer in the headlights.

Then they slide into their usual line of bullcrap - because its a default setting for them.

"Well, I don't have anything new to report. You know how it is getting in touch with government agencies..."

Absolutely no one thinks this to be anything other than crap.

Data leans in. "No. You were to look into getting corrections done to our list. You don't need an outside agency for this."

Noddy starts talking about how they'd spoken to Pooka, how they'd tried to get the government agency to make corrections.

"Again. No. This is about correcting our list."

At this point, I jump in. Pointing out the obvious. "Our list had typos. We need to fix that. There are also some additions that could be duplicates or could be true additions."

Noddy launches into ever more elaborate evasions and Data just keeps pouncing.

"What have you done since murph surfaced this issue? What has been done?"

Noddy refuses to concede the obvious - they've done absolutely nothing. 

We end the meeting. Mostly out of exhaustion. Pinning a bullsh!t artist is a lot of work.

*     *     *     *

Back in our team room, Noddy is trying to make nice. "murph, that list of corrections you had, do you still have that?"

I about hit the roof. I attached a copy of this list to the meeting which is still in Noddy's outlook calendar. The day after the meeting, I emailed them a revised copy of the corrections- along with the meeting notes and their two action items.

They have done nothing with any of this and now they are asking me if I might happen to have a copy.

Without a word. I do an outlook search and instantly find the message I'd sent them. I'm about to send it to them, when Noddy stops me. "Oh, is this it?" they gesture towards their screen.

I walk around to their monitor and see a spreadsheet of mine.

It is a spreadsheet about an entirely different subject matter. It would be easy to think that one spreadsheet might look like another, but this particular one of mine is absolutely, obviously, one that has nothing to do with the Arwafn element table.

I cannot hide my irritation at this point. "No. That is entirely the wrong kind of data. Look at it."

Noddy is trying to look busy and can't even manage a decent attempt. 

The issue I'd found is not a huge deal. The heavens will not fall if it is not corrected. But it absolutely should be corrected, and doing so would not have taken much effort on their part.

Worse than their abject failure to do anything is their attempt to make it seem like they have.

Noddy has utterly besh!tted themselves - yet they are still looking us in the eye and acting like they haven't.

I end up IM'ing Data and telling them about the whiff on the spreadsheet. 

They offer a succinct 'OMG.'

Which is pretty much how the rest of UX is feeling about the situation.

There's messing up, there is completely humiliating yourself-

- and then there's The Full Noddy.

Epic.

1 comment:

Unknown said...

Oh my. I felt myself start sweating just reading that.