Continued from Part II
I literally go from the airport, to Gumby's office, and thence to G's house to watch basketball. I'm absolutely shredded and a long weekend away from it all could not be better timed.
Hoops with the gang is sacred. I know next to nothing about basketball, but the gang puts up with me anyway. We eat a lot, drink a lot and loaf a lot - which is good for your mental state.
And my mental state needed a helping hand. I'd booked my PTO back when the West Virginia trip was not a threat - then the trip was moved to be right on top of Hoops.
Not cool. I ride out the weekend and soak up the camaraderie. Then I slouch back home to deal with Arwafn.
First there's the anxiety tag: Nerdhaven, like any business, has factions - and the loudest faction at present is the SMEs. The SMEs are frontline staff, up to their eyes in subject matter expertise, but also expected to perform a slew of customer contact, sales & support.
Since they have real jobs, they're not deeply plugged into what goes on back at Nerdhaven, and a lot of the work that our UX team does seems (to them, at least) to come from outer space. There are a lot of reasons for this: distance, iterative design, and a long history of unsatisfactory dealings with UX.
We don't build exactly what they ask for. Largely because every request out of the SMEs starts with a solution. They are smart people, and they know what they want.
Being in UX, we know that any solution causes more problems - and the trick is to look for root causes before you "just build what they ask for." It doesn't mean that UX folks have the answers (lord knows we don't) but our approach is to get more data to see if we can gain enough insight to make a 'big win' kind of solution.
The SMEs don't have time to wait for that kind of thing. They need answers yesterday - and are confident that their chosen approach will begin paying off immediately. They may be right - but one of the things we hear from our end users is classic alert fatigue: the users have more warnings then they have time to respond to. And a lot of the requests from SMEs amount to 'gimme a new alert option.'
Its difficult, and frequently frustrating for both UX and the SMEs and I don't see a good way out of it.
* * *
With that bit of background, you can appreciate the amount of anxiety tag that is ripping through the SMEs after the launch of Arwafn. They've done demos of the report in front of clients. They've described its many features and clients have oooohed and aaaaaaahed. And now the product has been flipped on for their clients and the result is a big, wet, thud.
And it's the SME on the horn or in the room with the client having to take the abuse, not me. Nobody in UX is on the phone with a baffled customer trying to explain why their numbers don't add up. Historically, the SMEs have to run with whatever features make it out of UX and they quickly share the skinny on what they've seen so they can go into client contact forewarned and forearmed.
I'm eventually cc'ed on a series of messages between the SMEs "Did you hear? This filter is a failure across the board!!" and that SME tells the next one, and the next - and soon you have a whirlwind of anxiety making its way to the higher ups.
The higher ups have seen this movie - and they rapidly direct their power on various minions who are ordered to bring them clarity about what the hell is going on. One of the higher ups is Gumby, new to Nerdhaven and that means I'm getting pulled into their office and other meetings to explain everything I know in detail.
I spell it out over and over again. I state my mistakes and assessment of where the product is at the moment. I spell out the issue with TURN codes and list out a half dozen other things that could be made to work better.
Then I do it again in front of the the execs. I have to go before the Wizard, who is mighty, but never really says much. I live in mortal terror of having to go before MGM, who is mightier still, but those fears are thankfully not realized.
Airing the TURN issue in front of this crowd means we are going to fix the issue. HockeyOne will be happy, Ajax will be relieved, and our client's data will at last be made whole.
And the exec level folks will make our lives an unending hell until all of this is done.
Product is leaving the company - completely unrelated to this situation, they just got a VP gig elsewhere, so this 'fix the Arwafn' effort is going to need a point person.
Absolutely no one volunteers. So, after a loooooong pause, Gumby raises a hand.
As the newest manager at Nerdhaven, Gumby has a free pass in the situation. They had nothing to do with what has happened up until now. Their role is enthusiastically endorsed by a roomful of cowards.
* * *
The boss is in this up to their neck, and UX is now on the hook in a larger sense. Gumby has to prevail.
And Nerdhaven has to fix Arwafn. All of it.
Friday, May 22, 2015
Thursday, May 21, 2015
Derailment, part II
Continuing from Part I
Arwafn is coming undone. And for a reason I first started hearing about a month after I came to Nerdhaven.
One of the base data elements of Arwafn is something I'll call TURN. TURN is a series of codes that factor into the key datapoint of the Arwafn report. Calling it the lynchpin of the report would not be overstating it. The customer sends Nerdhaven TURN codes and we group them, count them, and make them look beautiful.
Classically, a TURN code is a single character value. Like T, or U, or R, or N. And Nerdhaven's process is built on this standard.
Single character come in, single character go into database. Arwafn is happy.
But what if TURN comes in as multiple characters?
Instead of T, let's say it comes in as T<*
Then what?
This question had been posed and answered for me by Noddy. Noddy had raised an issue about multichar TURN codes in their usual manner: Airily, arrogantly "raising a concern" via email to a slew of senior staff. Runner and I were cc'ed so we would be aware of his brilliance.
Look at me!
That's Noddy, and the truth is - its hurts their message, since the resulting eyeroll drowns out the content. But the issue sounded serious.
According to Noddy, when a multi-character TURN code came in to a system I'll call Dragon, it would get truncated. Dragon would take just the first character and bin everything else.
T<* would become simply, T.
And that didn't sound so bad, and really wasn't bad, until new values were added to the TURN code like NO and TDD. These were values that were distinct from T, U, R, and N.
But if Noddy was right, Dragon would take an incoming code like TDD and convert it to T. NO would become N.
And that really would be a problem. Because now our system would be adulterating client data - and that is not cool. Arwafn would be inaccurate, and other features in our product would be wrong as well. TURN is deep, deep in the works. And Noddy wants us to know he's found this problem.
I remember Heater responding to Noddy's initial email. Heater had a few stories written up to deal with this issue - they'd never been given priority before, but maybe now the time was ri-
Dragonman kills that noise almost immediately. Dragonman is lead dev on the Dragon system, and his response took the wind out of Noddy & Heater.
Here's a count of all the incoming TURNs we've received in the past two months: out of millions of messages, we're only gettting a few hundred N values. Even if all of them were really 'NO' values, its a negligible amount of data.
In other words, Dragonman was saying this is not a problem.
Heater lives in fear of Dragonman, and Noddy never follows up on anything. So that was enough to make them drop the issue.
I'd remember talking with UXDir and they were frustrated that we weren't fixing this issue. "We shouldn't treat this like a tech support call! Who cares if there isn't a lot of volume now? We should make this right, so we future proof the system!"
I also remember talking to Product & suggesting that supporting longer TURN values might be a selling point for our product. Product said we didn't have any data to support that it would be worth it. Between that and the technical pushback, the idea was buried.
Yeah, we truncated the TURN codes, but it didn't matter enough to warrant allocating dev resources to fix it.
* * *
That answer became the party line: We truncate, but so what?
Months later, even I used that line to shoot down Suit when they tried to grandstand on the discovery that OMG, we're truncating TURN values!
Suit was engaging in that same chicken little exercise where "lookee everybody! I found a problem (and gosh, aren't I smart?)" and I was in no mood. I forwarded Dragonman's original response on TURN to Suit and cc'ed about everyone. This is a crap issue, leave it alone.
I felt proud of myself, too. I knew what was going on under the hood with TURN. It's no big deal.
* * *
Rewind to about five months ago. I'm on the phone with Goldeagle, one of our beta testers, and they are looking at their version of Arwafn and they are confused.
The numbers for this item just look wrong. It's way too high, and I think the count is off.
I'd seen this before, but it had always been a setting problem. The user had misconfigured things and broke the output. But a quick check showed that Goldeagle had the settings right. The numbers were just wrong. I told them I would figure out what the problem was.
This was weeks after Arwafn went live to over 100 customers. And this was definitely not good.
I asked BigDog to give me the under the hood data for Goldeagle's report. I do the math and everything adds up to the same crappy numbers Goldeagle was wondering about. And I notice a pattern in the data: the calculation is based on the occurrence of T's out of the total of T's, U's, and R's.
And there are no R's at all.
Which is odd, because Arwafn is report that spans a year or more. In the course of a year, for this particular datapoint - there should be at least an R here and there.
I check again. None.
As all the data I'm looking at is in the Guardian database, I wonder if we're screwing things up somewhere upstream: in Dragon, maybe.
But Dragon doesn't hold data, they take incoming values and slot them into our Guardian db. True, Dragon would be the place where we are truncating TURN codes, but truncating still passes something and I was seeing no Rs at all. Was there a process that was eating the Rs?
And why was it only for this one record? Arwafn has dozens of record lines, each crossing columns with other values. And only this one combination was showing this total lack of Rs.
What the hell?
At this point, I trot over to the other side of the building and visit HockeyOne. They are a code slinger, an expert on incoming client data, and they're one of the folks that are always asking me painful questions when I'm showcasing new features.
HockeyOne lives and breathes in another system that is upstream of Dragon. It's called Lizard. And Lizard is very low level stuff, grabbing up massive volumes of text files sent by our clients and flinging them into the jaws of Dragon.
HockeyOne is a guru of all things Lizard and I start asking them if they have any idea what could be happening to my missing Rs. HockeyOne looks at me with a sort of amused smile and says "have you looked at the KIM fields?"
No. KIM fields are not TURN fields, why the hell would I look there?
"Because if the TURN is too long to go into Dragon, we'll append it to the end of a KIM field and pass a blank TURN value."
Shut
the
front
door.
HockeyOne goes on to explain the problem. Lizard wants to pass all TURN values to Dragon, but it will get an error if it tries to pass a TURN that is longer than 1 character, since Dragon will only take a 1 char TURN code.
Rather than give up, Lizard will look for a neighboring field, KIM, and append the TURN value to it so the data has somewhere to go in Dragon. Lizard doesn't truncate the TURN, it puts it elswhere and passes nothing at all to the TURN code in Dragon.
I start to sweat. "Can you run a query to show me what kinds of extra noise is getting tacked on to Goldeagle's KIM field?"
HockeyOne says "sure!" and writes up byzantine query in a few minutes. They show me a list of values that are in the KIM field, but supposedly came in as values in TURN:
Whiskey Tango Foxtrot? I've never heard of these values coming across in TURN. I ask HockeyOne to make sure - absolutely sure - these values exist in the client's messages to our system. HockeyOne goes deep into the weeds to show me - that these are real values Goldeagle is sending to us AND that they are sending to as as TURN codes.
I ask for samples that prove this, so I can show them to others. HockeyOne has just demonstrated that we are NOT truncating TURN codes, we are moving them elsewhere. Which to downstream features like Arwafn, is essentially the same thing as deleting them, since Arwafn is looking at the TURN field, which ends up blank every time one of these multicharacter values bounces off Dragon's front door.
HockeyOne is excited. They've always hated the fact that Lizard was shoving data in weird places, and they've asked repeatedly for Dragon to take longer values and store them. They give me all kinds of examples of what is going on in Goldeagle's incoming messages - and I find front end examples of this data mangling so I can sell this as a real issue.
Then I get back with Goldeagle, and ask them what is going on with these other codes. What do they mean? and why are they being sent in a field that should only have 6 discrete values?
Goldeagle tells me that their system uses those other values to track TURN as well as an entirely different dimension called BLES. In their system, the code BLES = R and also has some additional meaning as well. UB is also the equivalent of R.
Which means I've found my missing R values, they are being sent as multichar values in TURN, which means Lizard isn't putting them into our db as TURN values, because Dragon won't let them.
Which means Arwafn is pretty much screwed at this point, because Arwafn only sees TURN values in the Guardian database. If multichar TURN values aren't landing in Guardian's TURN field, they might as well not exist - because to Arwafn - they don't.
I ask HockeyOne how often Lizard would move TURN values into KIM and send a blank value in TURN to Dragon. I mean, I'd always been told we truncate.
HockeyOne is pretty sure that we do this for a majority of our sites. Which means that Goldeagle has a lot of company. And a lot of our Arwafn customers are going to be missing data.
I flip out and go to Data and Heater and declare that the sky is falling. Data tells me that what I am saying is inaccurate. "We truncate."
I show them the messages. "No, we don't. At least not for this site."
My equivocation doesn't help my case. I may have just found an outlier. I email a wider group and provoke an exasperated response from Ajax, who is the sheriff of these kinds of data issues. Ajax is a supremely competent person and they tell me that I'm describing the situation wrong. That any moving of data into different fields would require changes that they would be aware of. And they have seen no such changes.
I take this as "murph, you are full of sh!t" and as I have only one site as a datapoint, I can't press much further. I get more data by hitting up OneTwentyEight to run HockeyOne's byzantine query against all of my beta sites.
More long TURN values show up, noisier ones like "NEG" and "POS" and a slew of things that look like equations. I run this past Data and Sprint and start to get some traction.
Sprint (who is awesome) runs ahead and sorts out a list of what chunks of work would need to be done to get these longer values into our system correctly.
We get Dragonman into the fold and a reasonable solution is proposed. Dragon will create a path for longer values to make it into Guardian. We draft up a list of work to present to Product.
We're on the verge of pitching this plan when all hell breaks lose on Arwafn.
I'm in West Virginia getting an alarmed email message from Product asking what the hell is going on with Arwafn. I'm wrung out from a long day of client visits yet completely unable to sleep - and so I lie in bed and type an epic mea culpa with my thumbs. We didn't go under the hood enough when we were in beta. This is over and above the TURN problem, but this starts the crazy train rolling.
I fly back home and drive straight to the office from the airport and see Gumby and Atlas clearly having a strategy meeting. I make myself available and soon we are in stage one of responding to an epic managerial sh!tshow.
The executives have heard that Arwafn isn't all it was cracked up to be and they are demanding answers.
The execs are flipping out about filters on the report not working - and they are pissed.
And at this point, they have no idea about the issues with TURN.
Its about to get a whole lot worse.
Arwafn is coming undone. And for a reason I first started hearing about a month after I came to Nerdhaven.
One of the base data elements of Arwafn is something I'll call TURN. TURN is a series of codes that factor into the key datapoint of the Arwafn report. Calling it the lynchpin of the report would not be overstating it. The customer sends Nerdhaven TURN codes and we group them, count them, and make them look beautiful.
Classically, a TURN code is a single character value. Like T, or U, or R, or N. And Nerdhaven's process is built on this standard.
Single character come in, single character go into database. Arwafn is happy.
But what if TURN comes in as multiple characters?
Instead of T, let's say it comes in as T<*
Then what?
This question had been posed and answered for me by Noddy. Noddy had raised an issue about multichar TURN codes in their usual manner: Airily, arrogantly "raising a concern" via email to a slew of senior staff. Runner and I were cc'ed so we would be aware of his brilliance.
Look at me!
That's Noddy, and the truth is - its hurts their message, since the resulting eyeroll drowns out the content. But the issue sounded serious.
According to Noddy, when a multi-character TURN code came in to a system I'll call Dragon, it would get truncated. Dragon would take just the first character and bin everything else.
T<* would become simply, T.
And that didn't sound so bad, and really wasn't bad, until new values were added to the TURN code like NO and TDD. These were values that were distinct from T, U, R, and N.
But if Noddy was right, Dragon would take an incoming code like TDD and convert it to T. NO would become N.
And that really would be a problem. Because now our system would be adulterating client data - and that is not cool. Arwafn would be inaccurate, and other features in our product would be wrong as well. TURN is deep, deep in the works. And Noddy wants us to know he's found this problem.
I remember Heater responding to Noddy's initial email. Heater had a few stories written up to deal with this issue - they'd never been given priority before, but maybe now the time was ri-
Dragonman kills that noise almost immediately. Dragonman is lead dev on the Dragon system, and his response took the wind out of Noddy & Heater.
Here's a count of all the incoming TURNs we've received in the past two months: out of millions of messages, we're only gettting a few hundred N values. Even if all of them were really 'NO' values, its a negligible amount of data.
In other words, Dragonman was saying this is not a problem.
Heater lives in fear of Dragonman, and Noddy never follows up on anything. So that was enough to make them drop the issue.
I'd remember talking with UXDir and they were frustrated that we weren't fixing this issue. "We shouldn't treat this like a tech support call! Who cares if there isn't a lot of volume now? We should make this right, so we future proof the system!"
I also remember talking to Product & suggesting that supporting longer TURN values might be a selling point for our product. Product said we didn't have any data to support that it would be worth it. Between that and the technical pushback, the idea was buried.
Yeah, we truncated the TURN codes, but it didn't matter enough to warrant allocating dev resources to fix it.
* * *
That answer became the party line: We truncate, but so what?
Months later, even I used that line to shoot down Suit when they tried to grandstand on the discovery that OMG, we're truncating TURN values!
Suit was engaging in that same chicken little exercise where "lookee everybody! I found a problem (and gosh, aren't I smart?)" and I was in no mood. I forwarded Dragonman's original response on TURN to Suit and cc'ed about everyone. This is a crap issue, leave it alone.
I felt proud of myself, too. I knew what was going on under the hood with TURN. It's no big deal.
* * *
Rewind to about five months ago. I'm on the phone with Goldeagle, one of our beta testers, and they are looking at their version of Arwafn and they are confused.
The numbers for this item just look wrong. It's way too high, and I think the count is off.
I'd seen this before, but it had always been a setting problem. The user had misconfigured things and broke the output. But a quick check showed that Goldeagle had the settings right. The numbers were just wrong. I told them I would figure out what the problem was.
This was weeks after Arwafn went live to over 100 customers. And this was definitely not good.
I asked BigDog to give me the under the hood data for Goldeagle's report. I do the math and everything adds up to the same crappy numbers Goldeagle was wondering about. And I notice a pattern in the data: the calculation is based on the occurrence of T's out of the total of T's, U's, and R's.
And there are no R's at all.
Which is odd, because Arwafn is report that spans a year or more. In the course of a year, for this particular datapoint - there should be at least an R here and there.
I check again. None.
As all the data I'm looking at is in the Guardian database, I wonder if we're screwing things up somewhere upstream: in Dragon, maybe.
But Dragon doesn't hold data, they take incoming values and slot them into our Guardian db. True, Dragon would be the place where we are truncating TURN codes, but truncating still passes something and I was seeing no Rs at all. Was there a process that was eating the Rs?
And why was it only for this one record? Arwafn has dozens of record lines, each crossing columns with other values. And only this one combination was showing this total lack of Rs.
What the hell?
At this point, I trot over to the other side of the building and visit HockeyOne. They are a code slinger, an expert on incoming client data, and they're one of the folks that are always asking me painful questions when I'm showcasing new features.
HockeyOne lives and breathes in another system that is upstream of Dragon. It's called Lizard. And Lizard is very low level stuff, grabbing up massive volumes of text files sent by our clients and flinging them into the jaws of Dragon.
HockeyOne is a guru of all things Lizard and I start asking them if they have any idea what could be happening to my missing Rs. HockeyOne looks at me with a sort of amused smile and says "have you looked at the KIM fields?"
No. KIM fields are not TURN fields, why the hell would I look there?
"Because if the TURN is too long to go into Dragon, we'll append it to the end of a KIM field and pass a blank TURN value."
Shut
the
front
door.
HockeyOne goes on to explain the problem. Lizard wants to pass all TURN values to Dragon, but it will get an error if it tries to pass a TURN that is longer than 1 character, since Dragon will only take a 1 char TURN code.
Rather than give up, Lizard will look for a neighboring field, KIM, and append the TURN value to it so the data has somewhere to go in Dragon. Lizard doesn't truncate the TURN, it puts it elswhere and passes nothing at all to the TURN code in Dragon.
I start to sweat. "Can you run a query to show me what kinds of extra noise is getting tacked on to Goldeagle's KIM field?"
HockeyOne says "sure!" and writes up byzantine query in a few minutes. They show me a list of values that are in the KIM field, but supposedly came in as values in TURN:
- BLES
- UB
- CLAB
Whiskey Tango Foxtrot? I've never heard of these values coming across in TURN. I ask HockeyOne to make sure - absolutely sure - these values exist in the client's messages to our system. HockeyOne goes deep into the weeds to show me - that these are real values Goldeagle is sending to us AND that they are sending to as as TURN codes.
I ask for samples that prove this, so I can show them to others. HockeyOne has just demonstrated that we are NOT truncating TURN codes, we are moving them elsewhere. Which to downstream features like Arwafn, is essentially the same thing as deleting them, since Arwafn is looking at the TURN field, which ends up blank every time one of these multicharacter values bounces off Dragon's front door.
HockeyOne is excited. They've always hated the fact that Lizard was shoving data in weird places, and they've asked repeatedly for Dragon to take longer values and store them. They give me all kinds of examples of what is going on in Goldeagle's incoming messages - and I find front end examples of this data mangling so I can sell this as a real issue.
Then I get back with Goldeagle, and ask them what is going on with these other codes. What do they mean? and why are they being sent in a field that should only have 6 discrete values?
Goldeagle tells me that their system uses those other values to track TURN as well as an entirely different dimension called BLES. In their system, the code BLES = R and also has some additional meaning as well. UB is also the equivalent of R.
Which means I've found my missing R values, they are being sent as multichar values in TURN, which means Lizard isn't putting them into our db as TURN values, because Dragon won't let them.
Which means Arwafn is pretty much screwed at this point, because Arwafn only sees TURN values in the Guardian database. If multichar TURN values aren't landing in Guardian's TURN field, they might as well not exist - because to Arwafn - they don't.
I ask HockeyOne how often Lizard would move TURN values into KIM and send a blank value in TURN to Dragon. I mean, I'd always been told we truncate.
HockeyOne is pretty sure that we do this for a majority of our sites. Which means that Goldeagle has a lot of company. And a lot of our Arwafn customers are going to be missing data.
I flip out and go to Data and Heater and declare that the sky is falling. Data tells me that what I am saying is inaccurate. "We truncate."
I show them the messages. "No, we don't. At least not for this site."
My equivocation doesn't help my case. I may have just found an outlier. I email a wider group and provoke an exasperated response from Ajax, who is the sheriff of these kinds of data issues. Ajax is a supremely competent person and they tell me that I'm describing the situation wrong. That any moving of data into different fields would require changes that they would be aware of. And they have seen no such changes.
I take this as "murph, you are full of sh!t" and as I have only one site as a datapoint, I can't press much further. I get more data by hitting up OneTwentyEight to run HockeyOne's byzantine query against all of my beta sites.
More long TURN values show up, noisier ones like "NEG" and "POS" and a slew of things that look like equations. I run this past Data and Sprint and start to get some traction.
Sprint (who is awesome) runs ahead and sorts out a list of what chunks of work would need to be done to get these longer values into our system correctly.
We get Dragonman into the fold and a reasonable solution is proposed. Dragon will create a path for longer values to make it into Guardian. We draft up a list of work to present to Product.
We're on the verge of pitching this plan when all hell breaks lose on Arwafn.
I'm in West Virginia getting an alarmed email message from Product asking what the hell is going on with Arwafn. I'm wrung out from a long day of client visits yet completely unable to sleep - and so I lie in bed and type an epic mea culpa with my thumbs. We didn't go under the hood enough when we were in beta. This is over and above the TURN problem, but this starts the crazy train rolling.
I fly back home and drive straight to the office from the airport and see Gumby and Atlas clearly having a strategy meeting. I make myself available and soon we are in stage one of responding to an epic managerial sh!tshow.
The executives have heard that Arwafn isn't all it was cracked up to be and they are demanding answers.
The execs are flipping out about filters on the report not working - and they are pissed.
And at this point, they have no idea about the issues with TURN.
Its about to get a whole lot worse.
Subscribe to:
Posts (Atom)