Arwafn is rolling out to the world, and while the internal reviews are good - the only score that counts is the one you get from the end users.
The early reports are the worst. Not because they are bad, but because they usually represent incomplete opinions, initial reactions - delight in something new. They are passed along by hopeful co-workers who want to believe the same thing I do: that it has worked. That our customers are pleased with what we've done.
Somehow, I always manage to fall for this. I supposed to be dispassionate, but after a year of working on the project, I want success. I want happy customers.
And so I flatter myself to think that this time, we did it right.
And then the second round of reviews roll in.
Second round is usually polite, deferential even. "Hey, I'm wondering about this bit here..."
By the third round, you know you have problems, and you start looking for patterns. This is not a drill. You'll need to fix this.
Bin the ego and lean into this, because it's going to hurt.
"Customer has some questions about this filter. It doesn't seem to work"
"...numbers aren't adding up. They want to know why..."Third, round - the co-workers have stopped being nice. They want to know what the problem is. And how those problems managed to make it through QA and beta testing.
There is no worse feeling in UX than watching a customer delight in seeing what they wanted to see - and then watch them realize their mistake.
Oh...I guess that isn't working.
* * *
The flaw in Arwafn that managed to make it all the way into production is the most maddening of all. Arwafn relies on customer supplied data - in the form of a data feed.
Data flows from a client's system and flows into Arwafn. Arwafn turns the tidal wave of data into an intelligible grid and allows the user to manipulate it in several ways. We've checked the design against the industry standards, we've verified that the added functionality was wanted by the customers - we've set up beta testers with their live data in Arwafn and let them play with it to get their responses.
And we've had QA try to beat it to death with everything short of a tire iron.
Arwafn came through all that. The board was green.
But we'd missed something - something that was admittedly pretty hard to quantify, but something that we could have noticed earlier.
The client data feeds are crap.
We were like engineers building a colossal aqueduct across miles of countryside and then aiming it at a dry lake.
Or, more accurately, a bog.
Some customers follow the agreed on specification for supplying the data and - for them - all is well. But for a still to be determined group of customers, they send us garbage - and for them, Arwafn is smelling pretty foul.
And it's no good pointing the finger back at our clients. We told them Arwafn was a new feature - we did not tell them it might work if your data isn't godawful.
Nor did we design the report to degrade gracefully. That principle I began learning with regards to crap browsers ages ago was noticeably absent when I baked out the stories to build this report.
It's a basic idea - and yet I'm re-learning it for the umpteenth time:
Okay, it works when A, B and C are present. What if I take away A?
Easy. It looks like it's still working - but isn't.
What if I take away B?
Same thing.
And so on. Any of those questions getting some real thought would have resulted in what NerdHaven is now rapidly deploying to the field. Switches to enable the functionality dependent on A.
Don't have A? We'll shut that off for you.
No B? We'll turn that off, too.
The rest will still work.
That's how it should have been designed from the get go, but in my naiveté thought "Of course, they'll all be sending us this information. How could they not?"
And they're not. Or rather, I wish they just weren't - that would have been easy to spot.
When we did beta sites, no data would have resulted in obvious failures. But that didn't happen. We had data flowing into our UI, and the dependent features worked as expected.
Did we ask users to go through our report line by line to ensure accuracy?
We did not.
Did we ask our internal experts to crawl around under the hood of our beta sites?
We did not.
All of this, every new expression of the problem, slams home into whatever professional pride I'd managed to save up. Three weeks ago, Runner and I were heroes. Now, I'm being forwarded emails from our SMEs where they are describing Arwafn's failings in scathing terms. I'd have hoped I would have been included in those messages when they were sent, but I get them secondhand. After they've been routed through management.
Atlas has been unfazed by this - they point out we've had data issues in the past, and the scope of the issue is being overblown. I feel genuinely ill. Atlas and Gumby finished up my performance reviews before this all blew up. I came out really well in their eyes. I was proud.
And now this. The folks who sang my praises are now being hauled in front of the executive leadership team.
And there is something else. Something that goes beyond a customer sending us crap data.
Something I'd heard about and went after.... just way too late.
