Salesperson's tech dream delivered by ill-equipped consultant who charged for the inevitable fix
‘I had a sharp new suit and a misplaced sense of confidence’ – and made a costly mistake
Who,Me? Before you shell out big bucks to engage the services of consultants, perhaps consider this week’s instalment of Who, Me? and the adventures of a reader we’ll call “Norman” for the duration of this tale.
Towards the end of the 2000s, Norman left an in-house software development “join the ranks of The Consultants.”
“My tech credentials were solid… assuming that is The Consultancy put me on projects that actually required them,” Norman explained.
And indeed, on his first job as a Consultant he was asked to work on “a technology that I had at best a passing experience in: - Microsoft Exchange.
The Consultancy didn’t much care that he wasn’t an Exchange expert. In Norman’s telling of the tale “The salespeople had done a great job of selling The Dream and it was of marginal consequence to them that I had virtually no idea what I was doing and certainly not the experience to foresee some of the things that could go wrong”
“I had a sharp new suit and a misplaced sense of confidence and superiority though, so I was ready.”
Norman’s consultancy task was to write a service that monitored an Exchange mailbox and forward certain messages to a document management system.
“The business logic was reasonably simple - if the email provided the required information in a specified format then it was sent on to DMS; if the required info was missing then the email was bounced to the sender with a helpful error message.”
Norman wrote the required code, which “flew through user acceptance testing with flying colours and was gaily deployed to Production.”
Until it broke.
- Yeah, we'll just take that first network handshake. What could possibly go wrong?
- You can never have too many backups. Also, you can never have too many backups
- In a time before calculators, going the extra mile at work sometimes didn't add up
Norman had long since been redeployed, but as luck would have it was still at the same client.
He could therefore observe “absolute pandemonium as what seemed like the entire IT team running about wildly complaining that their entire Exchange infrastructure was on its knees, and nobody could get any email.”
This was not Norman’s problem. But as the IT team triaged the mess, he overheard a phrase that suggested the mailbox processing service he’d built could be the culprit.
A little investigation proved his work was indeed the cause of the problem – but not because Norman had messed up.
That honour went to a business unit at Norman’s client, which had created a new automated process that sent emails to the mailbox being monitored by the service he built. But the automated process misconfigured the emails, so the required info was missing and added an auto responder on the mailbox they were using to send the emails.
Thus a game of autoresponder tennis had ensued.
“Their mailbox had sent an ‘invalid’ mail to my mailbox,” Norman explained. “My service had found the message and bounced it back. And their autoresponder responded with an equally invalid message. My service responded.”
And how it responded! Instead of just removing incoming messages from the mailbox it bounced them. This meant that every time Norman’s service ran, it not only processed the latest mail … and all the prior messages.
Norman described the result as “an exponential mailbox timebomb that was enough to completely overwhelm the entire environment.”
Norman fixed the problem by pausing the mailbox monitoring service, purging the target mailbox, and temporarily disabling the autoresponder on the other mailbox.
With that done he was triumphant. “I had successfully resolved the issue… that my code had created,” he admitted.
And then he had another win. “The Consultancy rustled up another hefty sale in the form of the required ‘change request’ to implement a two-line code fix to prevent this from happening again.”
Ain’t consultancy grand?
If you’ve ever charged a customer to make them pay for your own mistake, or paid to fix a consultant’s incompetence, tell all with an email to Who, Me? ®