No, this is not a picture of Phil Mickelson throwing up in a garbage bag after his double-bogey choke at the US Open. He’s just looking for his ball. Now if that’s not a sure way to disorient oneself, I don’t know what is. Same could be said about those elusive problems (gremlins?) that crop up from time to time, and hide right back into the codework as soon as the triage team is assembled to try to squash it.
Take for example Milliman, a global firm of consultants and actuaries with nearly half a billion dollars in worldwide revenue. Their division in Texas is a pure Microsoft shop, so they don’t have to worry about their application infrastructure components not being compatible with each other. Even being pure Microsoft, they were still plagued by an elusive performance gremlin playing hide-and-seek with their developers and operations staff.
There is absolutely no fun in this game of hide-and-seek. Every time a problem hits, they would organize a triage team of network, server, application and database specialists to look into the issue and attempt to reproduce the problem. By the time the triage team looked into the problem however, the problem had gone back into remission and was nowhere to be found. Until, of course, it hit again when certain conditions with the OS, server, middleware, database or a thousand other unknowns led to its reemergence.
Of course there is a happy ending to this story. Milliman’s IT staff tags and traces all end user transactions from browser all the way to SQL Server to record the servers, method calls and SQL queries associated with each slow performing transaction. As a result, once the elusive problem strikes, they can clearly see what went wrong with a couple of mouse clicks.
For their particular gremlin, they determined that the cause was related to a single method call in the .NET native code, out of the thousands of different method calls sprinkled throughout the application. They contacted Microsoft with the precise information and received a patch to take of the problem.
For more inspiration in dealing with your own gremlins, check out this case study.
