The Web APM Blog

"I'll do whatever she wants, and I have no idea what that is."

The above is Bill Clinton’s answer to a reporter’s question about his role if his wife is elected president of the US.  This statement also captures the sentiment of IT operations personnel with respect to their developers when deploying new applications.

As mentioned in my previous post, Elisabeth of Wallop wants her development team to have a glass wall into the production environment.  However, this wall should not be a one-way mirror.  Although IT ops principally worries about keeping the network and servers up, they should also be cognizant of the impact an OS patch they implemented or a change to the load balancing map might have on application performance.

They should have the insight into whether a performance problem is truly an application issue (or dare I say, a bug?) or if it is really an infrastructure problem.  To get this insight, they need a tool that monitors and diagnoses infrastructural issues as well as diving into the method call tree and the related database queries to see what is bogging down the transaction.

With such an ability, IT ops doesn’t have to arbitrarily assemble a triage team of network admins, developers, sys admins, DBAs etc. to play the blame-game before attempting to solve the problem.  With the complexity of Web applications, IT ops can no longer just install an application on their production platform per developers’ instructions and let it rip, i.e., the Bill Clinton "I'll do whatever she wants, and I have no idea what that is" approach.

To make it a true glass wall between development and ops, they have to both use the same monitoring and triage tool so they can share the same data and the same view of what went wrong.  The challenge here is that development-oriented tools like profilers or load testers are not appropriate for production use since they tends to be heavy on resource consumption, and production tools typically monitor an infrastructural silo and do not dive into the application.

Having such a lightweight, true end-to-end tool is the only way to build this glass wall between development and ops.

Break the glass ceiling, build a glass wall

In talking to Elisabeth of Wallop, she mentioned that she would like to build a glass wall between development and IT ops.  A wall because she doesn’t want developers to arbitrarily reach into production and change things.  There has to be a strictly enforced process by which new code is deployed in production.  But this wall has to offer transparency also. Developers should be able to get a clear, real-time view of the capability and performance of their application running in a real-life production infrastructure serving real users.

Having such a view into production allows the developers to quickly resolve any performance issues relating to their code.  In addition, it gives them ideas on how to continuously tune and improve their code.  With the complexity of Web applications (especially those built on SOA) it is virtually impossible to get a full grasp on the interdependencies of the multitude of factors that impact the performance that the end-users ultimately experience.  As a result, there is a perception of a glass ceiling to Web application performance that cannot be bridged by adding oomph to servers or accelerating the network.

To break the glass ceiling, the developers have to be able to see precisely how a real transaction initiated from real users’ browser traverses  the various tiers of the application infrastructure even down to the specific method calls and SQL queries that facilitate the transaction.  This tracing capability should also be aware of the impact of the physical infrastructure (for example, load balancers, network routing, servers, etc.) has on application performance. This, unfortunately, cannot be accomplished using existing development-oriented profiling or tracing tools as they are usually too bulky in terms of resource consumption to use on real user transactions.  Furthermore, they cannot figure out what’s going at the servers or Web tier.  Yes, developers can build traces into their code, but that tends to be time-consuming and make the code more bulky.

The key is to give developers a non-cumbersome view into the production environment so that better and more robust code can be delivered.  Elisabeth found a way, what’s yours?

Wallop Your Social Network

There's been a lot of news and speculations about the new superhot, yet stealthy, social networking spin-off of the "Evil Empire" in Redmond.  From the descriptions and screen shots available on the Web, it sure looks like a great service that will give the established social networking sites a run for their money (full disclosure: Wallop is a Symphoniq customer).  I said "will" because the site has not yet launched.  In another words, they are building buzz while hiding under an invisibility cloak until they are ready to pounce on the competition.  This is certainly the standard MO for the Microsofties, who tend to feel more comfortable laying low until the market is well-proven and then muscle in in a BIG way. 

Business wise, this is well and good.  However, from an IT perspective, their application and production infrastructure has to go from low-volume beta users during stealth mode to millions of hits per day in no time flat.  So I decided to find out how they can do that by talking to Elisabeth, Wallop’s head of development.

With such lofty goals and expectations, Elisabeth has to assure that the code her team developed can be deployed in production, and more importantly, scale quickly to satisfy the expectations of the millions of users that will flock to this service without sacrificing application performance.

She is fully aware that her users demands all sorts of cool whistles and bells, but are totally impatient in terms of waiting for the application to crank through the request regardless how computational intensive it is.  They expect the same lickity-split response time for loading the latest paparazzi photo of Lindsay Lohan, or have the application sort through the multi-variate linkage of thousands of acquaintances to figure out who’s good to go with when Spring Breaking in Cancun.

If the application takes a second longer, A.D.D. would most surely send them scurrying off into the welcoming arms of the next best thing.  Multiply this by tens or hundreds of thousands of concurrent users doing totally different (and sometimes unexpected) things to the application, you get an idea of what Elisabeth is dealing with.

This is the kind of problem that pouring on big bucks alone can’t solve.  Elisabeth knows full well that, despite the walk-on-water skills of her team and a gauntlet of pre-production tests, programming issues do exist (yes, s--t happens).  These issues are exacerbated by immature envelope-pushing application platforms, infrastructural quirks, out-of-bound usage by the users, or just gremlins; resulting in shabby performance or crashes.  That is why the Gartner Group reports that, on an average, 30% of developers’ time is spent on fixing application already in production.  This number is expected to be much higher for the cutting-edge Web 2.0 applications like Elisabeth’s.

I've got to catch my flight back home from Tech Ed, but I will talk about how Elisabeth built the performance groove into the Wallop application in my next post.

What’s under your kilt?

As a developer, you want to test your Web applications extensively--to make sure that all the application logic and algorithms are sound, let potential users run through the UI, make sure there are no memory leaks, hunt and kill a load of bugs, load test the application in the staging area, perform integration testing to ferret out conflicts with other applications, make sure all the system components required to run the application are there and that you're running the right version in the targeted infrastructure, and so on and so forth. (whew!)

But as Mr. Murphy would have predicted, once it goes live, you’ll get a call from your IT ops people passing on a customer complaint relating to some totally unexpected thing.  Just like the youngster in the picture, a real person always has an unexpected, illogical way of looking at things that can embarrass a Highlander in his full regalia.  Similarly, no amount of testing using inanimate tools can possibly find all the hidden problems that can pop out in live fire situations.  Throw in the complexity of the Web application infrastructure, and you’ll be spending a lot of time looking for hidden problems that might have nothing with your code.

So instead of moving on to the next project, the ops people are constantly dragging you back to fix their problems.  Moreover, their legacy server monitoring tools are not even the right tools to look for and diagnose Web application problems.  To avoid getting stuck, make sure you send the code to ops with the tools they need to keep it running.  No matter how good the code, apps don't run themselves.

Spring Training for Web Apps

In spring training, baseball players go through physical training to get their mind and body ready for the rigor of the regular season.  Then it is off to the Cactus (Arizona) or Grapefruit (Florida) league exhibition games where they actually start playing real games so they can get readjusted to “live fire” situations after a whole winter of not playing ball.  The preseason games also give management a chance to see how the players perform, and tune the roster and game plan to maximize the potential of the team.

For Web applications, I advocate that developers follow the same preseason approach prior to launch.  First, load test the heck out of the application to simulate what the application might have to endure.  Then, move into a pilot deployment phase (similar to the exhibition games) during which, a sample of end-users are allowed to use the application running on the production infrastructure.

In the pilot phase, a lightweight Web APM tool that monitors real user performance should be used to look for real, unexpected performance problems, errors, or broken links.  Real users typically have ways of using the application totally unforeseen by developers or architects.  And just like a slugger in spring training, the application has to be able to perform under real live-fire situation, not simulated load or pitches from a load testing tool or pitching machine.

Better yet, the tool used in this pre-deployment phase should track any problems found to the cause so that any issues arises from the interaction of the new application with the Web application infrastructure can be caught before full-scale deployment.

If all is well after that, go for the launch.  BUT make sure the operations team has the benefit of the same tool that developers used at the pre-deployment phase so that there is the continuity of information flow in case the developers are dragged back into fixing post-production problems.  Because this Web APM tool has to useable in a pre-deployment as well as a production setting, it has to be lightweight. 

In this pre-deployment phase, application developers and architects can also apply the knowledge they gain from real users using the application to tune and improve the performance of the application.

The budget for real-user monitoring and diagnostic Web APM tools should be set aside from the get-go out of the budget for the new application.  From a financial management point of view, there are two reasons why it should come out of the application’s budget instead of the operational budget.  First, these tools have a significant and direct impact on the potential for success of the entire project.  Furthermore, it is a lot easier for most enterprises to provide incremental funding under major application initiatives like employee portal, e-commerce, CRM, SFA, ERP, BPM, supply chain etc. to cover the cost of tools that increase the success of these projects than trying to squeeze it out of the overwhelmingly human-cost-oriented operations budget.

In fact, as I mentioned in an earlier post, 30% of developers’ time is spent dealing with fixing problems on applications already in production.  So there is significant hidden real and opportunity cost in not having these Web APM tools, and this hidden cost will act as a drag on the organization’s agility and cost structure as new applications are launched.

Of Web Apps and Developers

"The best-laid schemes o mice an men, Gang aft agley" ("Often go awry").  The words of Robert Burns sadly apply to Steinbeck’ characters Lennie and George as well as applications deployed on the information superhighway.   

As a developer, you try to make your application bullet-proof.  You start with a detailed requirement document that you thrashed out with the business owner.  You ponder every feature to squeeze out any technology risk.   You carefully architect the application.  You build it on robust servers and application platforms.  You make sure your coders follow proper programming techniques and cross-check their work.  You put up an extensive gantlet of tests and make sure the application passes with margin to spare.  You try out the application on potential users.  You load-test the application.  You regression-test any module that has to be changed.  You make sure that the application can be installed onto the target platform.  You train the help desk and technical support staff…

But what happens right after you put the application in production?  It fails.  404 errors, exceptions, this page or that refreshes slowly or doesn’t load at all, the color or font size is all wrong, buttons pushed and nothing happens, users can’t log in...  Cry for help from IT Ops, complaints, finger pointing, anger, denial, self-doubt…

You’re not alone. Gartner reported at Datacenter 2005 that 30% of developers’ time is spent troubleshooting deployed applications rather than building new ones.   The problems might have nothing to do with bugs, but the interaction between your application and the Web infrastructure and/or other applications running in the same infrastructure.  There might also be problems related to the end-users’ environment or the Internet that you can’t possibly test for.  There might also be intermittent problems that cannot be reproduced.  And the list goes on…

What you need is to make sure that the operations folks have an effective way to monitor and triage Web application problems.  These are generally operational tools that have nothing to do with developers.  However, if you and your team are constantly being dragged back into troubleshooting deployed applications, guess whose interest it is to make sure that the Web application infrastructure is properly managed.

"And will there be rabbits, George?"
"Yeah, Lennie. There'll be rabbits."

We all know how the story ends.  The same applies to your productivity.   If you run a team of, say, 5 developers each costing $150K/year fully loaded, the 30% rate mentioned above translates to a hefty $225K/year down the rabbit hole.  Now that’s a best-laid scheme gone awry.

There’ll be bugs, quirks, screw-ups, and lost revenue.  Don’t shoot the application; just make sure the operations staff has the right tool to handle whatever situation that comes up.  It’s complex out there.