The Web APM Blog

Hip-hop & the Art of Web Application Performance Management

The chorus of Remember The Name (a hip-hop song by Fort Minor made famous by its endless airtime on NBA broadcasts) reminds me of the plight IT faces in managing Web applications.  Here’s how the chorus goes:

This is ten percent luck, twenty percent skill
Fifteen percent concentrated power of will
Five percent pleasure, fifty percent pain

With all the moving parts involved in delivering Web applications, the above verses precisely describe the life of an IT professional dealing with Web performance issues, especially when he/she is under time pressure while users are complaining.  One IT professional after another has told me that managing Web applications is really lots of pain and very, very little pleasure.  Even worse, skill plays but a minor role in the process.  It certainly gives IT professional ample reasons to start washing down vast quantity of Vicodin® with Jack Daniels®.

How can this be? In one word: complexity.

This complexity is made worse by having application or infrastructural components that are outside the direct control of IT.  The old mantra: "You can’t manage it if you can’t measure it" is even more true with Web applications than client-server or mini/mainframe computing.  If one cannot see the application performance a real user is experiencing and be able to relate any issue to hot spots in the application infrastructure, then all one can count on to deal with application performance problems is "luck" and "concentrated power of will."

Maybe Fort Minor should start booking IT consulting gigs!

Cutting Through the Fog of War

One thing I came to appreciate from my trip to DC is that the government activities here function more or less like a large data "switch."

Data flows in from all over the world and beyond, and is processed, summarized and interpreted.  The processed data (or information) is then disseminated back to the field in the form of laws, regulations, policies or more importantly money so that action can be taken.

The government in DC does not take action itself.  The government apparatuses in the field do the work as it should be.  Yet, this machine, which is devoid of human intellect in order to minimize accompanying human weaknesses like partiality or common sense, does break down.

Take for example the Katrina disaster.  The House of Representatives report states that the tepid Federal government response was due to the Homeland Security Council’s (HSC) "failure to resolve conflicts in information and the 'fog of war,' not a lack of information."   It added that the crisis showed the government remains "woefully incapable" of managing information.

Since this is not a political blog, I would like to divert my writing from the occasional cataclysm to the daily finger-in-the-dyke facing IT personnel.  A Web application might not have collapsed in a total outage, but you certainly know that getting complaints about performance, broken links, or errors is quite common.

Usually complaints, especially those emanating from the higher reaches of the organizational hierarchy, are decidedly information content-free like: "How come the application was slow yesterday morning?"

While the politically correct response should be: "Sorry, I’ll go look into it;" the realistic respond should be: "I don’t know what you or the thousands of moving parts in the application infrastructure are doing, so realistically, I can’t help beyond giving you lip service."

However, if your best-of-breed Web APM tool has the requisite characteristics noted in my previous post, you don’t have to perform lip service, but can actually revisit the situation from yesterday morning, determine whether it was a real problem, note which of the moving parts went wrong, and dispatch the right expert to fix the problem.

You certainly don’t have to convene your version of the HSC consisting of your network person, server admin, developer, DBA, application architect, et al to debate, and perhaps, try to recreate in vain the problem.  The HSC approach of dealing with problems doesn’t work in the case of Katrina, and will not work for application availability and performance problems.

Besides avoiding the HSC approach, I would also suggest that you avoid the part-of-beast approach (see my earlier post).  Since the part-of-beast approach involves complex data flow and a management console designed to be all things to all people, it will only exacerbate confusion and amplify the "fog of war."

Your tool has to provide relevant real-time or historical information, and be designed especially for the workflow relating to managing the availability and performance of Web applications.  You certainly don’t want to be trapped like the gentlemen in the picture and get crowded out by useless data delivered at the wrong time for the wrong purpose.

More on Best of Breed

As I mentioned in my last post, a best-of-breed Web APM tool is far preferable to a part-of-beast tool in that it reflects the specific information needs and workflow of the beneficiaries of such a tool, namely, the IT personnel responsible for the development, deployment and operations of Web applications.  A Web APM tool that can be considered as best-of-breed should collect the right data, extract actionable information from the flood of data (instead of flooding the users), and present the information in an intuitive manner.  The tool should also support industry-standard reporting capabilities so that data can be sliced and diced for planning and business impact analysis.  In a nutshell, the best-of-breed Web APM tool should have four characteristics:

Top-down, real-time and historical view of the entire application infrastructure showing exactly how performance or lack thereof is delivered and impacted by the component parts.

Information that is pertinent and actionable to the job at hand, presented in an intuitive manner appropriate in timing and quantity that matches the users’ workflow.

Telegraph potential anomalies before they impact end-users’ satisfaction.

Summarize business impact information that can be easily sliced and diced by the beneficiaries of the tool for application performance tuning or operational improvements.

Coincidentally, if we make an acronym out of the above characteristics, it reminds me of a situation with a baby I encountered in one of the museums of the Smithsonian Institution while I was in DC.  I am posting the picture here to highlight the fact that, as far as the beneficiary of the sculpture (the baby in this instant in time) is concerned, the sculpture is pertinent, intuitive, meaningful, and above all, immediately and efficiently serve the needs of the job at hand.  I don’t think the more famous sculpture:  “Burghers of Calais” by Auguste Rodin (can also be seen at the Smithsonian) will have the same effect.  To our baby friend, the “Burghers of Calais” is just a collection of irrelevant, disjointed pieces tied together on a rigid bronze framework.  Now that’s an excellent allegory for a best-of-breed Web APM vs. part-of-beast product.

Managing Performance of .NET Web Apps

If you’re going to be in San Diego for Microsoft Management Summit 2006 next week, you should most definitely stop by to hear my friend Dan Garlewicz talk about "Proactively Manage Performance of .NET Web Applications" on the afternoon of Thursday, April 24th.  Dan has been around the block a lot of times, managing IT functions in leading-edge F500 companies like Nabisco, Philip Morris and Aventis.  Dan is also often quoted in press articles and industry case studies.

In his talk, Dan will draw upon his 20+ years of IT experience to reveal the secrets of managing complex and leading-edge (read, not mature and potentially frustratingly buggy) .NET Web applications.  His key takeaway is that one has to have the right people, process and technology working in concert in order to effectively deploy and manage such applications.  He will also share some of his thoughts on how to proactively manage these .NET Web applications so as to lower the total cost of ownership and improve end-user satisfaction.  For high value-added .NET Web applications deployed at the world’s leading pharmaceutical or consumer goods companies, reactive APM is just too costly (and career limiting) in terms of operating costs, legal exposures and lost business opportunities.

OK, I can’t drill-down any deeper without totally stealing Dan’s thunder.  But I can assure you that this is great, practical stuff that you cannot get anywhere.  I will surely be there taking notes.

See you there.

Using the Da Vinci Code to solve application problems

In Dan Brown’s novel "The Da Vinci Code," the Priory of Sion, a secret European society, is entrusted with the knowledge of the location of the Holy Grail of Christ. Among the leaders of the Priory are great minds of science and arts like Sir Isaac Newton, Botticelli, Victor Hugo and Da Vinci. In the novel, Brown provides an entertaining dissection of all the clues that Da Vinci had ingeniously hidden in plain view as to what is the Holy Grail and its location. Our hero and heroine, Robert and Sophie, have to be very attentive to the string of clues and follow then to the final "resting place" of the Grail.

I have always been a fan of Da Vinci, and his teaching that "simplicity is the highest form of sophistication" is spot-on in terms of product design. There are two more things we can learn from Da Vinci: (i) the solution to unfathomable mysteries or confusions are usually hidden in plain view, and (ii) follow the clues to the end.

The same method that Robert and Sophie used in the novel can be applied to managing complex Web applications. As you know, Web application problems--resulting from the interaction among the end-users’ computing environment, WWW and networking infrastructures, sophisticated applications, and the multi-tier Web application infrastructure--can be as complex and mysterious as the web of deception concocted by the great minds of the Priory over the centuries to protect their secrets. This is further complicated by the effect of server consolidation and virtualization, making it impossible to tell which infrastructural components are responsible for which transactions. The net-net is similar to the challenges facing Robert and Sophie--the Priory making things difficult to understand, and the unsavory elements of Opus Dei "consolidating" or hiding whoever or whatever that can help solve the problem.

Take for example a large consulting organization. Its end-users report intermittent application availability problems that cannot be reproduced by their IT operations and development staff. Taking a cue from Dan Brown’s book, they (i) monitor what’s hidden in plain view to the end-user by instrumenting the end-users’ browser to report on problems, and monitor application availability and performance at the end user level, and (ii) in case of end-user detected application performance problems, follow the clues by tacking and following the transaction from the end-users’ browser back to the source of the problem in real time as the incident is happening.

In short order, they discover that a .dll bug is the source of the availability problem. Before deploying this technique, they have been trying to find this problem by correlating (with great difficulties) silo or component oriented monitoring tools, brute-forcing a solution by re-booting servers and reinstalling applications and making configuration changes, taking the solution off-line and load-testing the hack out of it, etc. etc.

The solution sounds simple? Yes, it is, and again it points to the brilliance of Da Vinci--(i) simplicity, (ii) starting with what's visible in plain sight, and (iii) following the clues to a solution.

I hate my job and other attitude problems

IT’s function is to build and cure (firefight). In the US, IT is being pulled in a lot of different directions: developing compliance tools for government mandates like SOX, HIPAA etc.; integrating IT systems from acquired companies; integrating and Web-enabling business functions like SFA, CRM, ERP, etc.; building partner and employee portals; making sure that terabytes of emails get stored safely; defending against data marauders or virus spreaders; planning to recover from natural or man-made disasters; making sure new businesses get on-line, etc., etc. On top of all that, keeping the existing stuff running smoothly is still job one! And yes, no new hires.

This is the kind of job that cries out for productivity improvements! Well, turning a blind-eye to potential problems until they blow up, running around flustered and panicky, drinking or doing drugs on the job, or quitting are possible alternatives. However, one should consider the career limiting prospects of these alternatives.

Improving the productivity of personnel tasked with assuring the performance of Web application comes is several forms:

• Empowering the operations personnel so they can solve problems on their own based on business impact;
• Automate the triage function so the right people can be assigned to solve the right problem vs. having the entire team debate a problem or calling in application developers when it is not necessary; and
• Proactively tackle problems before real users complain and burn up valuable help-desk and IT time.

Beyond the above, productivity improvement can also be realized because IT is no longer operating in an interrupt-driven mode of fighting fires and dealing with (or more likely ignoring) complaints. They can plan their lives and perhaps even get to do more building instead of fire-fighting. Now, that’s a potent tonic for bad career limiting attitude!

So many tools, so small a budget

Ever since the bursting of the dot-com bubble, organizations are generally very conservative with their IT budget. At the same time, the cost of maintaining existing applications has grown to over 50% of IT budget according to some reports. For US-based organizations, IT budgets are further stressed by the need to acquire applications and systems to satisfy regulatory compliance requirements.

Given the above, what should be the priority for products that manage the performance and availability of Web applications? I would say quite high since an effective Web APM product can:

• Liberate IT from its fire fighting mode with dramatic impact on productivity (read, do more for less);
• Increase customer satisfaction and therefore revenue potential from e-commerce applications;
• Increase the productivity of users of Web-enabled enterprise applications;
• Lower help-desk cost by lowering the number of complaints;
• Speed up the transition to Web-enablement making the business more agile;
• Reduce the stress on IT, help-desk, and development staff; and
• The list just goes on and on.

Let's just look at the potential reduction in cost relating to handling end-user complaints. Just multiply (i) the average number of complaints handled by the help desk and IT per month, by (ii) the average time need to resolve a complaint. Take the result and multiply that by (iii) the cost of the personnel involved. On top of that, one can also factor in (iv) the opportunity and hard cost from the line items listed above. It is not hard to see how the investment in the right Web APM tools can have a very high ROI.

I would argue that spending priority be placed on Web APM tools to assure the quality of service of existing (or future) Web applications. This should happen before spending money on new Web-enablement projects or bringing new businesses online. Not the other way around as most organizations tend to do. It makes sense to spend money on paving the road before spending money on a sports car to run on it. Why buy a Porsche if the road is bumpy and unpaved?

Working too hard could kill

Working very hard brute-forcing a problem is detrimental to your health, even for this poor beaver. (I am referring to the poor furry animal crushed by the tree in the accompanying photo and not an MIT graduate, who I assume, would tackle a problem with grace and wit, not brute force.) 

All joking aside, brute force seems to be the de facto technique some IT organizations are using to tackle Web application performance problems.  Check the table below for the symptoms of reactive thinking.  Which camp are you in?

Reactive IT	Proactive IT
Learns about service disruptions after users complain.	Uses Application QoS tools and techniques to find problems before users call.
Projects application performance based on business volume.  For example, if sales are $N by 10 am, the application is probably working.	Asks the better question: How much more revenue will we book if we improve performance?
Relies on rebooting servers or re-installing applications to fix problems.	Diagnoses and fixes the real problem
Depends on task force approach to debate the source of problems tying up developers, network admins., and third party providers.	Reduces MTTFB (mean time to find someone to blame) using end-to-end monitoring to find the real problem
Blames service problems on the user: the PC, the net connection, or basic incompetence.	Ensures user satisfaction by monitoring performance from the end-user perspective.
Over-provisions (and, over-spends) on staffing, hardware and software to assure minimum application performance.	Provisions each application appropriately.  Uses staff time to think about how to use systems more efficiently.
Keeps people staring at the boxes and logfiles day and night just to be safe instead of doing creative work.	Measures the right metrics to keep performance at the right level for each application.

I invite you to share with your fellow IT professionals your past encounters with situations where working hard is just that… hard work with no apparent result at the Application Idol web site.  Happy sharing!