The Web APM Blog

Building a High-end Stereo? Make Sure All the Parts Work Together

Anyone who has ever tried to put together a high-end stereo system can tell you that the interconnection between the components like DVD, receiver, amplifier, speakers etc. are all standard... but.....  Although the parts all plug and play on the physical level, the real challenge lies in making sure that the individual components cooperate to deliver the best sound and video quality possible.  The configuration of the room (analogous to the Web cloud for Web apps) can also wreak havoc with the quality of the sound output. 

The challenge of building a stereo system is like the challenge of  orchestrating Web services into SOA applications.  You have no idea how reused or third party service modules will really act in the production environment.  You just can't predict how they will interact with each other and the underlying infrastructure.  While the application can be easily put together with all the required functionalities, the end result might not deliver the right performance or scale in production.  As a result, non-responsive applications can quickly erase the benefits you expect from a switch to SOA. 

What can you do if your company is already so far down the SOA adoption path that there’s no turning back and you’re stuck figuring out how to make it all run?  It's far to complicated to comprehensively test all the individual components in such complicated applications and infrastructure.  And you certainly cannot anticipate what your end users are being hit with especially with the unknown of the Web cloud between the servers and the end user’s browser.

There’s an easy approach to managing SOA in production.  Yours truly published an article in the April ‘08 edition of Software Test and Performance (URL:  http://www.stpmag.com/issues/stp-2008-04.pdf ) that you might want to check out.  The editors liked it so much that they put it on the cover! 

To answer the question, here's  a sneak preview: you would need (i) an early warning system measuring application performance from the real user’s perspective, and (ii) a holistic set of the right performance data so you can quickly response to any performance problem. 

So read the article and let me know what you think.

A Horse of a Different Color

Mr. Doug McClure, a veteran in the area of system management tools, presents a list of companies providing end-user performance management tools on his blog (http://dougmcclure.net/blog/).  This is certainly a great list, and I thank Doug for his effort. 

Indisputably new Web technologies, like SOA, virtualization, SaaS, Web 2.0 help organizations cut cost and improve business agility.  But the downside has been increasing complexity in Web applications and infrastructure with the side-effect of making life in IT miserable.

As Doug points out: “application performance is irrelevant if the end user isn’t getting the service he or she wants.”  With complex Web applications, there are many “moving parts” with unknown performance characteristics between the end user and the data center serving-up the application.  More and more of these “moving parts” are supplied by third-parties as companies outsource their data center and network functions.  Furthermore, with SOA and SaaS, even portions of the application are supplied by third party providers at run-time.
 
The bottom line is that end users might not be receiving adequate service levels even if the servers or the data center is running just fine.  Therefore, the only viable measurement of application performance is from the end users’ perspective.  And that drives the growth of the market for end-user performance monitoring tools.

But not all the tools on Doug’s list do the same thing. in a complex world, not all end-user performance management tools manage the same thing. 

Each one is a horse of a different color. For example, Tealeaf “TiVos” all interactions between users with the Web site to show how end users are using the Web site.  Coradiant, on the other hand, sniffs network traffic to estimate round-trip response time of http traffic making and gather data for network performance analysis. 

My company, Symphoniq, takes an application-oriented approach.  We designed a tool to non-intrusively monitor performance at the browser and tag and trace any ill-performing transactions from end-to-end, exposing errors, bugs, crashes, or bottlenecks that cause slow-downs.  The information that Symphoniq provides is actionable, meaning that you can rely on the information to automatically triage the problem so that the right domain expert can be assigned to tackle the right problem.

So it's not as simple as finding a horse to ride. You need to find the right horse to get your where you want to go. 

Innovators are Becoming Dinosaurs

My friend and fellow blogger Bernd Harzog posted an analysis of the market opportunity created by the adoption of virtualization.   He started by speaking to the fact that: “innovations in platforms always outstrip the ability of the major management vendors to keep up with them.”  He also cited examples of market innovators being created by platform changes over the past 20 years:

• Client/Server computing – Tivoli
• Windows Servers for production applications – NetIQ
• Web application response time management – Keynote
• J2EE Web Application Management – Wily (now part of CA)
• HTTP as the standard application level protocol – Coradient, and the other web appliances
• TCP/IP as the standard transport – NetQos, Network General, and the other TCP/IP appliances

Certainly, it is gratifying to see NetIQ being mentioned as the innovative leader in Windows Server management.  Yet, it is also sad to see that the company I founded is among this same list of dinosaurs whose days of glory have since passed because their technology is either:

• Commoditized with the passage of time.  For example, while the Keynote synthetic monitoring service is still very useful in testing the availability of Web sites, there are many different service or product offerings that can do “nearly” the same function and sometimes even for free.
• Rendered dysfunctional by new platforms or technologies being deployed.  For example, Web appliances based on sniffing technology cannot determine the performance experienced by end users if the content or the application is delivered by third-party content or application delivery networks like Akamai.  [Stay tuned for a future blog posting on this subject.]  As a result, while monitoring Web application performance is still a critical need, the adoption of Web monitoring appliances will slow down because the information they provide might be flawed or incomplete.
• Degraded in budgetary priority.  For example, monitoring software focusing on a particular silo of the infrastructure.  More often than not, these silo monitoring products can no longer provide actionable information for the management of complex Web and/or virtualized applications.  Bernd’s article provides amble examples why these products won’t work in a virtualized world.  As a result their prominence on IT’s budget is slipping.

Dr. Dobbs Discussing SOA

Recently your humble correspondent had the opportunity to talk SOA issues with Jonathon Erickson of "Dr. Dobbs Portal".  You can follow the conversation by clicking the following link. 

http://www.ddj.com/web-development/204802784

Lies, True Lies and Polling (aka heuristics, manipulation and make-believe)

People will vote for Hillary because she's got Bill
People will vote for Barack because he’s got charisma
People will vote for Rudy because he was there on 9/11
People will vote for Fred because he looks like a president

With nearly a year to go before election day, the long running 2008 presidential saga is certainly going to be the most expensive, punditized, analyzed, polled, debated, spinned and boring race in history. While campaign hot air fills the dead air on the 24-hour news channels and talk shows, I doubt that the electorate is becoming better informed or more caring.  In fact, the people might become so bored after being hit over the head by all this rhetoric for such a long time, they might just stay home on election day.

Seems like anyone with a phone can become a pollster.  If you can randomly call a hundred people (out of a total US population of 303,315,297 presently) and ask some questions about this or that, you might even get invited to talk live with Lou Dobbs or Bill O’Reilly depending on the left or right leaning of the polling results. 

Does the constant drum beat of “insightful” polling really help the election process?   Not much.  Between the potential  for bias in the polling questions and the interpretation of the result, one can certainly make all sorts of conclusion about how America feels about the candidates or what matters in the ’08 election.  Is being a woman going to hinder or help Hillary’s campaign?  Does a serial monogamist have the moral fortitude to occupy the most influential bully pulpit in the land?  Good questions--but can one really expect true insight from asking random people random questions?

Which brings us to the issue of monitoring real user’s experience of Web application.  So often we are also guilty of expounding on complex performance issues by relying on gross generalization--just like these political operatives with their polling questions.  Since the dawn of the WWW, we have regarded results from synthetic monitoring services as the standard for performance measurement while overlooking the possibility that the scripts might be flawed or outdated, and the machine generating the make-believe transactions might have a high bandwidth link to the Web -- or perhaps it's not even running a mainstream browser. 

Are we comfortable with relying on limited data to predict that Aunt Betsy in Des Moines can successfully refill her prescription using a shared computer at the local library?  Furthermore, do we have the visibility to determine what’s wrong, or how to tune the application or infrastructure to make them run better?   The answer is no, no and definitely no. 

The only way to deal with this challenge is to measure performance directly with each real user running real transaction in real time.  No heuristics, no fancy statistics and no make-believe.  That is why we still have an election day when each voter is being asked directly to make a definitive choice for president.  That’s what counts.

Norman Mailer is Dead--And Other Thoughts about Good and Evil

Norman Mailer is dead.  Like Hemingway half a generation before him, he cut a large swath through life and American literature.  While Hemingway’s persona is tempered by the larger than life heroism emanating from the international socialism movement between the world wars, Mailer’s was sculpted by the beat generation and the anti-Vietnam War movement.  It is like the “selfless” versus the “self” even though both are egotistic self-centered chauvinists who happen to be able to write, and write fabulously. 

Call it different UI (user interface) in IT lingo— underneath it’s the same drunken stupor, the same womanizing, and yet, different.  Perhaps the differing UI explains why Hemingway seems to have a larger worldwide following and better name recognition.

Hemingway was a Nobel Laureate and that should provide a significant PR edge over Mailer.  Beyond that, they both wrote about pasty regional conflicts in novels that affirmed their positions within the Parthenon of American literature.  The difference is that the war that Hemingway wrote of in For Whom the Bell Tolls is the war of good against evil, of the volunteers of the International Brigade marching towards certain demise against the Fascist military machine in the Spanish Civil War.  Yet, they persevere, sacrificing for International Socialism and the spirit of Man. 

Mailer, on the other hand wrote about the anti-Vietnam war march on the Pentagon—a war that is murky in cause and outcome.  There is no romanticism and heroism to counter the destruction and despair of war, any war. This is the reality where there is no “just” war with distinct “good” or absolute “evil” on either side of any conflict. 

For that matter, not even the Spanish Civil war is purely a struggle of good vs. evil.  In the view from the rear view mirror it is not clear that the Republicans are good and the Nationalists are bad.  Yet, thanks to the work of Hemingway and others, there’s certainly a patina of goodness layered on the Republican cause.

If good and evil are indistinct in war, they certainly give writers, historian and journalists ample opportunities to spin them in whatever way appropriate for their purpose or point of view.  The only reality is the human suffering endured by the populace —  be it in the struggle against Fascism, Communism, Imperialism, Colonialism, Capitalism, or any other “-isms” — that man in his unquenchable thirst for power chooses to create. 

In IT, unfortunately, we lack the Hemingways or Mailers to skillfully take a literary position on whether our applications are good or bad.  However, like war, we can count on the fact that certain users will be impacted by the effect of bad application performance, and the only real question is how much and how often.   You have to know, and more importantly, anticipate what users are experiencing in order to effectively manage and control the application and infrastructure.  Thinking that you don’t have a problem does not really mean much.

Sensible Monitoring for Sensitive Areas

How do you monitor “performance” as experienced by the end user?  In an earlier blog, we examined Mississippi State Ordinance §19-5-103 which essentially holds the adult entertainment industry responsible for certain biological reactions patrons might experience whilst appreciating "the show".

Without revisiting the details, the unanswered question is this: how can the law monitor audience response to “adult entertainment” without affecting the show itself? Such a monitoring system must be somehow individual, invisible, anonymous and yet painstakingly accurate—reputations are on the line here. 

Monitoring these events certainly poses a thorny challenge akin to the problem facing web operations as they attempt to monitor the end-user experience. The monitoring capability has to meet competing objectives:

1) Provide an accurate measure performance experienced by individual end-users under the thousands of different ways Web applications are delivered

2) Do so in a way that is individual, invisible, anonymous and accurate – in short, in a way that never impacts delivery of the application (unlike changing the source code or downloading monitoring agents onto the user’s PC).

You can easily appreciate the difficulty of this challenge for the Magnolia state adult entertainment establishments.  How do you detect (and document) customer response in a dimly-lit strip joint or massage parlor where the “action” might be obscured behind partitions or booths?  If enforcement required powering-up the house lights or rearranging the layout it would alter the “operating environment” and change the experience dramatically — akin to changing source code in the Web arena.  It seems unlikely that the law require patrons to wear a “turgidity” detection device (somewhat like downloading an agent onto the user’s browser for Web apps).

Luckily enough, the problem has been solved — for Web applications anyway. Our company, Symphoniq, has developed a deceptively simple solution that allows IT to monitor performance based on TRUE (the real user experience) for any Web application delivered by the F5 BIG-IP appliance. 
The lightweight, downloadable TrueView Express product is integrated with BIG-IP to measure application performance directly from the every browser without requiring client-side agents, changes to the code, or the purchase of additional appliances.  It can monitor millions of Web users located anywhere in the world without burdening the BIG-IP.  All told, a very simple to use solution to tackle a difficult challenge. Check it out at www.symphoniq.com.
 
Now if the law in Mississippi could have something to monitor “discernible turgidity” in an adult entertainment establishment, the law enforcement establishment in Mississippi would have a much easier time of it.

Mississippi Sex Shows & The Real User Experience

On a swing (so to speak) through the Southern States to preach the virtues of understanding TRUE (The Real User Experience) in the Web environment, I came upon this unexpected scene somewhere in the fields of Ol'e Miss.  Could it be legal to offer schools field trips to adult entertainment establishments down under (meaning the Mason-Dixon line not Australia)? 

While Yahoo didn’t give me an exact answer, I did get some enlightenment from Mississippi State Ordinance §19-5-103 entitled “Regulation of massage parlors and public displays of nudity.”  Interpreting this, such establishments in Mississippi can be fined for the expected stuff like too many of certain uncovered body parts.  But, the really interesting part is that in essence the State holds clubs accountable for the user experience as well.

That’s right.  Gentlemen's clubs can be punished if they create an environment where by “the covered human male genitals (are) in a discernibly turgid state.”   Now, let’s interpret the above: §19-5-103 stipulates that the establishment can be punished for the state of undress of the entertainer AS WELL AS the effect said entertainer has on the clientele (if they’re men anyway).

This is what I call legislating for TRUE (the real end-user experience)!   For example, even if the entertainer is fully clothed, but arouses the clientele through action alone, the State can still evoke the wrath of law upon the establishment.

TRUE enlightenment -- with lessons for us IT-types as well.  Why?
 

While 5-nines (99.999%) server-up-time is sexy in IT circles, TRUE -- the real user experience -- is what really matters.  Before you can rest assured that your application is running to spec, you need to know if those 5-nines are really delivering the user the experience he/she needs to fully appreciate the application... good performance so to speak.  And while nudity is considered the sexy part of adult entertainment, the reaction of the clientele (i.e., the end-user) has legal importance. 

In a world filled with complex, sexy Web apps., all the green lights flashing on servers can’t ever really predict that the end user is getting the desired level of application response time.  The service level threshold for a Web application must be determined solely on delivering the desired level of application response time — or in the case of an adult entertainment, the level of “satisfaction” as perceived by the end user. 

I ask you, what is the purpose of delivering the application if not to satisfy the desires of the end user?  With all that said, should one ever decided to frequent such as establishment, can one be allowed to claim any bills he or she might hang onto Sweet Lola’s G-string as a legitimate business expense?

Blue Suede Shoes

Well, it's one for the money

Two for the show

Three to get ready now

Go cat go

For eCommerce merchants, the months leading up to the hot Christmas season could be well described by the first verse of Blue Suede Shoes—the Carl Perkins song made famous by King Elvis.  Yet with the complexity of today's eCommerce sites, the "fat lady" might not be singing about “money” and “show” but a different verse of the same song:

Well, you can knock me down

Step on my face

Slander my name all over the place

Because every year, there will be a couple of household brands that get tripped up by sluggish performance or outage problems that not only cost them millions in lost sales, but way, way more in trashed brand image and decreased stock value due to bad publicity in Blog-sphere or the Wall Street Journal.  According to a recent 2007 poll by Harris Interactive

What to do?  Though they tested everything thoroughly before the lock-down to avoid anything that has a remote chance of going wrong, something will go wrong (according to Murphy's Law).  If an IT expert tells you there is no problem, it only means that from his/her perspective and based on what he/she can control there is no problem.  What about the web cloud that IT has no direct control?  It is called a cloud for a good reason—one cannot easily put an arm around this critical resources.  And what about the interaction between application and infrastructure in terms of differences in patch or .dll level? 

It's impossible to guarantee a trouble-free shopping season because of the complexity of the infrastructure and application including the screw-ups from third-party service providers and everything else which is beyond control of IT.  The only way to deal with the challenge of providing superior end user experience is to:

1.      Detect problems in real time from the real user’s perspective, and

2.      Isolate the cause of problems quickly so repair can be carried out before they snow-balled into crisis proportion, and

3.      Use the end user data to not only repair any problems, but also to optimized the infrastructure or application to minimize the chance of the same problems recurring.

Sounds difficult to implement the above procedure?  Not really.  You just have to get the right tools and take a systematic approach to managing Web application performance.

Oh yes, while you're thinking through the issue, enjoy this clip of Elvis singing Blue Suede Shoes.

"80 percent of U.S. adults who have had a negative experience with a company say they will never to go back to that company again, up from 68% in 2006."

Can Lock-down Prevent Lock-out?

Lock-down or Lock-out?  The Goal is Delivering Performance to Keep Transactions Flowing

'Tis the season for eCommerce sites to lock down applications and infrastructure to prepare for the holiday rush. Yes, it's the season when merchants hope that by freezing all changes to code and infrastructure no one will touch anything that might cause response-time issues.

Complex Systems Hide Performance Problems

Unfortunately, lock-down can't prevent customer lock-out.  Given the complexity of today’s eCommerce sites, hidden problems always lurk like time bombs everywhere in deployed and fully tested systems—infrastructure, application, or the interaction of these pieces. These problems are intermittent so they are hard to duplicate.  They are caused by the every changing Web.  And worse, they have the power to lock-out online shoppers -- presenting a brick wall that keeps shoppers from completing their transactions.

The Four Second Rule

Response time is a crucial part of online selling. To keep customers engaged, IT must meet or beat the goal of four-second page load times. When load-times exceed four seconds, customers often click to a competitor’s site. So, to keep response-times low, merchants try different approaches to gain control of the many distributed and moving pieces of technology that reside both inside and outside the firewall. Only if IT can control all these variables can they consistently meet shoppers’ expectations.

The basic nature of the Web works against IT’s need to control the system. In this regard, using a cloud to represent the Web is apt in many ways—like weather the Web cloud is unpredictable and fluid; and it’s just as impossible to control. Take for example what might happen to an online retailer who has just deployed new content using pictures or video to show new merchandise and perhaps a virtual mannequin to display the wares. To support these new applications perhaps they have added new infrastructure components like servers or network appliances.

Will it Really Work Under the Load of Real Holiday Traffic?

It worked in the test environment and even in production.  But given the fluid nature of the Web the real question is: How long will it continue working? When real holiday shoppers hit the site in droves will the new components perform as expected? What  unseen problems lurk in the system that that might rise-up and attack under the stress of real load?  When performance problems occur it’s highly likely that shoppers will find that even the most compelling shopping experience has transformed into a “perma-wait” state that only motivates them to click away and shop elsewhere.

Every Customer Counts

With the projected dismal growth in overall retail sales in the US caused by the housing credit crunch, online shopping is expected to be an ever more important slice of the overall retail pie this holiday season. According to a survey by Burst Media, four out of five shoppers are planning to research gifts online. And 50.7% of shoppers are planning to purchase gifts online versus only 37.6% who did so in 2006. eCommerce merchants just cannot afford to be lackadaisical about end-user response time. Tools and processes must be in place prior to and after the lock-down so that and problems with response time can be detected in real time and fixed quickly to keep the online shoppers happy. More in the next blog.