In Part One of BSMdigest's exclusive interview, VMware's Javier Soltero, talks about the founding of Hyperic and the management of web applications.

BSM: What drove you to develop Hyperic?

JS: Back in 2002, I had joined a company that was focusing on enterprise software called Covalent Technologies. Covalent was building products specifically around Apache with an emphasis on web infrastructure. We saw an opportunity when we started working with some of our larger web infrastructure customers. We saw that every one of them – ranging from enterprises to service providers to Internet companies – had this common sense of dissatisfaction with the management tools that were available for the types of web applications they were supporting.

That led us to investigate the problem and we discovered the people that operate web infrastructure tend to be a little different than the previous notion of the enterprise IT professional. The applications they support are subject to a completely different performance profile. There are many different places where you could get bottlenecks that you have to be able to diagnose and fix. Because of the change in the type of applications that people were building and the type of people that were supporting them, we saw a great opportunity to build a product that fit both the applications and the people who managed them.

I met the co-founders of Hyperic at a dot-com in the late 90s where we had experience building a management product for a very large consumer website. So even prior to getting to Covalent, we had firsthand experience as to what the life of a web operations person is like, and how their tools were not up to the task most of the time unless they chose to develop the tools themselves.

BSM: Tell me a little more about that. What were the traditional tools missing?

JS: You had two opposite ends of the spectrum. On the one end, you had enterprise BSM tools of the framework variety – the Big Four products that were originally built for managing the actual physical part of the infrastructure, namely servers, networking hardware and storage. The systems part of systems management is where the majority of the frameworks have their roots.

These frameworks were built to manage packaged applications like PeopleSoft and SAP that were essentially black boxes. They were monolithic and would scale upwards instead of outwards. They were managed in a completely different pattern and they required discrete skillsets in just managing that application. That is one extreme end of the management tools world.

The other extreme end was the set of point technologies developed mostly by web operations people that were frustrated with the fact that they had to keep these applications running and they didn’t know how. The primary challenge was that these tools were built by the same people that used them, which were not always software engineers experienced in architecting this type of complex software. So you had IT and ops people developing tools to do what they wanted to do, but with a different approach to building large-scale systems.

BSM: What is it about a web application that makes it so different from a traditional application, in terms of monitoring?

JS: The first part is the diversity of technology that powers it. In a web application architecture you have at a minimum two components, a web server and a database. But in reality, even in the simplest of cases like a PHP application, there is still a fair amount of custom application code sitting on top of the web server that you may or may not be able to manage. And every one of these layers of infrastructure are likely to have anything from bottlenecks to straight out faults to any kind of challenge that would require monitoring, so you can at least figure out the root of the problem that is bringing your app down.

The other part is the rate of change of these apps, particularly with consumer-facing web applications. One of the benefits of these types of apps is that you can change them whenever you like. Upgrading, adding features and making enhancements to a web application is typically orders of magnitude easier than rolling out a new version of the bigger traditional enterprise apps of the PeopleSoft and Siebel variety. Consequently, the fact that you can build them more quickly, and upgrade them, and enhance them at your discretion creates a far higher rate of change inside of the data center. If you are not able to keep up with the rate of change - if you are not able to augment your management and monitoring tools to be aware of new servers, new VMs, new Apache Tomcat instances - it is going to be very difficult for you to be aware of what is really happening inside of your application.

So I would say those two are the primary factors that make web applications different from a monitoring tool perspective, and they are the two factors that influenced our design of Hyperic more than anything.

BSM: What capabilities does Hyperic have to be able to solve those problems?

JS: First, it is a product that is built for a cross-functional IT professional – someone that cares about the hardware, the virtualization infrastructure, the middleware, and the application itself, and is responsible for the health of the entirety of that infrastructure and needs visibility across the entire infrastructure.

The second part is that the breadth and type of features it offers mirrors those that a person responsible for a web app would need. If you are just providing monitoring, if all you are doing is drawing graphs, you are only solving one part of the problem. Or if you are only sending out notifications for the problems that are happening at the moment, you are only solving one part of the problem. Ideally if I am trying to manage these things successfully, I want to get ahead of the problems. I want to anticipate when I am going to run out of capacity. I want to anticipate a minor problem before it snowballs into a massive outage. And more importantly, I want to be able to remediate the problem either with my intervention, or ideally without my intervention. These are things every IT professional tends to do with different degrees of sophistication, and the idea is to provide a one single solution that tries to cover as much of this vast problem space as possible. And at the same time make it flexible enough so that a user can customize, either by adding support for custom technology or by providing hooks into other systems.

BSM: Do most people that are building web apps understand that they need a different kind of monitoring tool?

JS: I think most people understand. The challenge is everybody comes into this discussion with a bias. Anyone who is evaluating any kind of monitoring tool is going to have a bias toward the area where they have the most competence. So if I am a network and systems guy and I hardly ever have any responsibility for or visibility into middleware and applications, I am going to want a tool that believes that the world begins and ends with the network and systems layer, and everything else is just a bunch of processes that I have to watch for.

However, if I am an applications and middleware person, and I treat hardware as something that needs to be there but is either someone else’s problem or assumed never to be the source of the problem, I am going to look for a product that will provide more visibility into the juicy middle of the application stack, while hopefully providing some sort of visibility into the hardware layer.

With respect to Hyperic, people who tend to be much more middleware and applications focused are generally a lot more drawn to Hyperic out of the gate, and people that come from a hardware and operating system perspective tend to have a little more difficulty understanding how Hyperic does what it does.

BSM: Was the Covalent Application Manager spun-off from Covalent before the company became part of SpringSource?

JS: Yes. The original Covalent that I was Chief Architect for, that employed myself and the other four co-founders of Hyperic, ceased to exist in April of 2004. At that point, the founders and I acquired the Covalent Application Manager technology, which had just been brought to market. We continued enhancing that with a focus on web applications and a completely different business model, and that is how Hyperic was born. The other half of that existing Covalent business was acquired by a different group of excellent people who kept the name Covalent but became a completely different company focused on providing world-class Apache support and other technologies. SpringSource acquired that second Covalent in 2008, and then acquired Hyperic in 2009. It is more of a coincidence.

BSM: That is what I was going to ask – so it was a coincidence that SpringSource happened to acquire both Covalent and Hyperic?

JS: We had a lot of familiarity between both teams. Hyperic was partner to both companies. They were an OEM partner to Covalent because they were selling the Hyperic product to some of the support services customers for Covalent 2.0. And Hyperic was also an OEM partner of SpringSource where the tc Server product and the Spring Enterprise solution in its original inception were built on an OEM version of Hyperic. Now, since we are all part of one company, Hyperic is the management platform for all of those pieces.

BSM: I find it interesting that you worked at Netscape as well. Were there any experiences at Netscape that you brought to Hyperic that helped you in some way?

JS: At Netscape I worked on infrastructure products for the very dawn of web infrastructure – things like web servers, LDAP directory servers and Java application servers. I think the core lesson there was that I saw firsthand how big enterprises approach the problem of deploying early stage Internet infrastructure and how that challenged everyone from the individuals responsible for managing it to the architects that were responsible for building applications that use it. And that influenced me and still does.

The other part of the Netscape experience that was valuable is the idea that when you have a big seismic shift in computing – like the browser was and like cloud technologies and virtualization are today – an enterprise customer is going to know deep inside that they need to participate; and they need to be forward thinking about these types of big shifts; and they need to be early adopters and invest in the technology and figure out how it maps into their world.

As a vendor, you need to be very diligent about how you ride this wave of change alongside your customers. If you get too far ahead of yourself, you are bringing products that don’t have the same degree of maturity or even ripeness into these high stress environments. You just have to know what you are in for. That was certainly the case with the early Internet technologies that Netscape brought to market. To a certain extent we live that today.

I find with BSM products in particular, the problem is so hard to solve that it takes three to fours years of actual product use in the field for you to hit your scalability target points, your functional objectives, etc. People don’t take huge risks on these types of products, and the very nature of these products mean that they sit at the intersection of all the change and complexity of the data center. And that is a tall order. At times that may seem like an impossible mission. How are you supposed to be the thing that brings order into chaos while keeping ahead of the game? And we live that every day.

BSM: You said people don’t like to take risks on management and monitoring tools. Does open source fit in well because it is lower risk?

JS: No, I think from that perspective open source might hurt, and I will tell you why in a second.

Where it does help is that you get more people using it, so if you do it right, you get much more feedback – unrestricted participatory feedback. If you are lucky (and you do it right), the users are interested in making the product better and so the feedback they are giving you will be far more honest. And it is not going to be encumbered by the typical vendor-customer relationship.

The reason why it hurts is the idea that open source as a purely cost reduction mechanism, or a way to get your hands on cheaper software, is a very dangerous proposition in the world of BSM. The reason is that this is a hard problem to solve. And solving this problem is very, very valuable. There is a reason why this is a $10 billion plus per year market, and that reason is that this is hard work. IT management is very difficult from a technical perspective, and if done right it is something that is going to provide a measurable impact to the customer that is worth money. So you cannot overly trivialize what it costs to build products that can do the things that our products do, or to provide the type of support our customers expect. This is something we have held core since we were an independent company and something that rings true at both SpringSource and VMware.

BSM: And yet it seems that open source is very prevalent in the management and monitoring space.

JS: Absolutely. The reason why it is so prevalent is there is tremendous value in getting people unfettered access to your technology and creating good feedback loops like Hyperic and a handful of other vendors have been able to create. And the other part is that you need to be able to empower the customer to extend that product easily and wrap it more closely to their own requirements, which is greatly enhanced by open source technology.

From a customer perspective, instead of looking at lower cost, you should look at the implicit insurance policy of open source. The technology is freely available and whether it is an open core arrangement or completely open source, the risk of that one vendor strangling you, not being able to move away from that, or you tying yourself too closely to the fortunes of any software company, is greatly mitigated by access to the product and source code.

BSM: What are you doing now for VMware?

JS: Most of what I have been doing is helping the company develop its cloud computing strategy, and linking up the SpringSource developer success with the infrastructure success that VMware has had, and the management software success that we continue to have as part of Hyperic.

In addition to that, I spend a lot of time talking to our largest customers, understanding how they are viewing virtualization and cloud computing, and what problems they are having.

BSM: Now that the technologies like Spring, Apache Tomcat, Hyperic and VMware are all under one company, are the next generations of these technologies becoming even more integrated than they are today?

JS: Yes, but not at the expense of continuing to provide support for technologies that fall outside of that. Hyperic has a continued commitment to breadth and choice beyond what VMware can offer you. You will see that Hyperic manages middleware in addition to Tomcat and the other things that are part of our VMware product catalog.

On the flip side, we intend to continue to develop more integrated solutions that leverage vSphere, which is the absolute best virtualization solution out there, as well as Tomcat, which we believe represents the future of Java middleware because of its lower footprint and scale.

Click here to read Javier Soltero's comments on virtualization and BSM in Part 2 of the interview.