The results of the APMdigest Big Data survey are in – revealing significant increases in the growth of IT environments, volume and diversity of performance metrics, and data monitoring frequency. The survey also shows "APM-generated Big Data" as a specific use case of the broader Big Data problem, typically requiring analysis and correlation of real-time data for operational impact.
The survey, sponsored by Netuitive, confirms three major factors are driving the growth of Big Data with respect to application performance management (APM).
1. Infrastructure Growth
One of the most interesting facts revealed by the survey is the fact that, in spite of wide adoption of virtualization technology to consolidate physical resources in recent years, IT infrastructure has continued to grow at a significant rate.
Almost half of the survey respondents (46%) said their IT infrastructure – comprising all servers, storage devices and network devices – either doubled or tripled over the past decade, while a quarter of respondents (26%) said IT infrastructure grew by four or fivefold.
These responses indicate that even if virtualization is minimizing purchases of physical servers, this is clearly not stopping the growth of the IT estate. Second, even if virtualization reduces the number of physical servers, it is not necessarily reducing the total number of servers. Now organizations must also monitor those virtual servers, and the specialized network and storage infrastructure to support virtualization, and this has contributed to the growth and complexity of performance data.
Growth in applications supported by this infrastructure also has a major impact on the Big Data boom, especially in terms of APM.
“The infrastructure complexity and scope increase is driven by virtualization and abstraction,” added Jonah Kowall, Research Director, IT Operations Management at Gartner. “This is more easily measured than the complexity of applications, which cannot be measured, but would likely show an order of magnitude increase over infrastructure. The complexity creates more monitoring which in turn creates more data.”
2. More Metrics
With the growth of IT infrastructure, as well as the growth of applications and their complexity, comes an increase in performance metrics. In fact, 83% of respondents agreed with Gartner’s estimate that metric collection has increased 300% or more in the last four years. And one respondent from a large corporation has seen the IT estate grow tenfold, to the point where they now collect over a billion metrics daily.
Several respondents also confirmed that APM tools are part of the huge explosion in metric collection, generating thousands of KPIs per application.
“Early on, we were just monitoring servers, with maybe 10 to 12 metrics per server,” explained Graham Gillen, Product Marketing Director for Netuitive. “Now we are monitoring servers, switches, storage devices, and even applications. Instead of ten metrics per server, it could be thousands of metrics for an application. This explosion in metrics is happening now because APM tools have really only been deployed in the last 2 to 3 years.”
3. Greater Frequency
Data monitoring frequency has also increased significantly, with monitoring intervals trending towards one to five minutes compared to 15 minutes only five years ago.
More than half of survey respondents (58%) agreed that frequency of collection increased significantly over the last decade. The most prevalent data collection frequency is about 5 minutes, though 23% of respondents collect data every minute or less. Greater monitoring frequency results directly in more data.
The obvious next question is: What do you do with all that performance data?
Click here to read: Facing the Big Data Challenge
Click on the infographic image to access the full render graphic, condensed survey results and press release.
Survey Methodology: The survey,was conducted by phone and online during June and July 2012, assessing the opinions of targeted IT professionals in the US and the UK.
