Analytics That Matter - For APM-Generated Big Data
September 20, 2012
Jean-François Huard, Ph.D.
Share this

“Big Data” is everywhere. What does it mean? Just as Cloud Computing bursted onto the scene a few years ago, it depends on whom you ask.

Traditionally, in the Business Intelligence (BI) world, Big Data included analyzing historical business data from large data warehouses with the purpose of identifying long-term trends that could be leveraged in consumer business strategies. In recent years, Big Data has been a term talked about in the IT industry as an application of technology to attack extremely large, unstructured data sets that can reside both within and outside of an organization. If you look at a recent definition of Big Data, it is a term applied to data sets whose size has grown beyond the capability of commonly used software tools to capture, manage and analyze within a tolerable period of times for different use cases.

Application Performance Management (APM) is an extremely relevant use case and has a developing “Big Data” problem. Several factors are contributing to the explosive growth and type of data that must be analyzed and/or correlated in application performance monitoring and business service management (BSM).

First, the number of components that make up today’s mission critical applications has exploded. Instead of hundreds of servers for an application, nowadays, because of virtualization, you can easily be talking about thousands of virtual servers and objects for web applications.

Secondly, the diversity of data that people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply understand traditional IT infrastructure performance based on server operating system, network traffic, and storage capacity. Application Performance analysis is now based on the relationships of IT infrastructure components, application performance metrics from applications and application servers, business activity monitors (BAM) data, customer experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the aggregated transactional data, there are new systems that capture transactions’ actual path encompassing the entire application stack.

Finally, the requirements for analysis speed and data granularity have also increased significantly. Mission critical application performance now requires real-time or near real-time data analysis. When we were doing server availability and performance monitoring 10 years ago, it was the norm to collect and analyze data every 15 minutes.  Today, this has evolved to data analysis every 5 minutes or less with sub-minute data collection where all transaction paths are collected for data analysis. When mapped out, it's easy to see the enormous growth particularly when you look at APM related storage requirements that are quickly growing from gigabytes to terabytes and tomorrow petabytes.

Where APM and Big Data Meet the Cloud

All this data requires extremely complex analysis and correlation in order to truly understand performance of critical applications.

One of Netuitive’s large enterprise customers reported that it monitors and correlates a billion infrastructure and application data points and business metrics daily as part of its global service delivery. This is what I am referring to as APM-generated Big Data.

In addition to the shear number of data points, IT operators are expected to provide real-time analysis to the business and long-term storage for post-mortem analysis, capacity planning and compliance.

So where does this lead us? This is where APM and Big Data meet the Cloud. The cloud can deliver cheaper and more flexible storage and computing power crucial to analytics for Big Data. It also has the capability to be much more elastic for your APM data storage and analytics needs. Organizations can actually think about storing years of collected and aggregated APM data for compliance and analysis purposes without the cost being prohibitive.

But what does this mean to vendors in the APM space? 

First of all, the analytics platform for APM data has to evolve to be able to process the growing number of different data sources across business, customer experience, applications and IT domains. Netuitives’ “Open” analytics platform is engineered to address virtually any data source in real-time.

Secondly, data storage and access time will be critical even as APM data volumes continue to explode, so not only does the technology need to be able to run in the cloud, but the traditional pull-based data collection architecture has to evolve into a push based model with an horizontally scalable computing and storage architecture in order to become virtually limitless in terms of scalability. This is critical for larger organizations as “real” time no longer means analysis every 5 to 15 minutes, but sub-minute analytics.

Lastly, because storage and computing costs should not significantly exceed the cost of analytics software for a solution to be viable, Netuitive is advancing its product architecture to leverage NoSQL columnar data store as a replacement to traditional database.

While our R&D challenges are complex, the goal is simple: provide APM Analytics that matter by enabling our enterprise customers to process billions of infrastructure, application, and business metrics from hundreds of thousands of managed elements at 10x less cost than existing infrastructures.

ABOUT Jean-François Huard

Jean-François Huard is Chief Technical Officer and Vice President of Research and Development at Netuitive, Inc. In this role he is responsible for leading the company’s vision and technology innovation effort.

Previously, Huard was Chief Network Architect and Vice President of Network Engineering at InvisibleHand Networks, a start-up company funded by Polaris Venture Partners. Earlier, he led the technology team at Xbind, Inc. Earlier in his career, Huard worked in network fault management at AT&T Bell Labs, and was a member of the research staff at the Center for Telecommunications Research.

Jean-François contributed to the definition of the international MPEG-4 standard, and was chair and technical editor of the IEEE P1520.2 working group. He has authored or co-authored many scientific papers published in technical journals and conferences, standard contributions, and has filed multiple patents.

Share this

The Latest

March 27, 2024

Nearly all (99%) globa IT decision makers, regardless of region or industry, recognize generative AI's (GenAI) transformative potential to influence change within their organizations, according to The Elastic Generative AI Report ...

March 27, 2024

Agent-based approaches to real user monitoring (RUM) simply do not work. If you are pitched to install an "agent" in your mobile or web environments, you should run for the hills ...

March 26, 2024

The world is now all about end-users. This paradigm of focusing on the end-user was simply not true a few years ago, as backend metrics generally revolved around uptime, SLAs, latency, and the like. DevOps teams always pitched and presented the metrics they thought were the most correlated to the end-user experience. But let's be blunt: Unless there was an egregious fire, the correlated metrics were super loose or entirely false ...

March 25, 2024

This year, New Relic published the State of Observability for Financial Services and Insurance Report to share insights derived from the 2023 Observability Forecast on the adoption and business value of observability across the financial services industry (FSI) and insurance sectors. Here are seven key takeaways from the report ...

March 22, 2024

In MEAN TIME TO INSIGHT Episode 4 - Part 2, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses artificial intelligence and AIOps ...

March 21, 2024

In the course of EMA research over the last twelve years, the message for IT organizations looking to pursue a forward path in AIOps adoption is overall a strongly positive one. The benefits achieved are growing in diversity and value ...

March 20, 2024

Today, as enterprises transcend into a new era of work, surpassing the revolution, they must shift their focus and strategies to thrive in this environment. Here are five key areas that organizations should prioritize to strengthen their foundation and steer themselves through the ever-changing digital world ...

March 19, 2024

If there's one thing we should tame in today's data-driven marketing landscape, this would be data debt, a silent menace threatening to undermine all the trust you've put in the data-driven decisions that guide your strategies. This blog aims to explore the true costs of data debt in marketing operations, offering four actionable strategies to mitigate them through enhanced marketing observability ...

March 18, 2024

Gartner has highlighted the top trends that will impact technology providers in 2024: Generative AI (GenAI) is dominating the technical and product agenda of nearly every tech provider ...

March 15, 2024

In MEAN TIME TO INSIGHT Episode 4 - Part 1, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses artificial intelligence and network management ...