Prelert announced a new feature of its Anomaly Detective machine learning engine that enables multidimensional analysis to be conducted on large volumes of data at high speed. This new feature, Stats Reduce, dramatically shrinks data transfer sizes, making it possible to perform the complex behavioral analysis of terabytes of data per hour.

Accurately identifying anomalous activities to detect the fingerprints of an advanced persistent threat or the cause of very complex IT performance issues requires a cross-correlated analysis of multiple data attributes. Performing this type of analysis at very large data scales has traditionally required a massive data transfer, which made real-time analysis impossible.

By leveraging the statistical aggregation functions already available in platforms like Splunk and Elasticsearch, Prelert’s Stats Reduce overcomes this challenge. The technology provides a 40x reduction in the amount of data that needs to be transferred and employs advanced analytics specifically designed to maintain data fidelity and return accurate results.

Stats Reduce has been tested against data sets from Prelert customers and the technology has been proven to return the same, accurate results whether operating on aggregated or raw data.

“Prelert is committed to providing the most accurate and robust insight into data in real-time, no matter how large or complex,” said Stephen Dodson Ph.D, Prelert’s CTO. “The scale of modern environments present challenges that require careful selection of methods and techniques, and we built our technology to align with these environments from the start. With Stats Reduce, the aggregation techniques we developed allow massive volumes of data to be analyzed in a distributed manner, enabling real-time multidimensional anomaly detection on Big Data.”

“Advanced threats and IT performance issues are becoming harder and harder to detect in part because they’re hidden in the massive amounts of machine data that IT systems generate every second,” said Dennis Drogseth, vice president at EMA. “Relying on the analysis of a single data source is no longer an adequate means to identify significant issues, as capturing increasingly complex interdependencies requires cross-correlated analysis of multiple data sets. Prelert’s ability to do this analysis in real time is a significant step toward providing more optimized and more secure service delivery.”

Keeping with Prelert’s mission to democratize data science and make it easy for everyday users, Stats Reduce is currently available in the latest version of Anomaly Detective, with push button functionality. Users must simply select “Use Stats Reduce” when deploying Anomaly Detective in the Splunk Enterprise environment. Stats Reduce will be available on other Big Data platforms in Q4 2014.