Monte Carlo released Incident IQ, a new suite of capabilities that help data engineers better pinpoint, address, and resolve data downtime at scale through the Monte Carlo Data Observability Platform.

Incident IQ automatically generates rich insights about critical data issues through root cause analysis, giving teams unprecedented visibility into the end-to-end health and trust of their data beyond the scope of traditional data quality solutions.

On average, companies lose over $15 million per year on bad data, with data engineers spending upwards of 40 percent - or 120 hours per week - of their time tackling broken data pipelines. In the same way that New Relic, DataDog, and other Application Performance Management (APM) solutions ensure reliable software and keep application downtime at bay, Data Observability solves the costly problem of data downtime, in other words, periods of time when data is missing, inaccurate, or otherwise unreliable.

To help companies eliminate data downtime, Monte Carlo built Incident IQ, the first end-to-end solution that conducts root cause analysis for data issues at each stage of the pipeline, from ingestion in the data warehouse or lake to analytics in your business intelligence dashboards. Incident IQ automatically generates historical insights about your data to identify patterns in query logs, trigger investigative follow-on query results, and monitor upstream dependency changes to pin-point exactly what caused the issue to occur, reducing the amount of data incidents by 90 percent at each stage of the pipeline.

Developed after reviewing thousands of real data incidents from our customers, Incident IQ gives data engineers access to insights about their code, their data, and their operational environment that allows them to quickly and collaboratively get to the root cause of data problems -- all in a single UI.

With Incident IQ, everything related to the data issue is captured in an elegant timeline with easy commenting, documentation, and collaboration features to create rich post-mortems. This level of detail, common in software engineering and DevOps tooling, helps data teams learn from past incidents and determine where to allocate future investment. Additionally, Incident IQ makes it easy to create and share high-level incident reporting with CTOs and CDOs, fostering greater data trust and ownership across the company.

Core capabilities of Incident IQ include:

- Central UI that connects the dots between correlated causes of data incidents, and surfaces a historical collection of data incidents for quick comparison.

- Access to example queries that pull sample data, as well as rich query logs, historical incidents, and quick links to Monte Carlo’s Lineage and Catalog features, making it easy to identify, root cause, and fix data issues all from the same interface.

- Automatic insights based on the statistical correlation between table fields in anomalous records (for instance, Incident IQ can surface if an increase in order_id null values correlates with a specific order source).

- Automatic, end-to-end lineage that maps impacted downstream BI dashboards to the furthest upstream tables, helping teams narrow the focus of root cause investigations.

- Automatic runbooks and workflows to make the incident resolution and triaging process easy, fast, and collaborative between data engineers and analysts.

- Comprehensive query logs that reveal periodic vs. ad hoc queries, changes in query patterns, and more.

“As companies become more data driven, it’s fundamental that organizations not only understand the health of their data, but also have the data observability necessary to trust it from end to end,” said Lior Gavish, CTO, Monte Carlo. “As the data stack fragments to incorporate new tools, it’s becoming increasingly difficult to identify when data pipelines break and take action to fix them. With Incident IQ, data practitioners and leaders alike can holistically understand and respond to issues faster, before they become a serious problem for the business. We believe these features will help customers eliminate hundreds of hours of data downtime and thousands to millions of dollars in savings each month, as well as enable data platform teams to scale with rich post-mortems that track performance and facilitate greater learning.”

Monte Carlo is a Data Observability partner for the FinTech, e-commerce, media, B2B software, and retail industries, counting data teams at Fox, Vimeo, ThredUp, and PagerDuty among their customers.

In February 2021, the company announced their $25M Series B funding, led by Redpoint Ventures and GGV Capital, and was named one of the 2021 Enterprise Tech 30.