Dataflow Complexity is the New Normal
April 07, 2017

Pat Patterson
StreamSets

Share this

The process of wrangling big data is fraught with pitfalls for enterprises. Data-driven enterprises are buckling under the burden of gathering, analyzing and making actionable an incredible and growing amount of data flowing in from a variety of sources. It's not just the amount of big data that is confounding data-driven companies: The speed at which data must be collected and analyzed, and the variety of data types (think: IoT sensors, log files, web clickstreams) are overwhelming enterprise data architectures, which are increasingly defined by a complex tangle of big data sources and processing systems. Topping all this is the problem of data drift, the unexpected changes that consistently plague big data sources and result in corrupt and unusable data.

In short, the complexity of data in motion is growing and risks undermining the success of the modern data-driven enterprise. A recent survey of data engineers and architects, conducted by StreamSets, sought to bring some perspective to the new reality in the enterprise, leading to some interesting insights about the enterprise data landscape.

As we expected, use of streaming data has become quite common, with a high number of respondents — 72 percent — collecting this data for a variety of uses. Of these, two-thirds (48 percent) collect a combination of batch and streaming data, since real-time data requires context to provide intelligence. In contrast, 28 percent move batch data only, and 24 percent ingest streaming data only.

Survey results also showed that enterprises are gathering data from a range of sources: 61 percent collect from transactional databases, 53 percent from log files, 42 percent from analytics databases, 27 percent from clickstream data and 18 percent from IoT devices.

Moving on from their use of streaming data, the survey reveals that enterprises are also experiencing a sense of data urgency — that is, expeditious analysis of their incoming data sets. In fact, according to the survey, 56 percent of respondents say they require data analysis within minutes of receiving the data, and 16 percent require analysis within seconds. The world has certainly evolved from the daily or weekly business intelligence report to a live dashboard, or even analysis that drives automated actions like website personalization that can have a direct impact on a business' effectiveness in engaging with its customers. These requirements put extreme pressure on enterprise data architectures not necessarily designed to deliver consumption-ready data with this type of speed.

Our survey responses indicate that enterprises funnel their data into a range of destinations, making them much more complicated and expensive to manage than ever before. In addition, respondents keep some of their data on premises (58 percent), some in private clouds (48 percent) and some in public clouds (27 percent). The combination of diverse data stores and multiple deployment models is a new phenomenon we call data sprawl, and it is a key driver of dataflow complexity.

The challenges of increased dataflow complexity are here and now and, given the unprecedented growth of data each day, must be considered the new normal. With this information as a bird's-eye view of the state of data in motion, savvy enterprises will adopt technologies and solutions that will help them evolve with the big data landscape.

Pat Patterson is Community Champion at StreamSets.

Share this

The Latest

April 21, 2017

In the spirit of Earth Day, which is Saturday, April 22, we recently asked IT professionals for the tips and tricks they're using to help keep their data centers as green as possible. Here are a few ideas inspired by the responses we got ...

April 20, 2017

Almost One-Third (28 percent) of IT workers surveyed fear that cloud adoption is putting their job at risk, according to a survey conducted by ScienceLogic ...

April 19, 2017

A majority of senior IT leaders and decision-making managers of large companies surveyed around the world indicate their organizations have yet to fully embrace the aspects of IT Transformation needed to remain competitive, according to a new study conducted by Enterprise Strategy Group (ESG) ...

April 18, 2017

The move to cloud-based solutions like Office 365, Google Apps and others is one of the biggest fundamental changes IT professionals will undertake in the history of computing. The cost savings and productivity enhancements available to organizations are huge. But these savings and benefits can't be reaped without careful planning, network assessment, change management and continuous monitoring. Read on for things that you shouldn't do with your network in preparation for a move to one of these cloud providers ...

April 17, 2017

One of the most ubiquitous words in the development and DevOps vocabularies is "Agile." It is that shining, valued, and sometimes elusive goal that all enterprises strive for. But how do you get there? How does your organization become truly Agile? With these questions in mind, DEVOPSdigest asked experts across the industry — including analysts, consultants and vendors — for their opinions on the best way for a development or DevOps team to become more Agile ...

April 12, 2017

Is composable infrastructure the right choice for your IT environment? The following are 5 key questions that can help you begin to explore the capabilities of composable infrastructure and its applicability within your own IT environment ...

April 11, 2017

What is composable infrastructure, and is it the right choice for your IT environment? That's the question on many CIOs' minds today as they work to position their organizations as "digitally driven," delivering better, deeper, faster user experiences and a more agile response to change in whatever vertical market you do business in today ...

April 10, 2017

As companies adopt new hardware and applications, their networks grow larger and become harder to manage. For network engineers and administrators, the continued emergence of integrated technology has forced them to reconfigure and manage networks in a more dynamic way ...

April 07, 2017

The complexity of data in motion is growing and risks undermining the success of the modern data-driven enterprise. A recent survey of data engineers and architects, conducted by StreamSets, sought to bring some perspective to the new reality in the enterprise, leading to some interesting insights about the enterprise data landscape ...

April 06, 2017

The biggest factor driving happiness is the quality of relationships IT professionals have with their coworkers, including users, peers, and managers, according to the 2017 IT Job Satisfaction report from Spiceworks ...