Teams tasked with developing and delivering software are under pressure to balance the business imperative for speed with high customer expectations for quality. In the course of trying to achieve this balance, engineering organizations rely on a variety of tools, techniques and processes. The 2020 State of Software Quality report, a survey of more than 600 engineering professionals worldwide, provides a snapshot of the key challenges organizations encounter when it comes to delivering quality software at speed, as well as how they are approaching these hurdles. This blog introduces its key findings.
Speed Versus Quality
The overwhelming majority of engineering organizations (70%) say software quality trumps delivery speed, yet critical production issues persist. That still leaves almost a third of respondents who cite the speed of a software release is more important than ensuring software quality, even if it means critical errors could pop up in production.
Regardless of whether respondents care more about speed or about quality, across the board they are encountering challenges in delivering reliable software. Over half of survey respondents (53%) indicated that they encounter critical or customer-impacting issues in production at least one or more times a month. Compounding the impact of these critical production errors is the frequency at which they are first detected by customers, rather than by internal error detection mechanisms. Nearly a quarter of survey respondents said that over 40% of critical production issues are first reported by end users or customers rather than by internal tools or processes.
When asked about their top application quality challenges, nearly half of participants (48%) cited manual processes and a lack of sufficient tooling as a major hurdle. This aligns with some of the other top challenges identified by participants, including monitoring processes and tools that fail to catch errors before customers (29%), and a lack of visibility, data and metrics (25%). Despite only 30% of respondents indicating they prioritize speed, 45% of all respondents said they were under pressure to move fast, leaving them with insufficient time to ensure quality.
Top Techniques, Tools and Metrics
Despite survey respondents reporting that quality is more important than speed, their key areas of DevOps investment indicate otherwise. Continuous Integration (CI) and Continuous Delivery (CD), both hallmarks of accelerated software delivery pipelines, were among the top areas of DevOps investment, at 54% and 42%, respectively. Meanwhile, testing/software quality and observability and monitoring, key components of software quality success, fell lower on the list, at 34% and 21%, respectively. In support of this finding, the overwhelming majority of survey participants also indicated they are releasing software at a high velocity. Over half of respondents (59%) said they release new code or features anywhere from bi-weekly to multiple times a day, while only 19% follow a quarterly or less frequent cadence.
From CI/CD and daily release schedules to feature flags and alerting, today's engineering teams are using a broad ecosystem of cutting-edge technologies and practices. Some of the top tools include manual testing (70%), automated testing (65%) and log management (62%). Interestingly, the 70% of respondents that prefer quality use automated testing pre-production compared to only 54% of those that cite speed is paramount over quality. Automated testing is about making things faster, but it also enforces policies and reduces the chance for human error. When asked about which tools participants plan to adopt in 2020, more than half (58%) said DevOps (i.e. pipeline automation), and another 45% reported microservices and containers. Analysis tools, both static and dynamic, were also at the top of the list.
Conclusion
Regardless of priorities, very few organizations have been able to find a balance that allows them to deliver software at speed without encountering critical errors in production on a regular basis. At the root of this problem seems to be a misalignment between processes and resources with primary engineering and business goals. This is compounded by failure to invest in the right tooling and culture that enables efficient error detection and resolution, regardless of where you are in the pipeline. We have reached a turning point where we need to be much more serious about how we address these issues going forward.
