The No-BS Guide to Logging - Part 2
A vendor-neutral checklist to help you get your log strategy straight
December 08, 2015

Sven Dummer
Loggly

Share this


Start with The No-BS Guide to Logging - Part 1

Coming off of the last post outlining the necessity for log management, the process of choosing logging software can seem daunting. The following are major elements of a good log strategy and can also serve as checklist items when you shop for a log management solution:

Collect, Aggregate, Retain

It's crucial to think about your data retention needs and the costs associated with storing them. How long do you need to keep the logs? Do you need them just for troubleshooting, or also for business intelligence type of analysis? Are there regulatory or audit requirements that require you to keep the logs for a certain period of time?

Your daily log volume might already be large, but keep in mind that it doesn't take much to multiply the volume temporarily. For example, a component failure and the resulting log messages in a complex system could easily quadruple the amount of log messages. An external event could have the same effect: if you run an online store, Black Friday might balloon your sales as well as your log volumes. If your log aggregation doesn't scale, you could lose your main troubleshooting foundation when you need it most.

Handle Log Diversity

Log files come in a variety of formats, some following standards and conventions, others completely custom. Your log solution should be able to parse and present the data in a comprehensive form in near real-time, and it should allow to define custom parsing rules. A desirable feature is the ability to add metadata.

Reveal What Matters

Just having a search tool is not enough. To make sense of your log data and the correlation between different data points, you need real-time indexing and parsing, grouping, along with powerful analytics, customizable dashboards, and data visualization. Your log analytics solution should provide a treasure map to the contents of your logs, not just a metal detector that you must use to scan indiscriminately.

Detect Anomalies

Given the volume and complexity of log data, you can't rely on searching for problems. Things you never anticipated happening are typically the type of problems that hurt the most. A good log analytics solution should be able to learn what is “normal” in your log data, and automatically identify and highlight any deviations from norms.

Make Your Own Apps Log

If you write your own code, your log management solution must be able to parse and analyze it. Consider using a well-established data format like JSON (our recommendation) or XML. Whatever you choose, make sure it's plain text format (not binary), that it is human-readable, and easy to parse. Your log solution should be able to easily receive the logs from your application and allow you to set up custom parsing rules if needed.

Be Alert(ed)

Just like every good monitoring application, every good log management solution should allow to send you and your teams alerts based on defined events, like error messages. It should be possible to send these alerts through common third party collaboration tools.

Don't Break the Bank

Cloud technologies made running distributed systems and elastic compute farms affordable for SMBs. The bill for the troubleshooting tools should be affordable, too. There are fully cloud-based SaaS solutions out there, as well as on-premise products and hybrids, which typically come at higher costs (including those for hardware and datacenter footprint).

Key criteria to decide if SaaS or on-premise solutions are right for you are the sensitivity and volume of your data. Security or privacy concerns or regulatory requirements may keep you from transferring data across public networks. Similarly, the sheer data volume could make this impossible or too expensive.

Sven Dummer is Senior Director of Product Marketing at Loggly.

Share this

The Latest

March 27, 2024

Nearly all (99%) globa IT decision makers, regardless of region or industry, recognize generative AI's (GenAI) transformative potential to influence change within their organizations, according to The Elastic Generative AI Report ...

March 27, 2024

Agent-based approaches to real user monitoring (RUM) simply do not work. If you are pitched to install an "agent" in your mobile or web environments, you should run for the hills ...

March 26, 2024

The world is now all about end-users. This paradigm of focusing on the end-user was simply not true a few years ago, as backend metrics generally revolved around uptime, SLAs, latency, and the like. DevOps teams always pitched and presented the metrics they thought were the most correlated to the end-user experience. But let's be blunt: Unless there was an egregious fire, the correlated metrics were super loose or entirely false ...

March 25, 2024

This year, New Relic published the State of Observability for Financial Services and Insurance Report to share insights derived from the 2023 Observability Forecast on the adoption and business value of observability across the financial services industry (FSI) and insurance sectors. Here are seven key takeaways from the report ...

March 22, 2024

In MEAN TIME TO INSIGHT Episode 4 - Part 2, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses artificial intelligence and AIOps ...

March 21, 2024

In the course of EMA research over the last twelve years, the message for IT organizations looking to pursue a forward path in AIOps adoption is overall a strongly positive one. The benefits achieved are growing in diversity and value ...

March 20, 2024

Today, as enterprises transcend into a new era of work, surpassing the revolution, they must shift their focus and strategies to thrive in this environment. Here are five key areas that organizations should prioritize to strengthen their foundation and steer themselves through the ever-changing digital world ...

March 19, 2024

If there's one thing we should tame in today's data-driven marketing landscape, this would be data debt, a silent menace threatening to undermine all the trust you've put in the data-driven decisions that guide your strategies. This blog aims to explore the true costs of data debt in marketing operations, offering four actionable strategies to mitigate them through enhanced marketing observability ...

March 18, 2024

Gartner has highlighted the top trends that will impact technology providers in 2024: Generative AI (GenAI) is dominating the technical and product agenda of nearly every tech provider ...

March 15, 2024

In MEAN TIME TO INSIGHT Episode 4 - Part 1, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses artificial intelligence and network management ...