0 Shares 866 Views

A Story About The Battle of Logging

Networking and computing systems generate a deluge of logs in seconds. Analyzing these voluminous logs without human interference plays a crucial role in autonomous management. Just as other computing systems, CDNs typically generate millions of logs per second. Dynamic real-time decision systems, and error control mechanisms highly depend on the successful log collection process in order to improve end-user experience and content delivery quality.

In this blog, we will compare some leading log management tools (a.k.a. log shippers) for not only CDNs systems but also for all other computing systems.

Widely-Utilized Log Management Tools:

  • Logstash
  • Elastic Beats
  • Fluentd
  • Telegraph
  • Flume
  • Rsyslog

Logstash

When someone asks for a log shipper, beyond doubt, Logstash is the top tool on the list. This is because it has many plugins for inputs, outputs, codecs, and filters, which enhance flexibility and compatibility.

PROS

  • The event routing process is in the form of an “if-else” structure. This means that if you need a non-complex routing process, going forward with Logstash might leverage the setup.
  • The number of plugins is mostly enough for any computing system and has power on log parser plugins. 
  • Logstash has a centralized repository for all plugins, making it easy to access and find a proper plugin. 
  • Logstash is the native tool of the ELK stack.

CONS

  • Logstash is written in JRuby, which is a Java implementation of the Ruby programming language. Therefore, it is needed to have Java runtime and adds up the additional cost.
  •  Performance might be an issue compared to other lightweight log shippers. But having a lot of features compensates for high memory usage with flexibility.
  • The persistent queue in Logstash might be helpful sometimes due to requiring less effort. But it might constraint developers as well when there are many logs to handle.

Elastic Beats

To solve some performance issues in Logstash, Elastic team proposed many Elastic Beats to run on end nodes. The main issue with the Logstash was leading to a lot of memory usage and low computational performance in heavy use cases. Thanks to Elastic Beats, hierarchical topologies can be used jointly with Logstash.

PROS

  • Since they proposed for specific routines on end nodes, they are quite a lightweight tool.
  • If you have servers with small capacity and relatively fewer resources, installing proper and minimized Elastic Beats instead of fully qualified Logstash is straightforward.
  • If you have a heavy-duty streaming process that leads to back-pressure and requires recovery, then Beats are compatible with ELK stack.

CONS

  • The ELK Stack needs Logstash as an aggregator besides Elastic Beats at the end. Therefore, if you plan to use only Beats, it might not be suitable for your system. 

Fluentd

Fluentd, an open-source project of the Cloud Native Computing Foundation (CNCF), is a most similar project to Logstash. According to a presentation in OpenStack Summit 2015 [1], overall Fluentd and Logstash performances are quite comparable. Moreover, Fluentd has a complementary tool for end devices, naming Fluent Bit, just as Elastic Beats.

PROS

  • In event routing, “tags” are used instead of the “if-else” structure, which makes it easier in complex event routing scenarios.
  • Fluentd has over 500 plugins.
  • Since Fluentd is supported by CNCF, it is definitely compatible for projects where Kubernetes, OpenTracing, or Prometheus are used.  
  • Fluentd has both in-memory and on-disk options, without fixed-size memory as in Logstash. 
  • Docker containers have built-in options for Fluentd, therefore no need for an extra plugin.
  • Since Fluentd is based on CRuby, no need for Java runtime. 

CONS

  • Plugins are not in the centralized repo, which might require extra effort to find a perfect match. 
  • Some plugins do not support multi-threading.
  • Even if Fluentd has many log parsing options, Logstash is more flexible in filtering, parsing, aggregation, etc.

Telegraph

Telegraph is a part of TICK (Telegraph, InfluxDB, Chronograph, and Kapacitor) stack, but it also suitable for ELK stack. It is mostly compared with the Metric Beat of the ELK due to their functional similarities.

PROS

  • It is pretty lightweight and written by Go like Beats, which makes it easier to set up.
  • It has more than 100 plugins for input streaming, including popular tools like Kafka, Redis, RabbitMQ, MySQL, MongoDB, PostgreSQL, Prometheus, Apache, Ngnix, etc.
  • It can easily be integrated with more than 30 outputs to monitor.

CONS

  • It does not become fully adaptable with ELK stack; thus, it is harder to set up, unlike Metric Beats.
  • It isn’t easy to set up with Logstash ad Redis to send data from Telegraph. 

Flume

Unlike the other log shippers, Flume is originally designed for collecting, aggregating, and forwarding a massive amount of log data. It is a part of Hadoop and stores output data in The Hadoop Distributed File System (HDFS).

PROS

  • It provides quite better CPU utilization compared with Fluentd and Logstash.
  • It is really successful to handle dense data streaming with low latency processing.
  • Compatibility with HDFS is a plus.
  • Lost and duplicate data problems are simply solved with Kafka.

CONS

  • Unfortunately, JVM is a memory footprint.
  • There are no many plugins like in Logstash and Fluentd, which decrease flexibility.
  • There are three points to config: source, channel, and sink, which might be difficult in some use-cases. 

Rsyslog

Rsyslog is an open-source and default software tool for UNIX systems for logging and forwarding data. It is an extended version of Syslog and was realized in 2004.

PROS

  • It is quite lightweight, simple, and fast.
  • It is written in C, so setup and configuration in UNIX systems are straightforward.

CONS

  • It is not applicable where back-pressure exists. In that cases, Beats or Fluent bit are more suitable.
  • If there are many logging requirements, it is not functional enough.

At Medianova, we are always looking for optimized and autonomous ways to shape our systems intelligently. Because we care for data, and we know the value of data. Get in touch with us to learn more about how Medianova can build and manage an optimized and dedicated CDN for you.

 

 

You may be interested

The Ultimate CDN (Content Delivery Network) Guide
CDN
5785 views
CDN
5785 views

The Ultimate CDN (Content Delivery Network) Guide

Nadia Benslimane - August 3, 2020

The Ultimate CDN Guide- Everything About Content Delivery Network You probably know what CDN (Content Delivery Network) stands for. You may also be aware of its full definition,…

The Essential CDN Glossary
CDN
9938 views
CDN
9938 views

The Essential CDN Glossary

Nadia Benslimane - November 21, 2019

Why Have We Decided To Create a CDN Glossary? Whether you are new to the world of CDN, or have been involved in it for years, there…

The Benefits of CDN for the Gaming Industry
CDN
1235 views
CDN
1235 views

The Benefits of CDN for the Gaming Industry

alikayikci - December 3, 2021

The Benefits of CDN for the Gaming Industry The gaming industry has witnessed a phenomenal growth in the last few years. The global nine gaming market only…

HTTP Live Streaming
CDN
571 views
CDN
571 views

HTTP Live Streaming

medianova - November 1, 2021

Introduction To HTTP Live Streaming What Is HTTP Live Streaming (HLS)? HLS (HTTP live streaming) is one of the most widely used video streaming protocols. Although live…

All You Need to Know About Round-Trip Time
CDN
642 views
CDN
642 views

All You Need to Know About Round-Trip Time

Gizay Yalçın - October 20, 2021

What exactly is RTT?  Round-trip time (RTT) is a networking metric that measures in milliseconds the time it takes for a data packet to be sent and…

Most from this category

The Ultimate CDN (Content Delivery Network) Guide
CDN
5785 views
5785 views

The Ultimate CDN (Content Delivery Network) Guide

Nadia Benslimane - August 3, 2020
The Essential CDN Glossary
CDN
9938 views
9938 views

The Essential CDN Glossary

Nadia Benslimane - November 21, 2019
SD-WAN in CDNs
CDN
415 views
415 views

SD-WAN in CDNs

Gizay Yalçın - October 20, 2021
Health Check Backup Origin
CDN
395 views
395 views

Health Check Backup Origin

Nilgün Ünsal - October 17, 2021
DDoS Attack Trends in 2021
CDN
682 views
682 views

DDoS Attack Trends in 2021

Elif Ak - October 17, 2021
Web Application Firewall
Uncategorized
167 views
167 views

Web Application Firewall

Aykut Teker - September 30, 2021