Introduction to Fluent Bit
Fluent Bit is a specialized event capture and distribution tool that handles log events, metrics, and traces. It is a lightweight and efficient data collector and processor, making it ideal for environments where resource consumption is a critical consideration.
Fluent Bit is crucial in modern logging and monitoring architectures, particularly within cloud-native environments. It acts as an intermediary between log sources and log storage or analysis systems. Fluent Bit collects logs from various sources, processes and filters them as needed, and then forwards them to destinations such as Elasticsearch, Kafka, or cloud storage services.
Key features of Fluent Bit include
- Lightweight and High Performance: Fluent Bit is designed to be resource-efficient, with a small memory footprint and low CPU usage, making it suitable for edge computing and IoT devices.
- Flexible Configuration: It offers a flexible configuration system that allows users to define inputs, filters, and outputs to tailor log processing to specific needs.
- Extensive Plugin Ecosystem: Fluent Bit supports a wide range of plugins for input, filtering, and output, enabling integration with various systems and formats.
- Scalability: Fluent Bit can be deployed in large-scale environments, from single-node setups to distributed systems and Kubernetes clusters.
Setting Up Fluent Bit
To install Fluent Bit on a Linux machine, you need to run the following command. This command automatically configures the YUM repository and installs Fluent Bit:
[root@fluentbit ~]# curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3917 100 3917 0 0 7446 0 --:--:-- --:--:-- --:--:-- 7446
================================
Fluent Bit Installation Script
================================
This script requires superuser access to install packages.
You will be prompted for your password by sudo.
[fluent-bit]
name = Fluent Bit
# Legacy server style
baseurl = https://packages.fluentbit.io/centos/$releasever/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
Fluent Bit 440 B/s | 866 B 00:01
Rocky Linux 9 - BaseOS 4.1 kB/s | 4.1 kB 00:00
Rocky Linux 9 - AppStream 4.7 kB/s | 4.5 kB 00:00
Rocky Linux 9 - Extras 3.6 kB/s | 2.9 kB 00:00
Package fluent-bit-3.0.5-1.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Installation completed. Happy Logging!
[root@fluentbit ~]#Once Fluent Bit is installed, you can verify the installation with the following command:
[root@fluentbit ~]# fluent-bit --version
Fluent Bit v3.0.5
Git commit:
[root@fluentbit ~]#Configuring Fluent Bit
Fluent Bit’s configuration is composed of three main sections: Inputs, Filters, and Outputs. Each section serves a different purpose in the log processing pipeline, allowing you to collect, process, and forward logs efficiently.
Configuration Structure
- Inputs: Define where Fluent Bit collects logs from. Common input plugins include
tailfor reading log files andsystemdfor reading systemd journal logs. - Filters: Allow you to modify, enrich, or exclude log records before they are sent to the output.
- Outputs: Specify where the processed logs should be sent, such as Elasticsearch, Kafka, or cloud storage services.
Examples of Basic Input Plugins and Their Configurations
1. Tail Input Plugin
The tail input plugin reads log files and is commonly used to collect logs from applications or system logs. Below is an example configuration:
[INPUT]
Name tail
Path /var/log/syslog
Tag syslog
Refresh_Interval 5
Rotate_Wait 30
DB /var/log/flb_kv.db
Mem_Buf_Limit 5MBName: Specifies the input plugin name (tailin this case).Path: The path to the log file(s) to be read.Tag: A tag to identify the logs collected by this input.Refresh_Interval: How often (in seconds) Fluent Bit checks for new data.Rotate_Wait: Time to wait (in seconds) to read new files after rotation.DB: Path to a database file to track file offsets.Mem_Buf_Limit: Maximum buffer size before data is flushed to output.
2. Systemd Input Plugin
The systemd input plugin reads logs from the systemd journal. This is useful for collecting system logs on systems that use systemd. Here’s an example configuration:
[INPUT]
Name systemd
Tag systemd
Systemd_Filter _SYSTEMD_UNIT=nginx.service
Read_From_Tail trueName: Specifies the input plugin name (systemdin this case).Tag: A tag to identify the logs collected by this input.Systemd_Filter: Filters logs by systemd unit (e.g.,nginx.service).Read_From_Tail: Starts reading from the end of the journal.
Filtering Logs
Filters are a critical component of Fluent Bit’s log processing pipeline. They allow you to modify and exclude log records based on specific criteria before forwarding them to the desired output. Filters help ensure that only relevant and properly formatted logs reach their destination, which can improve performance, reduce storage costs, and facilitate more efficient log analysis.
Importance of Filters
- Data Enrichment: Filters can add additional information to log records, such as metadata or tags, to provide more context.
- Data Reduction: By excluding unnecessary logs, filters can help reduce the volume of data sent to storage or analysis systems, saving bandwidth and storage space.
- Data Transformation: Filters can modify log records to ensure they meet the format requirements of the destination system, improving compatibility and readability.
- Selective Processing: Filters allow you to process only specific types of log records, which can be crucial in complex environments where different logs require different handling.
Basic Examples of Using Filters
1. Parser Filter
The parser filter is used to parse and restructure log records. For instance, you can use it to parse JSON logs or to apply a custom parser to a log field.
Example configuration:
[FILTER]
Name parser
Match *
Key_Name log
Parser jsonName: Specifies the filter plugin name (parser).Match: Defines which logs this filter should apply to (using a tag or wildcard).Key_Name: The key of the log record to apply the parser to.Parser: The name of the parser to use (e.g.,json).
2. Grep Filter
The grep filter includes or excludes log records based on matching criteria. This is useful for filtering logs that contain specific keywords or patterns.
Example configuration to include only logs containing the keyword “ERROR”:
[FILTER]
Name grep
Match *
Regex log ERRORName: Specifies the filter plugin name (grep).Match: Defines which logs this filter should apply to (using a tag or wildcard).Regex: The regular expression to match log records that should be included.
Example configuration to exclude logs containing the keyword “DEBUG”:
[FILTER]
Name grep
Match *
Exclude log DEBUGExclude: The regular expression to match log records that should be excluded.
Sending Logs to a Destination
Fluent Bit uses output plugins to send processed log data to various destinations, such as log storage systems, databases, or analysis tools. Configuring these output plugins ensures that your logs are forwarded to the appropriate location where they can be stored, visualized, or further analyzed.
Configuring Output Plugins
Fluent Bit offers multiple output plugins for various destinations such as Elasticsearch, S3, and Kafka. Each of these plugins requires unique configurations to connect and communicate with the target system.
Example Configuration for Sending Logs to Elasticsearch
Elasticsearch is a popular destination for storing and analyzing log data due to its powerful search and analytics capabilities. Below is an example configuration for sending logs to Elasticsearch:
[INPUT]
Name tail
Path /var/log/syslog
Tag syslog
Refresh_Interval 5
[FILTER]
Name parser
Match syslog
Key_Name log
Parser json
[FILTER]
Name grep
Match syslog
Regex log ERROR
[FILTER]
Name record_modifier
Match syslog
Record hostname ${HOSTNAME}
[OUTPUT]
Name es
Match syslog
Host elasticsearch.local
Port 9200
Index fluentbit
Type _doc
HTTP_User fluent_user
HTTP_Passwd fluent_password
tls On
tls.verify Off- Name: Specifies the output plugin name (
esfor Elasticsearch). - Match: Defines which logs should be sent to this output (using a tag or wildcard).
- Host: The hostname or IP address of the Elasticsearch server.
- Port: The port on which Elasticsearch is listening (default is 9200).
- Index: The Elasticsearch index where logs should be stored.
- Type: The document type (deprecated in newer Elasticsearch versions but still required in some setups).
- HTTP_User: Username for HTTP basic authentication (if required).
- HTTP_Passwd: Password for HTTP basic authentication (if required).
- tls: Enables TLS for secure communication.
- tls.verify: Disables TLS verification (useful for testing but not recommended for production).
Setting up Fluent Bit involves configuring input plugins to collect logs, filters to process and enrich them, and output plugins to forward the logs to desired destinations. This modular approach allows you to build a highly efficient and customizable logging pipeline.
By following these guidelines, you can effectively use Fluent Bit to manage your logging needs, ensuring that your logs are collected, processed, and stored efficiently. Whether you are working with simple log files or complex distributed systems, Fluent Bit provides the flexibility and performance required for modern logging solutions. Stay tuned for more advanced configurations and use cases in our next blog post.
Feel free to reach out with any questions or share your experiences with Fluent Bit in the comments below. Happy logging!
