This article documents the use of Filebeat, Kibana, and Elasticsearch to build a system for collecting and analyzing Nginx logs; Filebeat is responsible for delivering Nginx log data as a data source to Elasticsearch.
As an introduction, we will first explain the relationship between the software: Elasticsearch is a distributed full-text search and data analysis engine based on RESTful style; Kibana is a web system for visualizing Elasticsearch data; and Filebeat is responsible for delivering Nginx log data to Elasticsearch as a data source.
The server environment for this article: Ubuntu 20.04, with only one server node configured.
Environment preparation: JAVA
You need to install Java Runtime Environment (JRE) and Java Development Kit (JDK).
Reference ：How To Install Java with Apt on Ubuntu 20.04
The configuration file for Elasticsearch is located at:
Verify that the Elasticsearch service is up
Modifying Elasticsearch memory size
Elasticsearch 7.x is configured differently than before and requires a new file with the .options suffix in the /etc/elasticsearch/jvm.options.d directory. Modify the memory usage configuration as follows.
Reference: elasticsearch jvm-options
The source was added in the previous step, so you can install it directly. Because of version dependencies, install Elasticsearch first before installing Kibana.
The Elasticsearch configuration file is located at
Important configuration items.
Filebeat is a lightweight data collection and processing tool with the advantage of low resource consumption.
Filebeat is extended by Module to support files with different data formats. module-related commands.
Take Nginx for example, after starting Nginx’s Module, you can see that the file nginx.yml..disabled in the /etc/filebeat/modules.d directory has become nginx.yml.
Once the configuration is complete, use
filebeat setup to write the configuration and then start filebeat.
Removing redundant fields
Filebeat’s default output fields may be useless in some cases, causing redundancy. You can remove the redundant fields.
Add fields that need to be removed as needed.
Start the service
Start three components in sequence.
Nginx site configuration kibana access
Using the secondary directory of the Nginx site as an access path, for example.
Access to kibana is available through
Configure password for kibana
A password file based on HTTP Basic Authentication authentication will be generated. Add the following configuration to the Nginx site.
When the corresponding page is visited, it will prompt for password verification.
After opening the kibana page, click “Add Data” and select “Nginx Logs” (there will be instructions for the Filebeat module to parse Nginx logs), and follow the instructions to finish.