designetwork(EN)

IT technical memo of networking

Suppress Non-Zero Metrics log with Filebeat

As described in this article, Beats (Filebeat) is sending Fluentd in a simple log.

en-designetwork.hatenablog.com

I noticed that the following logs occurred frequently among them. It seems to be a mechanism of Beats' s Metrics monitoring, but in stable operation, we want to detect only abnormal logs by reducing logs that occur all the time.

$ sudo tail -f /var/log/filebeat/filebeat
2017-07-09T13:36:09+09:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=96 libbeat.logstash.publish.write_bytes=2224 libbeat.logstash.published_and_acked_events=16 libbeat.publisher.published_events=16 publish.events=17 registrar.states.update=17 registrar.writes=1
2017-07-09T13:36:39+09:00 INFO No non-zero metrics in the last 30s
2017-07-09T13:37:09+09:00 INFO No non-zero metrics in the last 30s
2017-07-09T13:37:39+09:00 INFO No non-zero metrics in the last 30s

Add settings to prevent this log from occurring.

Setting to suppress logs of Non-Zero Metrics

The setting to suppress the above log is as follows.

$ sudo vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths: ["/var/log/messages"]
#  symlinks: true
  fields:
    tagtype: linux
    tagapps: syslog
    taghost: centos7-m1
output.logstash:
  hosts: ["localhost:5044"]
#logging.level: debug
logging.metrics.enabled: false  //Add this setting

Logging is suppressed by invalidating logging.metrics.enabled . When changing the setting and restarting Filebeat, the log does not come out.

Setting to output finely

The logging interval can be changed.

logging.metrics.enabled: true
logging.metrics.period: 10s  //Default: 30s

By doing this, it is possible to output logs every ten seconds. Effective when you want to check log collection status in detail.

2017-07-09T14:00:12+09:00 INFO Harvester started for file: /var/log/messages
2017-07-09T14:00:22+09:00 INFO Non-zero metrics in the last 10s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=365 libbeat.logstash.published_and_acked_events=3 libbeat.publisher.published_events=3 publish.events=6 registrar.states.current=2 registrar.states.update=6 registrar.writes=1
2017-07-09T14:00:32+09:00 INFO No non-zero metrics in the last 10s
2017-07-09T14:00:42+09:00 INFO No non-zero metrics in the last 10s

Reference information

Information on the log is stated officially.

www.elastic.co

Also, since all settings including the default are described in /etc/filebeat/filebeat.full.yml , it is good to go through the first time.

$ sudo cat /etc/filebeat/filebeat.full.yml
######################## Filebeat Configuration ############################

# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see filebeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
...

Conclusion - Suppress Non-Zero Metrics log with Filebeat

Changed log output of Non-Zero Metrics by changing log output setting of Filebeat. Note that it is possible to increase the log output frequency as necessary.


This Blog is English Version of my JP's.

Sorry if my English sentences are incorrect.

designetwork