Suppress Non-Zero Metrics log with Filebeat

As described in this article, Beats (Filebeat) is sending Fluentd in a simple log.

en-designetwork.hatenablog.com

I noticed that the following logs occurred frequently among them. It seems to be a mechanism of Beats' s Metrics monitoring, but in stable operation, we want to detect only abnormal logs by reducing logs that occur all the time.

$ sudo tail -f /var/log/filebeat/filebeat
2017-07-09T13:36:09+09:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=96 libbeat.logstash.publish.write_bytes=2224 libbeat.logstash.published_and_acked_events=16 libbeat.publisher.published_events=16 publish.events=17 registrar.states.update=17 registrar.writes=1
2017-07-09T13:36:39+09:00 INFO No non-zero metrics in the last 30s
2017-07-09T13:37:09+09:00 INFO No non-zero metrics in the last 30s
2017-07-09T13:37:39+09:00 INFO No non-zero metrics in the last 30s

Add settings to prevent this log from occurring.

Setting to suppress logs of Non-Zero Metrics

The setting to suppress the above log is as follows.

$ sudo vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths: ["/var/log/messages"]
#  symlinks: true
  fields:
    tagtype: linux
    tagapps: syslog
    taghost: centos7-m1
output.logstash:
  hosts: ["localhost:5044"]
#logging.level: debug
logging.metrics.enabled: false  //Add this setting

Logging is suppressed by invalidating logging.metrics.enabled . When changing the setting and restarting Filebeat, the log does not come out.

Setting to output finely

The logging interval can be changed.

logging.metrics.enabled: true
logging.metrics.period: 10s  //Default: 30s

By doing this, it is possible to output logs every ten seconds. Effective when you want to check log collection status in detail.

2017-07-09T14:00:12+09:00 INFO Harvester started for file: /var/log/messages
2017-07-09T14:00:22+09:00 INFO Non-zero metrics in the last 10s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=365 libbeat.logstash.published_and_acked_events=3 libbeat.publisher.published_events=3 publish.events=6 registrar.states.current=2 registrar.states.update=6 registrar.writes=1
2017-07-09T14:00:32+09:00 INFO No non-zero metrics in the last 10s
2017-07-09T14:00:42+09:00 INFO No non-zero metrics in the last 10s

Reference information

Information on the log is stated officially.

www.elastic.co

Also, since all settings including the default are described in /etc/filebeat/filebeat.full.yml , it is good to go through the first time.

$ sudo cat /etc/filebeat/filebeat.full.yml
######################## Filebeat Configuration ############################

# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see filebeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
...

Conclusion - Suppress Non-Zero Metrics log with Filebeat

Changed log output of Non-Zero Metrics by changing log output setting of Filebeat. Note that it is possible to increase the log output frequency as necessary.