Beats (Filebeat) logs to Fluentd tag routing

Beats is a lightweight log shipper with a buffer and retransmission function (acknowledgment), and installing it on the server that generates logs makes it easy to analyze logs in Elasticsearch.

I usually use Fluentd (td-agent) as the main, but I felt troublesome installing td-agent on the log generation source server due to problems of dependency packages etc. If it is one of Beats Filebeat , it can be easily installed in a single package, and has a great merit.

However, even if I introduce Beats (Filebeat), I do not want to destroy existing log parsing configuration on Fluentd, so I verified the setting to incorporate Beats' log into Fluentd's tag routing.

Filebeat log forwarding settings

Assuming tag routing in Fluentd, set fields in filebeat as follows.

$ sudo vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths: ["/var/log/messages"]
#  symlinks: true
  fields:
    tagtype: linux
    tagapps: syslog
    taghost: centos7-m1
output.logstash:
  hosts: ["localhost:5044"]
#logging.level: debug
logging.metrics.enabled: false

Usage of fields is defined as optional for Filter and so it is considered suitable for this case.

www.elastic.co

Item names in fields can be freely defined. Since I am a tag use tagxxxx, clarify and prevent mixing with others.

The destination is Fluentd in the same server for the time being.

Optional Settings

symlinks: true Enable when symlinks
logging.level: debug Enable for debugging (default: info)
logging.metrics.enabled: false Prevent log of No non-zero metrics

Fluentd settings for Beats log

Use fluent-plugin-beats to receive Beats (Filebeat) logs on Fluentd (td-agent).

qiita.com

Also, install the plug-in as follows for log parsing / saving.

sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-beats --no-document
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-forest --no-document
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch --no-document
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer --no-document

Collect Filebeat's log with the following settings. (include from td-agent.conf)

$ vi /etc/td-agent/conf/td_beats.conf
### Log Collect from Beats
<source>
  @type beats
  tag "beats.collect"
</source>

### Reform for tag routing
<match beats.collect>
  @type record_reformer
  tag ${fields['tagtype']}.${fields['tagapps']}.${fields['taghost']}
</match>

### For Parse
<filter linux.syslog.*>
  @type parser
  format syslog
  key_name message
</filter>

### General match
<match *.*.*>
  type forest
  subtype copy
  <template>
    <store>
      @type elasticsearch
      host localhost
      port 9200
      logstash_format true
      logstash_prefix ${tag_parts[0]}.${tag_parts[1]}
      type_name ${tag_parts[0]}
      flush_interval 20
    </store>
    <store>
      @type file
      path /var/log/td-agent/${tag_parts[0]}/${tag_parts[1]}_${tag_parts[2]}.log
    </store>
  </template>
</match>

Setting explanations

Log Collect from Beats

Receive logs with Beats plugin. Port 5044 is used by default.

Reform for tag routing

Rewrite tag with contents of fields . Processing as a log of Beats ends here, and it is routed again within Fluentd with a new tag. (due to match directive)

For Parse

Parses the received log. Since it is a log of /var/log/messages here, simply expand it in syslog format. It is a point to reduce the set amount by generalizing and commonizing with tag.

General match

Transmit and save the log of which parsing is completed to Elasticsearch, file. Although it was set with the same file this time, it is good to make it separate file and apply at the end of include. Optimization such as file buffer / chunk setting etc is not implemented yet.

Log reception result

You can receive, parse and save logs as expected.

$ tail /var/log/td-agent/linux/syslog_centos7-m1.log.20170709.b553d9a17802a264d
2017-07-09T11:57:37+09:00   linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"systemd","message":"Stopping LSB: data collector for Treasure Data..."}
2017-07-09T11:57:38+09:00   linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"td-agent","message":"Stopping td-agent: td-agent[  OK  ]"}
2017-07-09T11:57:38+09:00   linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"systemd","message":"Starting LSB: data collector for Treasure Data..."}
2017-07-09T11:57:38+09:00   linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"td-agent","message":"Starting td-agent: [  OK  ]#015td-agent[  OK  ]"}
2017-07-09T11:57:38+09:00   linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"systemd","message":"Started LSB: data collector for Treasure Data."}

Conclusion - Beats (Filebeat) logs to Fluentd tag routing

By using the item of fileds of Filebeat, we set a tag to use in Fluentd so that tag routing can be done like normal Fluentd log. As a result, when sending logs with Filebeat, you can also aggregate, parse, save, or elasticsearch by conventional Fluentd.