Beats is a lightweight log shipper with a buffer and retransmission function (acknowledgment), and installing it on the server that generates logs makes it easy to analyze logs in Elasticsearch.
I usually use Fluentd (td-agent) as the main, but I felt troublesome installing td-agent on the log generation source server due to problems of dependency packages etc. If it is one of Beats Filebeat , it can be easily installed in a single package, and has a great merit.
However, even if I introduce Beats (Filebeat), I do not want to destroy existing log parsing configuration on Fluentd, so I verified the setting to incorporate Beats' log into Fluentd's tag routing.
- Filebeat log forwarding settings
- Fluentd settings for Beats log
- Log reception result
- Conclusion - Beats (Filebeat) logs to Fluentd tag routing
Filebeat log forwarding settings
Assuming tag routing in Fluentd, set fields in filebeat as follows.
$ sudo vi /etc/filebeat/filebeat.yml filebeat.prospectors: - input_type: log paths: ["/var/log/messages"] # symlinks: true fields: tagtype: linux tagapps: syslog taghost: centos7-m1 output.logstash: hosts: ["localhost:5044"] #logging.level: debug logging.metrics.enabled: false
Usage of fields is defined as optional for Filter and so it is considered suitable for this case.
Item names in fields can be freely defined. Since I am a tag use tagxxxx, clarify and prevent mixing with others.
The destination is Fluentd in the same server for the time being.
Optional Settings
symlinks: true
Enable when symlinks
logging.level: debug
Enable for debugging (default: info)
logging.metrics.enabled: false
Prevent log of No non-zero metrics
Fluentd settings for Beats log
Use fluent-plugin-beats to receive Beats (Filebeat) logs on Fluentd (td-agent).
Also, install the plug-in as follows for log parsing / saving.
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-beats --no-document sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-forest --no-document sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch --no-document sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer --no-document
Collect Filebeat's log with the following settings. (include from td-agent.conf)
$ vi /etc/td-agent/conf/td_beats.conf ### Log Collect from Beats <source> @type beats tag "beats.collect" </source> ### Reform for tag routing <match beats.collect> @type record_reformer tag ${fields['tagtype']}.${fields['tagapps']}.${fields['taghost']} </match> ### For Parse <filter linux.syslog.*> @type parser format syslog key_name message </filter> ### General match <match *.*.*> type forest subtype copy <template> <store> @type elasticsearch host localhost port 9200 logstash_format true logstash_prefix ${tag_parts[0]}.${tag_parts[1]} type_name ${tag_parts[0]} flush_interval 20 </store> <store> @type file path /var/log/td-agent/${tag_parts[0]}/${tag_parts[1]}_${tag_parts[2]}.log </store> </template> </match>
Setting explanations
Log Collect from Beats
Receive logs with Beats plugin. Port 5044 is used by default.
Reform for tag routing
Rewrite tag with contents of fields . Processing as a log of Beats ends here, and it is routed again within Fluentd with a new tag. (due to match directive)
For Parse
Parses the received log. Since it is a log of /var/log/messages
here, simply expand it in syslog format. It is a point to reduce the set amount by generalizing and commonizing with tag.
General match
Transmit and save the log of which parsing is completed to Elasticsearch, file. Although it was set with the same file this time, it is good to make it separate file and apply at the end of include. Optimization such as file buffer / chunk setting etc is not implemented yet.
Log reception result
You can receive, parse and save logs as expected.
$ tail /var/log/td-agent/linux/syslog_centos7-m1.log.20170709.b553d9a17802a264d 2017-07-09T11:57:37+09:00 linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"systemd","message":"Stopping LSB: data collector for Treasure Data..."} 2017-07-09T11:57:38+09:00 linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"td-agent","message":"Stopping td-agent: td-agent[ OK ]"} 2017-07-09T11:57:38+09:00 linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"systemd","message":"Starting LSB: data collector for Treasure Data..."} 2017-07-09T11:57:38+09:00 linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"td-agent","message":"Starting td-agent: [ OK ]#015td-agent[ OK ]"} 2017-07-09T11:57:38+09:00 linux.syslog.centos7-m1 {"host":"CentOS7-M1","ident":"systemd","message":"Started LSB: data collector for Treasure Data."}
Conclusion - Beats (Filebeat) logs to Fluentd tag routing
By using the item of fileds of Filebeat, we set a tag to use in Fluentd so that tag routing can be done like normal Fluentd log. As a result, when sending logs with Filebeat, you can also aggregate, parse, save, or elasticsearch by conventional Fluentd.