designetwork(EN)

IT technical memo of networking

Visualize NetFlow with ElastiFlow (Elasticsearch + Logstash + Kibana)

f:id:daichi703n:20190316180848p:plain

Earlier, I reviewed the open source (OSS) NetFlow collector, as summarized in this article.

designetwork.daichi703n.com

When I investigated again, I found ElastiFlow , a NetFlow collector and visualizer based on Elasticsearch + Logstash + Kibana (ELK stack).

github.com

The software is not completely open source (OSS), but is free under the author's Robert Cowart Public License.

Required skills

Please note that this article requires the following knowledge in addition to basic knowledge of network devices (Router, Switch). (To utilize OSS on Linux, not on appliance)

For details, please refer to other sites. Recently, even network engineers are not only concerned with conventional CLI operation of Cisco, Juniper, etc., F5 BIG-IP, GUI operation of various appliances, but also IaaS, Linux, programming skills ...

  • Basic operations of VM
  • Basic Linux operations
  • Basic Docker operations
  • Basic knowledge of Elasticsearch, Logstash, Kibana

NetFlow exporter

Use NetFlow v5 with VyOS as the NetFlow exporter. When sent with NetFlow v9 it was not processed by Logstash...

Regarding NetFlow setting of VyOS, I referred to this article.

qiita.com

Install ElastiFlow

ElastiFlow can be built by installing a plugin template for the ELK stack. There is an official description of system requirements and installation.

ElastiFlow Installation

Elasticsearch, Logstash, and Kibana can each be used with an open source license (not the X-Pack Elastic license). The procedure for building the ELK stack is omitted because many articles have been published.

In this case, ElastiFlow is built with Docker so that it can be easily used for testing.

Environment

The host that runs the ElastiFlow Docker container is as follows.

  • CentOS: 7.3.1611 (vSphere6.5 VM)
  • RAM: 8GB
  • CPU: 2vCPU (Intel(R) Xeon(R) CPU E31220 @ 3.10GHz)

Docker uses the latest version of the community edition. * In old v1.13.x, an error occurs at Build. (Correspondence is possible by modifying Dockerfile) Omitting because there are many articles about Docker environment construction.

$ docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:33:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 06:02:24 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ docker-compose version
docker-compose version 1.14.0, build c7bdf9e
docker-py version: 2.4.2
CPython version: 2.7.5
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

Launch ElastiFlow container

Clone, build, and launch the ElastiFlow repository.

$ git clone https://github.com/robcowart/elastiflow
$ cd ./elastiflow
$ ./docker_build.sh
Sending build context to Docker daemon  821.8MB
Step 1/11 : FROM docker.elastic.co/logstash/logstash-oss:6.1.3
 ---> bfbbe5b25fe6
Step 2/11 : ARG BUILD_DATE
 ---> Using cache
 ---> 31982c13d210
Step 3/11 : LABEL org.opencontainers.image.created="$BUILD_DATE"       org.opencontainers.image.authors="rob@koiossian.com"       org.opencontainers.image.url="https://github.com/robcowart/elastiflow-docker/elastiflow-logstash"       org.opencontainers.image.documentation="https://github.com/robcowart/elastiflow-docker/elastiflow-logstash/README.md"       org.opencontainers.image.source="https://github.com/robcowart/elastiflow"       org.opencontainers.image.version="v3.4.1_6.1.3"       org.opencontainers.image.vendor="Robert Cowart"       org.opencontainers.image.title="ElastiFlow™ - Logstash"       org.opencontainers.image.description=""
 ---> Running in b035b0d67d0c
Removing intermediate container b035b0d67d0c
 ---> 8dbda1604495
Step 4/11 : ENV ELASTIFLOW_ES_HOST="http://127.0.0.1:9200"
 ---> Running in 51935ee56198
Removing intermediate container 51935ee56198
 ---> 11da570d161a
Step 5/11 : RUN $HOME/bin/logstash-plugin install logstash-codec-sflow     && $HOME/bin/logstash-plugin update logstash-codec-netflow     && $HOME/bin/logstash-plugin update logstash-input-udp     && $HOME/bin/logstash-plugin update logstash-input-tcp     && $HOME/bin/logstash-plugin update logstash-filter-dns     && $HOME/bin/logstash-plugin update logstash-filter-geoip     && $HOME/bin/logstash-plugin update logstash-filter-translate
 ---> Running in 1c080aa843d5
Validating logstash-codec-sflow
Installing logstash-codec-sflow
Installation successful
Updating logstash-codec-netflow
Updated logstash-codec-netflow 3.10.0 to 4.2.1
Updating logstash-input-udp
Updated logstash-input-udp 3.2.1 to 3.3.4
Updating logstash-input-tcp
Updated logstash-codec-json_lines 3.0.5 to 3.0.6
Updated logstash-codec-multiline 3.0.9 to 3.0.10
Updated logstash-input-tcp 5.0.3 to 6.0.2
Updating logstash-filter-dns
Updated logstash-filter-dns 3.0.7 to 3.0.12
Updating logstash-filter-geoip
Updated logstash-filter-geoip 5.0.3 to 6.0.0
Updating logstash-filter-translate
Updated logstash-filter-translate 3.0.4 to 3.2.3
Removing intermediate container 1c080aa843d5
 ---> a021db633283
Step 6/11 : WORKDIR /usr/share/logstash/config
 ---> Running in 38e6b342d99e
Removing intermediate container 38e6b342d99e
 ---> f17a3a0602b9
Step 7/11 : COPY --chown=logstash:logstash ./docker_assets/logstash.yml ./
 ---> fe94fcf606a9
Step 8/11 : COPY --chown=logstash:logstash ./docker_assets/pipelines.yml ./
 ---> 1b607617f663
Step 9/11 : WORKDIR /etc/logstash/elastiflow
 ---> Running in 46dae18bc5f7
Removing intermediate container 46dae18bc5f7
 ---> 148f2f7f7ba1
Step 10/11 : COPY --chown=logstash:root ./logstash/elastiflow/ ./
 ---> 47f4642be2f5
Step 11/11 : ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
 ---> Running in 5f221fd60c53
Removing intermediate container 5f221fd60c53
 ---> f3d7a9193483
Successfully built f3d7a9193483
Successfully tagged robcowart/elastiflow-logstash-oss:3.4.1_6.1.3

Modify the parameters of docker-compose.yml as necessary. In my environment, I switched to memory reduction and use data volume containers.

$ git diff
diff --git a/docker-compose.yml b/docker-compose.yml
index af66265..434ca39 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -28,12 +28,12 @@ services:
         hard: -1
     network_mode: host
     volumes:
-      - /var/lib/elastiflow_es:/usr/share/elasticsearch/data
+      - elastiflow-elasticsearch-oss-data:/usr/share/elasticsearch/data
     environment:
       # JVM Heap size
       #   - this should be at least 2GB for simple testing, receiving only a few flo
       #   - for production environments upto 31GB is recommended.
-      ES_JAVA_OPTS: '-Xms2g -Xmx2g'
+      ES_JAVA_OPTS: '-Xms1g -Xmx1g'

       cluster.name: elastiflow

@@ -70,7 +70,7 @@ services:
     network_mode: host
     environment:
       # JVM Heap size - this MUST be at least 3GB (4GB preferred)
-      LS_JAVA_OPTS: '-Xms3g -Xmx3g'
+      LS_JAVA_OPTS: '-Xms2g -Xmx2g'

       # ElastiFlow global configuration
       ELASTIFLOW_DEFAULT_APPID_SRCTYPE: "__UNKNOWN"
@@ -82,3 +82,8 @@ services:
       ELASTIFLOW_NETFLOW_IPV4_PORT: 2055
       ELASTIFLOW_SFLOW_IPV4_PORT: 6343
       ELASTIFLOW_IPFIX_TCP_IPV4_PORT: 4739
+
+volumes:
+  elastiflow-elasticsearch-oss-data:
+    driver: local

Start a set. Logstash uses the image built earlier. If firewall is set, allow the port to be used.

$ sudo docker-compose up
Starting elastiflow-elasticsearch-oss ...
Starting elastiflow-elasticsearch-oss ... done
Starting elastiflow-kibana-oss ...
Recreating elastiflow-logstash-oss ...
Recreating elastiflow-logstash-oss
Recreating elastiflow-logstash-oss ... done
Attaching to elastiflow-elasticsearch-oss, elastiflow-kibana-oss, elastiflow-logstash-oss
elastiflow-elasticsearch-oss    | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elastiflow-elasticsearch-oss    | OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=1
elastiflow-elasticsearch-oss    | [2019-03-16T05:08:33,729][INFO ][o.e.e.NodeEnvironment    ] [B0ID7Xo] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/VG1-var)]], net usable_space [59.5gb], net total_space [83.9gb], types [xfs]
...

When the following log of Logstash is output, an index for NetFlow is created in Elasticsearch.

elastiflow-logstash-oss         | [2019-03-16T05:30:32,652][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"order"=>0, "version"=>30401, "index_patterns"=>"elastiflow-3.4.1-*", "settings"=>{"index"=>{"number_of_shards"=>3, "number_of_replicas"=>1, "refresh_interval"=>"10s", "codec"=>"best_compression"}}, "mappings"=>{"_default_"=>{"numeric_detection"=>true, ...

It is docker-compose up for log confirmation, but it isdocker-compose up -d for continuous startup. Also, set restart: always indocker-compose.yml and set it to start automatically after restarting the OS.

Import data set on Kibana

Access Kibana and import the data of Template, Index pattern, Visualize, Dashboard.

Access https://<Docker host>:5601 from Web browser, import elastiflow/kibana/elastiflow.kibana.6.6.x.json(latest version) by Management > Saved Objects > Import .

f:id:daichi703n:20190316165718p:plain

f:id:daichi703n:20190316165731p:plain

Check NetFlow traffic

If NetFlow record generation / transmission / reception is working properly, it will be available on Kibana's dashboard.

Various dashboards are provided, which makes it easy to visualize and analyze network traffic based on NetFlow records. There are many parts that can be helpful when creating your own Visualize and Dashboard. Details of Dashboard are as described in Official.

f:id:daichi703n:20190316173141p:plain

f:id:daichi703n:20190316173208p:plain

f:id:daichi703n:20190316173225p:plain

f:id:daichi703n:20190316173245p:plain

Conclusion - Visualize NetFlow with ElastiFlow (Elasticsearch + Logstash + Kibana)

ElastiFlow was used as a NetFlow collector and visualizer to visualize the network. There are some examples of using open source (OSS) Elasticsearch + Logstash + Kibana in NetFlow visualization, but ElastiFlow has a rich dashboard, and it is possible to start analysis equivalent to commercial products immediately.


This Blog is English Version of my JP's.

Sorry if my English sentences are incorrect.

designetwork