Saturday, January 20, 2018

AWS VPC Flow Logs grok Pattern


Amazon Web Services(AWS) can generate VPC flow logs, format below
2 123456789010 eni-abc123de 172.31.9.69 172.31.9.12 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK

For more information on flow logs and grok filter plugin refer below links
https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

grok patterns can be tested using below links
http://grokdebug.herokuapp.com
http://grokconstructor.appspot.com/do/match#result

%{NONNEGINT:version} %{NONNEGINT:accountid} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NONNEGINT:srcport} %{NONNEGINT:dstport} %{NONNEGINT:protocol} %{NONNEGINT:packets} %{NONNEGINT:bytes} %{NONNEGINT:starttime} %{NONNEGINT:endtime} %{NOTSPACE:action} %{NOTSPACE:log-status}

Test using grokdebugger

Test using grokconstructor

You can also consider INT instead of NONNEGINT


Found few patterns by googling which looked like below, were not working on grokconstructor website.
%{NUMBER:version} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}

Tested on grokdebugger


Tested on grokconstructor

We can use the extracted variables from grok filter plugin in Kibana search or enhance data using logstash filter plugins geoip, dns, date etc.

Working in or using Python virtualenv

Install Python virtualenv on Ubuntu using below command
apt-get -y install python-virtualenv

Create virtualenv
$virtualenv test_env1
New python executable in test_env1/bin/python

$. test_env1/bin/activate
Or
$source test_env1/bin/activate

Exit virtualenv
$deactivate

Switch between virtualenv’s
$workon test_env2

List all available virtualenv’s
$workon


virtualenvwrapper comes with few handy commands

$pip install virtualenvwrapper

virtualenvwrapper supports extra commands like
mkvirtualenv
cdvirtualenv
rmvirtualenv