Saturday, January 20, 2018

AWS VPC Flow Logs grok Pattern

Amazon Web Services(AWS) can generate VPC flow logs, format below
2 123456789010 eni-abc123de 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK

For more information on flow logs and grok filter plugin refer below links

grok patterns can be tested using below links

%{NONNEGINT:version} %{NONNEGINT:accountid} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NONNEGINT:srcport} %{NONNEGINT:dstport} %{NONNEGINT:protocol} %{NONNEGINT:packets} %{NONNEGINT:bytes} %{NONNEGINT:starttime} %{NONNEGINT:endtime} %{NOTSPACE:action} %{NOTSPACE:log-status}

Test using grokdebugger

Test using grokconstructor

You can also consider INT instead of NONNEGINT

Found few patterns by googling which looked like below, were not working on grokconstructor website.
%{NUMBER:version} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}

Tested on grokdebugger

Tested on grokconstructor

We can use the extracted variables from grok filter plugin in Kibana search or enhance data using logstash filter plugins geoip, dns, date etc.

Working in or using Python virtualenv

Install Python virtualenv on Ubuntu using below command
apt-get -y install python-virtualenv

Create virtualenv
$virtualenv test_env1
New python executable in test_env1/bin/python

$. test_env1/bin/activate
$source test_env1/bin/activate

Exit virtualenv

Switch between virtualenv’s
$workon test_env2

List all available virtualenv’s

virtualenvwrapper comes with few handy commands

$pip install virtualenvwrapper

virtualenvwrapper supports extra commands like