Saturday, January 20, 2018

AWS VPC Flow Logs grok Pattern

Amazon Web Services(AWS) can generate VPC flow logs, format below
2 123456789010 eni-abc123de 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK

For more information on flow logs and grok filter plugin refer below links

grok patterns can be tested using below links

%{NONNEGINT:version} %{NONNEGINT:accountid} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NONNEGINT:srcport} %{NONNEGINT:dstport} %{NONNEGINT:protocol} %{NONNEGINT:packets} %{NONNEGINT:bytes} %{NONNEGINT:starttime} %{NONNEGINT:endtime} %{NOTSPACE:action} %{NOTSPACE:log-status}

Test using grokdebugger

Test using grokconstructor

You can also consider INT instead of NONNEGINT

Found few patterns by googling which looked like below, were not working on grokconstructor website.
%{NUMBER:version} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}

Tested on grokdebugger

Tested on grokconstructor

We can use the extracted variables from grok filter plugin in Kibana search or enhance data using logstash filter plugins geoip, dns, date etc.