Saturday, January 20, 2018

AWS VPC Flow Logs grok Pattern


Amazon Web Services(AWS) can generate VPC flow logs, format below
2 123456789010 eni-abc123de 172.31.9.69 172.31.9.12 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK

For more information on flow logs and grok filter plugin refer below links
https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

grok patterns can be tested using below links
http://grokdebug.herokuapp.com
http://grokconstructor.appspot.com/do/match#result

%{NONNEGINT:version} %{NONNEGINT:accountid} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NONNEGINT:srcport} %{NONNEGINT:dstport} %{NONNEGINT:protocol} %{NONNEGINT:packets} %{NONNEGINT:bytes} %{NONNEGINT:starttime} %{NONNEGINT:endtime} %{NOTSPACE:action} %{NOTSPACE:log-status}

Test using grokdebugger

Test using grokconstructor

You can also consider INT instead of NONNEGINT


Found few patterns by googling which looked like below, were not working on grokconstructor website.
%{NUMBER:version} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}

Tested on grokdebugger


Tested on grokconstructor

We can use the extracted variables from grok filter plugin in Kibana search or enhance data using logstash filter plugins geoip, dns, date etc.

No comments:

Post a Comment