Amazon Web Services(AWS) can generate VPC flow logs, format below
2 123456789010 eni-abc123de 172.31.9.69 172.31.9.12 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK
For more information on flow logs and grok filter plugin refer below links
https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
grok patterns can be tested using below links
http://grokdebug.herokuapp.com
http://grokconstructor.appspot.com/do/match#result
%{NONNEGINT:version} %{NONNEGINT:accountid} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NONNEGINT:srcport} %{NONNEGINT:dstport} %{NONNEGINT:protocol} %{NONNEGINT:packets} %{NONNEGINT:bytes} %{NONNEGINT:starttime} %{NONNEGINT:endtime} %{NOTSPACE:action} %{NOTSPACE:log-status}
Test using grokdebugger
Test using grokconstructor
You can also consider INT instead of NONNEGINT
Found few patterns by googling which looked like below, were not working on grokconstructor website.
%{NUMBER:version} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}
Tested on grokdebugger
Tested on grokconstructor
We can use the extracted variables from grok filter plugin in Kibana search or enhance data using logstash filter plugins geoip, dns, date etc.
Nice article, Which you have shared here about the AWS. Your article is very informative and useful to know more about the AWS VPC Flow Logs grok Pattern. Simply2cloud offers the Best AWS Training in Delhi
ReplyDeleteI read your post and trust me its really helpful for us.
ReplyDeleteaws training institute in delhi
aws training institute in noida
This is a great high resolution screen which you have shared for the users. Making a website is not an easy task but managing a good website is really hard work.good content about aws vpc flow logs grok pattern. oracle fusion hcm training india
ReplyDeleteYou have shared such a wonderful post which about grok. Thanks for such information. Keep it up. Policy Creation Software Online
ReplyDelete