3 Replies Latest reply on Mar 19, 2015 8:45 AM by akeller

    Cacti Templates and Graylog Parsing


      Hi there,


      since I really like OpenSource tools, here comes two ways we made monitoring the MWG a bit easier:


      Cacti • View topic - McAfee WebGateway 7.5 Graphs -> Graphs and Installation setup to get the MWG monitored
      with all private SNMP-MIB information available.


      As for collecting logs in Graylog the "efficient" way (aka structured/indexed logs), I used the very good guide
      about how to export logs in "NITRO/SIEM" format from here: Best Practices: Configuring Syslog on Web Gateway 7.x


      I did adjust the output in 2 ways: we erased the first and second field (first was just the apliance name, second was date),
      so our ruleset looks like this:




      The logs come in a pretty structered way (although not GELF) and can be parsed with the new Graylog 1.0 with a

      grok-pattern like that:


      \|auth_user=%{USER:mwg_user}\|src_ip=%{IPV4:mwg_srcip}\|server_ip=%{IPV4:mwg_ser verip}\|host=%{HOST:mwg_host}\|url_port=%{NUMBER:mwg_urlport}\|status_code=%{NUM BER:mwg_statuscode}\|bytes_from_client=%{NUMBER:mwg_bytesFROMclient}\|bytes_to_c lient=%{NUMBER:mwg_bytesTOclient}\|categories=%{DATA:mwg_categories}\|rep_level= %{DATA:mwg_replevel}\|method=%{WORD:mwg_method}\|url=%{URI:mwg_url}\|media_type= %{DATA:mwg_mediatype}\|application_name=%{DATA:mwg_appname}\|user_agent=%{DATA:m wg_useragent}\|block_res=%{NUMBER:mwg_blockcode}\|block_reason=%{DATA:mwg_blockr eason}\|virus_name=%{DATA:mwg_virusname}\|hash=%{DATA:mwg_hash}\|filename=%{DATA :mwg_filename}\|filesize=%{NUMBER:mwg_filesize}\|


      You do need to import some grok-patterns beforehand in Graylog and make a new "input" in order to use the grok-extractor.
      Importing the grok patterns is fairly straight forward: copy and paste "grok-patterns" from here Grok Debugger to a text file
      and import that text-file into Graylogs "import pattern file" button.


      You´ll get indexed logs like this:


      It works for me mayhaps for you as well .

        • 1. Re: Cacti Templates and Graylog Parsing
          Jon Scholten

          Very cool stuff!

          • 3. Re: Cacti Templates and Graylog Parsing

            Since i can´t seem to edit my original post, here´s some adjustment to the grok filter since I noticed some messages weren´t parsed at all:


            \|auth_user=%{DATA:mwg_user}\|src_ip=%{IPV4:mwg_srcip}\|server_ip=%{IPV4:mwg_ser verip}\|host=%{DATA:mwg_host}\|url_port=%{NUMBER:mwg_urlport}\|status_code=%{NUM BER:mwg_statuscode}\|bytes_from_client=%{NUMBER:mwg_bytesFROMclient}\|bytes_to_c lient=%{NUMBER:mwg_bytesTOclient}\|categories=%{DATA:mwg_categories}\|rep_level= %{DATA:mwg_replevel}\|method=%{WORD:mwg_method}\|url=%{DATA:mwg_url}\|media_type =%{DATA:mwg_mediatype}\|application_name=%{DATA:mwg_appname}\|user_agent=%{DATA: mwg_useragent}\|block_res=%{NUMBER:mwg_blockcode}\|block_reason=%{DATA:mwg_block reason}\|virus_name=%{DATA:mwg_virusname}\|hash=%{DATA:mwg_hash}\|filename=%{DAT A:mwg_filename}\|filesize=%{NUMBER:mwg_filesize}\|


            Also, there´s a limitation with the grok-implementation in Graylog that you can not use the grok-conversions for field, ie turning strings into integer or floats.
            That might might be added somewhen, the syntax change would maybe be like that: %{NUMBER:mwg_bytesTOclient:int}, etc.