WR: How to Add a Log Column in Webgateway and How to Report on it Using Web Reporter

Version 1

    Introduction

    Web Reporter has default data points for reporting. If you want additional info, there are four custom columns available for reporting on additional data. However, these additional columns have some restrictions.

     

    • Only available in detail data / not available in summary data
    • Only for advanced reports (requiring a premium license)
    • No special data type recognition. For example, byte values will not calculate sum or average values, IP addresses cannot be filtered by a netmask, etc.

     

    For general information about setting up MWG and WR log sources, please see this article: https://community.mcafee.com/docs/DOC-4928

    Overview

     

    There are 3 primary steps for getting Web Reporter to report on additional info.


      1. Configure the Web Gateway to include the desired field in the access log.
      2. Update the Web Gateway’s access log header to reflect the change made in step one.
      3. On Web Reporter, modify your log source by adding a User-Defined Column for this new log header, so WR can understand this particular access.log format.

     

    Example using destination IP

    We often see that administrators want to run reports based on, or to at least include the destination IP, so we will use destination IP as an example for adding a custom column.

     

     

    **Before going any further, it is important to know that if there is any misconfiguration on the Web Gateway side, then none of your access logs will be processed by Web Reporter until this is corrected. Any affected logs may not be able to be repaired. Therefore, we recommend testing the changes on a practice access log first. Instructions for creating a practice log can be found here, under the “Creating a Customized log” section:


                           More information about customizing and managing your log files on MWG, please see this article:  https://community.mcafee.com/docs/DOC-4812

     

    I. Configure the Web Gateway to include the desired field in the access log

    In the Web Gateway UI go to: Policy > Log handler (bottom left corner) > Access log...highlight Write Access.log rule and click edit so the edit rule window appears.

                                  MBO ONE.png


                           

                                    -    In the Edit Rule window, click on 4. Events,highlight the "Set User-Defined .logline" and click edit.

                                     

                             MBO TWO.png

     

     

                       

                                  -    "Edit Set Property" Window appears: click the lower Add button in this window - below where it says "To concatenation of these strings"

                            

                             THREE.png

     

     

     

    - "Enter a string" Window appears: select Use Property and from drop down box select IP.ToString(IP). With IP.ToString(IP) highlighted, Click on "Parameters" to the right of it

                             FOUR.png


     

     

                             -  Select "Use Property" (bottom Right corner) and from the drop down box select URL.Destination IP...click OK 3 times then click finish. Save changes.

                             

                             FIVE.png

     

     

                             SIX.png

     

     

    * Important: before proceeding, stop right here and have a look at your event column and note where this new log line has been placed. It should be at the very end and it should read:

    +IP.ToString (URL.Destination.IP) -- as seen in the two screenshots below:

     

                             SEVEN.png

     

                           

                             EIGHT.png

     

     

     

                             *To ensure that there is a space between Block.ID and URL.Destination.IP in the log, we need to add a delimiter to the Block ID line.

                                  Edit events > Edit Set Property Window appears, highlight the line just below Number.ToString(Block.ID)

                                 

                                  Delimit ONE.png

     

     

     

                            

                             …and click edit so the “Enter a string” window appears. Here, we want to add a white space following the double quote so it looks just like this:

     

                                 Delimit TWO.png

     

                             FINISHED MWG SIDE.png

                                                                    

    *Do NOT save your changes just yet, as the header needs to be modified. Continue with step II below.

     

     

    II. Update the Web Gateway’s access log header

    We must now modify the header so it matches the order of your events column as seen above. Since our new log column is last in the events list, it must be last in the headers line, label it server_ip.

     

         In the Web Gateway UI go to:

     

    Policy > Settings > File System Logging > Access Log Configuration... under "File System Logging Settings" you will see the Log header box -- add server_ip to the end of it.

     

     

     

    As an example, I was using the default Write access.log rule, so the header for it would now look like this:

     

    time_stamp "auth_user" src_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res" server_ip

     

                        HEADER one.png

     

    Following the advice below about headers will prevent much frustration, as ANY type of error with headers will prevent Web Reporter from understanding the log format, and it will not process any corresponding logs.

     

        •     Other than underscore (_), no other special characters are permitted
        •     Header names cannot contain spaces. Use an underscore wherever a space is desired (server_ip)
        •     If the field logged was wrapped in double quotes, then wrap the header name in double quotes as well
        •     Avoid duplicate names – do not give it name that already exists in the Log header
        •     As a general rule of thumb, the example below is always a safe approach to header names:

     

                                     “mwr_header_name”

     

     


     

     

    III. On Web Reporter, modify the log source by adding a User-Defined Column for the new log header

     

    Now that the Web Gateway is logging the destination IP and the access log header has been updated, you must modify your log source inside of Web Reporter. Note that, if you have multiple Web Gateway log sources in Web Reporter, you have to make sure to update all of them.

    Add a User-Defined Column for this new log header (server_ip) so that WR can use this particular access.log format and so that you can report using it. To modify the header, you can take the following steps:

     

        • Log into your Web Reporter

     

        • Navigate to Administration -> Setup ->Log Sources -> Log Sources.

     

        • Select your log source, and hit ‘Edit’.

     

        • On the ‘Edit Log Source’ screen, click the‘User-Defined Columns’ tab.

     

        • Assuming you aren’t already using it, check the checkbox for User-Defined 1 to ‘Populate this column’, and enter a Log file header of server_ip (assuming of course, you used server_ip as your header on the MWG side).

     

                             User_Def_1.png

    Now, any further log files coming into Web Reporter from this log source should have this new log entry parsed and added to the User-Defined 1 column. You can use this when running advanced reports on detail data. It’s important to note that all data prior to when this change was made will not have this data available to report on.