Best Practice: Configuring McAfee Web Reporter log source for McAfee Web Gateway

Version 2

    Introduction

    This document will explain how to configure the McAfee Web Gateway to push access log data to the McAfee Web Reporter for analysis. 

    If you wish to have logs collected from the McAfee Web Gateway (log pulling) please see our knowledge base article:
    http://kc.mcafee.com/corporate/index?page=content&id=KB76963

     

     

     

    Configuring the McAfee Web Gateway

     

    To properly configure McAfee Web Gateway for reporting purposes follow these steps.


    1. Logon to the Web Gateway admin user interface and navigate to:
    Policy > Settings > Engines > File System Logging > Access Log Configuration. Expand “Settings for Rotation, Pushing, and Deletion”.

     

    NOTE:  DO NOT CONFIGURE log pushing from the Configuration >  [[Appliance Name]] > Log File Manager section as this will result in unwanted logs getting sent to Web Reporter.  See Troubleshooting section below.


    2. Under Auto Pushing select the “Enable auto pushing” check box and configure the URL to the Web Reporter.


    3. In the “Destination" field enter the Web Reporter log processing URL.  For example, ftp://WebReporterIP:9121, http://WebReporterIP:9111/logloader.


    4. Create a username and password unique to this function and enter them under the “User name” section.  Note:  The username and password defined here will be needed later in the Web Reporter configuration (below). If you have multiple Web Gateways pushing logs to one Web Reporter server, please review the following KB for details on using variables as usernames: http://kc.mcafee.com/corporate/index?page=content&id=KB76899

     


    5. It is recommended to setup the Web Gateway to automatically push the logs immediately after rotation. For that keep the “Enable pushing log files directly after rotation” checked.  

     

    If you would like to use time based push intervals instead, uncheck “Enable pushing log files directly after rotation” and set your “Push interval” hours and minutes.

    Save Changes in the Web Gateway UI after configuring the Auto Pushing section.

     

     

                         autopushing.jpg              

    Configuring the McAfee Web Reporter

     

     

    How to properly configure McAfee Web Reporter to accept these incoming logs.

     

     

     

    1. Logon to the Web Reporter admin user interface and navigate to: Administration > Setup > Log Sources.  Click Add to create a new log source.

     

     

     

     

    2. Give this log source a name, note that there cannot be spaces in the name.

     

     

    3. Select “Accept incoming log files".

     

     

    4. In the log format drop down make sure “McAfee Web Gateway (Webwasher) – Auto Discover” is selected.

     

     

    5. For the “Logon name” and “Password” fields use the same username and password created in the Web Gateway section (#4 above).

     

     

     

    logsource.jpg

     

    Validating your configuration

     

    To confirm your Web Gateway to Web Reporter configuration is operating properly generate traffic until your next log push occurs.  Alternatively, force a log push from the Web Gateway by clicking "Rotate and push logs" from the Configuration > [[Appliance Name]] > page.   On Web Reporter, check the Jobs section of your log source; under Administration > Setup > Log Sources > Jobs.

     

     

    JobsSuccess.jpg

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    Also you should see information starting to show up under the Quick View section of the Web Reporter interface.

     

     

    QuickViewTest.jpg

     

     

    Common issues and Troubleshooting

     

    Mismatched Password

     

    The usernames and passwords must match exactly on both the Web Gateway and Web Reporter, for log pushing and reporting to operate properly. If you accidentally mistype the password you will not see new data coming into the Web Reporter. Check the mwg-logmanager.errors.log via the Web Gateway UI under Troubleshooting > Appliance name > Log Files > mwg errors >  mwg-logmanager.errors.log and you will see entries like the following.

     


    [06/Jun/2013:15:35:04 UTC] Cannot push '/opt/mwg/log/user-defined-logs/access.log/access1306061535.log.gz' to 'ftp://10.10.76.16:9121/access1306061535-10.10.76.10.log.gz'

    Detailed reason(s):

    command 'curl -g -q -f  -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs  -u wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061535.log.gz ftp://10.10.76.16:9121/access1306061535-10.10.76.10.log.gz' failed with error code 67

    Error output is 'curl: (67) Access denied: 530'

    SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'

     

     

     

     

     

     

    Note: You will not see errors on the Web Reporter as it is simply not receiving data via the configured log source.

     

     

    Misconfigured Port


    If the destination URLs port is entered incorrectly, such as port 9111 (Web Reporter http port) is entered for the ftp URL you will see the following in the mwg-logmanager.errors.log

     

     

     

     

     

     

     

     

     

    [06/Jun/2013:16:04:02 UTC] Cannot push '/opt/mwg/log/user-defined-logs/access.log/access1306061600.log.gz' to 'ftp://10.10.76.16:9111/access1306061600-10.10.76.10.log.gz'

    Detailed reason(s):

    command 'curl -g -q -f  -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs  -u wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061600.log.gz ftp://10.10.76.16:9111/access1306061600-10.10.76.10.log.gz' failed with error code 56

    Error output is 'curl: (56) FTP response reading failed'

    SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'

     

     

     

    Global Log File pushing configured

     

     

    Configuring auto pushing on the Web Gateway under 'Configuration > Log File Manager' instead of 'Policy > Settings > Engines > File System Logging' will result in unwanted files being sent to the Web Reporter of which cannot be reported. What you’ll see under Administration > Setup > Log Sources > Jobs is that many of your jobs are failing. In the details of the job you can see that the log name was not "access........log". Only the access.logs from mwg can be imported into Web Reporter.

     

     

                        LogSourceFail.jpg.jpg

     

                        Note the File name here – mwg-monitor.errors1305290000-10.10.76.10.. etc – this is a log the Web Reporter cannot process.
                        If you have configured Log Pushing under Configuration > Log File Manager please refer to the steps at the beginning of this doc to properly configure log pushing for the access log only.

     

    Log header does not match log lines

     

    In case you see all your jobs completed as successful, but there is still no data in your reports, it is possible that the log data import failed due to mismtached log headers and log lines. This sometimes happens when you try to modify your log file format (adding or removing columns) and the header does not line up with the fields that are being written.

     

    On the Web reporter side you would see that the logs got uploaded and the header was detected (job successful), but when you look at the details of the job, you would see that all lines errored out and were ignored.

     

     

    recordsrejected.jpg

     

    More information for "empty report" situations can be found in this KB: http://kc.mcafee.com/corporate/index?page=content&id=KB67289

     

     

    Web Reporter ports not allowed

     

    Traffic is not reaching the Web Reporter server at all. Assume that your firewall is not allowing ports 9121/9111/9112 you will not be able to logon to the Web Reporter interface from another host, log processing jobs/new report data will not show up and in the mwg-logmanager.errors.log output you will see information like the following (similar to mismatched port configuration).

     

     

     

     

     

     

     

     

    [06/Jun/2013:16:13:04 UTC] Cannot push '/opt/mwg/log/user-defined-logs/access.log/access1306061605.log.gz' to 'ftp://10.10.76.16:9111/access1306061605-10.10.76.10.log.gz'
    Detailed reason(s):
    command 'curl -g -q -f  -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs  -u wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061605.log.gz ftp://10.10.76.16:9111/access1306061605-10.10.76.10.log.gz' failed with error code 56
    Error output is 'curl: (56) FTP response reading failed'
    SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'

     

     

     

     

    Related Information

     

     

    Best Practice Documents

     

    Logs and log file management
    https://community.mcafee.com/docs/DOC-4812


    How to add and report on custom columns

    https://community.mcafee.com/docs/DOC-4929