cancel
Showing results for 
Search instead for 
Did you mean: 

[EOL] Web Reporter: Configuring log file pushing in McAfee Web Gateway (MWG)

[EOL] Web Reporter: Configuring log file pushing in McAfee Web Gateway (MWG)

Introduction

This document will explain how to configure the McAfee Web Gateway to push access log data to the McAfee Web Reporter for analysis.  If you wish to have logs collected from the McAfee Web Gateway (log pulling) please see our knowledge base article: http://kc.mcafee.com/corporate/index?page=content&id=KB76963

Configuring the McAfee Web Gateway

To properly configure McAfee Web Gateway for reporting purposes follow these steps.


1. Logon to the Web Gateway admin user interface and navigate to:

Policy > Settings > Engines > File System Logging > Access Log Configuration. Expand “Settings for Rotation, Pushing, and Deletion”.

NOTE:  DO NOT CONFIGURE log pushing from the Configuration >  [[Appliance Name]] > Log File Manager section as this will result in unwanted logs getting sent to Web Reporter.  See Troubleshooting section below.

2. Under Auto Pushing select the “Enable auto pushing” check box and configure the URL to the Web Reporter.

3. In the “Destination" field enter the Web Reporter log processing URL.  For example, ftp://WebReporterIP:9121, http://WebReporterIP:9111/logloader.

4. Create a username and password unique to this function and enter them under the “User name” section.  Note:  The username and password defined here will be needed later in the Web Reporter configuration (below). If you have multiple Web Gateways pushing logs to one Web Reporter server, please review the following KB for details on using variables as usernames: http://kc.mcafee.com/corporate/index?page=content&id=KB76899

5. It is recommended to setup the Web Gateway to automatically push the logs immediately after rotation. For that keep the “Enable pushing log files directly after rotation” checked. 

If you would like to use time based push intervals instead, uncheck “Enable pushing log files directly after rotation” and set your “Push interval” hours and minutes.

Save Changes in the Web Gateway UI after configuring the Auto Pushing section.

autopushing.jpg

Configuring Web Reporter

How to properly configure McAfee Web Reporter to accept these incoming logs.

1. Logon to the Web Reporter admin user interface and navigate to: Administration > Setup > Log Sources.  Click Add to create a new log source.

2. Give this log source a name, note that there cannot be spaces in the name.

3. Select “Accept incoming log files".

4. In the log format drop down make sure “McAfee Web Gateway (Webwasher) – Auto Discover” is selected.

5. For the “Logon name” and “Password” fields use the same username and password created in the Web Gateway section (#4 above).

logsource.jpg

Validating your configuration

To confirm your Web Gateway to Web Reporter configuration is operating properly generate traffic until your next log push occurs.  Alternatively, force a log push from the Web Gateway by clicking "Rotate and push logs" from the Configuration > [[Appliance Name]] > page.   On Web Reporter, check the Jobs section of your log source; under Administration > Setup > Log Sources > Jobs.

JobsSuccess.jpg

Also you should see information starting to show up under the Quick View section of the Web Reporter interface.

QuickViewTest.jpg

Common issues and Troubleshooting

Mismatched Password

The usernames and passwords must match exactly on both the Web Gateway and Web Reporter, for log pushing and reporting to operate properly. If you accidentally mistype the password you will not see new data coming into the Web Reporter. Check the mwg-logmanager.errors.log via the Web Gateway UI under Troubleshooting > Appliance name > Log Files > mwg errors >  mwg-logmanager.errors.log and you will see entries like the following.


[06/Jun/2013:15:35:04 UTC] Cannot push '/opt/mwg/log/user-defined-logs/access.log/access1306061535.log.gz' to 'ftp://10.10.76.16:9121/access1306061535-10.10.76.10.log.gz'

Detailed reason(s):

command 'curl -g -q -f  -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs  -u wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061535.log.gz ftp://10.10.76.16:9121/access1306061535-10.10.76.10.log.gz' failed with error code 67

Error output is 'curl: (67) Access denied: 530'

SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'

Note: You will not see errors on the Web Reporter as it is simply not receiving data via the configured log source.

Misconfigured Port

If the destination URLs port is entered incorrectly, such as port 9111 (Web Reporter http port) is entered for the ftp URL you will see the following in the mwg-logmanager.errors.log

[06/Jun/2013:16:04:02 UTC] Cannot push '/opt/mwg/log/user-defined-logs/access.log/access1306061600.log.gz' to 'ftp://10.10.76.16:9111/access1306061600-10.10.76.10.log.gz'

Detailed reason(s):

command 'curl -g -q -f  -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs  -u wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061600.log.gz ftp://10.10.76.16:9111/access1306061600-10.10.76.10.log.gz' failed with error code 56

Error output is 'curl: (56) FTP response reading failed'

SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'

Global Log File pushing configured

Configuring auto pushing on the Web Gateway under 'Configuration > Log File Manager' instead of 'Policy > Settings > Engines > File System Logging' will result in unwanted files being sent to the Web Reporter of which cannot be reported. What you’ll see under Administration > Setup > Log Sources > Jobs is that many of your jobs are failing. In the details of the job you can see that the log name was not "access........log". Only the access.logs from mwg can be imported into Web Reporter.

                    LogSourceFail.jpg.jpg

                    Note the File name here – mwg-monitor.errors1305290000-10.10.76.10.. etc – this is a log the Web Reporter cannot process.
                    If you have configured Log Pushing under Configuration > Log File Manager please refer to the steps at the beginning of this doc to properly configure log pushing for the access log only.

Log header does not match log lines

In case you see all your jobs completed as successful, but there is still no data in your reports, it is possible that the log data import failed due to mismtached log headers and log lines. This sometimes happens when you try to modify your log file format (adding or removing columns) and the header does not line up with the fields that are being written.

On the Web reporter side you would see that the logs got uploaded and the header was detected (job successful), but when you look at the details of the job, you would see that all lines errored out and were ignored.

recordsrejected.jpg

More information for "empty report" situations can be found in this KB: http://kc.mcafee.com/corporate/index?page=content&id=KB67289

Web Reporter ports not allowed

Traffic is not reaching the Web Reporter server at all. Assume that your firewall is not allowing ports 9121/9111/9112 you will not be able to logon to the Web Reporter interface from another host, log processing jobs/new report data will not show up and in the mwg-logmanager.errors.log output you will see information like the following (similar to mismatched port configuration).

[06/Jun/2013:16:13:04 UTC] Cannot push '/opt/mwg/log/user-defined-logs/access.log/access1306061605.log.gz' to 'ftp://10.10.76.16:9111/access1306061605-10.10.76.10.log.gz'
Detailed reason(s):
command 'curl -g -q -f  -k -s -S --connect-timeout 30 -m 300 --ftp-create-dirs  -u wradmin:***** -T /opt/mwg/log/user-defined-logs/access.log/access1306061605.log.gz ftp://10.10.76.16:9111/access1306061605-10.10.76.10.log.gz' failed with error code 56
Error output is 'curl: (56) FTP response reading failed'
SHA1Hash of password is '940787ecca1e4710059774a6bbdcd08fb66b1029'

Related Information

Comments
msiemens

I have an existing WR fetching log files from three MWGv6 servers. Can I fetch/push logs from MWGv7 into the same WR database? What issues will I encounter?

I'd like to use ePO/CSR with my MWGv7 gateways but CSR isn't compatible with ePOv5.

sroering

Yes, you can use both MWG 6 and MWG 7 with the same Web Reporter Server, just create a log source for each MWG appliance.  The only "issue" is more data in your database.

If you need CSR 2.1 will be compatible with ePO 5.

goldenvxr

Great article.  One question though.  In a clustered environment, does this configuration change?  For example, if I configure the Management appliance to push logs to the Reporting Server, will the cluster send logs from one appliance or all in the cluster?

The configuration of pushing logs is part of the policy, when its configured correctly in UI > Policy > Settings. The policy is synchronized in a CM cluster so all of your appliances in this cluster will get the same policy and push the logs to the same server!

Version history
Revision #:
1 of 1
Last update:
‎05-02-2013 06:44 AM
Updated by: