McAfee Web Gateway can easily be used in conjunction with a reporting tool (Content Security Reporter, Web Reporter, Enterprise Security Manager or other) to monitor the categories and reputations of websites on a periodic or ongoing basis. One way to accomplish this is with a periodic cron job that runs on the gateway itself, coupled with a list of the sites to be monitored. My preference is to use wget (sitelist is cleaner and I know how to insert a delay between requests so that if "page views" are enabled for your log source, you don't have a problem with missing sites in your reports), but curl could also be used. Here are the details.
Creating your list of sites to monitor
Log on to the MWG console using SSH. Create your sitelist using vi or some other editor. You could even edit your sitelist off box and move it to the gateway using scp. For example, you could use winSCP to transfer a sitelist created in notepad++. I named my sitelists monitorsitelist.curl and monitorsitelist.wget I stored them in the \home directory, but they could be placed almost anywhere
The sitelist format is different between curl and wget
Setting up a periodic job to access the sites and thereby generate a log entry
My proxy is at 192.168.11.122 with proxy port 9090 I use sitemonitor as the username this is leveraged in the MWG ruleset. If your ruleset is set up properly you don't need to setup a password or account for the cronjob to use. (Only the MWG IP address is permitted to "authenticate" with that username (see rules))
crontab -u root -e
If using wget:
# Run at midnight every day (includes 20 second delay between site requests)
0 0 * * * wget -e http_proxy=192.168.11.122:9090 --proxy-user=sitemonitor --proxy-password=null -i /home/sitemonitorlist.wget -w 20 -O- &>/dev/null
If using curl:
# Run at midnight every day
0 0 * * * curl -x http://192.168.11.122:9090 --proxy-user sitemonitor:null -K /home/sitemonitorlist.curl -s >/dev/null
Set up an MWG rulset to handle the requests generated from the cronjob
Place the attached ruleset or similar before other authentication rules in your ruleset hierarchy. The ruleset only operates on requests generated from the proxy IP address and will only mark the username "sitemonitor" as authenticated. Note that you do need to add sitemonitor to the user database because of how the authentication rule is structured.
Set up reporting
Set up your reporting tool to have monitoring dashboards for the logs associated with the proxy IP and/or the sitemonitor username. Attached is an ePolicy Orchestrator dashboard export and an ePolicy Orchestrator report export for use with Content Security Reporter. These exports include the associated queries to generate reports and other exports. As with all ePO queries and reports they can be automated and scheduled to periodically deliver results via email, download or file share.
Here is what my Site Monitor dashboard looks like:
Attached is a Site Monitor report output, and a CSV output from one of the queries.
As always the standard disclaimer applies that posts on this forum by McAfee employees are provided on an as is basis without any implied obligations for support, warranty, or correctness. Comments and suggestions welcome.
One further note on this. I prefer not to do stop cycle in the ruleset (because then I won’t catch it if my rules would normally block on the category or reputation), and I also don’t want to block in the ruleset (because then my reporting will then indicate that I am sometimes blocking the monitored sites, when I am not, for normal clients).
However, allowing the rest of the rules to run can create some other undesirable effects. The block pages, logging, and rule tracing all use “most recently used” parameter settings for categories, geolocation, and reputations, so if URL.Geolocation, URL.Categories or URL.Reputation properties are checked elsewhere in your rulesets, and the parameters for the last one checked don’t match what you used in the site monitor ruleset, you might not get the expected results in your reports.
One setting that I found caused particular problems was the reverse DNS setting. If I check trusted source / GTI online for www.testingmcafeesites.com/testcat_uncat.html , it correctly reports the site as uncategorized. However, if I check the same site on the web gateway with URL.Categories with reverse DNS enabled it will report the site as a Business and Software/Hardware site. With that setting disabled, the site is correctly reported by MWG as Uncategorized.