cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Understanding syslog - send logs to your SIEM or other syslog server

Introduction

This document outlines the most common steps and considerations when configuring Web Gateway 7.x/8.x to send access log and audit log data to a syslog server. After reading this document, you should have a good understanding of:

  1. The required configuration changes
  2. Common issues encountered when using syslog

Prerequisites

Before beginning to configure syslog, you should decide what type of configuration suits your environment best. Below, some common variations are explained in order to simplify the decision process.

Decide how to send

Syslog lets you send data using UDP or TCP. UDP is much more commonly used (some syslog servers may not even have TCP listeners), but TCP is naturally more flexible. UDP typically uses port 514, but TCP has no standardised port. UDP provides considerably lower overheads in terms of network traffic, but TCP provides flow control, error recovery, and support for queueing.

UDP and TCP configurations can be easily identified by the number of @ symbols in the forwarding line:

  • UDP configuration – first line sends access log events; second line sends access and audit log events:
daemon.info @x.x.x.x:514
daemon.info;auth.=info @x.x.x.x:514
  • TCP configuration – as above, but using TCP and an example port
daemon.info @@x.x.x.x:6514
daemon.info;auth.=info @@x.x.x.x:6514

Decide what severity to send

The recommended severity is always 6 (Info)! However, the below table shows the available list for reference. The descriptions are provided as examples; the RFC for syslog only recommends that appropriate severities are chosen.

Value

Severity

Message keyword

Description

0

Emergency

emerg

A panic condition (system is unusable)

1

Alert

alert

A condition that should be corrected immediately

2

Critical

crit

A critical condition such as a hardware error

3

Error

err

Error conditions

4

Warning

warning

Warning conditions

5

Notice

notice

Conditions that are not an error, but may require special handling

6

Informational

info

Informational messages that do not require immediate attention

7

Debug

debug

Messages typically only expected during debugging

 

For more info see: http://tools.ietf.org/html/rfc5424#page-11

Decide what to send

What kind of data do you want to send to the syslog server? All access logs and audit logs? Only access logs, or only blocked requests?

Decide what format to send

Which format does your syslog server require data to be presented in? You may wish to check with your server admin to see which format should be used by MWG:

Default format

Accepted by McAfee Content Security Reporter (which does accept other modified formats too). CSR simply requires that the format (log header) be present in the Log Source configuration.

McAfee SIEM

Accepted by McAfee SIEM. The McAfee SIEM format requires additional log fields that are not present in the default format.

CEF format

Accepted by other report servers such as ArcSight. The devices parsing CEF format output can be directional; if a virus is found (for example) different rules can be used to formulate the syslog output sent to the server.

Configuring the syslog daemon (rsyslog.conf)

Before configuring Web Gateway rules, the syslog daemon configuration must be updated. This should only be done in the GUI using the File Editor. The changes must be made on a per-appliance basis, so different appliances in a cluster can have different configurations here.

Use the GUI file editor

When configuring syslog, it is tempting to jump on the command line and edit /etc/rsyslog.conf directly. This will not end well! The MWG Coordinator will overwrite this file and you will lose changes. Use the File Editor on the Configuration page.

Don't write to disk!

It is extremely important that syslog does not write access or audit log data to disk. The default syslog configuration will happily do this and fill the /var partition on the appliance. To avoid this, look for this line in ryslog.conf:

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

If you only plan to send access logs, replace it with this line:

*.info;daemon.!=info;mail.none;authpriv.none;cron.none                -/var/log/messages
  • The addition of “daemon.!=info” excludes the writing of anything from daemon facility with a priority of info to the messages file, thus maintaining disk space. It needs to be placed after “*.info” to allow other info-level messages to be written as normal.
  • Adding a dash to the beginning of the file path stops rsyslog from syncing the file after every message, protecting performance.

If you plan to send both access and audit logs, use the below line instead:

*.info;daemon.!=info;auth.!=info;mail.none;authpriv.none;cron.none                -/var/log/messages
  • As with the first example, an exclusion is added for info-level messages from the auth facility.

Send to syslog

To send using UDP, add a rule to capture messages matching daemon.info (access logs) and auth.info (audit logs) to the destination server:

daemon.=info;auth.=info @x.x.x.x:514

This rule should be placed at the bottom of the file.

For TCP logging, things are a little more complex but a lot more configurable. The section between ### begin forwarding rule ### and ### end of the forwarding rule ### contains a complete explanation and example of a queue and TCP forwarding rule. When the comment symbols are removed, they look like this:

Remote Logging (we use TCP for reliable delivery)

An on-disk queue is created for this action. If the remote host is
down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/spppl/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
#remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@remote-host:514

As you will see, the settings are well explained, but the defaults are fine here. The only line you should edit is the final one for the remote host. As with UDP, specify the messages to capture and then add the desired server:

daemon.=info;auth.=info @x.x.x.x:6514

Configuring the rules

Now that the syslog daemon has been configured, we can configure the rules in the McAfee Web Gateway to start passing messages to it. To do this, we simply need to create the contents of the message, and then configure an event to send the newly created message. As mentioned above, there are various different formats which your syslog server may require the message to be in. Below are the most common.

Default format

For the default format, all that is required is to create a rule which applies the conditions in which you want data to be sent to the syslog server. In our case, we're sending ALL access log data to the syslog server. The only change required is to create an additional rule to send the logline to syslog.

Name: Send to syslog
Criteria: Always
Action: Continue
Event: Syslog (6, User-Defined.logLine)

 

McAfee SIEM (Nitro)

The McAfee SIEM format includes additional log fields that the McAfee SIEM will parse. See the Online Ruleset Library (in the Content Security Portal) for the most up-to-date McAfee SIEM ruleset. Below is an example log entry:

McAfeeWG|time_stamp=[01/Jan/2015:02:12:31 +0800]|auth_user=jsmith|src_ip=10.10.69.1|server_ip=172.224.247.54|host=www.mcafee.com|url_port=80|status_code=301|bytes_from_client=279|bytes_to_client=1149|categories=Bus..., Software/Hardware|rep_level=Minimal Risk|method=GET|url=http://www.mcafee.com/|media_type=text/html|application_name=|user_agent=Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)|block_res=0|block_reason=|virus_name=|hash=|filename=|filesize=753|

Below is screenshots of the rule sets installed in the log handler:

CEF Format

The CEF format is much different from the default McAfee Web Gateway format. See the Online Ruleset Library (in the Content Security Portal) for the most up-to-date CEF format. The CEF format will include the column metadata (i.e. what the column represents) in the log line. The CEF format is a generic format that a large number of SIEM vendors support including Arcsight and Splunk.

CEF:0|McAfee|Web Gateway|7.3.2|301|Proxy--|2|rt=Sep 02 2013 16:55:57 cat=Access Log dst=12.234.121.129 dhost=www.mcafee.com suser=jsmith src=10.10.69.1 requestMethod=GET request=http://www.mcafee.com/ app=HTTP cs3=HTTP/1.1 cs3Label=Protocol/Version cs4=Business, Software/Hardware cs4Label=URL Categories cs6=Minimal Risk cs6Label=Reputation fileType=text/html out=1182 requestClientApplication=Mozilla/5.0 Firefox/23.0 cs1= cs1Label=Virus Name cn1=0 cn1Label=Block Reason cs5=Default cs5Label=Policy

Below are some screenshots of what the rules will look like:

Attachments

Below are links to the rulesets referenced in the screenshots above. They can be imported using the Ruleset library.

McAfee SIEM (Nitro) logging ruleset

CEF syslog format ruleset

Send the audit log to Syslog

The Audit log is used to track changes made within the Web Gateway GUI, including login/logout/login failure events. To send audit log data with syslog, simply tick “Write audit log to syslog” under Configuration > Appliances > Log File Manager > Settings for the Audit log. This setting is appliance-specific.

Format

Audit logs are generated in CEF format; for example:

Nov 23 19:22:33 localhost CEF: 0|McAfee|WebGateway|1|USER_LOGIN|USER_LOGIN|3|Timestamp=23/Nov/2016:19:22:33.289 User=admin Action=USER_LOGIN Source_Type=USER Source_ID=10.10.69.2 Appliance=gsd-mwg1 User-Agent=Java/1.8.0_111 Role=Super Administrator

Facility and Severity

Audit events are sent by the “auth” facility, and using information severity (6). For this reason, the configuration examples above all match for auth.=info.

Common Issues

Filling /var partition

It is common for the /var partition to be filled if the steps in ‘Don’t write to disk’ are not followed – this will usually not be seen immediately. You can check if the messages file is being written to via the GUI and the CLI:

GUI: Open the messages file located in Troubleshooting > Log files > system > messages

CLI: run the command

tail /var/log/messages

In both cases, the presence of either access or audit log lines indicates that they have not been correctly excluded.

Messages not received on syslog server

If messages are not received on the destination server, the first check should be for firewalls that may block the specified ports. To verify that the messages are being sent, run a simple tcpdump on the CLI against the port(s) you configured:

tcpdump port 514

If you see data, rsyslog is doing its job and you should look into the network side – bear in mind that UDP will not show any responses! If you see nothing, it’s possible that the rules are not being triggered properly. You can check a Rule Trace to see if something fails to match in the Logging cycle.

Message size

Web Gateway will, by default, truncate messages to 2,000 characters before sending them to the server. To increase the maximum message size, you can add this line to the first line of rsyslog.conf, with a size value in kilobytes:

$MaxMessageSize 4k

Additional Uses

Usage of syslog is not limited to forwarding logs! Syslog can also be used for monitoring the health of Web Gateway, using the Best Practice guide Notifications and Alerting Options.

Links

Content & Cloud Online Ruleset Library - https://contentsecurity.mcafee.com/ruleset_library/

The Syslog Protocol (RFC5424) - http://tools.ietf.org/html/rfc5424

Additional rsyslog configuration parameters: http://www.rsyslog.com/doc/rsyslog_conf_global.html

Changelog

2019-05-03 - Updated text, clarified some steps, and expanded on others

2017-12-12 - Formatted and updated branding

2016-11-23 - Added steps to send audit log to Syslog.

2014-12-31 - Updated McAfee SIEM & CEF Format sections to reference the McAfee Content & Cloud Security Online Ruleset Library.

Labels (1)
Comments

If I disable writing to /var/log/messages, will I still be able to write to the default log path? In other words: I want syslog + local logging.

Thanks

MWG will still write to the access log (/opt/mwg/log/user-defined-logs/access.log/access.log). The goal of what I wrote above was to not write to /var/log/messages, which is on the /var partition and is much smaller.

Take note of the respective events:

Syslog(6,User-Defined.logLine) - What sends the log line message to syslog daemon (used in syslog rules)


FileSystemLogging.WriteLogEntry(User-Defined.logLine)<Access Log Configuration) - What sends the log line message to a log file (used in normal logging rules)

Best,

Jon

Is it possible to use other facilities than 'daemon'? Let's say I want to send out different logs (access.log, access_denied.log etc.) and have them end up in different logs on the syslog server. I usually do this by using different facilities.

Hi Everyone,

How to exclude the Kernel/system/hardware syslogs like system reboot,hard disk failure etc..

I have configured syslog at level 6.

Hi Haaris,

If you followed the guide, this should *exclude* such events. Using "daemon.info @syslogserver:514" should limit the messages sent to your syslog server.

If you wanted to send all events you would use something like "*.* @syslogserver:514".

Best Regards,

Jon

I have configured daemon.info. @syslogserver:514 in rsyslog.conf but still I m getting reboot messages/hardware/kernel messages.

Its a long shot but hoping someone could help!

I've configured as per the above to log from the MWG to the McAfee SIEM.

Working pretty good except for the fact that the "BytesFromClient" and "BytesTo Client" both record as "0" for all entries.

Anyone came across this before?

Doesn't work. Lots of non-existing objects.

Which part doesnt work? Or what ruleset are you importing? Please start up a new discussion thread if it might be lengthy.

Lots of stuff... can't remember... solved it by using some other ruleset from community... Hate MWG... I've never seen such undocumented crap, wonder how they can sell it as corporate solution...

you're just too stupid to use it.

Yep, I'm stupid...

We're adding a second syslog server.  Would configuring it be the same as the first, just adding  a second line of "daemon.info @syslogserver:514"?

Yes, that's right.

"daemon.info @syslogserver:514", to send over UDP

"daemon.info @@syslogserver:514", to send over TCP so that no messages are lost

Dear all, i´ve just configured MWG to send events to a SIEM, configuring "Write audit log to syslog" under Configuration > Appliances > Log File Manager > Settings for the Audit Log. and adding "*.* @syslogserver:514" in rsyslog.conf, but im not seeing that any audit event is going from MWG to the SIEM, (using tcpdump -i eth0 port 514) however the audit.log is getting the events OK, and, when i do tcpdump -i lo port 514, i seeing the audit events reaching localhost! please help me! im missing something else?

Thanks in advance!

Hi Matthew and YD,

If you're using two syslog servers with the same log format, then using the same config entry is fine:

daemon.info @syslogserver1:514

daemon.info @syslogserver2:514

BUT if you're sending two different log formats, you might want respective the format sent to its respective syslog server. In that case it gets a little more complex, however, its doable.

We would do something like this instead

# Format 1 (nitro), Logging rule uses severity level 6 aka Info in the Event

daemon.info;daemon.!=notice @syslogserver1:514

# Format 2 (CEF), Logging rule uses severity level 5 aka notice in the Event

daemon.info @syslogserver2:514

The "daemon.info" part means that we send everything from the "daemon" facility, with severity info or lower (0-6).

By using "daemon.info;daemon.!=notice" we're saying send everything from the daemon facility, with severity info or lower, but exclude daemon.notice specifically (the part that makes this specific is the !=).

Hope this helps!

Best Regards,

Jon

Hi Teo!

What version are you on? I did just notice there was a bug regarding the audit logging over syslog, this is fixed in 7.6.2.6:

McAfee Corporate KB - Web Gateway 7.6.2.6 Release Notes PD26796

"Audit log information could not be recorded using syslog due to a problem with parsing time zone values. (1165003)"

Best Regards,

Jon

Hi,

I need the rule set for CEF format but this link will ask for a username and password that i can't access, anyone can provide this info to us:

CEF syslog format ruleset

Please clarify. I have no special configuration for access.log (use default one). But when I set in /etc/rsyslog.conf daemon.info @x.x.x.x:514 I see messages in access.log format and CEF. Can't understand why access log goes to rsyslogd.

Hi vvadym,

I'd suggest creating a new thread and include some screenshots.

The only way you'd see access.log entries in syslog, is if your rules tell it to send to syslog using the event Syslog(X, User-Defined.XXX). Perhaps you have it in your rules twice? Or perhaps there is a gap in logic.

The logic gap could happen if you have a criteria tied to your CEF rule, but no logic applied to your "Send to Syslog" rule.

For example, in the CEF rules I describe above, there is no logic gap. There is two rules, one with the criteria: VirusFound equals true, another with criteria: VirusFound equals false. This accounts for both TRUE and FALSE situations. If I were to disable the one of them, then what you describe could happen.

The simple fix is to merge the send to syslog rule with the CEF rules by adding the Syslog even at the end of your events in each rule.

Hope that helps.

Anyway to configure syslog to send audit.log with a hostname other than localhost?

This is an improvement we will release with 7.7.2 version. This is already published as Beta release in contentsecurity.mcafee.com portal.

-Sergej

This is actually what I've received from the QRadar support which I put to single line:

# send access log to qradar *.info;

daemon.!=info;

mail.none;authpriv.none;

cron.none -/var/log/messages *.info;mail.none;

authpriv.none;

cron.none

@@[IP]:514

But to be able to send access logs via syslog I guess it should be like that...?

#prevent storing on disk?

*.info;daemon.!=info;mail.none;authpriv.none;cron.none -/var/log/messages

#send accesslogs matched by above ruleset through syslog?

daemon.info @@10.110.96.56:514

We are using syslog at level 6 to write to a SIEM and a third party reporting tool. However, the most recorded destination is our own proxy address as a HEAD request. I think this is because we use a PAC file with failover to our DR site and the clients reach out to set if that destination is reachable with a HEAD request to see if valid headers come back. Is there any way to exclude this type of request from syslog without losing other data?

Contributors
Version history
Revision #:
3 of 3
Last update:
‎05-06-2019 05:53 AM
Updated by:
 

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community