cancel
Showing results for 
Search instead for 
Did you mean: 

Parsing logs from a file

Jump to solution

Hi All,

I am new to NiroSecurity. I wanted to know how to go about configuring NitroSecurity to parse logs from a file. Say for example IIS or apache logs that are logging to a file. Is there an agent that can read and parse logs from a file. I went through the user guide and did not find any information regarding this. In the case of IIS and apache the guide mentions that these can be configured to send logs as syslogs. However the requirement is to read logs from a file and not send them as syslog. Any information regarding this would be really helpfull.

Thanks in advance

Rohan

1 Solution

Accepted Solutions

Re: Parsing logs from a file

Jump to solution

Hi Rohan,

We have been researching a simliar way to retrieve application log files and the Linux agent seems to be the way to go. I came across this in my research. Hope it helps:

McAfee Linux Event Collector 9.1.3 provides you with the capability to add a local agent to your system to push several types of data to the McAfee Event Receiver.

The installer is available by calling McAfee Support at 800-937-2237.

-------------------------

Supported Versions

Ubuntu 10.04 Uses mcafee-linux-event-collector_9.1.1.0-358_1004_amd64.deb

Ubuntu 12.04 Uses mcafee-linux-event-collector_9.1.1.0-358_1204_amd64.deb

Redhat 5.8  Uses mcafee-linux-event-collector_9.1.1.0-358.el5.x86_64.rpm

Redhat 6.2  Uses mcafee-linux-event-collector_9.1.1.0-358.x86_64.rpm 

Fedora 16   Uses mcafee-linux-event-collector_9.1.1.0-358.x86_64.rpm

Suse 11   Uses mcafee-linux-event-collector_9.1.1.0-358.x86_64.rpm

------------------------- 

Installing the Agent

Run the installer by double clicking the .deb or .rpm from the gui or

using rpm -i package.rpm from the command line for rpm and dpkg -i package.deb for deb

End-User License is here:

/usr/share/doc/mcafee/EULA McAfee - Corporate-August 2010.rtf

-------------------------

Configuring the Agent

To date filetail is the only plugin "type" that is supported, but you can have as many filetail sections as you want.

The file to be tailed must be on the local system not a mounted file.

The path to your conf file is below you can change the default path of the conf file by changing the path in the init script.

/etc/mcafee/mcafee_event_collector.conf

bookmark_dir= Is directory where bookmark file is saved and is configurable.

debug_level= Is the level of debug output by the collector options are error,info,warning,and debug. 

log_path= Is the direcotry where the log is written. 

sleep= If a file has not been modified since the agent was last shutdown, on startup will put the file in a watch list and check on it from time to time. If there are files in the watch list, the agent will check it every x number of seconds.

inactive_sleep= If there are no files in the watch list, the agent will sleep y number of seconds, before waking and checking for files in the watch list.

rec_ip= Is the IP of the receiver to send events to.

rec_port= Is the port of the receiver is listining on. 

rec_encrypt= Changin this value enables or disables encryption 0=off 1=on

type= Is the plugin type.  (To date filetail is the only plugin "type" that is supported, but you can have as many filetail sections as you want.)

subtype = Is a subtype of the plugin. ( To date big_fix is the only subtype that is supported.) Big_fix logs with a date at the top of a File with this subtype option it takes that date and appends it to the beggining of each event.

hostid = Put a value here if you would like to use a Host ID on the receiver. 

ft_dir = Directory where plugin will look for files to tail. 

ft_filter = Filter for what file to tail ie. mesages or log.*

ft_delim =  Delemiter for collector to know when a new event has happend ie. <newline>, <space>, <tab>, Regular expressions are also supported. 

ft_delim_end_of_event = Delemiter to start at the begginging or the end of the event 0=begginging 1=end Default is 1

ft_start_top = This tells us to start at the top of the file 0=no 1=yes

See example Configuration file at bottom of this docuemnt. 

-------------------------

Running the Agent

Once you have completed editing the file, restart your Event Collector service with this cmd:

/etc/init.d/mcafee_event_collecotr restart or

service mcafee_event_collector restart

start and stop are also options.

you can also run the Agent manualy run /usr/bin/event_collector -h to see your options

To enable auto learning for the agent run event_collector manually from command line with the -a option

-------------------------

Example Configuration File with two filetail sections with one using a hostid. 

##############

# Collector

##############

bookmark_dir=/var/lib/mcafee/bookmark

debug_level=error

log_path=/var/log/mcafee/event_collector.log

sleep=5

inactive_sleep=300

##############

#       Receiver

##############

rec_ip=10.0.0.0

rec_port=8081

rec_encrypt=0

##############

#       Plugin

##############

type=filetail

hostid=

ft_dir=/var/log

ft_filter=log.1

ft_delim=<newline>

ft_delim_end_of_event=1

ft_start_top=1

type=filetail

hostid=mesages

ft_dir=/var/log

ft_filter=messages

ft_delim=<newline>

ft_start_top=1

# Sample Big fix logging

type=filetail

subtype=big_fix

hostid=

ft_dir=/var/log

ft_filter=*.log

ft_delim=At \d*:\d*:\d* -\d*

ft_delim_end_of_event=0

ft_start_top=1

-------------------------

5 Replies
haroot
Level 9
Report Inappropriate Content
Message 2 of 6

Re: Parsing logs from a file

Jump to solution

Hi Rohan,

File based logs can be collected using Mcafee's Windows/linux Agent. Below is the KB article which includes the details of using McAfee windows Agent for IIS as well as DNS log collection.

http://kc.mcafee.com/corporate/index?page=content&id=KB74849&actp=search&viewlocale=en_US&searchid=1...

Cheers

Re: Parsing logs from a file

Jump to solution

Thanks Haroot that was helpful.

Do you have a link to a KB article or documentation for reading logs from a linux system remotely? I would need to read and parse application from a linux system remotely and don't seem to see any documentation on how to proceed with this integration.

Thanks in advance.

Rohan

haroot
Level 9
Report Inappropriate Content
Message 4 of 6

Re: Parsing logs from a file

Jump to solution

Hi Rohan,

You can access the KB site http://kc.mcafee.com/corporate/index?page=home  and search for LInux data source. One good thing about McAfee Siem is that you are not limited to one method of collection. As an alternative to Agent you can use collection mechanisn such as scp,sftp to collect the logs from the data sources.

Hope this is helpful.

Haroot

Re: Parsing logs from a file

Jump to solution

Hi Rohan,

We have been researching a simliar way to retrieve application log files and the Linux agent seems to be the way to go. I came across this in my research. Hope it helps:

McAfee Linux Event Collector 9.1.3 provides you with the capability to add a local agent to your system to push several types of data to the McAfee Event Receiver.

The installer is available by calling McAfee Support at 800-937-2237.

-------------------------

Supported Versions

Ubuntu 10.04 Uses mcafee-linux-event-collector_9.1.1.0-358_1004_amd64.deb

Ubuntu 12.04 Uses mcafee-linux-event-collector_9.1.1.0-358_1204_amd64.deb

Redhat 5.8  Uses mcafee-linux-event-collector_9.1.1.0-358.el5.x86_64.rpm

Redhat 6.2  Uses mcafee-linux-event-collector_9.1.1.0-358.x86_64.rpm 

Fedora 16   Uses mcafee-linux-event-collector_9.1.1.0-358.x86_64.rpm

Suse 11   Uses mcafee-linux-event-collector_9.1.1.0-358.x86_64.rpm

------------------------- 

Installing the Agent

Run the installer by double clicking the .deb or .rpm from the gui or

using rpm -i package.rpm from the command line for rpm and dpkg -i package.deb for deb

End-User License is here:

/usr/share/doc/mcafee/EULA McAfee - Corporate-August 2010.rtf

-------------------------

Configuring the Agent

To date filetail is the only plugin "type" that is supported, but you can have as many filetail sections as you want.

The file to be tailed must be on the local system not a mounted file.

The path to your conf file is below you can change the default path of the conf file by changing the path in the init script.

/etc/mcafee/mcafee_event_collector.conf

bookmark_dir= Is directory where bookmark file is saved and is configurable.

debug_level= Is the level of debug output by the collector options are error,info,warning,and debug. 

log_path= Is the direcotry where the log is written. 

sleep= If a file has not been modified since the agent was last shutdown, on startup will put the file in a watch list and check on it from time to time. If there are files in the watch list, the agent will check it every x number of seconds.

inactive_sleep= If there are no files in the watch list, the agent will sleep y number of seconds, before waking and checking for files in the watch list.

rec_ip= Is the IP of the receiver to send events to.

rec_port= Is the port of the receiver is listining on. 

rec_encrypt= Changin this value enables or disables encryption 0=off 1=on

type= Is the plugin type.  (To date filetail is the only plugin "type" that is supported, but you can have as many filetail sections as you want.)

subtype = Is a subtype of the plugin. ( To date big_fix is the only subtype that is supported.) Big_fix logs with a date at the top of a File with this subtype option it takes that date and appends it to the beggining of each event.

hostid = Put a value here if you would like to use a Host ID on the receiver. 

ft_dir = Directory where plugin will look for files to tail. 

ft_filter = Filter for what file to tail ie. mesages or log.*

ft_delim =  Delemiter for collector to know when a new event has happend ie. <newline>, <space>, <tab>, Regular expressions are also supported. 

ft_delim_end_of_event = Delemiter to start at the begginging or the end of the event 0=begginging 1=end Default is 1

ft_start_top = This tells us to start at the top of the file 0=no 1=yes

See example Configuration file at bottom of this docuemnt. 

-------------------------

Running the Agent

Once you have completed editing the file, restart your Event Collector service with this cmd:

/etc/init.d/mcafee_event_collecotr restart or

service mcafee_event_collector restart

start and stop are also options.

you can also run the Agent manualy run /usr/bin/event_collector -h to see your options

To enable auto learning for the agent run event_collector manually from command line with the -a option

-------------------------

Example Configuration File with two filetail sections with one using a hostid. 

##############

# Collector

##############

bookmark_dir=/var/lib/mcafee/bookmark

debug_level=error

log_path=/var/log/mcafee/event_collector.log

sleep=5

inactive_sleep=300

##############

#       Receiver

##############

rec_ip=10.0.0.0

rec_port=8081

rec_encrypt=0

##############

#       Plugin

##############

type=filetail

hostid=

ft_dir=/var/log

ft_filter=log.1

ft_delim=<newline>

ft_delim_end_of_event=1

ft_start_top=1

type=filetail

hostid=mesages

ft_dir=/var/log

ft_filter=messages

ft_delim=<newline>

ft_start_top=1

# Sample Big fix logging

type=filetail

subtype=big_fix

hostid=

ft_dir=/var/log

ft_filter=*.log

ft_delim=At \d*:\d*:\d* -\d*

ft_delim_end_of_event=0

ft_start_top=1

-------------------------

Re: Parsing logs from a file

Jump to solution

Thanks a lot Chris for this. I will check this option and let you know how it goes.

Regards,

Rohan