cancel
Showing results for 
Search instead for 
Did you mean: 
McAfee Employee jebeling
McAfee Employee
Report Inappropriate Content
Message 1 of 2

Example Bash Script for Log Pull from Web Gateway Cloud Service?

Jump to solution

Are there any examples of Linux scripts that can be used to retrieve logs from the Web Gateway cloud service so that they can be imported into other reporting tools?

 

1 Solution

Accepted Solutions
McAfee Employee jebeling
McAfee Employee
Report Inappropriate Content
Message 2 of 2

Re: Example Bash Script for Log Pull from Web Gateway Cloud Service?

Jump to solution

Why yes Jeff, I'm glad you asked. 😉  Details about the API can be found in the Reporting section of the documentation for Web Gateway Cloud Service  here . Included in the documentation are all the available additional filters that can be used in addition to the two required timestamp filters. Also included are the available versions with a listing of included fields and the supported return format types (csv and xml).

I am by no means a scripting wizard but the attached code is fully functional and should give the reader a good idea of how to get the log files from WGCS in bulk or selectively, What you do with the retrieved files is up to you and your selected reporting tool vendor. This code is iintended to be an example and  comes without any warranty or support. I'm sure there are much better ways to implement the scripting for example override via command line arguements as opposed to a file, but I didn't have time for that. If you do use the code or improve upon it please post  back here so that others may benefit.

There are three files in the attached zip. logpull.conf, logpull.sh and logpull.add.conf

Note that none of the files include any credentials. In order to download log files from WGCS you need to use the admin (not user) credentials that are used to log in to manage.mcafee.com. These credentials should be placed in a .netrc file in the home directory of the user that is running the job. 

The .netrc file format looks like this:

machine msg.mcafeesaas.com

    login <your wgcs admin email>

    password <your wgcs admin password>

logpull.sh is the actual bash script if you run it just by name it will pull a log file using the parameters in logpull.conf. If you run it followed by a second filename, if that filename exists the variables set in that file will override the variables in the logpull.conf file

The logpull.conf script provides an initial configuration file that logpull.sh uses as a base configuration for all runs. logpull.conf can be modified with your customerid and other parameters for your scheduled jobs. If logpull.conf is not present in the directory with logfile.sh, logfile.sh will create logpull.conf with default parameters including a bogus customerid. 

Logpull.sh is designed to be run as a periodic  cron job. You can run it at any time interval you like, but the default setting is to only pull a log covering the endtime of the last successfull pull plus one second to the current time less 5 minutes. If the pull is successful then the conf file is updated so that the next pull will have a start time one second later than the end time of the previous successful pull. 

There is also a file named logpull.add.conf which can be used as a basis for creating override files for running custom jobs for example, adding additional filters or changing the time range for a pull. You can create files with any names you like. Using an override settings file is simple, just run ./logpull.sh <setting override file name>. Since this is intended for ad-hoc log pulls the script does not update lastSuccessEpoch if you specify an additional config file on the command line.

One last note, there is some minimal error checking and logging included. Execution logs go to logpull.log and there is a check to see if the curl command returned without error and that the log file is at least 400 bytes. If either of those checks fail logpull.conf variable lastSuccessEpoch isn't updated

 

1 Reply
McAfee Employee jebeling
McAfee Employee
Report Inappropriate Content
Message 2 of 2

Re: Example Bash Script for Log Pull from Web Gateway Cloud Service?

Jump to solution

Why yes Jeff, I'm glad you asked. 😉  Details about the API can be found in the Reporting section of the documentation for Web Gateway Cloud Service  here . Included in the documentation are all the available additional filters that can be used in addition to the two required timestamp filters. Also included are the available versions with a listing of included fields and the supported return format types (csv and xml).

I am by no means a scripting wizard but the attached code is fully functional and should give the reader a good idea of how to get the log files from WGCS in bulk or selectively, What you do with the retrieved files is up to you and your selected reporting tool vendor. This code is iintended to be an example and  comes without any warranty or support. I'm sure there are much better ways to implement the scripting for example override via command line arguements as opposed to a file, but I didn't have time for that. If you do use the code or improve upon it please post  back here so that others may benefit.

There are three files in the attached zip. logpull.conf, logpull.sh and logpull.add.conf

Note that none of the files include any credentials. In order to download log files from WGCS you need to use the admin (not user) credentials that are used to log in to manage.mcafee.com. These credentials should be placed in a .netrc file in the home directory of the user that is running the job. 

The .netrc file format looks like this:

machine msg.mcafeesaas.com

    login <your wgcs admin email>

    password <your wgcs admin password>

logpull.sh is the actual bash script if you run it just by name it will pull a log file using the parameters in logpull.conf. If you run it followed by a second filename, if that filename exists the variables set in that file will override the variables in the logpull.conf file

The logpull.conf script provides an initial configuration file that logpull.sh uses as a base configuration for all runs. logpull.conf can be modified with your customerid and other parameters for your scheduled jobs. If logpull.conf is not present in the directory with logfile.sh, logfile.sh will create logpull.conf with default parameters including a bogus customerid. 

Logpull.sh is designed to be run as a periodic  cron job. You can run it at any time interval you like, but the default setting is to only pull a log covering the endtime of the last successfull pull plus one second to the current time less 5 minutes. If the pull is successful then the conf file is updated so that the next pull will have a start time one second later than the end time of the previous successful pull. 

There is also a file named logpull.add.conf which can be used as a basis for creating override files for running custom jobs for example, adding additional filters or changing the time range for a pull. You can create files with any names you like. Using an override settings file is simple, just run ./logpull.sh <setting override file name>. Since this is intended for ad-hoc log pulls the script does not update lastSuccessEpoch if you specify an additional config file on the command line.

One last note, there is some minimal error checking and logging included. Execution logs go to logpull.log and there is a check to see if the curl command returned without error and that the log file is at least 400 bytes. If either of those checks fail logpull.conf variable lastSuccessEpoch isn't updated

 

More McAfee Tools to Help You
  • How-to: Endpoint Removal Tool
  • Support: Endpoint Security
  • Visit: Business Service Portal
  • More: Search Knowledge Articles
  • ePolicy Orchestrator Support

    • Download the new ePolicy Orchestrator (ePO) Support Center Extension which simplifies ePO management and provides support resources directly in the console. Learn more about ePO Support Center