Showing results for 
Search instead for 
Did you mean: 
Level 9

Quick and dirty log retrieval, don't judge.

While I have become quite adept at logging into each of our proxies ten times an hour to troubleshoot and 'grep' logs I have a much harder time trying to teach a few of our other guys to do so.

The other day I finally tired of having one kind fellow ask me how to break out of a 'tail' with a panicked look in his I just gave in and wrote up a quick ruby script. I am no programmer and I worry a bit about how hard this could get torn apart here, but here it is.

If you don't like ruby and don't have it installed this might not be for you. If you 'do' have ruby installed then you can likely script better than me and this serves you nothing. But, if you 'are' like me and dabble in ruby and want to save yourself some time this is an easy way out.

I wrote this in ruby 1.9.3, and you will need to install three gems:

gem install highline

gem install net-ssh

gem install colorize


#!/usr/bin/env ruby

require 'highline/import'

require 'net/ssh'

require 'colorize'

user = ask("Username: ") {|q| q.echo = true}

pass = ask("Password: ") {|q| q.echo = '*'}

#user = 'readonlyusername'

#pass = 'readonlyuserpassword'

hosts = ['ip','ip','ip','ip']

filter = ARGV[0]

access_log_path = '/opt/mwg/log/user-defined-logs/access.log/access.log'

fileout ="results-#{ARGV[0]}-#{"%Y-%m-%d-%H-%M-%S")}.txt","w")

fileout_name = "results-#{ARGV[0]}-#{"%Y-%m-%d-%H-%M-%S")}.txt"

hosts.each do |host|

  host = host.chomp

  puts "Retrieving logs from #{host}".yellow

  Net::SSH.start( host, user, :password => pass ) do |ssh|

    fileinfo = ssh.exec!("ls -lh #{access_log_path}").red

    output = ssh.exec!("grep #{filter} #{access_log_path} | tail -n 45")

    fileout.puts "\r\n#{host} | #{fileinfo}#{output}".cyan



puts "Output file:   #{fileout_name}".green

Essentially to run this you need to make one edit for your environment and you are up. I used 'highline' to hide the password you type in at the prompt so it's not in your bash history.

(I later went back and created a read-only account on my proxies and statically stored the username and password in the script, which McAfee will tell you not to do and I would "not" recommend unless you have very "very" tight control of where this file ends up.)

Save the above code into a text file such as 'proxy-logs.rb'. Place the ip's for your proxies into the 'hosts = ['ip','ip','ip','ip']' values (make more or less as needed, this should also work with a single host). Then, once ruby and the three gems are installed, simply type: ruby proxy-logs.rb filter  at the command prompt.

Whatever you type as the filter will be used to 'grep' out the last 45 matching lines in each access.log file. It also tells you when access.log was last written to and how large it is, we have ours set to roll at 150 MB so if there are no results but I can see it just rolled I know why.

The script will create an output file named 'results-whateveryoutypedasyourfilter-year-month-day-hour-minute-second.txt  in the directory you ran it from. (ie '')

It will also give you prompts as it works so you know what it's doing, as well as the output filename.

There are a lot of things this needs as it's not really 'done' but it's working just fine for my co-worker thus far. It needs code cleanup, I will likely add a rescue loop in case one of the proxies times out during retrieval, etc etc. I have a ton of ideas but most of them are going into an entirely new tool to have many more functions directed at the web gateways. Comments, concerns, and smart remarks welcome.

:edited for spelling:

Message was edited by: consoul on 1/24/13 6:48:56 PM CST
5 Replies
Level 9

Re: Quick and dirty log retrieval, don't judge.

A co-works of mine is a ruby enthusiast and he added this to a 'sinatra' page be built for another one of my tools. Someone at McAfee want to use this as a template and one-up us? This is hugely beneficial for me, and I assume it will be for others as well.

Here is an example search after we added more colorization, I have covered the logins and portions of our ip scheme:

Sinatra App.png

McAfee Employee

Re: Quick and dirty log retrieval, don't judge.

I don't do ruby, but this makes it look verrry appealing.

I'd love to have something like this in the MWG GUI.

::cough:: product managment ::cough::  



Level 15

Re: Quick and dirty log retrieval, don't judge.

Very nice. I'm impressed.

Now do it as a standalone off-box .Net app that uses the REST interface to pull the logs instead of SSH.

Then I'll be super-impressed.

0 Kudos
Level 9

Re: Quick and dirty log retrieval, don't judge.

I haven't had the pleasure of working with the REST interface yet, and only know as much about it as I have seen in questions here on the forum.

Do you know what the long-term goal for REST is? Just an API of sorts?

0 Kudos
McAfee Employee

Re: Quick and dirty log retrieval, don't judge.

Yeah, the REST api just allows for better integration between products and can also be used for custom use cases (like integration with a custom ticketing system for example to grant access to a requested site).

As far as integration between products, as of right now Web Reporter 5.2.1 can pull logs from MWG 7.3. As of 7.2 ePo can send lists to MWG for use in your rules.



Message was edited by: jscholte (added product integration examples) on 1/29/13 2:00:22 PM CST
0 Kudos