cancel
Showing results for 
Search instead for 
Did you mean: 
jont717
Level 12

Web Reporter question

It looks like our Web Reporter does not report on this data from our log files:

reporter.png

This site is in our global whilelist.  It seems like the Web Reporter has no data on any of our HTTPS sites that are in the Global Whitelist.   IBM.com is another one:

reporter2.png

When I search for www.ibm.com or ibm.com in the Web Reporter, I get no results.  This site is also in the Global Whitelist. 

But clearly it is being logged in my access.log files. 

0 Kudos
16 Replies
sroering
Level 13

Re: Web Reporter question

Have you checked the log parsing errors? for those files?

The potential problem I see are the empty "" pairs.  You should always include a dash in the place of empty strings.  "-"

At least that is my best guess without testing. This should be straight forward to test if you put the header and those records into a new log file, then manually import.

0 Kudos
eelsasser
Level 15

Re: Web Reporter question

Shawn, correct me if I'm wrong but I think the reason that reporter doesn't see it is because the status code is 0.

The question is, why is the status code 0?

I get the same results when I turn off SSl scanning.

0 Kudos
sroering
Level 13

Re: Web Reporter question

I guess I've never looked at what happens to status code 0.  I know WR will ignore 407s, but I was unaware of any others that are explicitly ignored. 

A quick check of status code 0 means that the response was empty (not even headers provided).  I suppose there is no reason that these should be ignored.  If correct, should probably file a bug for this.

But in addtion to that, putting empty quote pairs is never good.  It's always best practice to explictly provide a dash instead of null.

0 Kudos
eelsasser
Level 15

Re: Web Reporter question

Nevermind. That's not it.

I created a log of 15 records of:

#time_stamp "auth_user" src_ip status_code cache_status "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res" "geolocation" file_scanned

[18/Mar/2011:16:08:50 -0400] "-" 192.168.2.10 0 TCP_MISS "CONNECT https://www.apple.com/ HTTP/1.1" "" "-" "-" 15613 "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203" "" "0" "US-United States" false

And reporter saw them correctly. It's gotta be something else.

0 Kudos
jont717
Level 12

Re: Web Reporter question

I am hoping to have this traffic readable in WR.  If anything, it would be nice to at least see these sites being hit and at least see the client IP address. 

Let me know if you need more information or what you want me to try.  Thanks for the help.

0 Kudos
sroering
Level 13

Re: Web Reporter question

jont717 wrote:

I am hoping to have this traffic readable in WR.  If anything, it would be nice to at least see these sites being hit and at least see the client IP address. 

Let me know if you need more information or what you want me to try.  Thanks for the help.

My recommendation is still the same as the first reply.

Copy the log header to a new text file

Append several log lines that failed to import

On the log lines, insert a dash between any empty "" pairs ---->  "-"

manually import that log

check for errors on the log parsing job status and server.log.

If that resolved the problem, then getting the file to import could be problematic.  You'd have to fix the logs using a utility like sed.  If the access logs are too large, notepadd++ might be able to do it with a text replace.

0 Kudos
jont717
Level 12

Re: Web Reporter question

I believe it has to do with the status_code being 0 or the CONNECT

In other sites that are whitelisted, they do not have a status_code of 0 and they come into the Web Reporter just fine.  And they have all the same "-" or "" as the ones that are not being logged.

Message was edited by: jont717 on 3/21/11 10:33:25 AM EDT
0 Kudos
jont717
Level 12

Re: Web Reporter question

No matter what I do, I cannot get this to log in the Web Reporter.  I know I can search by IP address in the WR because i have done it before.  As you can see, I put dashes on every " ". 

#time_stamp "auth_user" src_ip server_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"

[21/Mar/2011:09:55:56 -0400] "-" 172.16.xxx.249 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

[21/Mar/2011:10:21:50 -0400] "-" 172.16.xxx.145 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

This below does not log either...

#time_stamp "auth_user" src_ip server_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"

[21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom4_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

[21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom5_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

[21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom6_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

[21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom7_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

[21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom8_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

[21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom9_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

Message was edited by: jont717 on 3/21/11 10:40:21 AM EDT
0 Kudos
sroering
Level 13

Re: Web Reporter question

jont717 wrote:

No matter what I do, I cannot get this to log in the Web Reporter.  I know I can search by IP address in the WR because i have done it before.  As you can see, I put dashes on every " ". 

#time_stamp "auth_user" src_ip server_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"

[21/Mar/2011:09:55:56 -0400] "-" 172.16.100.249 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

[21/Mar/2011:10:21:50 -0400] "-" 172.16.100.145 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

Are these being reported as errors?  I know there is a bug in the "ignored records" counter, so ignored records is always 0.

0 Kudos