1 2 Previous Next 16 Replies Latest reply on Mar 28, 2011 11:50 AM by sroering

    Web Reporter question

    jont717

      It looks like our Web Reporter does not report on this data from our log files:

       

      reporter.png

       

      This site is in our global whilelist.  It seems like the Web Reporter has no data on any of our HTTPS sites that are in the Global Whitelist.   IBM.com is another one:

       

      reporter2.png

      When I search for www.ibm.com or ibm.com in the Web Reporter, I get no results.  This site is also in the Global Whitelist. 

       

      But clearly it is being logged in my access.log files. 

        • 1. Re: Web Reporter question
          sroering

          Have you checked the log parsing errors? for those files?

           

          The potential problem I see are the empty "" pairs.  You should always include a dash in the place of empty strings.  "-"

           

          At least that is my best guess without testing. This should be straight forward to test if you put the header and those records into a new log file, then manually import.

          • 2. Re: Web Reporter question

            Shawn, correct me if I'm wrong but I think the reason that reporter doesn't see it is because the status code is 0.

            The question is, why is the status code 0?

            I get the same results when I turn off SSl scanning.

            • 3. Re: Web Reporter question
              sroering

               

               

              I guess I've never looked at what happens to status code 0.  I know WR will ignore 407s, but I was unaware of any others that are explicitly ignored. 

               

              A quick check of status code 0 means that the response was empty (not even headers provided).  I suppose there is no reason that these should be ignored.  If correct, should probably file a bug for this.

               

              But in addtion to that, putting empty quote pairs is never good.  It's always best practice to explictly provide a dash instead of null.

              • 4. Re: Web Reporter question

                Nevermind. That's not it.

                I created a log of 15 records of:

                #time_stamp "auth_user" src_ip status_code cache_status "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res" "geolocation" file_scanned

                [18/Mar/2011:16:08:50 -0400] "-" 192.168.2.10 0 TCP_MISS "CONNECT https://www.apple.com/ HTTP/1.1" "" "-" "-" 15613 "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203" "" "0" "US-United States" false

                 

                And reporter saw them correctly. It's gotta be something else.

                • 5. Re: Web Reporter question
                  jont717

                  I am hoping to have this traffic readable in WR.  If anything, it would be nice to at least see these sites being hit and at least see the client IP address. 

                   

                  Let me know if you need more information or what you want me to try.  Thanks for the help.

                  • 6. Re: Web Reporter question
                    sroering

                    jont717 wrote:

                     

                    I am hoping to have this traffic readable in WR.  If anything, it would be nice to at least see these sites being hit and at least see the client IP address. 

                     

                    Let me know if you need more information or what you want me to try.  Thanks for the help.

                     

                    My recommendation is still the same as the first reply.

                    Copy the log header to a new text file

                    Append several log lines that failed to import

                    On the log lines, insert a dash between any empty "" pairs ---->  "-"

                    manually import that log

                    check for errors on the log parsing job status and server.log.

                     

                    If that resolved the problem, then getting the file to import could be problematic.  You'd have to fix the logs using a utility like sed.  If the access logs are too large, notepadd++ might be able to do it with a text replace.

                    • 7. Re: Web Reporter question
                      jont717

                      I believe it has to do with the status_code being 0 or the CONNECT

                       

                      In other sites that are whitelisted, they do not have a status_code of 0 and they come into the Web Reporter just fine.  And they have all the same "-" or "" as the ones that are not being logged.

                       

                      Message was edited by: jont717 on 3/21/11 10:33:25 AM EDT
                      • 8. Re: Web Reporter question
                        jont717

                        No matter what I do, I cannot get this to log in the Web Reporter.  I know I can search by IP address in the WR because i have done it before.  As you can see, I put dashes on every " ". 

                         

                        #time_stamp "auth_user" src_ip server_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"

                        [21/Mar/2011:09:55:56 -0400] "-" 172.16.xxx.249 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

                        [21/Mar/2011:10:21:50 -0400] "-" 172.16.xxx.145 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

                         

                         

                        This below does not log either...

                         

                        #time_stamp "auth_user" src_ip server_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"

                        [21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom4_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

                        [21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom5_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

                        [21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom6_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

                        [21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom7_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

                        [21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom8_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

                        [21/Mar/2011:09:48:02 -0400] "" 172.16.xxx.222 68.236.99.34 304 "GET http://www.isohomevalue.com/homevaluenet/_public/images/Zoom9_off.gif HTTP/1.1" "" "-" "" 0 "" "" "0"

                         

                        Message was edited by: jont717 on 3/21/11 10:40:21 AM EDT
                        • 9. Re: Web Reporter question
                          sroering

                          jont717 wrote:

                           

                          No matter what I do, I cannot get this to log in the Web Reporter.  I know I can search by IP address in the WR because i have done it before.  As you can see, I put dashes on every " ". 

                           

                          #time_stamp "auth_user" src_ip server_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"

                          [21/Mar/2011:09:55:56 -0400] "-" 172.16.100.249 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

                          [21/Mar/2011:10:21:50 -0400] "-" 172.16.100.145 65.197.19.159 0 "CONNECT https://65.197.19.159/ HTTP/1.0" "-" "-" "-" 0 "-" "-" "0"

                           

                          Are these being reported as errors?  I know there is a bug in the "ignored records" counter, so ignored records is always 0.

                          1 2 Previous Next