6 Replies Latest reply: Jan 4, 2013 6:36 AM by heberrio RSS

    Web reporter not show in the Quick View

    heberrio

      Hello,

       

      I can see in the log files on the gateway that is has been pushed successfully. 

       

      erro1.PNG

       

        erro2.PNGerro3.PNG

      Any Ideas?

        • 1. Re: Web reporter not show in the Quick View
          consoul

          Not sure if you are running this on linux or windows server. Recently our WebReporter has stopped working on a regular basis, after years of stable (yet sub-par) performance.

           

          If you are on Linux, try the following:

           

          service webreporter_x64_control stop

          service webreporter_x64_idb_control stop

           

          service webreporter_x64_control start

          service webreporter_x64_idb_control start

           

          Then check to see if your db inserts start back up.

           

          (Note: You will have to re-open the web console if you had it open at the time, simply refreshing doesnt seem to do it for me after an app restart. Assuming it has to do with the Java clients connection settings.)

          • 2. Re: Web reporter not show in the Quick View
            sroering

            If log jobs are showing in Web Reporter, then check the status. If the status is successful, that really only means that Web Reporter looked at the header (first record) and decided it had the necessary log structure.  If the log records don't match the header (a common mistake on Web Gateway 7), then the log job could result in 100% errors. I suspect this is your problem.  Take a careful look at the access logs to see that the header matches the records.

             

            Sorry, I know this sometimes causes confustion.

             

            heberrio wrote:

             

            Hello,

             

            I can see in the log files on the gateway that is has been pushed successfully. 

             

            erro1.PNG

             

              erro2.PNGerro3.PNG

            Any Ideas?

            • 3. Re: Web reporter not show in the Quick View
              sroering

              Hello,

               

              Sorry that you seem to be experiencing issues, but I want to make a few things clear. 

               

               

              1) The issue reported by herberrio doesn't appear to be performance related.

               

              2) Restarting the service does not resolve any performance problems unless something is misconfigured, such as setting the java memory too high (KB73295). See 7th bullet point. Large JVM memory allocation (more than 4 GB) overloads the garbage collector over time, and simply isn't necessary for anyone.

               

              3) Restarting the service is only necessary if the JVM memory is exhausted. This may seem contradictory to point 2, but if Web Reporter needed more than 4GB of RAM, something is wrong. An example is creating a detail data based report for all records.  This is the analogous to creating a PDF document or HTML page directly from your Web Gateway access logs.  Unfortunately there is no safety switch to prevent people from making reports so large if they choose 'all results' on their query.

               

              4) Restarting the service may give 'perceived' performance increase for a short time due to empty buffers.  Web Reporter has always been extremely efficient at taking records from an access log file and reading them into memory.  Once that buffer is full, log parsers go to sleep to prevent the system from running out of memory.  The bottle neck in performance is always due to other problems. 

               

              5) There are different types of performance issues and we can help you identify them and likely resolve them. Perhaps you could start a new thread and provide us with some specific details to the problem(s) you are seeing.

               

              6) The issue with refreshing the browser when the Web Reporter service is restarted I could likely explain, but won't help resolve the problem. I think it happens when the browser doesn't restart the JVM on the client side. Then the Web Reporter client cannot re-establish the connection.  Restarting your browser would also close the JVM and essentially work-around the issue.  If you remove everything after ".../reporter/" in the URL, it should reload the applet without needing to restart the browser.

               

              consoul wrote:

               

              Not sure if you are running this on linux or windows server. Recently our WebReporter has stopped working on a regular basis, after years of stable (yet sub-par) performance.

               

              If you are on Linux, try the following:

               

              service webreporter_x64_control stop

              service webreporter_x64_idb_control stop

               

              service webreporter_x64_control start

              service webreporter_x64_idb_control start

               

              Then check to see if your db inserts start back up.

               

              (Note: You will have to re-open the web console if you had it open at the time, simply refreshing doesnt seem to do it for me after an app restart. Assuming it has to do with the Java clients connection settings.)

              • 4. Re: Web reporter not show in the Quick View
                consoul

                sroering, thanks for your reply. My reply was however like the OP, not about performance but about logs not being processed. Whenever I see that logs are simply not being processed (and often get the message at the bottom of quickview reports about there being a backlog) I restart the services. This usually remediates the issue.

                • 5. Re: Web reporter not show in the Quick View
                  sroering

                  Consoul,

                   

                  In the basic sense, Web Reporter does 2 functions, import logs and make reports. However, there are a lot of components involved and issues are not always so simple.  Next time you have issues with logs not being processed, please contact support to report the issue.  Submit a feedback so we can look at the data.

                   

                  If Web Reporter stops working randomly, I can only think of two possibilities without looking at any data.  1) out of memory issue as I described earlier. So until you understand the cause of out-of-memory error, you cannot fix the issue.  2) Problem repairing connection to the database.  Typically broken connections are automatically repaired and don't have any lasting impact.  But there have only been a couple of these issues over the life of this product and they were very specific and not commonly encountered.  If you report problems to support we can help you get them solved.

                   

                  Since version 5.2.0, all log parsing performance issues should have been resolved. We haven't seen any new issues since then.  Personally I'm expecting some improvement in the next release, based on testing I've done in the past, but it is more or less the side effect of a larger clean-up project. But the benifit is only likely to be noticed by large enterprises with more than 10 proxies and/or 200 million log records per day.

                  • 6. Re: Web reporter not show in the Quick View
                    heberrio

                    hi,

                     

                    The error was related to the access.log and thus used the following procedure.

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                    McAfee Web Gateway 7.0 and Web Reporter 5.1

                     

                    Integration Guide

                     

                     

                    Preparing Web Reporter to accept Logfiles from Web Gateway 7


                    Logfiles from one or more Web Gateway instances are usually uploaded (pushed)to the Web Reporter 5.1 System. There are 3 protocols available to do this:FTP, HTTP and HTTPS. This means that depending on the protocol you choose anaccording TCP Port on the Web Reporter will be used.

                    For FTP (recommended): TCP/9121

                    For HTTP: TCP/9111

                    For HTTPS: TCP/9112

                    In our sample configuration we willconfigure Web Reporter to accept incoming logs from the McAfee Web Gateway 7Systems. To do so please go to "Administration > Setup > Log Sources"Screen and click the "Add" Button.

                     

                    Figure 6: LogSource Definition Screen.

                     

                    Configure the Log source as shown inFigure 6. Select an appropriate Name for the Log source and set the Mode to “AcceptIncoming Logfiles”.

                    Important:It is mandatory that you pick “McAfee Web Gateway (Webwasher) – Auto Discover”in the Log Format drop down. Other Log formats (also Custom definable Logformats) will not import correctly and will cause parsing errors during import.

                    The “Logon Name” and Password that canbe configured here is not the Web Interface login of the McAfee Web Gateway.You can specify any User Name/Password here that will be used by the WebGateway to log on to the Web Reporter via FTP to upload the Log File Data.

                    If you want to push all imported Data toan archival system, please go to the “Post Processing” tab and select theaccording local directory or FTP location, otherwise the log files can bedeleted from the Web Reporter Systems Hard Disk. (Keep in mind that dependingon the Auto Deletion setting on your Web Gateway Appliance the originalLogfiles are still stored on the Web Gateway Server).

                     

                    Optional:Importing Logfiles from an Intermediate FTP System:

                    Some customers prefer to have a stagedLogfile Archive Server (e.g. a FTP Server) where Logfiles are pushed from theMcAfee Web Gateway to in first place to store and archive the original Logfilewith all its details to comply with local regulations. The Logfiles are thenusually pulled by the McAfee Web Reporter from that intermediate FTP Server. Inthis case select “Collect Log files from FTP Server and configure accordingly.

                     

                    1.   Setting up Web Gateway 7 topush Log Data to the Web Reporter System

                     


                    It is required to have the Log file structure set to factory defaults.

                    The Logfile Structure is configured underPolicy > Rule Sets > Log Handler and then open the property Screen of the“Wrote Access.log Rule and go to the Events Section of that rule, as shown inFigure 7

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                    Figure7: Logfile Structure Setting for the Access Log.

                    The Following Strings must be included:

                    Figure 8: Default Log FileDefinition

                    Please also make sure that the LogfileHeader is set correctly under Policy > Settings > File SystemLogging > Access Log Configuration.

                    Figure 9: Access Log Header Linedefinition

                     

                    The following fields must be specified:

                    time_stamp "auth_user"src_ip status_code "req_line" "categories""rep_level" "media_type" bytes_to_client"user_agent" "virus_name" "block_res"

                     

                    On the same screen (scroll down a littlebit) you will find the settings to rotate and push the Access.log. Expand the “Settingsfor Rotation, Pushing and Deletion” Section and check the “Enablespecific settings for user defined log” checkbox.

                     

                    It is a good idea to have the Logfrequently rotated and then pushed to the Web Reporter Server via FTP. Figure10 shows the settings where you will need to change the IP Address192.168.178.111 to your Web Reporter Server IP. For initial testing the Logfilecan rotated every 1 Megabyte, for production environments it is recommend toselect a higher value (like 10 Megabytes for smaller companies up to 1000 usersand 100 Megabytes or higher for large enterprises).

                     

                    Figure 10: Rotation and Log File pushingproperties of the access.log File.

                     

                     

                     

                     

                     

                     

                    2.   Troubleshooting

                     


                    In case the Web Reporter is not populated with data as expected it is a goodidea to check the following:

                     

                     

                    1. Has there been enough data collected on the Web Gateway Access Logto perform a log file rotate and push event? Check the log file size as shownin figure 11.

                     

                    Figure 11: viewactive and rolled log files. In this case the active log has not been rolledsend to the Web Reporter yet, as it has not reached the configure 1 Megabytesize limit yet.

                     

                     

                    1. Can the Web Gateway Appliance reach the Web Reporter Server? Checkany intermediate Firewalls for logged block events.

                     

                    Note: Some Firewall will blockthe FTP traffic originating from the Web Gateway appliance as it uses highsource ports. In this case you might need to change the Firewall rule from anFTP type rule to a generic TCP-Port based rule.

                     

                     

                    1. Check the “Jobs” Screen on the Web ReporterUser Interface is the Import Jobs are appearing (so data is arriving at the WebGateway) as shown in Figure 12.

                     

                    Note that a minor “Error Rate” in most cases is acceptable as WebReporter will truncate extremely long URLs and count this as an error in theJob Screen.

                     

                    Figure12: Import Jobs Result Screen

                     

                     

                    1. Check The “Log File Parsing” Log on the Web Reporter User

                     

                    “Administration> Tools > Logging” and then select “LogParsing” form the drop down. This file might contain additionalInformation that is required by the Technical Support.

                     

                     

                    1. Check the “mwg-logmanager.errors.log” that is (if errors occur) will be written to the mwg-errors folder

                     

                    Figure 13:mwg-logmanager.errors.log

                     

                    ‘Curl’ the tool that is used to push thelog files will give you more details on problems like:

                    Figure 14: detailed error output inside themwg-logmanager.errors.log