4 Replies Latest reply on Jan 16, 2012 9:17 AM by maitane

    Problems with Web Reporter log jobs

    maitane

      Hi,

       

      Since some days ago we´re noticing that Web Reporter is not processing log jobs properly. It takes several time to process each of the jobs (actually more than an hour for just one 100Mb job). This is causing that hundreds of jobs are waiting to be processed and so our MWG are alerting us that they can´t push the logs.

       

      Also MWG are alerting us that /opt has exceded the limit  (Screenshot attached)

       

      This is part of the server.log file:

      2012-01-15 00:00:03,562 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:141)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:112)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.discoverServer(NamingContext.java:1265)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1402)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:596)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:589)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at javax.naming.InitialContext.lookup(InitialContext.java:392)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerConnection.doL oginInternal(Unknown Source)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger.invoke( Unknown Source)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at $Proxy121.doLoginInternal(Unknown Source)

      2012-01-15 00:00:03,562 ERROR [STDERR]  at securecomputing.smartfilter.server.util.internal.InternalUseOnly.createAdminSer ver(Unknown Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.server.project.services.runReport.MasterRunReportTa sk.getScheduledReport(Unknown Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.server.project.services.runReport.MasterRunReportTa sk.init(Unknown Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool.runI t(Unknown Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerEngine.runIt(Un known Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.Scheduler.runIt(Unknown Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer.checkEven tTasks(Unknown Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SPoller.poll(Unkn own Source)

      2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SBasePoller.run(U nknown Source)

      2012-01-15 00:00:08,390 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer] run task 'SystemMaintenanceTask' now.

      2012-01-15 00:00:08,390 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool] Init with data=

      2012-01-15 00:00:08,390 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] before method doLoginInternal

      2012-01-15 00:00:08,390 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@9bc5db

      java.lang.OutOfMemoryError: Java heap space

      2012-01-15 00:00:08,406 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] after method doLoginInternal

      2012-01-15 00:00:11,906 ERROR [STDERR] java.lang.NoClassDefFoundError: Could not initialize class java.net.PlainDatagramSocketImpl

      2012-01-15 00:00:11,906 ERROR [STDERR]  at java.lang.Class.forName0(Native Method)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at java.lang.Class.forName(Class.java:169)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.DatagramSocket.createImpl(DatagramSocket.java:297)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.DatagramSocket.<init>(DatagramSocket.java:209)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:141)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:112)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.discoverServer(NamingContext.java:1265)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1402)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:596)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:589)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at javax.naming.InitialContext.lookup(InitialContext.java:392)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerConnection.doL oginInternal(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger.invoke( Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at $Proxy121.doLoginInternal(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.util.internal.InternalUseOnly.createAdminSer ver(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.project.services.systemMaintenance.SystemMai ntenanceTask.init(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool.runI t(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerEngine.runIt(Un known Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.Scheduler.runIt(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer.checkEven tTasks(Unknown Source)

      2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SPoller.poll(Unkn own Source)

      2012-01-15 00:00:11,921 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SBasePoller.run(U nknown Source)

      2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer] run task 'SystemMaintenanceTask' now.

      2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool] Init with data=

      2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] before method doLoginInternal

      2012-01-15 00:00:11,921 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@ccb8db

      java.lang.OutOfMemoryError: Java heap space

      2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] after method doLoginInternal

       

      Can anybody help us?

       

      Thanks in advance.

      Regards.

        • 1. Re: Problems with Web Reporter log jobs
          sroering

          You are describing two issues.

           

          1) Web Reporter is backed up on log parsing (looks like the service is barely alive because you ran out of memory) Any idea how long Web Reporter has been not working properly?

           

          2) The Web Gateway out of disk space on /opt partition. The partition is 36GB, so that is a lot of log data. How many days of logs are on the appliance?  Can you remove any of the older logs?

           

          Web Reporter doesn't need a lot of memory under normal circumstances, so it is important to figure out why it ran out of heap space. I would like to take a look at your backup.xml for problems, but I would not recommend posting it here because it would contain sensative information.  I recommend opening a service request with Technical Support so we can look at it properly.

           

          You should kill the Web Reporter process (java.exe owned by SYSTEM account on windows), and then start the service. That should hopefully get the log parsing going again, but I would need to see the backup and server.logs to give you a proper diagnosis.

           

           

          Toll Free:      +1.800.700.8328
          International:  +1.651.628.1500
          United Kingdom: +44 (0) 870 460 4755
          Australia:      +61 1300 559 109

          Support Portal: https://mysupport.mcafee.com

          • 2. Re: Problems with Web Reporter log jobs
            maitane

            Hi again,

             

            We´re working this way since a week ago aprox. and it is corresponded with the update to 7.1.0.5 version. I understand these are two different issues but, couldn´t be they related?  I mean, we´ve notice both issues at the same time.

             

            We´re going to export the logs manually from the appliances and then we´re going to delete them.

             

            Also we´re going to open a SR to try to find out the Web Reporter issue. If I have not  understand you wrong I sould send the server.log file and also the backup.xml file,
            Isn´t it?

             

            Thanks for your help.

            • 3. Re: Problems with Web Reporter log jobs
              sroering

              Please send us these files with the ticket.

               

              1) .../reporter/conf/backupauto/backup.xml

              2) .../reporter/log/server.log*  (All the server logs so I can look at the history too)

               

              3) directory listing of the files in the log parsing queue.

              dir .../reporter/tmp/logparsing/processing/* > logqueue.txt

               

              I'll watch for the ticket and take a look at the data once we have it.

              • 4. Re: Problems with Web Reporter log jobs
                maitane

                Ok,

                 

                I´ve just opened the SR. The ticket number is 3-1936170762

                I´m going to collect all the files and then I´ll send you them.

                 

                Thanks again for your quick help.