cancel
Showing results for 
Search instead for 
Did you mean: 
maitane
Level 7

Problems with Web Reporter log jobs

Hi,

Since some days ago we´re noticing that Web Reporter is not processing log jobs properly. It takes several time to process each of the jobs (actually more than an hour for just one 100Mb job). This is causing that hundreds of jobs are waiting to be processed and so our MWG are alerting us that they can´t push the logs.

Also MWG are alerting us that /opt has exceded the limit  (Screenshot attached)

This is part of the server.log file:

2012-01-15 00:00:03,562 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:141)

2012-01-15 00:00:03,562 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:112)

2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.discoverServer(NamingContext.java:1265)

2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1402)

2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:596)

2012-01-15 00:00:03,562 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:589)

2012-01-15 00:00:03,562 ERROR [STDERR]  at javax.naming.InitialContext.lookup(InitialContext.java:392)

2012-01-15 00:00:03,562 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerConnection.doLoginInternal(Unknown Source)

2012-01-15 00:00:03,562 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger.invoke(Unknown Source)

2012-01-15 00:00:03,562 ERROR [STDERR]  at $Proxy121.doLoginInternal(Unknown Source)

2012-01-15 00:00:03,562 ERROR [STDERR]  at securecomputing.smartfilter.server.util.internal.InternalUseOnly.createAdminServer(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.server.project.services.runReport.MasterRunReportTask.getScheduledReport(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.server.project.services.runReport.MasterRunReportTask.init(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool.runIt(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerEngine.runIt(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.Scheduler.runIt(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer.checkEventTasks(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SPoller.poll(Unknown Source)

2012-01-15 00:00:08,390 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SBasePoller.run(Unknown Source)

2012-01-15 00:00:08,390 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer] run task 'SystemMaintenanceTask' now.

2012-01-15 00:00:08,390 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool] Init with data=

2012-01-15 00:00:08,390 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] before method doLoginInternal

2012-01-15 00:00:08,390 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@9bc5db

java.lang.OutOfMemoryError: Java heap space

2012-01-15 00:00:08,406 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] after method doLoginInternal

2012-01-15 00:00:11,906 ERROR [STDERR] java.lang.NoClassDefFoundError: Could not initialize class java.net.PlainDatagramSocketImpl

2012-01-15 00:00:11,906 ERROR [STDERR]  at java.lang.Class.forName0(Native Method)

2012-01-15 00:00:11,906 ERROR [STDERR]  at java.lang.Class.forName(Class.java:169)

2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.DatagramSocket.createImpl(DatagramSocket.java:297)

2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.DatagramSocket.<init>(DatagramSocket.java:209)

2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:141)

2012-01-15 00:00:11,906 ERROR [STDERR]  at java.net.MulticastSocket.<init>(MulticastSocket.java:112)

2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.discoverServer(NamingContext.java:1265)

2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1402)

2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:596)

2012-01-15 00:00:11,906 ERROR [STDERR]  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:589)

2012-01-15 00:00:11,906 ERROR [STDERR]  at javax.naming.InitialContext.lookup(InitialContext.java:392)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerConnection.doLoginInternal(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger.invoke(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at $Proxy121.doLoginInternal(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.util.internal.InternalUseOnly.createAdminServer(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.server.project.services.systemMaintenance.SystemMaintenanceTask.init(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool.runIt(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerEngine.runIt(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.Scheduler.runIt(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer.checkEventTasks(Unknown Source)

2012-01-15 00:00:11,906 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SPoller.poll(Unknown Source)

2012-01-15 00:00:11,921 ERROR [STDERR]  at securecomputing.smartfilter.common.core.util.scheduler.poller.SBasePoller.run(Unknown Source)

2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerTimer] run task 'SystemMaintenanceTask' now.

2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.common.core.util.scheduler.SchedulerThreadPool] Init with data=

2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] before method doLoginInternal

2012-01-15 00:00:11,921 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@ccb8db

java.lang.OutOfMemoryError: Java heap space

2012-01-15 00:00:11,921 INFO  [securecomputing.smartfilter.server.gateway.interfaces.AdminServerMerger] after method doLoginInternal

Can anybody help us?

Thanks in advance.

Regards.

0 Kudos
4 Replies
sroering
Level 13

Re: Problems with Web Reporter log jobs

You are describing two issues.

1) Web Reporter is backed up on log parsing (looks like the service is barely alive because you ran out of memory) Any idea how long Web Reporter has been not working properly?

2) The Web Gateway out of disk space on /opt partition. The partition is 36GB, so that is a lot of log data. How many days of logs are on the appliance?  Can you remove any of the older logs?

Web Reporter doesn't need a lot of memory under normal circumstances, so it is important to figure out why it ran out of heap space. I would like to take a look at your backup.xml for problems, but I would not recommend posting it here because it would contain sensative information.  I recommend opening a service request with Technical Support so we can look at it properly.

You should kill the Web Reporter process (java.exe owned by SYSTEM account on windows), and then start the service. That should hopefully get the log parsing going again, but I would need to see the backup and server.logs to give you a proper diagnosis.

Toll Free:      +1.800.700.8328
International:  +1.651.628.1500
United Kingdom: +44 (0) 870 460 4755
Australia:      +61 1300 559 109

Support Portal: https://mysupport.mcafee.com

0 Kudos
maitane
Level 7

Re: Problems with Web Reporter log jobs

Hi again,

We´re working this way since a week ago aprox. and it is corresponded with the update to 7.1.0.5 version. I understand these are two different issues but, couldn´t be they related?  I mean, we´ve notice both issues at the same time.

We´re going to export the logs manually from the appliances and then we´re going to delete them.

Also we´re going to open a SR to try to find out the Web Reporter issue. If I have not  understand you wrong I sould send the server.log file and also the backup.xml file,
Isn´t it?

Thanks for your help.

0 Kudos
sroering
Level 13

Re: Problems with Web Reporter log jobs

Please send us these files with the ticket.

1) .../reporter/conf/backupauto/backup.xml

2) .../reporter/log/server.log*  (All the server logs so I can look at the history too)

3) directory listing of the files in the log parsing queue.

dir .../reporter/tmp/logparsing/processing/* > logqueue.txt

I'll watch for the ticket and take a look at the data once we have it.

0 Kudos
maitane
Level 7

Re: Problems with Web Reporter log jobs

Ok,

I´ve just opened the SR. The ticket number is 3-1936170762

I´m going to collect all the files and then I´ll send you them.

Thanks again for your quick help.

0 Kudos