6 Replies Latest reply on Dec 28, 2015 1:09 PM by penoffd

    9.5.1 Memory Leak on 5205 ESM Appliance?

    rbroom

      Has anyone seen a progressive memory leak on a 5205 ESM after upgrading to 9.5.1?

       

      On a typical day (averaging about 900 EPS) I can run out of RAM and swap in about 13 hours.  I'm mitigating it with "service dbserver restart", but I'm wondering if anyone else is experiencing this.  When I exhaust memory system performance falls of course, and some functions start failing (like GUI logins).

       

      Leak issues aside, I think performance going from 9.5.0 MR7 to 9.5.1 MR1 improved performance on my 5205.  I'm hoping other folks in the same boat are seeing that as well.

        • 1. Re: 9.5.1 Memory Leak on 5205 ESM Appliance?
          jdell

          I noticed an extremely slow performance after as well. I restarted the dbserver like you mentioned and it all came back with much better response time. Looks to be an issue with 9.5.1 MR1. I am running a virtual appliance.

          • 2. Re: 9.5.1 Memory Leak on 5205 ESM Appliance?
            pepelepuu

            rbroom I highly doubt you have a memory leak based on your description. I would usually suggest contacting technical support, but I'm presuming you might not have a contract with them.

            What you should keep in mind as you upgrade, to the newer versions is they new code is meant for faster, more capable processor, which I suspect may ultimately be your problem, and moving forward, with version 10.0 coming up it will ease the demand on your computing resources. 900 EPS is nothing for that device. With the updates, and upgrade you have new correlation rules, different database schema etc...So, you may be getting to the point where a device upgrade will be necessary. However, in the mean time, go through your rules and do some tuning.

            So I would be interested in viewing your /var/log/messages, /usr/local/ess/nitroerror.log.

            This being such an old model, it would be worth while to review the output from running getstats-healthstats.pl

            • 3. Re: 9.5.1 Memory Leak on 5205 ESM Appliance?
              rbroom

              Thanks @pepelepuu .  I've had a support ticket open for performance since 9.5.0 mr7.  As I mentioned above I think performance improved for me  to 9.5.1, however that ends as memory is used (8gb RAM and 4gb swap).

               

              I agree a likely issue is new code reducing performance of older appliances. I'd hoped to last another year (to EOL) before a forklift upgrade, and thought running under 20% of rated capacity (which is 5K EPS) might work.

               

              I like the thought that it's not memory related, I'm just going by what I'm seeing: everything working, memory going down, things stop working well or at all.

               

              I'll review the log data. Barring obvious errors and issues, can you suggest things to look for?  I'll share what I can. (Since someone  seems to have a similar issue.)

              • 4. Re: 9.5.1 Memory Leak on 5205 ESM Appliance?
                rbroom

                Update for folks tracking this: Intel released a patch for the memory leak.  It's in 9.5.1 MR2, which dropped last week.  I've just completed my upgrade and already see an improvement.  The swap consumption rate is way down.  I'll need to run for several days to see if the slow rise levels out somewhere, but I have to say the GUI responsiveness is MUCH better.

                 

                If I get a chance tomorrow during peak, I'll try to capture the output of getstats-healthstats.pl.  I didn't do it before the upgrade, but I expect these numbers to be better than they would have been.  Perhaps it will be useful for someone to compare to their own system.

                • 5. Re: 9.5.1 Memory Leak on 5205 ESM Appliance?
                  penoffd

                  We encountered what I suspect is a memory leak upon upgrading to 9.5.1,   Our ESM will come to a complete stop at peak times with memory totally maxed out.  I opened a ticket with support but have yet to see a response beyond having the ticket excalated.

                   

                  Stopping and starting the dbserver service will get things going again for a while.

                   

                  I'm going to patch/upgrade to MR2 tonight and see if that helps.

                  • 6. Re: 9.5.1 Memory Leak on 5205 ESM Appliance?
                    penoffd

                    Successfully patched our appliances to MR2 and all is well now.  No memory leaks and the system appears to be running well. Memory utilization is hanging around 50%-60% at peak times.

                     

                    Would have been nice if the left hand knew what the right hand was doing.  I had an open ticket on this that was escalated to Engineering, all the while the patch had already been released.  They need to communicate between the teams.