3 Replies Latest reply on Nov 19, 2014 4:13 AM by povlhp

    LSASS high CPU

    povlhp

      I did my first install of Vulnerability Manager on Win 2008R2, with SQL 2008R2 SP1 Express.

       

      I have one issue, LSASS keeps eating 15-50% of the CPU on a 2 CPU system. No matter if it is scanning, generating reports etc. I installed a hotfix from Microsoft to fix the PBFDK2 slow hashing DLL. No difference.

       

      After I restarted and did 2 scans, I have spent 38 minutes idle, but 50+ minutes in lsass.exe. So something is wrong here. Is it the database ? The API ? What is causing continous user validations, and not keeping authentication state ? Any fixes to this ?

        • 1. Re: LSASS high CPU
          povlhp

          Had the server running over the weekend, and lsass was no longer CPU grabbing.

          I am now up to 3 CPUs.

           

          Started new scan, and right away, I have lsass eating around 30-33% CPU = 100% of one core. So it is the limiting factor in the scanning. There are lots of CPU left, but the single-threaded lsass is the bottleneck. Memory usage is 16MB, so we are not talking memory issues here.

           

          Is there any way to fix this problem ? Should I install an older database to get a worse but faster credentials hashes ? Or what is the solution ?

          Will there be a fix to recycle the connections, causing fewer validations ? Seems like a bad design to keep opening db connections, instead of keeping a pool.

          • 2. Re: LSASS high CPU
            povlhp

            Seems like I keep running into issues. Documentation clearly states that I can run the database server with Windows Only authentification. Yet the API Server fails to connect to the SQL server if I disable username/password authentification.

            • 3. Re: LSASS high CPU
              povlhp

              Found the solution to the LSASS bottleneck.

              Downgrade SQL server to SQL Server 2005 express. Seems like the issue is with the username/password validation against the SQL 2008 server is caused by slow PBKDF2 algorithm used to validate password hashes. This gives more security, and the whole idea of PBKDF2 is to have a slow password hash algorithm, to prevent attacks on the hashes. So by going to SQL 2005 I allowed for weaker faster password hashes in logons, and the bottleneck is removed.

               

              But, the main issue here is still, that Foundstone does so many validations towards the SQL server, instead of keeping a connection pool. The install documentation says I can disable mixed mode after installation, this implicitly means I can use integrated authentication, but this is not true.  At least there is no guide to what to to on the MVM side, only how to do on the SQL Server side. Integrated Auth (Kerberos) should also avoid the very slow hashes.

               

              So until a redesign, it seems like we are stuck with SQL 2005 to get any sort of performance.

               

              PS: I did find a "slow PBKDF2" hotfix from MSFT, but it did not help.