5 Replies Latest reply on May 21, 2014 1:23 PM by fuzziest

    Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

    fuzziest

      I'm trying to launch a JNLP file and keep getting an error

      cannot-create-javavm.png

       

      If I disable HIPS by unchecking the "Enable Host IPS" checkbox, the program launches successfully.

       

      enable-host-ips.png

       

      But if I keep "Enable Host IPS" checked, and instead I uncheck all the HIPS Engines (Help -> Troubleshooting -> Functionality), I still get the unable to create Java Virtual Machine error.

       

      hips-engines.png

      There are no HIPS events being triggered and there is nothing helpful that I can find indicating the problem in the HIPS log files even in Debug mode.

       

      Any ideas on where else I can look to troubleshoot?

        • 1. Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't
          greatscott

          Open a ticket with McAfee. They will probably have you set all your McAfee default policies for IPS, but also with your Client UI policy set to debug. Reproduce issue, note the exact time, and run a MER.

          • 2. Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't
            fuzziest
            "They will probably have you set all your McAfee default policies for IPS"

             

            Thanks, that helped resolve the problem (sort of). So I tried setting the IPS Rules policy to only the McAfee default and it still didn't run. So I did the opposite. I applied every policy EXCEPT for the McAfee Default policy and it worked!

             

            Then I started looking for differences between the McAfee Default IPS signature settings and the Effective Policy when McAfee default wasn't applied. They were both the same. But I was only checking the signatures containing the word "java".

             

            Then I checked the Application Protection Rules and searched for "java". The difference when applying the McAfee default and not applying it was that javaw.exe was not on the application protection list (only java.exe and javaws.exe were).

             

            So for now, I'm going to have to assume it has something to do with IPS blocking javaw.exe. I even disabled all the signatures containing the word "java" but it still didn't run with the McAfee Default policy applied so it seems like protecting javaw.exe is causing the block.

            • 3. Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't
              Kary Tankink

              Your IPS Rules and Trusted application policies should be running with McAfee Default policy, along with any custom policy, per McAfee Best Practices.  These are multi-slot policies. 

               

              PD22894 - Host Intrusion Prevention 8.0 for ePO 4.5 ProductGuide

              https://kc.mcafee.com/corporate/index?page=content&id=PD22894

               

              Page37

               

              Assigning multiple instances of the policy

              Assigningone or more instances of the policy to a group or system in the ePolicyOrchestrator

              SystemTree provides for single policy multi-purpose protection.

               

              TheIPS Rules policy and the Trusted Applications policy are multiple-instancepolicies that can

              havemore than one instance assigned. A multiple-instance policy can be useful foran IIS

              Server,for example, where you might apply a general default policy, a server policy,and an

              IISpolicy, the latter two configured to specifically target systems running as IISservers. When

              assigningmultiple instances, you are assigning a union of all the elements in eachinstance of

              thepolicy.

               

              NOTE: The McAfee Default policy for both IPS Rules andTrusted Applications are updated when

              content is update.McAfee recommends that these two policies always be applied to make sure

              protectionas up to date as possible.

               

               

               

              Check for Signature violations for Javaw.exe, once HIPS is injecting into that process.

               

              KB67056 - Third-party application stops working or isimpaired after McAfee Host Intrusion Prevention is installed or content isupdated

              https://kc.mcafee.com/corporate/index?page=content&id=KB67056

              • 4. Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't
                fuzziest

                Thanks KB67056 has some good info. I did most of what that article said to do. The main problem was that there was no info in the log files even in debug mode that I could find useful information from.

                 

                You make a good point in that we shouldn't be disabling the McAfee Default policy since that is the one that gets the updated content.

                 

                So first I'll try enabling the McAfee Default policy and add javaw.exe to the Trusted Application policy.

                 

                If that doesn't work, I will try to do like KB67056 says:

                 

                     Disable Signature 432

                 

                Message was edited by: fuzziest Didn't work. I had 432 disabled, put java.exe, javaw.exe, and javaws.exe as trusted applications. It still gave me the could not create Java VM error. Then once I disabled McAfee Default IPS policy, it started working again. on 5/21/14 7:31:14 AM HST
                • 5. Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't
                  fuzziest

                  So this is what I had to do to get it running with the McAfee Default Policy applied.

                   

                  I copied the javaw.exe entry from the McAfee Default policy application protection list tab to our custom policy for the group.

                  Then I modified the javaw.exe entry on the custom policy and set Inclusion Status to "Excluded".

                   

                  javaw.png

                   

                  Now it works.

                  It makes sense to do it this way, but it looks kind of funny when I view the "Effective Policy" for the group (shown in screenshot above).

                  Two of the same rule, but one shows Included and one shows Excluded.

                  I would expect it to only show the "effective" rule taking effect, which in this case would be the "excluded" one.

                  Otherwise, how would I know which rule is being applied?

                   

                  Anyway, it works so I'm happy.