cancel
Showing results for 
Search instead for 
Did you mean: 
fuzziest
Level 9

Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

I'm trying to launch a JNLP file and keep getting an error

cannot-create-javavm.png

If I disable HIPS by unchecking the "Enable Host IPS" checkbox, the program launches successfully.

enable-host-ips.png

But if I keep "Enable Host IPS" checked, and instead I uncheck all the HIPS Engines (Help -> Troubleshooting -> Functionality), I still get the unable to create Java Virtual Machine error.

hips-engines.png

There are no HIPS events being triggered and there is nothing helpful that I can find indicating the problem in the HIPS log files even in Debug mode.

Any ideas on where else I can look to troubleshoot?

0 Kudos
5 Replies
greatscott
Level 12

Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

Open a ticket with McAfee. They will probably have you set all your McAfee default policies for IPS, but also with your Client UI policy set to debug. Reproduce issue, note the exact time, and run a MER.

0 Kudos
fuzziest
Level 9

Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

"They will probably have you set all your McAfee default policies for IPS"

Thanks, that helped resolve the problem (sort of). So I tried setting the IPS Rules policy to only the McAfee default and it still didn't run. So I did the opposite. I applied every policy EXCEPT for the McAfee Default policy and it worked!

Then I started looking for differences between the McAfee Default IPS signature settings and the Effective Policy when McAfee default wasn't applied. They were both the same. But I was only checking the signatures containing the word "java".

Then I checked the Application Protection Rules and searched for "java". The difference when applying the McAfee default and not applying it was that javaw.exe was not on the application protection list (only java.exe and javaws.exe were).

So for now, I'm going to have to assume it has something to do with IPS blocking javaw.exe. I even disabled all the signatures containing the word "java" but it still didn't run with the McAfee Default policy applied so it seems like protecting javaw.exe is causing the block.

0 Kudos
McAfee Employee

Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

Your IPS Rules and Trusted application policies should be running with McAfee Default policy, along with any custom policy, per McAfee Best Practices.  These are multi-slot policies. 

PD22894 - Host Intrusion Prevention 8.0 for ePO 4.5 ProductGuide

https://kc.mcafee.com/corporate/index?page=content&id=PD22894

Page37

Assigning multiple instances of the policy

Assigningone or more instances of the policy to a group or system in the ePolicyOrchestrator

SystemTree provides for single policy multi-purpose protection.

TheIPS Rules policy and the Trusted Applications policy are multiple-instancepolicies that can

havemore than one instance assigned. A multiple-instance policy can be useful foran IIS

Server,for example, where you might apply a general default policy, a server policy,and an

IISpolicy, the latter two configured to specifically target systems running as IISservers. When

assigningmultiple instances, you are assigning a union of all the elements in eachinstance of

thepolicy.

NOTE: The McAfee Default policy for both IPS Rules andTrusted Applications are updated when

content is update.McAfee recommends that these two policies always be applied to make sure

protectionas up to date as possible.

Check for Signature violations for Javaw.exe, once HIPS is injecting into that process.

KB67056 - Third-party application stops working or isimpaired after McAfee Host Intrusion Prevention is installed or content isupdated

https://kc.mcafee.com/corporate/index?page=content&id=KB67056

0 Kudos
fuzziest
Level 9

Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

Thanks KB67056 has some good info. I did most of what that article said to do. The main problem was that there was no info in the log files even in debug mode that I could find useful information from.

You make a good point in that we shouldn't be disabling the McAfee Default policy since that is the one that gets the updated content.

So first I'll try enabling the McAfee Default policy and add javaw.exe to the Trusted Application policy.

If that doesn't work, I will try to do like KB67056 says:

     Disable Signature 432

Message was edited by: fuzziest

Didn't work. I had 432 disabled, put java.exe, javaw.exe, and javaws.exe as trusted applications. It still gave me the could not create Java VM error. Then once I disabled McAfee Default IPS policy, it started working again. on 5/21/14 7:31:14 AM HST
0 Kudos
fuzziest
Level 9

Re: Unchecking "Enable Host IPS" allows program to run, but unchecking all HIPS engines doesn't

So this is what I had to do to get it running with the McAfee Default Policy applied.

I copied the javaw.exe entry from the McAfee Default policy application protection list tab to our custom policy for the group.

Then I modified the javaw.exe entry on the custom policy and set Inclusion Status to "Excluded".

javaw.png

Now it works.

It makes sense to do it this way, but it looks kind of funny when I view the "Effective Policy" for the group (shown in screenshot above).

Two of the same rule, but one shows Included and one shows Excluded.

I would expect it to only show the "effective" rule taking effect, which in this case would be the "excluded" one.

Otherwise, how would I know which rule is being applied?

Anyway, it works so I'm happy.

0 Kudos