I have a couple of customers who use a variety of custom applications created internally. Customer is very cautious and want to test each and every custom application prior to enhanced mode so that user-end experience would be smooth.
I am adding the custom application to IPS - IPS Rules - Application Protection Rules. Watching in the log for any type of malicious hooking or activities someone is trying to perform on those applications.
Is this the right process I am following or there is another way to achieve this?
Funny thing is that most folks believe that host-based IPS in general and McAfee Host IPS specifically will have some really bad effects on custom written applications. At least for MFE Host IPS, this isn't generally true.
There are a couple of areas within the product itself that MIGHT cause conflicts:
1. Exploits (mostly in the High category) - Generally none of this is going to affect a custom application because, frankly, we're not watching their applications. However, if a customer writes an application that interfaces with an application that we do protect then there's some level of testing. For example, if you write an application that makes a suspicious call into say... Internet Explorer or Outlook then you'd need to test that those work.
2. Behavioral (mostly in the medium category) - These are triggered based on activities that are fairly well defined and are expressed in our literature as "Shielding and Enveloping". These can cause interference with other applications where there are boundaries. For example, executing applications directly from the temp directory or running an application directly from an outlook email.
These are the two most common areas of conflict but the reality is that these areas happen far less frequently than most folks think. I still think it is a fine thing to test new versions of applications and to run a pilot group of users with new software.
Just remember that, you're looking for two things when testing:
a) Was the end user experience interrupted in any way?
b) Was the function of the program actually compromised?
If you can't tell either of those things then you probably have an event that is just fine to block. These happen often and usually require no intervention or tuning. The port scan and Adobe scripting vulnerabilities come to mind. Usually these are blocked (from innocent causes) and no one and no program notices the API was stopped.