I've been testing the duplication of a MWG upload/download process in terms of backgrounding the portion of the process that is time-intensive while releasing the original download/process to deliver content to/from the client. This is based on the discussion at
I've run into some interesting outcomes so I'd like to get verification of a couple of items.
1) Is it required that the ruleset that utilizes the property AntiMalware.MATD.IsBackgroundScan equals true to handle the initiation of the background process be a Top Level ruleset?
2) For the "Init Offline Scan" ruleset, with criteria Antimalware.MATD.InitBackgroundScan (#) equals <boolean>, what is the correct logic to activate the background/offline scan and is it expected that this rule will trigger the Error Handler? I have seen both true and false values referenced in the forum and I've also seen documentation about how to make specific errors result in a fail open situation..which suggests that that may be an expected outcome in certain situations.
Current situation/challenges/what I would like to accomplish:
- I have a monitoring rule set which exists after the standard Common Rules rule set and before the standard Gateway Anti-Malware rule set
- In some cases, this monitoring rule set may result in significantly delayed delivery of downloads to the requesting client
- I want to implement logic that will make it possible to achieve:
- complete traversal of existing monitoring ruleset as that ruleset contains rulesets/rules which log specific data about Embedded Objects
- limited direct/perceived impact to the requesting client
- My thought was that I could potentially accomplish that by using the MATD properties for background scanning to first trigger a background "scan" event and then catch that traffic at the top of the overall rule set and shovel it into a monitoring rule set
Any input on items #1 and #2 listed above would be greatly appreciated. Also, if there are version-specific considerations, that would also be useful to know.
1) It is indeed required, and I suggest putting it towards the top of your ruleset. "IsBackgroundScan" is a special property that will only be set to true when MWG itself, sent a transaction to the background -- so there is no risk of moving it there. You should really only have one "Handle Offline Scan" ruleset.
2) (from my experience) You will only encounter the error handler if you do not have a corresponding "Handle Offline Scan" ruleset in place at the top of your ruleset. Also, you need to ensure that the overall structure of the rules is similar to the original (i.e. the last rule in the "Handle Offline Scan" ruleset is set to block).
When you say "monitoring", I think "monitoring the MWG's health", but in the context of your goals, it sounds like your doing something different.
To simplify, are you trying to send a potentially big download to an Offline Scan and bypass GAM and Composite Opener? Or am I misunderstanding?
Another interpretation might be that you have a suspect user that you are monitoring and when under "monitoring" this user has special rules that apply.
I seem to have this partially working, but not completely. I have succeeded in eliminating the invocation of the error handler when the test connection (which has embedded objects) hits the Offline Scanning With Immediate File Availability rule, but the top level rule that has criteria of Antimalware.MATD.IsBackgroundScan equals true isn't activating (according to rule engine tracing output). Since all indications are that this concept is successful in other environments, I probably need to test against a simplified configuration.
For context, my core configuration has two top level rulesets -- a Debug ruleset and a Main ruleset. All user functionality (with the exception of enabling rule engine tracing) is handled in the Main ruleset. In my testing against a representation of our production ruleset, I added MATD - Handle Offline Scan as a top level ruleset after Debug, but before Main. The Init Offline Scan ruleset was added to the overall Main ruleset immediately prior to where the original Monitoring ruleset would have been invoked.
The Monitoring ruleset I mentioned is not associated with MWG system health monitoring. The focus of this ruleset is to facilitate the generation of log events associated with Embedded Object processing. This ruleset exists between the "default" Common Rules and Gateway Anti-Malware rulesets. From a MWG processing standpoint, this means that Enable Composite Opener has been invoked when this ruleset is traversed. The overall ruleset includes multiple rulesets with various criteria defined to determine whether or not each ruleset is traversed and log events are generated @ this point (rather than during the Logging Cycle).
There are some specific situations where the invocation of a ruleset in the overall Monitoring ruleset will significantly degrade performance from the client's standpoint. For context, a 1.1GB download of a compressed (zip) upgrade file which contains hundreds of thousands of Java class files will download to the proxy in < 10 minutes, but the Monitoring ruleset processing will delay overall delivery so that client receipt of the complete file takes nearly an hour. I could put a bypass in place for specific downloads or count the # of times we traverse the Embedded Object cycle and potentially bail out once a certain number is reached, but what I would prefer to do is to provide the file to the client in a timely manner, but also generate log events for the embedded objects by having the original download/connection traverse the Monitoring ruleset. The generated log events are conveyed to the enterprise SIEM for review by our security operations team and there are other mitigating controls in place should a specific download be identified as malicious after it has reached the client.
I think I need to see this one. Can you remind of of an SR you've opened in the past? I can give you a call if I have that.