8 Replies Latest reply on Oct 18, 2012 4:54 AM by btlyric

    Rule Order + Properties (7.x and Streaming Media)


      Looking for info about best practices for the order of rules. This question is specifically related to streaming media.


      For example, it's obvious that the implementation of a set of debug rules to do rule tracing should go at the top of the rule set and you need to do SSL Scanner before you go to looking at content categories, but here's the real world example that I was looking at tonight:


      Common Rules

           - caching rules

           - progress indication (data trickling)

           - enable opener

                - skip streaming (subset of later rules)

                - enable composite opener


      Gateway Anti-Malware

           - skip streaming


      While debugging an issue, rule tracing output indicated that although the composite opener cannot open certain types of  data, that data was still being passed on to the web caching, progress indication and enable opener rule sets.


      I think that the following logic makes sense, but if it doesn't, I need to know in the next 6 hours.


      Common Rules

           - stream detector rule set with top level criteria = Applies to "Responses"  that sets user-defined.streaming = true for traffic that meets specific criteria associated with streaming media in subordinate rules

           - caching rules with top level criteria user-defined.streaming does not equal true

           - progress indication with top level criteria user-defined.streaming does not equal true

           - enable composite opener with top level criteria user-defined.streaming does not equal true


      Gateway AntiMalware with top level criteria user-defined.streaming does not equal true

        • 1. Re: Rule Order + Properties (7.x and Streaming Media)



          I think, that these rules could work, there are only several comments:

          • not necessary to set user-defined property - if you'll call streaming detector with the same config as before, the cached results will be used, not calculated once again
          • how many rules are in your policy, that could be applied to streaming media? maybe it would be better to finish ruleset early?
          • 2. Re: Rule Order + Properties (7.x and Streaming Media)

            Thanks for the reply.




            We were advised by MFE professional services to add in stream detection rules before Enable Composite Opener and duplicate some of those in the Gateway Anti-Malware rule set.


            For clarification:


            Are you saying that if I have a rule that says something to the effect of:


            Skip Streaming Media, criteria:

            Cycle.Name equals "Response" AND StreamDetector.IsMediaStream<Default> equals true


            and that rule is activated


            then if I have a subsequent rule that is identical, the rule engine will use the cached results from the previous check and won't re-run its internal check?


            Also, If there are multiple initial rules that examine/determine the state, is it more efficient to set a user-defined property for future rules or to repeat the rules at each stage?


            I currently have 4 specific rules dedicated to the identification of  streaming media data -- on the surface it seems to me that it would be more efficient/less cumbersome from a configuration standpoint to run those rules once and set a property value so that I can then skip caching, data trickling, enable opener, certain types of monitoring and AV checks rather than adding those 4 rules into each rule set as a stop rule set. Or should we be using stop cycle?


            Another thing that I've been trying to figure out is when in the transaction process it is determined that Body.IsSupportedByOpener is true or false -- I was under the impression that that was triggered by attempting Composite Opener, but recent examination of rule tracing indicates that that is not the case.

            • 3. Re: Rule Order + Properties (7.x and Streaming Media)



              MWG caches results for calculation of individual properties (most, but not all). But you're right - when you have complex conditions, like you showed here (consisting from several parts), then user-defined properties could be a better choice.


              Regarding your last question - "Enable Composite Opener" doesn't triggers opening of file, it only specfies that file can be opened in future. Actual opening will happen when we'll ask for properties, like Body.Text, Body.IsSupportedByOpener, Body.IsCorrupted, etc., or when we're trying to switch to processing of embedded objects. This is done to prevent opening of files when we won't use any embedded objects from it, plus it gives us much better performance.

              • 4. Re: Rule Order + Properties (7.x and Streaming Media)

                Given a rule with criteria Body.IsSupportedByOpener equals false, will that trigger a check to try try to open a file?


                I'm trying to figure out why, when I have a set of defined rules that attempt to identify/classify Streaming Media, I'm hitting a subsequent monitoring rule that is looking at whether or not the opener is supported + user-defined.streaming does not equal true, seems to result in the transaction being logged to the specific log file I have configured to track specific file types/sizes/etc.


                Is there any available documentation about how connections traverse the rule set?


                Thanks in advance,



                • 5. Re: Rule Order + Properties (7.x and Streaming Media)

                  Yes, when we use Body.IsSupportedByOpener property, then we'll try to open the file. You can also use another property - MediaType.HasOpener - it will calculated during media type detection, and will be true if we have a registered opener for given mime type, and for calculation of this property we won't try to open file - this will much faster...


                  Regarding documentation - I don't remember right now, sorry...

                  • 6. Re: Rule Order + Properties (7.x and Streaming Media)

                    So, of course, your answer leads to another question from me...


                    If calling Body.IsSupportedByOpener (or MediaType.HasOpener) properties will result in an attempt to open the file, how does that differ from the Enable Composite Opener event?


                    I think this is related to how I ended up trying to track down how this works.

                    • 7. Re: Rule Order + Properties (7.x and Streaming Media)

                      "Enable Composite Opener" just sets the flag, that will allow opening of archives - it won't open it itself. Actual opening will be deffered until data will required.


                      MediaType.HasOpener property won't try to open file at all - this property is filled when we're performing media type detection, and in this case, we just getting media type for file, and then check, do this type is in the list of media types for which we have openers.


                      But Body.IsSupportedByOpener is different from MediaType.HasOpener because it will try to get opener for current file, and will try to initialize it. If initialization was performed without errors, then we say that we can open file. This property can be used to distinguish files for which we "have opener", but when the opener doesn't support some of algorithms. For example, we have opener for 'application/msword' file type, but we don't support processing of .doc files created by MS Word versions before Word 97 (MS Word 5, 6,...). Or, another example, we have opener for 'application/zip', but we don't support Shrink compression method yet, so we'll return Body.IsSupportedByOpener == false if your .zip archive will contain files compressed with this method

                      • 8. Re: Rule Order + Properties (7.x and Streaming Media)

                        Good explanation. Thanks. Now I just have to figure out why specific traffic is hitting a Body.IsSupportedByOpener rule when it shouldn't be doing so.