7 Replies Latest reply on Nov 25, 2013 3:44 AM by asabban

    7.4.0: Bug with error message and DCC?

    cryptochrome

      Hi,

       

      I just tested the dynamic content classifier (DCC). I am dynamically blocking the category 'Pornography' in the response cycle. The blocking works nice, but the default error message is telling me this:

       

      http://d.pr/i/uDYL+

      As you can see in the last line of the error message, it is saying it was NOT blocked by DCC.

       

      The log file does in fact show that DCC has blocked.

       

      Is this a bug?

       

      Thanks

        • 1. Re: 7.4.0: Bug with error message and DCC?
          cryptochrome

          I might add: I am using the default error messages, no customizations. Stock 7.4.0 installation (upgraded from earlier versions).

          • 2. Re: 7.4.0: Bug with error message and DCC?

            Bug? I can't say. Wierd? definately.

            I haven't tried this in 7.4 yet, but have observed strangeness with 7.3.2.3.

             

            I think it has something to do with the cycles. When the request comes in first, in the request cycle, it is uncategorized.

            When the content comes in the response cycle, the content is categorized and if there is a block right there, then you should get  DCC.ResultComputedByDCC=true.

             

            However, URL.Host is then cached with that category for the next 10 or 15 minutes. Any subsequent requests won't do another lookup and will pull the results from cache.

            What i don't know is if the DCC.ResultComputedByDCC (and RequestSentToCloud) is set to true on subsequent requests when pulling from cache. I would have to test, once i get try it on 7.4.

             

            You say the logs tell you it does, but block page does not. I believe that because sometimes properties don't get sent to the block pages like you think they should. It is likely something to do with <MostRecentlyUsed> on subsequent requests that are cached.

             

            But to bring up another pet peeve of mine, it annoys me that you have to have 2 rulesets for DCC. The challenge is I want to DCC as many categories as i possible can and log the fact they are DCC'd...even the ones i would want to allow, like games or business.

            However, if i have some group exceptions, like GroupA is allowed to go to Games, but group B is not, i have to duplicate those rules in the DCC rule set.

             

            So, as a solution, i've worked out a way to accomodate that.

             

            URL Filtering

            [URL Filtering: The primary flow of blocking sites by categories.
            This rule set incorporates Dynamic Content Classification.
            If you do not want to use DCC, uncheck Response Cycle and disable the first 2 DCC rules.]

            Enabled
            Applies to: [] Requests [] Responses [] Embedded Objects
            Always
            EnabledRuleActionEventsComments
            Enabled DCC: Enable Dynamic Content Classification
            1: Cycle.Name equals "Response"
            2: AND (List.OfCategory.IsEmpty(URL.Categories<URL Filter: Default>) equals false
            3: AND DCC.ResultComputedByDCC<URL Filter: Default> equals false)
            Stop Rule SetWhen this rule is enabled, conted returned from site will be categorized by the DCC engine.
            If this rule is disabled, uncheck the Response Cycle checkmark in the URL Filtering rule set.
            Enabled DCC: X-DCC-Categories Header
            1: List.OfCategory.IsEmpty(URL.Categories<URL Filter: Default>) equals false
            2: AND DCC.ResultComputedByDCC<URL Filter: Default> equals true
            ContinueHeader.Block.Add("X-DCC-Categories",String.Concat(List.OfCategory.ToString(URL.C ategories<URL Filter: Default>),"(DCC)"))This is used in the log to determine if the category was computed by DCC. If DCC is disabled, then disable this rule as well.
            Enabled URL Filter: Allowed Domains or URL Filter: Allowed URLs
            1: URL.Host.BelongsToDomains(URL Filter: Allowed Domains°) equals true
            2: OR URL matches in list URL Filter: Allowed URLs°
            Stop Rule SetDomains or URLs in these lists are always allowed for everyone and not blocked by category.
            Enabled URL Filter: Blocked Domains or URL Filter: Blocked URLs
            1: URL.Host.BelongsToDomains(URL Filter: Blocked Domains°) equals true
            2: OR URL matches in list URL Filter: Blocked URLs°
            Block<URL Blocked>Statistics.Counter.Increment("BlockedByURLFilter",1)<Default>Domains or URLs in these lists are always blocked for everyone.
            Enabled SafeSearch: Strict and SafeSearch: Bypass URLs
            1: Cycle.Name equals "Request"
            2: AND URL does not match in list SafeSearch: Bypass URLs
            ContinueEnable SafeSearch Enforcer<SafeSearch: Strict>SafeSearch Enforcer is set to Strict.
            There are some rare instances where SafeSearch may need to be bypassed for specific URLs in order to fix some applications.
            Enabled GTI: Block HighRisk and MediumRisk Reputations
            1: Cycle.Name equals "Request"
            2: AND (URL.IsHighRisk<URL Filter: Default> equals true
            3: OR URL.IsMediumRisk<URL Filter: Default> equals true)
            Block<URL Blocked>Statistics.Counter.Increment("BlockedByURLFilter",1)<Default>Blocks urls with bad reputation.
            Enabled Exception #1: Categories for Exception #1: Users or Exception #1: Groups (e.g. SocialNetworking)
            1: URL.Categories<URL Filter: Default> at least one in list Exception #1: Categories
            2: AND (Authentication.UserName is in list Exception #1: Users
            3: OR Authentication.UserGroups at least one in list Exception #1: Groups)
            Stop Rule SetPeople in the Exception #1: Users or Groups lists will be allowed to these categories.
            SocialNetworking is just an example.
            Enabled Exception #2: Categories for Exception #2: Users or Exception #2: Groups (e.g. WebMail)
            1: URL.Categories<URL Filter: Default> at least one in list Exception #2: Categories
            2: AND (Authentication.UserName is in list Exception #2: Users
            3: OR Authentication.UserGroups at least one in list Exception #2: Groups)
            Stop Rule SetPeople in the Exception #2: Users or Groups lists will be allowed to these categories.
            WebMail is just an example.
            Enabled Exception #3: Categories for Exception #3: Users or Exception #3: Groups (e.g. Cloud Storage)
            1: URL.Categories<URL Filter: Default> at least one in list Exception #3: Categories
            2: AND (Authentication.UserName is in list Exception #3: Users
            3: OR Authentication.UserGroups at least one in list Exception #3: Groups)
            Stop Rule SetPeople in the Exception #3: Users or Groups lists will be allowed to these categories.
            Personal Network Storage is just an example.
            Enabled Exception #4: Categories for Exception #4: Users or Exception #4: Groups (e.g. Executives)
            1: URL.Categories<URL Filter: Default> at least one in list Exception #4: Categories
            2: AND (Authentication.UserName is in list Exception #4: Users
            3: OR Authentication.UserGroups at least one in list Exception #4: Groups)
            Stop Rule SetPeople in the Exception #4: Users or Groups lists will be allowed to these categories.
            The categories listed are just an example.
            Disabled Uncategorized: Block URLs
            1: List.OfCategory.IsEmpty(URL.Categories<URL Filter: Default>) equals true
            Block<URL Blocked>This rule allows all URLs that are uncategorized by the GTI web database.
            THIS IS NOT RECOMMENED.
            Enabled URL Filter: Override Blocked Categories
            1: URL.Categories<URL Filter: Default> at least one in list URL Filter: Override Blocked Categories
            Stop Rule SetCategories that override a block if site is in multiple categories.
            Enabled URL Filter: Default Blocked Categories
            1: URL.Categories<URL Filter: Default> at least one in list URL Filter: Default Blocked Categories
            Block<URL Blocked>Statistics.Counter.Increment("BlockedByURLFilter",1)<Default>This is the default list of categories that are blocked for everyone. Anyone needing access to these categories should be defined as a Stop Rule Set exception above this rule.


            The interesting thing to note here, is when a category gets DCC'd, it sets a block header for the block page to use, if needed.

            Header.Block.Add ("X-DCC-Categories", String.Concat (List.OfCategory.ToString (URL.Categories<URL Filter: Default>), "(DCC)"))

             

            Then i use this in the logs and block page:

             

            String.ReplaceIfEquals (String.ReplaceIfEquals (Header.Block.Get ("X-DCC-Categories"), "", List.OfCategory.ToString (URL.Categories<MostRecent>)), "", "-")

             

            This means:

            If X-DCC-Categories has something in it, display it, like "Games(DCC)"

            If it's empty ("") then display the URL.Categories.

            If both are empty, display "-"

             

            And the page displays as:

            URL Categories: Games(DCC)

             

            Another reason to use the Header.Block.Add instead of User-Defined variables, is so you can transport block pages between machines. Anybody who has ever tried it knows what i mean.

            • 3. Re: 7.4.0: Bug with error message and DCC?
              Jon Scholten

              Quick answer is, no this is not a bug.

               

              By default DCC is only supposed to apply to uncategorized URLs (so we can attempt to find a categorization). Playboy.com IS categorized, so DCC should not apply.

               

              Best,

              Jon

              • 4. Re: 7.4.0: Bug with error message and DCC?
                cryptochrome

                Jon, that is strange. Because DCC is definitely hitting for categorized URLs in my case. I have a very simple rulebase that contains absolutely no URL-filtering rules at all, with the exception of the default DCC ruleset. My requests to playboy.com are being blocked for pornography. The logfile (access_denied.log) prints the rule that blocked the request and it clearly states it was the DCC rule.

                 

                Am I missing something here?

                 

                Thanks!

                • 5. Re: 7.4.0: Bug with error message and DCC?
                  asabban

                  Hello,

                   

                  I am not sure but I think there is misunderstanding either on my side or on your side :-)

                   

                  If you look at the DCC setting there is a hint:

                   

                  2013-11-19 10_02_01-Edit Settings.png

                   

                  So even if you turn off all cetegorizations and ONLY turn on the DCC setting, MWG will STILL ask for the URL in the cloud. In the logs you will see that the categorization was performed by DCC, but "DCC only" in this case means "ask the cloud, use DCC is there is no category in the cloud". Since playboy.com is categorized in the cloud (I am very sure it is...) I think the detection as "Pornography" is coming from the cloud, not DCC.

                   

                  Best,

                  Andre

                  1 of 1 people found this helpful
                  • 6. Re: 7.4.0: Bug with error message and DCC?
                    cryptochrome

                    Hi Andre,

                     

                    that makes sense, thank you.

                     

                    However, this still seems inconsistent to me: If I only have the DCC rule, then only DCC should block, not a (categorized) cloud lookup result. I know this is hair splitting

                     

                    Cheers

                    Sascha

                    • 7. Re: 7.4.0: Bug with error message and DCC?
                      asabban

                      Hi Sascha,

                       

                      I was confused, too. So I agree with you :-) If you want to rely on DCC only (for testing or so) you can only prevent MWG from talking to GTI (or find an uncategorized URL) and try again. At the moment you will get Cloud + DCC when you use DCC only As far as I now the list of categories shown in the block page have an "*" attached when DCC did the categorization rather than the cloud lookup. Maybe that helps :-)

                       

                      Best,

                      Andre