4 Replies Latest reply on Nov 22, 2016 10:52 PM by Regis

    GTI server outage ugliness

    Regis

      Saw an issue around 7:20pm central time (UTC -0600) tonight...

       

      Anyone else seeing a GTI server outage?

       

      Users reported antimalware engine errors (14002) Internal antivirus filter error.

       

      To recover,  support had me go into: 

      • Gateway antimalware rule,  wherever Gateway anti malware is called to get to Gateway anti-malware settings>  Advanced > Enable GTI file reputations and disabling the GTI file reputation lookups.
      • Any Url.categories(<default>) call  ... go in there and modify that default to uncheck "Use online GTI web reputation and categorization if local yields no results"

       

      temporarily while they work the issue.

        • 1. Re: GTI server outage ugliness
          jacek

          Did you check this thread? 14002 Internal antivirus filter error - Internal Anti-Malware Engine Error

          Please check if there is enough free space on partitions.

          • 2. Re: GTI server outage ugliness
            Regis

            Thanks for the reply jacek.   I'd seen that one but no disk space issues in this case.

             

            It turned out to be an intermittent flakiness of one of the two internet providers that was only affecting a subset of sites from this location. 

             

            When the problematic provider was downed, everything went quickly back to normal.    The GTI servers were being routed through the problematic (but not totally down) provider at the time and unfortunately that slowed all web gateways to a standstill.    I'll be preparing documentation for the local team on how to disable GTI lookups  should such a case impacting GTI server reachability occur again that doesn't otherwise impact internet reachability generally.

             

            My "are you sure about this GTI server outage you're talking about affecting me, cus, this is the local ISP issue we just discovered and corrected?"   followup to platinum was returned with an indication that "no, the GTI outage did not affect you."    Which wasn't terribly reassuring, but disabling GTI was helpful when we needed it as we wound our way to isolating the issue with one of our ISP's.

             

            Reachability of the GTI servers is really really important and by default there doesn't seem to be any automagic fail open on it.    First time in 4-5 years though that I've seen it.

            • 3. Re: GTI server outage ugliness
              Jon Scholten

              Hey Regis,

               

              MWG does fail-open for the transaction, but as a whole it will keep trying even if past transactions failed (because it reallllly wants to rate them URLs).

               

              The default transaction timeout is 6 seconds, but was recently made configurable (7.6.2):

               

              3 attempts * 2s timeout for each attempt = 6 seconds.

               

              This setting is in the advanced section of the URL Filter settings.

               

              This wont eliminate the "hey im talking to a dead ISP" problem , but it will reduce the slowness quite a bit.

               

              Best Regards,

              Jon

              1 of 1 people found this helpful
              • 4. Re: GTI server outage ugliness
                Regis

                Nice.  Thanks Jon.