Thanks for the reply jacek. I'd seen that one but no disk space issues in this case.
It turned out to be an intermittent flakiness of one of the two internet providers that was only affecting a subset of sites from this location.
When the problematic provider was downed, everything went quickly back to normal. The GTI servers were being routed through the problematic (but not totally down) provider at the time and unfortunately that slowed all web gateways to a standstill. I'll be preparing documentation for the local team on how to disable GTI lookups should such a case impacting GTI server reachability occur again that doesn't otherwise impact internet reachability generally.
My "are you sure about this GTI server outage you're talking about affecting me, cus, this is the local ISP issue we just discovered and corrected?" followup to platinum was returned with an indication that "no, the GTI outage did not affect you." Which wasn't terribly reassuring, but disabling GTI was helpful when we needed it as we wound our way to isolating the issue with one of our ISP's.
Reachability of the GTI servers is really really important and by default there doesn't seem to be any automagic fail open on it. First time in 4-5 years though that I've seen it.
1 of 1 people found this helpful
MWG does fail-open for the transaction, but as a whole it will keep trying even if past transactions failed (because it reallllly wants to rate them URLs).
The default transaction timeout is 6 seconds, but was recently made configurable (7.6.2):
3 attempts * 2s timeout for each attempt = 6 seconds.
This setting is in the advanced section of the URL Filter settings.
This wont eliminate the "hey im talking to a dead ISP" problem , but it will reduce the slowness quite a bit.
Nice. Thanks Jon.