Web Gateway communication resilliency with ICAP server
I have a few questions for anyone who has experience with deploying a set of Web Gateways with a set of ICAP servers (for DLP scanning via reqmod). My scenario includes three DCs, (active/standby/standby) with 3 Web Gateways and 3 ICAP Servers (NDLP Prevents) per DC.
I am particularly interested in knowing the number of times a Web Gateway will retry connecting to each ICAP server defined in policy settings before returning a block page indicating Error ID 16000 or 16003 (Reqmod server unavailable/ICAP client filter error: Maximum connection limit reached).
In my testing, I have been unable to influence the Web Gateway to ICAP server connection retry/timeout behavior via the Timeouts settings in Configuration > Proxies
On the same note, I have tried adding a proxy control setting to my DLP with ICAP policy but that has no effect either. (Policy > Settings > Proxy Control > Created new entry for ICAP Connection Timeout)
There is an LTM in the environment which is responsible for an automated failover decision among the Data Centers.
My intention is to avoid consecutive failover among the DCs in a scenario where there is a spike in HTTP traffic (and associated ICAP traffic load on the NDLP Prevents). Rather than serve a block page based on ICAP server down or too busy (error id 16000/16003) Ideally, I would like to continue retrying the ICAP server for a minute or two before serving the block page. This may slightly degrade user experience, however it would add to the stability of the DLP filtering solution.
If none of this is possible, I guess the only other option is to obtain additional ICAP server capacity for each DC to handle potential traffic spikes.
Thanks in advance - I really look forward to hearing your feedback.