This is possible, you'd need to have some sort of sentinel request going through the proxies that is monitored (watching the http status). If that sentinel request fails, then you enable the next hop proxy in the rules perhaps via PDStorage and based on the proxy location (to ensure location1 doesnt try to use itself as its own next hop).
Note: Moved out of Moderation Queue.
OK. That's what I already used in the past. This is very slow (wait for timeouts..) and I don't find a way to switch back if direct internet connection is back up again.
You can set a lower timeout for the "Sentinel" requests, that way if it ends in a timeout, it will die after say 10 seconds.
To determine if the connection is backup again, I would keep monitoring the sentinel requests, and reset the flag if they start working again.
I came up with a ruleset that checks "http://www.msftncsi.com/ncsi.txt ", if there is a 200 AND the body matches the expected response, then the test passes. If not, the test fails and a counter is incremented. If there is three failed tests, a flag is set in PDStorage to re-route traffic.
The check is continuous, and the flag is reset if the connection goes backup.
I came up with these rules pretty quickly. The rules to check the response code and body must go in the Log Handler because if there is a 502, then technically there is no response cycle.
The sentinel request is done using watch and wget:
The rules are written to work based on the system hostname, so it should scale for the cluster.
Let me know if you have any questions or concerns.