External list timeout

[slizka]

Hi,

Is there any way how to use external list in policy so that when it's not accessible the rule would be kind of skipped? Or better prevent "deleting" the list from proxy when the hosting webserver is not reachable?

Regards

Ales

[snoehler]

Hi Ales,

I think there are two different ways/options. Depending on your needs/possibilities:

1. Use a second server to test either just the connection to the hosting web server (ICMP or something) or if the actual file is still accessible.
- If this test fails, use the REST Interface to remove the list from the web gateway.
- If the hosting server comes back, push the list back to the web gateway

2. Use the Periodic Rule Engine Trigger combined with PDStorage. E.g.
- Let the Periodic Rule Engine Trigger test the connection to the file every 60 secs
- If the connection is successful: set the PDStorage.Property to true
- If the connection fails: set the Property to false
- Use the Property as a Criteria in the Rule where you are using the list

Note:
Periodic Rule Engine Trigger can be found under: Configuration >> Proxiest (HTTP(S), FTP, SOCKS, ICAP...) >> Scroll all the way down to "Advanced Settings" >> Periodic Rule Engine Trigger


What exactly are you trying to archive? Do you have any example usecase?
In case you would like to have any details on one of the above mentioned methods, let me know!

Best
Steffen

[slizka]

Hi Steffen,

Thanks for replying so quickly 🙂

It's just simple case that I'm trying to solve. Customer want to use external list of malicious domains that is hosted on some webserver. It's actually following URL https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist

But from time to time the server is down which then causes rule engine error because it's unavailable, so I want to store the list locally and keep using it until the proxy can connect there again.

The second option sounds to be better for me as I don't want to rely on some additional server, that might go down 🙂 I've tried to check MWG documentation, but there's quite nothing, at least nothing useful, about the PDstorage or the rule engine trigger. I'll later check also community to find out more about these features, but for now could you please help me setting the rules?

e.g. which property should I use to find out if the connection was successful? And which PDStorage property should be set to true/false?

Br. Ales

[snoehler]

Hi Ales,

I see. The best would be to subscribe to that list instead of using the External List Feature. This way you wont have any issues when the web server would be unreachable for a while.
The list will be saved on the web gateway and updated on a regular basis. However I have checked the content of the hosted list and unfortunatly the list is not in a proper format so that you can subscribe to it.

Since I think it's always better to use a subscribed list instead of an external one (perfomance perspective as well as availablity factor/risk to get blocked just because the external list is not available), I would suggest to:
-use an internal web server
-grab the list from the orginal source using a cronjob and modify the list to match a format the web gateway can use to subscribe

-host the proper list on the internal web server
-subscribe that list.

This way:
The list will be updated on a regular basis (hourly, daily, whatever you prefer);
And you won't have any issues once the external webserver hosting the list is not available for a few hours...

There are some other options to set this up, but they are really ugly and prone to errors..
But would it be possible for you to raise a service request and let me know the SR Number? Then I can pick this up and discuss this further and more detailed in order to find the best solution for you/your customer

Best
Steffen