Say that I have a wildcard list called FOO.
In FOO, I have regex URL.Raw values like
I also have an IP range list called FOO IP.
Is it possible to configure MWG to check if the destination URL is in list FOO and if it is in list FOO, check if client IP is in list FOO IP.
I also have more wildcard lists -- BAR, ABC, DEF, GHI and more IP range lists called BAR IP, ABC IP, DEF IP and GHI IP.
Is it possible to configure MWG to look at all of the wildcard lists and then, based on which one there's a match in, identify that list and subsequently check if client IP is in list XYZ IP?
It seems like there's something that could work via a combination of ListOfString.MapInList or ListOfStringMapMatchInList plus string.replace, user-defined properties or List.LastMatches.
Rather than having to specifically define 5+ sets of rules, if I could define a single (or even a few) rule(s) that could handle all of the situations, that would reduce the rule set complexity and increase extensibility.
Using MapInList assumes there will be a 1:1 alignment positionally between 2 tables. they have to have the same numberof elements and the indexed position of them must line up exactly.
This is good for simple lookups like how i used to do country codes to country names on a block page:
|position||List 1||List 2|
|3||AE||AE:United Arab Emirates|
|5||AG||AG:Antigua and Barbuda|
|19||BA||BA:Bosnia and Herzegovina|
7.3 now has a MapType List that has a key column and a value column that has the same effect.
There are 2 challenges you face.
1) You need 2 lists that have the same number of elements and they must be 1:1 aligned.
You may think, ok i'll pad with duplicate entries.
2) you can't have duplicate entries.
3) maintining a pair of lists in this manner is really hard. You are going to have some helpdesk user adding entries to the bottom and mis-aligning the indexes and not knowing why things blow up. Trust me, when you are gone, no one will know why it's there. (just like most firewall rules i've seen over time)
In the case of lines 1 & 2, you cannot have duplicates. It won't let you.
Even if you could have duplicates, lines 4 & 5 would match on line 4 always as a key and not ever match on line 5.
"Is it possible to configure MWG to check if the destination URL is in list FOO and if it is in list FOO, check if client IP is in list FOO IP."
Is exactly what is accomplished by the condition:
A. URL matches in FOO
B. Client.IP is in range list FOO IP
If A is false, B never gets evaluated.
Yes, you need one line of rules for each pair of URL/IP combo. I suspect you are trying to get that whole process down to one rule. Commendable. I get it. But this isn't the use case for MapInList. (At least from what you described)
I think what you are trying to also accomplish is more like a database repository of lists.
You have a Client.IP and you want a list of sites they can go to.
You kinda want something like:
URL matches in list (SELECT allowedSites FROM whiteList WHERE client_ip=$Client.IP$)
One rule for everything, it's done. The database does the rest. Now you just have to write some workflow around getting rows into some tables. Wouldn't that be cool?
Well, you can do this. External Lists.
I've used them with web services. That is, you request a URL to a web server that may have a URL like:
The service returns a flat list of entries that you use in the rule.
You can have the source of the list as a web service as above, a file depostied on the MWG's local disk (XML usually, where you can use xPath to select the list elements), a LDAP query on remote server, or an external PostgrSQL database through the network, or a SQLite3 database on the local MWG disk.
This is the fundamental method of how our SaaS service uses MWG to apply multitenancy polices to user traffic. The rules are static for all the proxies, but the requests force lookups to external sources based on ip and user and groups. All of the self-service portal administration for each customer are just workflow to edit various lists in various tables. (there's a little more to it than that, but it's the basic design)
Intrigued yet? you should check it out.
Yes. I am intrigued.
Do you have any XML examples that you can share?
The problem I'm trying to solve is one where I have a bunch of groups of clients who are permitted access to different services, but I don't want to have rule after rule after rule of URL is in list FOO and client.ip is in list FOO IP.
An individual client may be permitted to multiple destinations so I figured if I could do a lookup against the destination URL and then use the name of whatever list that matched in, I could more dynamically handle the rules. I'd still have a bunch of lists, but the rule configuration wouldn't need to change.
Without external lists, the closest I was able to get to was:
Permit FOO, URL matches in list Permit URL FOO, Continue, Set User-Defined.Permit = Rules.CurrentRule.Name
Permit BAR, URL matches in list Permit URL BAR, Continue, Set User-Defined.Permit = Rules.CurrentRule.Name
Permit Access, Client.IP is in range list List.OfIPRange.ByName(User-Defined.Permit), Stop Rule Set
The .ByName(variable) function was the other way i was going to mention.
I think your biggest problem with external lists or .ByName is you are using Client.IP as a key.
If it were a username or a user group, that's an exact string match.
I can have a list called 'Allowed Sites: joe' with URLs they are allow to go to:
And the only condition I need after authentication is:
URL.Host matches in list List.OfWildcard.ByName('Allowed Sites: "+Authentication.Username)
The key is already known and can lookup right there.
But you kinda double indexing by seeing if the IP is in a IPRangeList first, then see if the Range applies to a URL List
I can't think of any other way to do it with ByName, than how you describe above.
You also are approaching it starting with the URL. I would think that starting with the Client.IP would be more logical/easier. I'll have to think about that.
(Word of caution with .ByName(variable). If you export a rule set that uses .ByName, the associated lists do not get exported with it because 'variable' is a soft reference. Importing the rule set into another configuration may not maintain referential integrity with the lists you intend to reference in 'variable'. this has bitten me before)
As for the External List example, i think you would need to create a web app to do this the right way. You feed it a Client.IP and it returns a list.
The app would have to have logic to do the range comparison and return the correct flat file list. I'm sure there would be a database involved on the web app's side, too that you would have to maintain.
I've probably run out of ideas other than implementing a real database.
Originally, I used client.ip as my key value, but I was looking at switching it around because the URLs in each URL list will be unique to each list, whereas the client IPs may be included in multiple lists.