cancel
Showing results for 
Search instead for 
Did you mean: 
fwmonitor
Level 7

Rate limiting

Hi all,

how to restrict a user to 6000 requests/minute but allow short request bursts? NumberOfConnections doesn't suit well because it shows only a current value (which I want to allow to be high during short peaks).

What about PStorage?

regards

0 Kudos
7 Replies
asabban
Level 17

Re: Rate limiting

Hello,

I don't think PDStorage will work. You can increment a counter in PDStorage for every request a user does, but if you set the lifetime of PDStorage to 1 minute the counter will be reset only if there was no rule executed looking into PDStorage, otherwise the lifetime will be reset so the values from PDStorage won't be deleted.

All I think that could work would be a custom statistics counter. You increase the counter for every request that comes in from that user. In front of increasing the counter you place a rule that blocks if the counter is larger than 6000. Every minute you call another rule that resets the counter to 0.

This will limit the number of requests per minute to 6000, but it will not work like "6000 requests in the LAST 60 seconds"... if you make 6000 requests at 09:48:59, the counter resets at 09:49:00 and you make another 6000 requests at 09:49:10 you have 1200 requests within a minute, however you cannot do more than 6000 requests betweeb 09:49:00 and 09:50:00. I am not sure if this is suitable.

... and I strongly recommend to do such a thing only for one or a handful of users. A counter needs to be created individually for each user, so you don't want to use this globally. If you are looking for a global solution then I think this is not possible at the moment.

Best,

Andre

0 Kudos
fwmonitor
Level 7

Re: Rate limiting

PD Storage can be emptied completely by Incident.5 every minute or every 10 minutes, could it be a way to implement this functionality?

on every request the number of request will be checked for client.ip and it more than 6000 the request will be blocked.

I assume the type should be a map/hash to be able to store IP+Count pair, right?

on 09.09.13 04:16:31 CDT
0 Kudos
feickholt
Level 10

Re: Rate limiting

>I don't think PDStorage will work. You can increment a counter in PDStorage for every request a user does, but if you set the lifetime of PDStorage to 1 minute >the counter will be reset only if there was no rule executed looking into PDStorage, otherwise the lifetime will be reset so the values from PDStorage won't be >deleted.

We've sucesfull implemented such counting using PDs Storage. We store frist time the user requests the proxy in a local PDs variable. Then we

incr a counter for each request. Before any new request we check the time and if the time is older than one minute we look in to the counter. If it's exceed

a predefined value the client will get block page and it will be locked for 15 minutes. After 15 minutes he is able to use the proxy again.

In the block page the user will be informed about the issue.  (in most case the user will be blocked in case of missconfigured nokia software trying to get

it's updated.

This works great for a cluster with 20000 concurrent users. (We use 7.2.... so we do not use the global PDs since the variabel does not syncronize in the right way.

Frank

0 Kudos
fwmonitor
Level 7

Re: Rate limiting

Hi Frank,

very interesting!

feickholt schrieb:

We store frist time the user requests the proxy in a local PDs variable. Then we

incr a counter for each request.


So you store for each user a time of the first request and a count. How you update a timestamp? Or you leave it expire as whole using PDs Timeout value?

Which type do you use for PDs: List or Number?

on 09.09.13 05:00:00 CDT
0 Kudos
feickholt
Level 10

Re: Rate limiting

Yes we set the time the first time we saw a reqeust. We reset the time and counter in the first request we saw after 1 minute.

We use 3 pds vor each user. Time (number), Cnt (number) and last url (string)

0 Kudos
fwmonitor
Level 7

Re: Rate limiting

Frank I think I've got it. Is it what you suggested? Here is a simplified ruleset:

Rate_Limiting.png

feickholt
Level 10

Re: Rate limiting

Yes this is what we use in a short form. We also log such events and add a list for clients we should  not monitor.

But yes this is essential configuration!

One thing! Be sure that the <Request> definitions for PDS lifetime is at least 3 days.... Otherwise you might have performance problems on mondays. (only in large environments with more than 20000 CLients and runtime groups with more than 8  proxies...).

If you're interested in this issue you can contact me using email.

Thanks

Frank

0 Kudos