7 Replies Latest reply on Sep 23, 2013 1:36 AM by feickholt

    Rate limiting

    fwmonitor

      Hi all,

       

      how to restrict a user to 6000 requests/minute but allow short request bursts? NumberOfConnections doesn't suit well because it shows only a current value (which I want to allow to be high during short peaks).

       

      What about PStorage?

       

      regards

        • 1. Re: Rate limiting
          asabban

          Hello,

           

          I don't think PDStorage will work. You can increment a counter in PDStorage for every request a user does, but if you set the lifetime of PDStorage to 1 minute the counter will be reset only if there was no rule executed looking into PDStorage, otherwise the lifetime will be reset so the values from PDStorage won't be deleted.

           

          All I think that could work would be a custom statistics counter. You increase the counter for every request that comes in from that user. In front of increasing the counter you place a rule that blocks if the counter is larger than 6000. Every minute you call another rule that resets the counter to 0.

           

          This will limit the number of requests per minute to 6000, but it will not work like "6000 requests in the LAST 60 seconds"... if you make 6000 requests at 09:48:59, the counter resets at 09:49:00 and you make another 6000 requests at 09:49:10 you have 1200 requests within a minute, however you cannot do more than 6000 requests betweeb 09:49:00 and 09:50:00. I am not sure if this is suitable.

           

          ... and I strongly recommend to do such a thing only for one or a handful of users. A counter needs to be created individually for each user, so you don't want to use this globally. If you are looking for a global solution then I think this is not possible at the moment.

           

          Best,

          Andre

          • 2. Re: Rate limiting
            fwmonitor

            PD Storage can be emptied completely by Incident.5 every minute or every 10 minutes, could it be a way to implement this functionality?

             

            on every request the number of request will be checked for client.ip and it more than 6000 the request will be blocked.

             

            I assume the type should be a map/hash to be able to store IP+Count pair, right?

             

            on 09.09.13 04:16:31 CDT
            • 3. Re: Rate limiting
              feickholt

              >I don't think PDStorage will work. You can increment a counter in PDStorage for every request a user does, but if you set the lifetime of PDStorage to 1 minute >the counter will be reset only if there was no rule executed looking into PDStorage, otherwise the lifetime will be reset so the values from PDStorage won't be >deleted.

               

              We've sucesfull implemented such counting using PDs Storage. We store frist time the user requests the proxy in a local PDs variable. Then we

              incr a counter for each request. Before any new request we check the time and if the time is older than one minute we look in to the counter. If it's exceed

              a predefined value the client will get block page and it will be locked for 15 minutes. After 15 minutes he is able to use the proxy again.

              In the block page the user will be informed about the issue.  (in most case the user will be blocked in case of missconfigured nokia software trying to get

              it's updated.

              This works great for a cluster with 20000 concurrent users. (We use 7.2.... so we do not use the global PDs since the variabel does not syncronize in the right way.

               

              Frank

              • 4. Re: Rate limiting
                fwmonitor

                Hi Frank,

                 

                 

                very interesting!

                feickholt schrieb:

                 

                We store frist time the user requests the proxy in a local PDs variable. Then we

                incr a counter for each request.


                 

                 

                So you store for each user a time of the first request and a count. How you update a timestamp? Or you leave it expire as whole using PDs Timeout value?

                 

                Which type do you use for PDs: List or Number?

                 

                on 09.09.13 05:00:00 CDT
                • 5. Re: Rate limiting
                  feickholt

                  Yes we set the time the first time we saw a reqeust. We reset the time and counter in the first request we saw after 1 minute.

                   

                  We use 3 pds vor each user. Time (number), Cnt (number) and last url (string)

                  • 6. Re: Rate limiting
                    fwmonitor

                    Frank I think I've got it. Is it what you suggested? Here is a simplified ruleset:

                    Rate_Limiting.png

                    1 of 1 people found this helpful
                    • 7. Re: Rate limiting
                      feickholt

                      Yes this is what we use in a short form. We also log such events and add a list for clients we should  not monitor.

                      But yes this is essential configuration!

                       

                      One thing! Be sure that the <Request> definitions for PDS lifetime is at least 3 days.... Otherwise you might have performance problems on mondays. (only in large environments with more than 20000 CLients and runtime groups with more than 8  proxies...).

                       

                      If you're interested in this issue you can contact me using email.

                       

                      Thanks

                      Frank