1 2 Previous Next 10 Replies Latest reply: Jun 28, 2013 1:02 AM by feickholt RSS

    PDs experiences


      PDs analyse


      Hi Folks,

      During the last month I used more and more the PDStoragefeature, to implement a few feature currently not implemented in the WebGateway(concurrent user count /min, user count/day, block massive request from oneuser to one url....).  I like to share myexperiences about PDs, since I found not very much in information anddocumentation about such great feature.

      During my work I found some bugs (Global PDS will not be synchronized,

      PDS deletion using events does not work as expected –this might related to our Proxy environment currently 12 devices in Centralmanagement – 3 Cluster with 4 devices – 50000 users – large number ofPDsVariables ).

      Also I missed some features (example: Searching Variablesusing Regular expression...).

      First I wrote some Blockpages to analyze and dump thePDS, but this was very slow (currently we use about 80000 global variables and equivalentnumber local variables. ).

      Also using this block page I had only access to theglobal variables.

      I found no way to make a dump for the Local PDS.

      MC did not make me (sorry Andre) any hope that there willbe any GUI to manage the PDs in the near future.

      So I tried to analyze the content file stored in the filesystem using a little Perl script.

      I had no clue about the file structure, so I tried to analyzethe file using a hex dump.

      And 3 days later... yeah!!!! I got it! Now most of thestructure is clear to me.

      There a some little parts I don’t understand, but it’senough to dump and analyze the variables.


      I like to offer all of you a Perl module to dump thewhole PDs (Local and Global).

      You can search using regular expression for variables andvalues. This is VERY fast!

      Dumping our PDs (12MB) will take only about 3-4sec on ourproxy. The script can be easily installed on the proxy itself. (Just copy themodule and the Perl script in one director)


      Please feel free to contact me if you have anysuggestions or remarks.

      You are invited to find out the last unknown PDs values,also to write a gui for easier access….. J





      Here is an overview about the script possibilities.



      usage: ./PDs.pl

           -i<input>   Content file fromMcWebgateway

                               (can be found on/opt/mwg/plugin/data/bucketmap/content)

                               if you don't specify the file/opt/mwg/plugin/data/bucketmap/content will be used.

           -v              print MGW Version which haswritten the PDs

           -g              global PDs

           -l              local PDs

           -d             use readable Timeformat

           -s             remote site

                           this will copy the file to /tmpand analyse it.

                          Using -s will ignore -i

           -O<format>     OutputFormat

                       %PN  : PD Variable Name

                       %PV  : PD Variable Value

                       %PD  : PD Variable Expire Date

                          Default: "%PN = %PD"

           -r <regular>    Regular Expression to filter output

                          Filer applies on Variable name (Global and Local PDs) or Client IP(local PDs)


      This Software is BETA!

      You can omit r, if you omit -g and -l, -g will be used.


      Not all possible PDsData Types might be readable.

      Remarks and bugs can be send to frank@eickholt.com


      If you are interested please feel free to contact me. Ilike to share my experience

      Maybe someone can write a nice GUI using thisinformations.





      zfmwr|sgs|LAST_REQUEST = Fri, 12 Apr 2013 08:15:27 GMT

      imcst|sgs|LAST_HTTPREQUEST =http://www9.dict.cc/inc/dict.css?version=277

      phbdc|sgs|CLIENT_IP =

      sgori|bhc|LAST_REQUEST = Tue, 19 Mar 2013 08:10:37 GMT

      gbqxu|bhc|CLIENT_IP =

      wdiay|sgs|LAST_HTTPREQUEST = https://plus.google.com

      erotv|sgs|LAST_REQUEST = Thu, 21 Mar 2013 09:51:25 GMT

      wvlsf|sgs|LAST_HTTPREQUEST = http://config.messenger.msn.com/config/msgrconfig.asmx?op=GetOlcConfig

      wvfin|sgs|LAST_REQUEST = Fri, 12 Apr 2013 13:33:11 GMT

      gbvtr|sgs|CLIENT_IP =……


      #./PDs.pl -l | more :

      WebADs.activ = false :

      WebADs.activ = false :

      WebADs.activ = false :

      WebADs.activ = false :

      WebADs.activ = false :

      Request.url.blocked.reason = NONE

      Request.url.Cnt.Date = 1365774695

      WebADs.activ = false


      # ./PDs.pl -g-r "eick"

      eick|sgs|CLIENT_IP =

      eick|bhc|LAST_HTTPREQUEST =

      eick|sgs|LAST_REQUEST = Thu, 11 Apr 2013 15:39:31 GMT

      eick|bhc|CLIENT_IP =

      eick|sgs|LAST_HTTPREQUEST = https://webmailerng.1und1.de

      eick|bhc|LAST_REQUEST = Tue, 19 Mar 2013 12:02:00 GMT


      # ./PDs.pl -g -r "eick" -O "%PN (%PD):(%PV)"

      eick|sgs|CLIENT_IP (1368718771): (

      eick|bhc|LAST_HTTPREQUEST (1372248120):(

      eick|sgs|LAST_REQUEST (1368718771): (Thu, 11 Apr 201315:39:31 GMT)

      eick|bhc|CLIENT_IP (1372248120): (

      eick|sgs|LAST_HTTPREQUEST (1368718771):(https://webmailerng.1und1.de)

      eick|bhc|LAST_REQUEST (1372248120): (Tue, 19 Mar 201312:02:00 GMT)

      # ./PDs.pl -g -d -r "eick" -O "%PN (%PD):(%PV)"

      eick|sgs|CLIENT_IP (16.05.2013 15:39:31): (

      eick|bhc|LAST_HTTPREQUEST (26.06.2013 12:02:00):(

      eick|sgs|LAST_REQUEST (16.05.2013 15:39:31): (Thu, 11 Apr2013 15:39:31 GMT)

      eick|bhc|CLIENT_IP (26.06.2013 12:02:00): (

      eick|sgs|LAST_HTTPREQUEST (16.05.2013 15:39:31):(https://webmailerng.1und1.de)

      eick|bhc|LAST_REQUEST (26.06.2013 12:02:00): (Tue, 19 Mar2013 12:02:00 GMT)

        • 1. Re: PDs experiences

          This is EXCELLENT! Thank you!


          I also tried writing block pages to get PD Storage values out of MWG, but that didn't help for looking at items not related to my source IP.


          I have been slowly implementing PD Storage rules to deal with various situations, but I was concerned that at some point things would break and it would be hard to troubleshoot (and, as you know, the file structure is not easily examined) and I've been seeing periodic errors to the effect of:


          [BucketMapPlugin] [ParsingError] 'CBucketMapStorage::Parse': Error while parsing: 'Could not read user map! Will use default configuration!'.

          [BucketMapPlugin] [ParsingError] 'Parsing CUserMap from string': Error while parsing: 'Inserted bytesize is not equal with sent bytesize! Inserted bytesize: 564833 sent bytesize: 564813'.


          Hearing that you've got 50K users across 12 MWGs and are using PD Storage for multiple things gives me much more confidence in continuing to implement such rules.


          Currently using PD Storage in combination with coaching for certificates that are untrusted, unverified or expired and for sites that are uncategorized and unverified. Also testing using it as part of a Progressive Lockout rule set -- right now I am just tracking the # of blocked sites per user for a specific period of time.


          If you have any PD Storage rules (or ideas) that you can share, that would be great. If you don't want to post to the forum, feel free to send me a PM,

          • 2. Re: PDs experiences

            Here are some PDs examples we use

            - For each user we store their last Requestdate and Requested URL.

            Using the PD Script we are able to find out how many concurrent users use the proxy for last requested time (1h, 1day, 30 days)

            We use nagios to graph such  values.



            - we use the proxy to generate dynamic PAC -Files.

            Unfortunalty there  a some misconfigured firefox plugins which might fetch the pac file 1000 times/s. This results in a DoS. Now we use the PDS to count such request. And if there are to many requests in a minute we block the request to the pac file for a short time.


            -  The same mechanisem we use for misconfigured NOKIA suite. We require proxy authentication for any  internet access.

            If the suite has no or wrong credentials configured it tries to connect the ovi store with  >1000r/s. We count every repeated request.

            If this number is higher than a predefined value in 1 minute we blockt the internet access for the client for 15 minutes. 

            The client receives a block page with the information that he has to reconfigure the software and he is able to use the internet again in

            xx seconds.  Otherwise he will be blocked again. This is really great!!!!


            - We use the PDS to set also some variables if the user will see ADs or not. He is able to set this using a dedicated block page....



            This is all in place.

            And I have some other ideas which might be realized using PDs.





            • 3. Re: PDs experiences

              Here is an updated Version! The old script contains a bug.If there are only global variables the program failed in trying to find local PDs variables.

              (Thanks Pavel!)


              Nachricht geändert durch feickholt on 02.05.13 02:44:55 CDT


              Nachricht geändert durch feickholt on 02.05.13 05:43:42 CDT


              Nachricht geändert durch feickholt on 02.05.13 05:45:41 CDT
              • 4. Re: PDs experiences

                works like a charm, very helpful, thank you Frank!


                additionally I use Troubleshooting > PDStorage Troubleshooting > Log PDStorage Events


                Can somebody explain the difference between PDStorage.CleanUp und DeleteAllXXXData ?


                PDStorage.Cleanup                                  Cleans up persistently stored data.

                PDStorage.DeleteAllXXXData                 Deletes all permanently stored user data.

                • 5. Re: PDs experiences

                  good question....

                  since deleting PDStorage does not work for me i don' t know the difference....


                  Maybe MC can explain this.

                  • 6. Re: PDs experiences
                    Jon Scholten

                    Hi All,


                    To fwmonitor: Cleanup will look for data this is expired and remove it. DeleteAll will delete all global or user data (stored by IP or username).


                    To feickholt: How are you storing the data, in user storage or global? What event are you using to delete it?


                    Please consider this, PDStorage is an inactivity timeout, not a timeout based on when the data was inserted.


                    Meaning if you insert a value into PDstorage it will not necessarily be removed after the timeout defined in the settings is reached. It will only be removed if the value has not been accessed in the alloted time.


                    For more info see:





                    • 7. Re: PDs experiences

                      I know the timeout related things.


                      I know if I access the Variable again (read is enough) the timeout starts again.

                      But  our failure is also on our test CM. There is no traffic and no read access....

                      if i remove the Varible using remove all or remove var there

                      is no longer the variable variable. But 10minutes later they are all back... (I tried flush and all combinations).

                      I can see using my programm that the varibale is also away after next HD sync, but I have only to wait they will be back again. This is valid for local and global variables.

                      The variable is deleted after reaching the defined valid timeout. This works as expected.

                      There is a bug open about this issue.


                      I can live with that, but i would be easier if it all works as defined. So I have store timestamps also and use them to decide if a variable is still valid for me.




                      • 8. Re: PDs experiences
                        Jon Scholten

                        Hi Frank,


                        I'm a little confused by your description, but I think I get what you are saying.


                        The cluster sync seems to be causing issues with deleting the data from PDStorage? So you delete the data from one node (with PDStorage.DeleteGlobalData event), then the MWG's sync up, and the data is back?




                        • 9. Re: PDs experiences

                          Yes you are right. The curious thing is that the variable is delete already on the snapshort file on the MWG. It's back after the next sync.


                          For easier debugging I set PD Save interval and send interval to 0. (in our test Cluster - 2 WG5500)

                          Now I can verify using the script the data will be written direct to disk.

                          I also enable in troubleshooting for coordinator.

                          This shows me that every PD change will be propagated directly to the other node.


                          Nachricht geändert durch feickholt on 03.05.13 00:57:33 CDT
                          1 2 Previous Next