Skip navigation
McAfee Secure sites help keep you safe from identity theft, credit card fraud, spyware, spam, viruses and online scams
1863 Views 10 Replies Latest reply: Jun 27, 2013 5:55 AM by feickholt RSS 1 2 Previous Next
feickholt Apprentice 51 posts since
Nov 16, 2012
Currently Being Moderated

Apr 15, 2013 3:15 AM

PDs experiences

PDs analyse

 

Hi Folks,

During the last month I used more and more the PDStoragefeature, to implement a few feature currently not implemented in the WebGateway(concurrent user count /min, user count/day, block massive request from oneuser to one url....).  I like to share myexperiences about PDs, since I found not very much in information anddocumentation about such great feature.

During my work I found some bugs (Global PDS will not be synchronized,

PDS deletion using events does not work as expected –this might related to our Proxy environment currently 12 devices in Centralmanagement – 3 Cluster with 4 devices – 50000 users – large number ofPDsVariables ).

Also I missed some features (example: Searching Variablesusing Regular expression...).

First I wrote some Blockpages to analyze and dump thePDS, but this was very slow (currently we use about 80000 global variables and equivalentnumber local variables. ).

Also using this block page I had only access to theglobal variables.

I found no way to make a dump for the Local PDS.

MC did not make me (sorry Andre) any hope that there willbe any GUI to manage the PDs in the near future.

So I tried to analyze the content file stored in the filesystem using a little Perl script.

I had no clue about the file structure, so I tried to analyzethe file using a hex dump.

And 3 days later... yeah!!!! I got it! Now most of thestructure is clear to me.

There a some little parts I don’t understand, but it’senough to dump and analyze the variables.

 

I like to offer all of you a Perl module to dump thewhole PDs (Local and Global).

You can search using regular expression for variables andvalues. This is VERY fast!

Dumping our PDs (12MB) will take only about 3-4sec on ourproxy. The script can be easily installed on the proxy itself. (Just copy themodule and the Perl script in one director)

 

Please feel free to contact me if you have anysuggestions or remarks.

You are invited to find out the last unknown PDs values,also to write a gui for easier access….. J

 

Regards

Frank

 

Here is an overview about the script possibilities.

 

-----

usage: ./PDs.pl

     -i<input>   Content file fromMcWebgateway

                         (can be found on/opt/mwg/plugin/data/bucketmap/content)

                         if you don't specify the file/opt/mwg/plugin/data/bucketmap/content will be used.

     -v              print MGW Version which haswritten the PDs

     -g              global PDs

     -l              local PDs

     -d             use readable Timeformat

     -s             remote site

                     this will copy the file to /tmpand analyse it.

                    Using -s will ignore -i

     -O<format>     OutputFormat

                 %PN  : PD Variable Name

                 %PV  : PD Variable Value

                 %PD  : PD Variable Expire Date

                    Default: "%PN = %PD"

     -r <regular>    Regular Expression to filter output

                    Filer applies on Variable name (Global and Local PDs) or Client IP(local PDs)

 

This Software is BETA!

You can omit r, if you omit -g and -l, -g will be used.

 

Not all possible PDsData Types might be readable.

Remarks and bugs can be send to frank@eickholt.com

 

If you are interested please feel free to contact me. Ilike to share my experience

Maybe someone can write a nice GUI using thisinformations.

-----

 

Examples:

#./PDs.pl

zfmwr|sgs|LAST_REQUEST = Fri, 12 Apr 2013 08:15:27 GMT

imcst|sgs|LAST_HTTPREQUEST =http://www9.dict.cc/inc/dict.css?version=277

phbdc|sgs|CLIENT_IP = 10.123.253.74

sgori|bhc|LAST_REQUEST = Tue, 19 Mar 2013 08:10:37 GMT

gbqxu|bhc|CLIENT_IP = 10.108.117.151

wdiay|sgs|LAST_HTTPREQUEST = https://plus.google.com

erotv|sgs|LAST_REQUEST = Thu, 21 Mar 2013 09:51:25 GMT

wvlsf|sgs|LAST_HTTPREQUEST = http://config.messenger.msn.com/config/msgrconfig.asmx?op=GetOlcConfig

wvfin|sgs|LAST_REQUEST = Fri, 12 Apr 2013 13:33:11 GMT

gbvtr|sgs|CLIENT_IP = 172.18.96.148……

 

#./PDs.pl -l | more

10.135.4.230 :

WebADs.activ = false

10.135.4.229 :

WebADs.activ = false

10.135.4.227 :

WebADs.activ = false

10.135.4.216 :

WebADs.activ = false

10.135.4.201 :

WebADs.activ = false

10.165.65.1 :

Request.url.blocked.reason = NONE

Request.url.Cnt.Date = 1365774695

WebADs.activ = false

 

# ./PDs.pl -g-r "eick"

eick|sgs|CLIENT_IP = 10.185.222.42

eick|bhc|LAST_HTTPREQUEST = https://92.43.21.138

eick|sgs|LAST_REQUEST = Thu, 11 Apr 2013 15:39:31 GMT

eick|bhc|CLIENT_IP = 10.161.41.63

eick|sgs|LAST_HTTPREQUEST = https://webmailerng.1und1.de

eick|bhc|LAST_REQUEST = Tue, 19 Mar 2013 12:02:00 GMT

 

# ./PDs.pl -g -r "eick" -O "%PN (%PD):(%PV)"

eick|sgs|CLIENT_IP (1368718771): (10.185.222.42)

eick|bhc|LAST_HTTPREQUEST (1372248120):(https://92.43.21.138)

eick|sgs|LAST_REQUEST (1368718771): (Thu, 11 Apr 201315:39:31 GMT)

eick|bhc|CLIENT_IP (1372248120): (10.161.41.63)

eick|sgs|LAST_HTTPREQUEST (1368718771):(https://webmailerng.1und1.de)

eick|bhc|LAST_REQUEST (1372248120): (Tue, 19 Mar 201312:02:00 GMT)

# ./PDs.pl -g -d -r "eick" -O "%PN (%PD):(%PV)"

eick|sgs|CLIENT_IP (16.05.2013 15:39:31): (10.185.222.42)

eick|bhc|LAST_HTTPREQUEST (26.06.2013 12:02:00):(https://92.43.21.138)

eick|sgs|LAST_REQUEST (16.05.2013 15:39:31): (Thu, 11 Apr2013 15:39:31 GMT)

eick|bhc|CLIENT_IP (26.06.2013 12:02:00): (10.161.41.63)

eick|sgs|LAST_HTTPREQUEST (16.05.2013 15:39:31):(https://webmailerng.1und1.de)

eick|bhc|LAST_REQUEST (26.06.2013 12:02:00): (Tue, 19 Mar2013 12:02:00 GMT)

Attachments:
  • btlyric Apprentice 184 posts since
    Aug 1, 2012
    Currently Being Moderated
    1. Apr 16, 2013 9:21 PM (in response to feickholt)
    Re: PDs experiences

    This is EXCELLENT! Thank you!

     

    I also tried writing block pages to get PD Storage values out of MWG, but that didn't help for looking at items not related to my source IP.

     

    I have been slowly implementing PD Storage rules to deal with various situations, but I was concerned that at some point things would break and it would be hard to troubleshoot (and, as you know, the file structure is not easily examined) and I've been seeing periodic errors to the effect of:

     

    [BucketMapPlugin] [ParsingError] 'CBucketMapStorage::Parse': Error while parsing: 'Could not read user map! Will use default configuration!'.

    [BucketMapPlugin] [ParsingError] 'Parsing CUserMap from string': Error while parsing: 'Inserted bytesize is not equal with sent bytesize! Inserted bytesize: 564833 sent bytesize: 564813'.

     

    Hearing that you've got 50K users across 12 MWGs and are using PD Storage for multiple things gives me much more confidence in continuing to implement such rules.

     

    Currently using PD Storage in combination with coaching for certificates that are untrusted, unverified or expired and for sites that are uncategorized and unverified. Also testing using it as part of a Progressive Lockout rule set -- right now I am just tracking the # of blocked sites per user for a specific period of time.

     

    If you have any PD Storage rules (or ideas) that you can share, that would be great. If you don't want to post to the forum, feel free to send me a PM,

  • fwmonitor Newcomer 40 posts since
    Dec 2, 2011
    Currently Being Moderated
    4. May 2, 2013 8:01 AM (in response to feickholt)
    Re: PDs experiences

    works like a charm, very helpful, thank you Frank!

     

    additionally I use Troubleshooting > PDStorage Troubleshooting > Log PDStorage Events

     

    Can somebody explain the difference between PDStorage.CleanUp und DeleteAllXXXData ?

     

    PDStorage.Cleanup                                  Cleans up persistently stored data.

    PDStorage.DeleteAllXXXData                 Deletes all permanently stored user data.

  • Jon Scholten McAfee SME 852 posts since
    Nov 3, 2009
    Currently Being Moderated
    6. May 2, 2013 11:14 AM (in response to feickholt)
    Re: PDs experiences

    Hi All,

     

    To fwmonitor: Cleanup will look for data this is expired and remove it. DeleteAll will delete all global or user data (stored by IP or username).

     

    To feickholt: How are you storing the data, in user storage or global? What event are you using to delete it?

     

    Please consider this, PDStorage is an inactivity timeout, not a timeout based on when the data was inserted.

     

    Meaning if you insert a value into PDstorage it will not necessarily be removed after the timeout defined in the settings is reached. It will only be removed if the value has not been accessed in the alloted time.

     

    For more info see:

    https://community.mcafee.com/message/268114#268114

     

    Best,

    Jon

  • Jon Scholten McAfee SME 852 posts since
    Nov 3, 2009
    Currently Being Moderated
    8. May 2, 2013 4:18 PM (in response to feickholt)
    Re: PDs experiences

    Hi Frank,

     

    I'm a little confused by your description, but I think I get what you are saying.

     

    The cluster sync seems to be causing issues with deleting the data from PDStorage? So you delete the data from one node (with PDStorage.DeleteGlobalData event), then the MWG's sync up, and the data is back?

     

    Best,

    Jon

1 2 Previous Next

More Like This

  • Retrieving data ...

Bookmarked By (1)

Legend

  • Correct Answers - 5 points
  • Helpful Answers - 3 points