cancel
Showing results for 
Search instead for 
Did you mean: 
feickholt
Level 10

Solution: How often does an entry of a list matched in the past.

Hi Folks,

You like to know which element in a List is used?

How often?

Find the elements which are never used?

Ok here is my solution :-)

First of all you need the following two rules

Find List Matches
[✔] Enabled [✘] Disabled in Cloud
Applies to: [] Requests [] Responses [] Embedded Objects
Always
EnabledRuleActionEventsComments
[✔] Enabled Store matching in PDs (last USed, Cnt)
Always
ContinueSet User-Defined.Blocked.by = "<enter Name of List>"
Set User-Defined.TEMP.String =
     "LIST_USE|" +
     User-Defined.Blocked.by +
     "|Last_used|" +
     List.LastMatches
PDStorage.AddGlobalData.String(User-Defined.TEMP.String,DateTime.ToISOString)<PDS Keep Forever>
Set User-Defined.TEMP.String =
     "LIST_USE|" +
     User-Defined.Blocked.by +
     "|Count|" +
     List.LastMatches
Set User-Defined.TEMP.cnt =
     PDStorage.GetGlobalData.Number(User-Defined.TEMP.String)<PDS Keep Forever> +
     1
PDStorage.AddGlobalData.Number(User-Defined.TEMP.String,User-Defined.TEMP.cnt)<PDS Keep Forever>
Set User-Defined.TEMP.String =
     "LIST_USE|" +
     User-Defined.Blocked.by +
     "|First_used|" +
     List.LastMatches
[✔] Enabled Set first Use
1: PDStorage.HasGlobalData(User-Defined.TEMP.String)<PDS Keep Forever> equals false
2: OR PDStorage.GetGlobalData.String(User-Defined.TEMP.String)<PDS Keep Forever> less than or equals "0"
ContinuePDStorage.AddGlobalData.String(User-Defined.TEMP.String,DateTime.ToISOString)<PDS Keep Forever>

This rule you have to place right after the place where the list is used in the policy.

For easier finding you should give the user defined variable blocked_by a unique name.Normally I use the ListName.

Ok this was the Policy Part.

Now the tricky part....

A few month ago I posted a PDStorage analyzer... (you find all here )

You need to have a linux or cygwin installation with perl.

Also you need to have a trusted SSH Access from this installation to your Proxy. (ssh access without login - example: use ssh-copy-id to exange keys)

Using this you can do the following....

>  ./PDs.pl -g -s <IP-PROXY>  | grep LIST_USE

LIST_USE|GLB_BLACKLIST (Host)|Last_used|ADS.CNN.COM = 2016-05-13 13:11:32

LIST_USE|GLB_BLACKLIST (Host)|First_used|ADS.CNN.COM = 2016-05-01 13:07:34

LIST_USE|GLB_BLACKLIST (Host)|count|ADS.CNN.COM = 244

LIST_USE|GLB_WHITELIST (Pattern)|Last_used|regex(^(http|https|ftp)://[^/]*.onenote.com.*) = 2016-05-30 07:33:01

LIST_USE|GLB_WHITELIST (Pattern)|First_used|regex(^(http|https|ftp)://[^/]*.onenote.com.*) = 2016-01-20 17:53:34

LIST_USE|GLB_WHITELIST (Pattern)|count|regex(^(http|https|ftp)://[^/]*.onenote.com.*) = 233433

...


Now you have all informations you are looking for......

The Listname,

The matched entry,

The last and first seen date

and how often this entry matches during this time period.

If your Global PDs is not syncronized over all proxies you'll have to ask every proxy and merge the results together.

If it is syncronized you have to ask only one proxy.

One restriction.

You can only see entries requested during the last 99 days. Older entries will be deleted from the PDs...

A workaround might be to request all entries every 3 month and store them for later use.

Regards

Frank


0 Kudos
1 Reply
feickholt
Level 10

Re: Solution: How often does an entry of a list matched in the past.

For those people which would not like to use perl and the package I wrote I have another solution using blockpages...

:-)

Here's the rule set

Rule Sets
ADMIN_Proxy generated statistic pages
[✔] Enabled [✘] Disabled in Cloud
Applies to: [] Requests [] Responses [] Embedded Objects
1: URL.Host matches proxy.admin
/lists
[✔] Enabled [✘] Disabled in Cloud
Applies to: [] Requests [] Responses [] Embedded Objects
1: URL.Path equals "/lists"
EnabledRuleActionEventsComments
[✔] EnabledShow all ListUse PDs Values
Always
Block<ListUse>
/PD
[✔] Enabled [✘] Disabled in Cloud
Applies to: [] Requests [] Responses [] Embedded Objects
1: URL.Path equals "/PD"
EnabledRuleActionEventsComments
[✔] EnabledReturn PD Variable Value
1: URL.HasParameter("PDValue") equals true
Block<PRINT>Set User-Defined.PD.key = URL.GetParameter("PDValue")
Set User-Defined.PD.PrintKey = URL.HasParameter("PrintKey")
Set User-Defined.PD.value.string = PDStorage.GetGlobalData.String(User-Defined.PD.key)<PDS Keep 30 days>
Set User-Defined.PD.value.number = PDStorage.GetGlobalData.Number(User-Defined.PD.key)<PDS Keep 1 day>

You have to enter

http://proxy.admin/lists in your browser (use the proxy you like to check as explicit proxy defined in your network setting.)


You have to define 2 blockpages.


ListUse

<!--Content-->

<script language="JavaScript">

var PDStor = "$PDStorage.GetAllGlobalData$";

var VALUES = PDStor.split(", ");

if (VALUES.length == 1 && VALUES[0] == "") {

  document.write("<b>" + "No VALUES found" + "</b>");

} else {

  VALUES.sort();

  document.write("<font size=-1><table><tr><th>List</th><th>Entry</th><th>First Seen</th><th>Last  Seen</th><th>Count</th>");

  for (var i = 0; i < VALUES.length; i++) {

      var v = VALUES;

      var list = v.split ("|");

      if (list[0]=="LIST_USE" && list[2] == "Last_used") {

  document.write("<tr><td>"+list[1]+"</td><td>"+list[3]+"</td><td>");

  var value = httpGet ("/PD?PDvalue=LIST_USE|" +list[1]+"|First_used|"+ list[3]);

    document.write( value);

  document.write( "</td><td>");

  var value = httpGet ("/PD?PDvalue=LIST_USE|" +list[1]+"|Last_used|"+ list[3]);

    document.write( value);

  document.write( "</td><td>");

  var value = httpGet ("/PD?PDvalue=LIST_USE|" +list[1]+"|count|"+ list[3]);

    document.write( value);

  document.write( "</td></tr>");

    }

}

  }

document.write("</table></font>");

</script>

<!--/Content-->

-----------------------------------------------------------------

And  PRINT

---------------------------------------------------------------------

#<script language="JavaScript">

if ("$User-Defined.PD.PrintKey$" == "true") {

  document.write ("$User-Defined.PD.Key$:");

}

if  ("$User-Defined.PD.value.string" == "") {

document.write ("$User-Defined.PD.value.number$");

} else {

document.write ("$User-Defined.PD.value.string$");

}

</script>#

----------------------------------------------------------------------

This should be defined in a dedicated schema without any Headers and Footers

This schema should only use

-----------------------------------------

<html>

$CONTENT$

</html>

---------------------------------------

in it's html file

In case on any question....  feel free to ask me :-)

Frank

0 Kudos