1 Reply Latest reply on May 30, 2016 8:40 AM by feickholt

    Solution: How often does an entry of a list matched in the past.

    feickholt

      Hi Folks,

       

      You like to know which element in a List is used?

      How often?

      Find the elements which are never used?

       

      Ok here is my solution :-)

       

      First of all you need the following two rules

      Find List Matches
      [✔] Enabled [✘] Disabled in Cloud
      Applies to: [] Requests [] Responses [] Embedded Objects
      Always
      EnabledRuleActionEventsComments
      [✔] Enabled Store matching in PDs (last USed, Cnt)
      Always
      ContinueSet User-Defined.Blocked.by = "<enter Name of List>"
      Set User-Defined.TEMP.String =
           "LIST_USE|" +
           User-Defined.Blocked.by +
           "|Last_used|" +
           List.LastMatches
      PDStorage.AddGlobalData.String(User-Defined.TEMP.String,DateTime.ToISOString)<PDS Keep Forever>
      Set User-Defined.TEMP.String =
           "LIST_USE|" +
           User-Defined.Blocked.by +
           "|Count|" +
           List.LastMatches
      Set User-Defined.TEMP.cnt =
           PDStorage.GetGlobalData.Number(User-Defined.TEMP.String)<PDS Keep Forever> +
           1
      PDStorage.AddGlobalData.Number(User-Defined.TEMP.String,User-Defined.TEMP.cnt)<PDS Keep Forever>
      Set User-Defined.TEMP.String =
           "LIST_USE|" +
           User-Defined.Blocked.by +
           "|First_used|" +
           List.LastMatches
      [✔] Enabled Set first Use
      1: PDStorage.HasGlobalData(User-Defined.TEMP.String)<PDS Keep Forever> equals false
      2: OR PDStorage.GetGlobalData.String(User-Defined.TEMP.String)<PDS Keep Forever> less than or equals "0"
      ContinuePDStorage.AddGlobalData.String(User-Defined.TEMP.String,DateTime.ToISOString)<PDS Keep Forever>

       

      This rule you have to place right after the place where the list is used in the policy.

      For easier finding you should give the user defined variable blocked_by a unique name.Normally I use the ListName.

       

      Ok this was the Policy Part.

       

      Now the tricky part....

      A few month ago I posted a PDStorage analyzer... (you find all here PDs experiences)

       

      You need to have a linux or cygwin installation with perl.

      Also you need to have a trusted SSH Access from this installation to your Proxy. (ssh access without login - example: use ssh-copy-id to exange keys)

       

      Using this you can do the following....

      >  ./PDs.pl -g -s <IP-PROXY>  | grep LIST_USE

       

      LIST_USE|GLB_BLACKLIST (Host)|Last_used|ADS.CNN.COM = 2016-05-13 13:11:32

      LIST_USE|GLB_BLACKLIST (Host)|First_used|ADS.CNN.COM = 2016-05-01 13:07:34

      LIST_USE|GLB_BLACKLIST (Host)|count|ADS.CNN.COM = 244

      LIST_USE|GLB_WHITELIST (Pattern)|Last_used|regex(^(http|https|ftp)://[^/]*.onenote.com.*) = 2016-05-30 07:33:01

      LIST_USE|GLB_WHITELIST (Pattern)|First_used|regex(^(http|https|ftp)://[^/]*.onenote.com.*) = 2016-01-20 17:53:34

      LIST_USE|GLB_WHITELIST (Pattern)|count|regex(^(http|https|ftp)://[^/]*.onenote.com.*) = 233433

      ...


      Now you have all informations you are looking for......

       

      The Listname,

      The matched entry,

      The last and first seen date

      and how often this entry matches during this time period.

       

      If your Global PDs is not syncronized over all proxies you'll have to ask every proxy and merge the results together.

      If it is syncronized you have to ask only one proxy.

       

      One restriction.

      You can only see entries requested during the last 99 days. Older entries will be deleted from the PDs...

      A workaround might be to request all entries every 3 month and store them for later use.

       

      Regards

      Frank


        • 1. Re: Solution: How often does an entry of a list matched in the past.
          feickholt

          For those people which would not like to use perl and the package I wrote I have another solution using blockpages...

           

          :-)

           

          Here's the rule set

           

          Rule Sets
          ADMIN_Proxy generated statistic pages
          [✔] Enabled [✘] Disabled in Cloud
          Applies to: [] Requests [] Responses [] Embedded Objects
          1: URL.Host matches proxy.admin
          /lists
          [✔] Enabled [✘] Disabled in Cloud
          Applies to: [] Requests [] Responses [] Embedded Objects
          1: URL.Path equals "/lists"
          EnabledRuleActionEventsComments
          [✔] EnabledShow all ListUse PDs Values
          Always
          Block<ListUse>
          /PD
          [✔] Enabled [✘] Disabled in Cloud
          Applies to: [] Requests [] Responses [] Embedded Objects
          1: URL.Path equals "/PD"
          EnabledRuleActionEventsComments
          [✔] EnabledReturn PD Variable Value
          1: URL.HasParameter("PDValue") equals true
          Block<PRINT>Set User-Defined.PD.key = URL.GetParameter("PDValue")
          Set User-Defined.PD.PrintKey = URL.HasParameter("PrintKey")
          Set User-Defined.PD.value.string = PDStorage.GetGlobalData.String(User-Defined.PD.key)<PDS Keep 30 days>
          Set User-Defined.PD.value.number = PDStorage.GetGlobalData.Number(User-Defined.PD.key)<PDS Keep 1 day>

           

           

          You have to enter

           

          http://proxy.admin/lists in your browser (use the proxy you like to check as explicit proxy defined in your network setting.)


          You have to define 2 blockpages.


          ListUse

          <!--Content-->

          <script language="JavaScript">

          var PDStor = "$PDStorage.GetAllGlobalData$";

          var VALUES = PDStor.split(", ");

          if (VALUES.length == 1 && VALUES[0] == "") {

            document.write("<b>" + "No VALUES found" + "</b>");

          } else {

            VALUES.sort();

            document.write("<font size=-1><table><tr><th>List</th><th>Entry</th><th>First Seen</th><th>Last  Seen</th><th>Count</th>");

            for (var i = 0; i < VALUES.length; i++) {

                var v = VALUES[i];

           

                var list = v.split ("|");

           

                if (list[0]=="LIST_USE" && list[2] == "Last_used") {

            document.write("<tr><td>"+list[1]+"</td><td>"+list[3]+"</td><td>");

            var value = httpGet ("/PD?PDvalue=LIST_USE|" +list[1]+"|First_used|"+ list[3]);

              document.write( value);

           

           

            document.write( "</td><td>");

            var value = httpGet ("/PD?PDvalue=LIST_USE|" +list[1]+"|Last_used|"+ list[3]);

              document.write( value);

            document.write( "</td><td>");

            var value = httpGet ("/PD?PDvalue=LIST_USE|" +list[1]+"|count|"+ list[3]);

              document.write( value);

           

           

            document.write( "</td></tr>");

              }

          }

            }

          document.write("</table></font>");

          </script>

          <!--/Content-->

          -----------------------------------------------------------------

          And  PRINT

          ---------------------------------------------------------------------

          #<script language="JavaScript">

          if ("$User-Defined.PD.PrintKey$" == "true") {

            document.write ("$User-Defined.PD.Key$:");

          }

          if  ("$User-Defined.PD.value.string" == "") {

          document.write ("$User-Defined.PD.value.number$");

          } else {

          document.write ("$User-Defined.PD.value.string$");

          }

          </script>#

          ----------------------------------------------------------------------

          This should be defined in a dedicated schema without any Headers and Footers

          This schema should only use

          -----------------------------------------

          <html>

          $CONTENT$

          </html>

          ---------------------------------------

          in it's html file

           

           

           

          In case on any question....  feel free to ask me :-)

           

          Frank