0 Replies Latest reply on Aug 11, 2014 6:59 AM by feickholt

    Count distict user using a proxy or cluster

    feickholt

      Here is a short solution we do this in our proxy installation

       

      Question

      You like to know how many distinct users uses you proxies. (in a different time) .

       

      Solution

      Use SNMPTraps to send user connections to a unix server. This servers stores the data in a redis db for later analysis.


      Requirement

      Proxy: 

      • snmptrap enabled

      Linux server:

      • snmptrapd enabled
      • redis

      Installation


      Proxy Part


      • Configuration - <node> - SNMP – SNMP Trap Sinks

      Define at least  SNMP Trap destination. All other values might be leave as they are.

      1.png

      Define a new Rule in Logging section


       

      Rule Sets

       

      CNT-USER: Send username if not seen during the last 2 minutes

      [] Enabled
      Applies to: [
      ] Requests [] Responses [] Embedded Objects
      1: Response.StatusCode is in list RETURN_CODES_2Cnt
      2: AND PDStorage.HasUserData(String.Concat("UserCnt.",Authentication.UserName))<PDS keep 2 Minutes> equals false

      Enabled

      Rule

      Action

      Events

      Comments

      [] Enabled

      SEND USER per SNMPtrap
      Always

      Continue

      Set User-Defined.TEMP.String =
      "USERCNT|" +
      Authentication.UserName +
      "|" +
      System.HostName +
      "|" +
      User-Defined.region +
      "|" +
      User-Defined.region.pillar +
      "|" +
      User-Defined.region.country
      SNMP.Trap.Send.User(7,User-Defined.TEMP.String)

      [] Enabled

      Store use in local PDS
      Always

      Continue

      1. PDStorage.AddUserData.String(String.Concat("UserCnt.",Authentication.UserName)," ")<PDS keep 2 Minutes>

       


       

      What does this rule do?

       

      The rule will called of the return from the last request is 200. So we will analyze only successful requests.

      Then we check if there is already a Local PDS variable called “UserCnt.<username>”. This variable has a TTL of 2 minutes.

      If there variable is not known the following steps will be executed

      • We build a string containing

      USERCNT|<username>|<Proxyname>|<Region>|<Pillar>|<Country>

      • Send the string using SNMPTRAP to our linux server
      • Store the username in a local PDS Variable. This Variable has a lifetime of 2minutes and prevent to send the same userstring during the next 2 minutes.

      You might modify the string to your requirement. You should at least use username and proxyname

       

      Unix Part

      • Install redis-db

      Check your linux distribution how to install redis and install it.

       

      • Configure SNMPTRAPD.conf

      Configure at least

      disableAuthorization yes

      logOption n

      #Notifications Processing

      traphandle default <path to your script>/snmphandle


      • Enable SNMPTRAPD

       

      Check how to start smptrapd. This might be using /etc/init.d/snmptrapd start or using /etc/init.d/snmpt start. Check your documentation for the right way.

       

      • Add snmphandle script
        for easier understanding I removed the parts, containing Region, Pillar, Country information.

      #!/bin/sh

       

      if [ -f /home/eick/snmp/snmp.debug.true ]; then debug=1; fi

       

      read host; read ip

      if [ "$host" = "" ]; then

      exit

      fi

       

      db="/tmp/MC"

      while read oid val

      do

        if [ "$oid" = "SNMPv2-SMI::enterprises.1230.2.7.4.10.5" ]; then

           TYPE=`echo $val | sed 's/\"//g' |cut -d'|' -f1 `

           # check if snmp value contains USERCNT information

           if [ "$TYPE" = "USERCNT" ]; then

             # extract Username, PROXY, and if required Region, Pillar Country

      CWID=`echo $val | cut -d'|' -f2 `

      HOST=`echo $val | cut -d'|' -f3 `
      # you can add additional parts in the samve way

             # REGION=`echo $val | cut -d'|' -f4 `

             # Check if username was send

             if [ "$CWID" != "" ]; then

       

      # 1min 5min 1h 6h 1d 7d 1w 1m

      echo "multi " > /tmp/.$$.rdis

       

                   # you may define TTL however you like

      for time in 60 300 3600 21600 86400 604800 2592000; do

      echo "

      set h.$HOST.$time.$CWID $val

      expire h.$HOST.$time.$CWID $time

      #set r.$REGION.$time.$CWID $val

      #expire r.$REGION.$time.$CWID $time

      " >> /tmp/.$$.rdis

      done

       

      echo "exec " >>   /tmp/.$$.rdis

      cat /tmp/.$$.rdis | redis-cli > /dev/null

       

      rm /tmp/.$$.rdis

             fi

           fi

        fi

      done

       

      #----------------------

       

      Now we have user entry with different TTL Values in the REDIS Database.

      Format is:

      • h.<proxy>.<ttl queue>.<username>

      The ttl values will be reset if user will be seen again.

       

      Analysis


      What can we do now:

      • Find out all proxy we’ve datas for

      redis-cli keys h.*.60.* | awk -F. '{print $2}' | sort | uniq



      • Count all users seen on proxy1 using the proxy during the last 3600 seconds (1hour)

      redis-cli keys h.proxy1.3600.* | wc –l

       

      • Find out proxy where user was active and how long it was ago

      # get all proxies you’ve datas for

      proxy=`redis-cli keys h.*.*.$1 | awk -F. '{print $2}' | sort | uniq`

       

      echo "$1 last seen on proxy (value = maximum econds ago)"

      for p in $proxy; do

         echo -n "$p: "

         ti=`redis-cli keys h.$p.*.$1 | awk -F. '{print $3}' | sort -n | tac | tail -1`

      ttl=`redis-cli ttl h.$p.$ti.$1`

         echo "$ti (`expr $ti - $ttl` sec. ago)"

      done

      Output:
      # ./find_user IMSOO

      IMSOO last seen on proxy (value = maximum econds ago)

      BYMCG1: 604800 (271286 sec. ago)

      BYMCG2: 3600 (2790 sec. ago)

      BYMCG3: 3600 (850 sec. ago)

      BYMCG4: 604800 (274743 sec. ago)

       

       

       

      Ideas

      • Use values for RRD graphs. (Distinct Users for 5 minutes, 1h, 1 day…)