cancel
Showing results for 
Search instead for 
Did you mean: 
feickholt
Level 10

Count distict user using a proxy or cluster

Here is a short solution we do this in our proxy installation

Question

You like to know how many distinct users uses you proxies. (in a different time) .

Solution

Use SNMPTraps to send user connections to a unix server. This servers stores the data in a redis db for later analysis.


Requirement

Proxy: 

  • snmptrap enabled

Linux server:

  • snmptrapd enabled
  • redis

Installation


Proxy Part


  • Configuration - <node> - SNMP – SNMP Trap Sinks

Define at least  SNMP Trap destination. All other values might be leave as they are.

1.png

Define a new Rule in Logging section


Rule Sets

CNT-USER: Send username if not seen during the last 2 minutes

[] Enabled
Applies to: [
] Requests [] Responses [] Embedded Objects
1: Response.StatusCode is in list RETURN_CODES_2Cnt
2: AND PDStorage.HasUserData(String.Concat("UserCnt.",Authentication.UserName))<PDS keep 2 Minutes> equals false

Enabled

Rule

Action

Events

Comments

[] Enabled

SEND USER per SNMPtrap
Always

Continue

Set User-Defined.TEMP.String =
"USERCNT|" +
Authentication.UserName +
"|" +
System.HostName +
"|" +
User-Defined.region +
"|" +
User-Defined.region.pillar +
"|" +
User-Defined.region.country
SNMP.Trap.Send.User(7,User-Defined.TEMP.String)

[] Enabled

Store use in local PDS
Always

Continue

  1. PDStorage.AddUserData.String(String.Concat("UserCnt.",Authentication.UserName),"")<PDS keep 2 Minutes>

 


 

What does this rule do?

The rule will called of the return from the last request is 200. So we will analyze only successful requests.

Then we check if there is already a Local PDS variable called “UserCnt.<username>”. This variable has a TTL of 2 minutes.

If there variable is not known the following steps will be executed

  • We build a string containing

USERCNT|<username>|<Proxyname>|<Region>|<Pillar>|<Country>

  • Send the string using SNMPTRAP to our linux server
  • Store the username in a local PDS Variable. This Variable has a lifetime of 2minutes and prevent to send the same userstring during the next 2 minutes.

You might modify the string to your requirement. You should at least use username and proxyname

Unix Part

  • Install redis-db

Check your linux distribution how to install redis and install it.

  • Configure SNMPTRAPD.conf

Configure at least

disableAuthorization yes

logOption n

#Notifications Processing

traphandle default <path to your script>/snmphandle


  • Enable SNMPTRAPD

Check how to start smptrapd. This might be using /etc/init.d/snmptrapd start or using /etc/init.d/snmpt start. Check your documentation for the right way.

  • Add snmphandle script
    for easier understanding I removed the parts, containing Region, Pillar, Country information.

#!/bin/sh

if [ -f /home/eick/snmp/snmp.debug.true ]; then debug=1; fi

read host; read ip

if [ "$host" = "" ]; then

exit

fi

db="/tmp/MC"

while read oid val

do

  if [ "$oid" = "SNMPv2-SMI::enterprises.1230.2.7.4.10.5" ]; then

     TYPE=`echo $val | sed 's/\"//g' |cut -d'|' -f1 `

     # check if snmp value contains USERCNT information

     if [ "$TYPE" = "USERCNT" ]; then

       # extract Username, PROXY, and if required Region, Pillar Country

CWID=`echo $val | cut -d'|' -f2 `

HOST=`echo $val | cut -d'|' -f3 `
# you can add additional parts in the samve way

       # REGION=`echo $val | cut -d'|' -f4 `

       # Check if username was send

       if [ "$CWID" != "" ]; then

# 1min 5min 1h 6h 1d 7d 1w 1m

echo "multi " > /tmp/.$$.rdis

             # you may define TTL however you like

for time in 60 300 3600 21600 86400 604800 2592000; do

echo "

set h.$HOST.$time.$CWID $val

expire h.$HOST.$time.$CWID $time

#set r.$REGION.$time.$CWID $val

#expire r.$REGION.$time.$CWID $time

" >> /tmp/.$$.rdis

done

echo "exec " >>   /tmp/.$$.rdis

cat /tmp/.$$.rdis | redis-cli > /dev/null

rm /tmp/.$$.rdis

       fi

     fi

  fi

done

#----------------------

Now we have user entry with different TTL Values in the REDIS Database.

Format is:

  • h.<proxy>.<ttl queue>.<username>

The ttl values will be reset if user will be seen again.

Analysis


What can we do now:

  • Find out all proxy we’ve datas for

redis-cli keys h.*.60.* | awk -F. '{print $2}' | sort | uniq



  • Count all users seen on proxy1 using the proxy during the last 3600 seconds (1hour)

redis-cli keys h.proxy1.3600.* | wc –l

  • Find out proxy where user was active and how long it was ago

# get all proxies you’ve datas for

proxy=`redis-cli keys h.*.*.$1 | awk -F. '{print $2}' | sort | uniq`

echo "$1 last seen on proxy (value = maximum econds ago)"

for p in $proxy; do

   echo -n "$p: "

   ti=`redis-cli keys h.$p.*.$1 | awk -F. '{print $3}' | sort -n | tac | tail -1`

ttl=`redis-cli ttl h.$p.$ti.$1`

   echo "$ti (`expr $ti - $ttl` sec. ago)"

done

Output:
# ./find_user IMSOO

IMSOO last seen on proxy (value = maximum econds ago)

BYMCG1: 604800 (271286 sec. ago)

BYMCG2: 3600 (2790 sec. ago)

BYMCG3: 3600 (850 sec. ago)

BYMCG4: 604800 (274743 sec. ago)

Ideas

  • Use values for RRD graphs. (Distinct Users for 5 minutes, 1h, 1 day…)