Support Doc: User-Agents Explained

     

    Introduction

     

    You might be asking yourself, "Okay, why do I care about the user-agent?".  Well, the user-agent can be important and can help you out in lots of situations.  Lets say iTunes keeps prompting users to enter in their credentials, or a windows media player stream simply will not play.  You might need to bypass these programs from certain rules to get them functioning the way they should be.  You can easily do that with 1 header, and that is the user-agent header.

     

     

    What is a User-Agent?

     

    A User-Agent is a header in an HTTP request that identifies the client software originating the request.

     

     

    Why is this header important?

     

    This is important because it identifies what piece of software may be making a request so you can properly identify where it is coming from.  This could be anything from the browser, to windows media player.  They all have a user-agent to identify what software it is.

    Why would I need to create a rule to match on the user-agent?

    There are times where you need to match the user-agent criteria for a rule.  These times usually include bypassing a specific rule set within web gateway that may be causing issues with the user-agent.  The beauty about Web Gateway 7 is that you don’t have to bypass all of the rules.  Say you are having an issue where a specific user-agent is not working with authentication (which is one of the most common times) so you only want to bypass authentication.  You can easily do that.  There will be some examples of this and other common scenarios at the end.

     

     

    How do I find out what the user-agent is for a request?

     

    There are a few ways to find out this sort of information.  Below I will list and explain some common ways to find this out:

     

    Access Logs

     

    It is possible to log the user-agent in your access logs.  By default, the web gateway logs the user agent to the access logs.

     

     

    Example entry:

     

    Access Log Example.png

     

    Online Resources

    There are lots of online resources available to find out the user-agent.  A lot of them are made publicly available.  It is possible to do a Google search, and there is also a very useful website that has a list of a lot of the common user-agents.  This URL is http://www.useragentstring.com.  Also, if you would like to find out what your current browser user-agent is you can go to this website:  http://whatsmyuseragent.com

     

     

    TCP-Dump

    It is possible to find the user-agent header for a GET request in a TCP Dump by following the TCP Stream of a request.  You can do this using a packet tracing software, such as ‘Wireshark’.  You would do this by finding the specific request you want, right clicking on it, and selecting “Follow TCP Stream”.  This would then bring up a window that shows the request and its headers.  You can also use a filter within Wireshark to search for a specific user-agent.  For example, you could use a filter such as: 

     

    http.user_agent matches "Mozilla"
    

     

    The web gateway has capabilities to gather a TCP Dump so that you can open it with Wireshark to analyze.  For help with gathering a TCP Dump please check this KB article:

    https://kc.mcafee.com/corporate/index?page=content&id=KB75056

     

     

    Examples

    Here are some common user-agents that we come across quite frequently:

     

     

    Firefox 16.0.1

    Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1
    
    **All Firefox browser user-agents will have “Firefox/” and then the corresponding version number**
    

     

    Internet Explorer 10

    Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
    
    **All IE browser user-agents will have “MSIE” and then the corresponding version number**
    

     

    Google Chrome 25

    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22
    
    **Note that there is "AppleWebKit" in the Google Chrome user-agent, not to be confused with an Apple\iPhone header**
    


    Windows Media Player 10

    Windows-Media-Player/10.0.0.xxxx
    
    NSPlayer/10.0.0.xxxx WMFSDK/10.0
    
    **Windows Media Player will always use both of these user-agent strings**
    

     

    iTunes 9.1.1

    iTunes/9.1.1
    

     

    Safari on the iPhone

    Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7
    
    ** Most iPhone apps will have “iPhone” somewhere in the user-agent string**
    

     

     

     

    Note:  Most browser user-agents have “Mozilla/” at the beginning, but do not confuse this with Mozilla Firefox.  This does not mean the user-agent is for Firefox, it is just what all browser user-agents start with.

     

     

    How do I create a rule based on the user-agent header in Web Gateway 7?

     

    There are several reasons for why you might want to create a rule to match a user-agent.  You might want to either whitelist or block a specific user-agent.  For example, if you didn’t want users to use the Internet Explorer 6 browser, you could block the user-agent.  Another reason is that some user-agents do not know how to respond to proxy authentication.  Some common ones we have encountered in the past that have trouble with authentication are: Silverlight, iTunes, Windows Media Player, RealPlayer, Adobe Updater, Adobe Flash.  These user-agents typically do not know how to respond to a 407 proxy authentication request and simply do not answer when the web gateway asks for it, so they need to be bypassed.

     

    The web gateway does not have a specific property for the user-agent so it needs to be entered manually.

     

      • First you would select the header.request.get property from the list

     

    Figure A.png

     

     

      • Then, you would press the “parameters” button

     

    Figure B.png

     

     

      • Now, here is where you have to manually type in:  User-Agent

     

    Figure C.png

     

     

      • Next, you have a choice to either have it match a single User-Agent, or have it match in a list of user-agents.  The choice is yours of which you want to do, but lists are always recommended because then if you have new entries you want to add later, all you have to do is add it to the already existing list.  Here is an example of a list of user-agents it can match in.

     

    Figure D.png

     

     

      • The action you want to choose is completely up to you.  You can either block it, stop rule set, or stop cycle.  It all depends on what you are trying to accomplish with the rule.

     

     

    Conclusion

     

    Hopefully this has given you a better understanding of what the user-agent header is exactly, and what purpose it has.  Also, hopefully this gets you on the right path to creating a rule based on the user-agent property to use within your web gateway appliance.

     

     

    Rule Set Examples

     

    Bypass a user-agent from authentication


    Example1.png

     

     

     

     

    Bypass a user-agent from the SSL Scanner

     

    Example2.png

     

     

    Create a Global Block list of User-Agents

     

    Example3.png

     

    TCP Dump Troubleshooting Examples

     

    Here is an example of what you might see in a TCP Dump that shows when a user-agent doesn't respond to proxy NTLM authentication requests, and also how to find the user-agent in a GET request.

     

    Program is not responding to NTLM authentication

    Here you can see that iTunes keeps trying to go to the iTunes store, and we keep asking for authentication.  However, each time, iTunes keeps making the same request and never responds with any credentials.  In a normal situation it would complete the NTLM handshake by first sending back the same request with an NTLM negotiate, which the web gateway would then respond back with an NTLM challenge, and then iTunes would respond back with an NTLM authenticate and provide the NTLM credentials to the web gateway.

     

    iTunes TCP Dump.png

     

     

    How to find the user-agent in a GET request

    Here you can see I have selected a GET request in a TCP Dump, and under the "Hypertext Transfer Protocol" section below you can see the User-Agent header.  I did this using a program called "Wireshark".

     

    User-Agent Header TCP Dump.png