Best Practice: Implementing and Understanding Bandwidth Control

Version 12

     

    Introduction

    With the release of 7.6.2, MWG’s ability to control bandwidth has been greatly enhanced.  MWG now includes a classful means to control your company’s bandwidth. Traffic can now be categorized (not in the URL sense) into different classes and prioritized accordingly. This is different from the 7.1.5 feature called Bandwidth Throttling (White paper: MWG Bandwidth Throttling ). 7.7.0 adds support for transparent and proxyHA modes (See: Defining applicable Network Interfaces).

     

     

    What value does Bandwidth Control add?

    Often organizations (small or big) might not have a means to intelligently control bandwidth. Your web policy is the ideal egress point to make decisions based on user/group/ip/category/application. By managing the network traffic bandwidth in this manner, the following can be achieved:

    • Provide guaranteed minimum bandwidth rate to certain traffic classes.
    • Restrict the bandwidth usage of certain traffic classes to a maximum limit.
    • Define priority within traffic classes to share the excess available bandwidth.

     

    Bandwidth Control allows you to control data as it goes in and out of the MWG. The events available for Bandwidth Control are Bandwidth.FromClient (Upload - Client to MWG), Bandwidth.ToClient (Download - MWG to Client), Bandwidth.FromServer (Download - Server to MWG), Bandwidth.ToServer (Upload - MWG to Server). The directionality of the events can be found in the diagram below:

     

    Nuts and Bolts (Concept)

    A Bandwidth class can be identified by a name along with few parameters relevant to classify and control bandwidth. Bandwidth classes can be grouped together to form a class hierarchy (parent-child representation). A child class can have another class as its parent (the parent class should already exist). Each class must have the following parameters:

    • Minimum Bandwidth: Minimum amount of bandwidth guaranteed for the traffic in the class i.e the class receives at least the amount of bandwidth, if bandwidth is available.
    • Maximum Bandwidth: Maximum amount of bandwidth allowed for traffic in the class. It does not matter how much bandwidth is available, a class can never receive more bandwidth than its maximum.
    • Priority: When sharing excess bandwidth with classes in the same hierarchy, the class with the lower priority # gets the first opportunity to use excess bandwidth. For Eg,

    If a class request less than allocated rate for a child, the excess will be offered to another child with the lowest priority # and so on.

      • If a minimum bandwidth is configured for a parent class, it must be greater than or equal to the sum of the minimum bandwidths of all its child classes.
      • The sum of the minimum bandwidth of all child classes should not exceed the maximum bandwidth defined for the parent class.
      • The sum of the minimum of the children classes would also match the minimum of parent class, so that the parent distribute the leftover bandwidth (maximum – minimum) among the children class.
      • This feature utilizes the native Linux Traffic Control as an underlying mechanism to implement bandwidth control.
      • HTB classful qdisc is used for granular control over traffic. HTB uses the concept of tokens and buckets. It also only enforces leaf classes’ configuration.

     

     

    Setting your goals & Bandwidth Tree example

    To start testing the waters with Bandwidth Control you’ll need to map out what you want to control and what experience you want users to have. In our bandwidth tree, only "leaf" classes can be used in the configuration, parent classes cannot be used. The leaf classes are noted in the diagram below with the leaf icon.

     

    Here is some example goals:

    • Limit YouTube and the Streaming Media category to a Max bandwidth of 10,000 Kbit/s
    • Limit the category Social Networking to a Max bandwidth of 5,000 Kbit/s
    • Ensure that the YouTube category has priority over the Streaming Media class

     

     

     

    Defining the Bandwidth Classes

    Under Configuration > Bandwidth Control we must check the box for "Enable Bandwidth Control". Once Bandwidth Control is enabled, we must define the classes we defined in our goals. IMPORTANT: The definitions must be created on each proxy you wish to utilize Bandwidth Control.

    • Define a Root class with a max and min bandwidth of 20,000 Kbit/s, the priority is 0 because there is no other competing bandwidth classes.
    • Define two child classes under Root called Streaming and Social Networking
      • Streaming – Max = 10000, Min = 5000, Priority = 0
      • Social Networking – Max = 5000, Min 2500, Priority = 0
    • Define two child classes under Streaming called YouTube and Streaming Media
      • YouTube – Max = 10000, Min = 3500, Priority = 50
      • Streaming Media – Max = 5000, Min = 1500, Priority = 75

     

    The resulting class definitions should appear as you see in the screenshot below:

     

     

    Define applicable network interfaces

    Again under Configuration > Bandwidth Control we must specify the network interfaces which will impose bandwidth controls. To do this we click add, then select the interface we wish to use. For Explicit Proxy, ProxyHA, transparent router, and WCCP, simply specify the interface you wish to control. Depending on the interface configuration for ProxyHA you might need to specify multiple interfaces on the active director or redundant director node (anything with directory priority higher than 0).

     

     

    Bridge mode

    For bridge mode we need to specify the interfaces differently because of the bridge interface on the active proxy.

      • On the active or redundant director nodes (director priority greater than 0), specify the inbound and possibly the outbound interface.

      • On the scanning nodes (director priority set to 0), specify the interface for which traffic is received from the director on.

     

     

     

    Example Ruleset

    To use the Bandwidth classes, we must use the Bandwidth.(FromClient|ToClient|FromServer|ToServer) events in the rule configuration. Bandwidth Control events are last match wins, meaning the last rule which applies Bandwidth Control is the class that is applied. IMPORTANT: In our example this is important because YouTube is a subset of Streaming Media, so to prioritize YouTube separately from Streaming Media, we need to adjust the rule accordingly.

     

    >>> Ruleset Download <<<

     

    1. Bandwidth Control: YouTube - This rule is first because YouTube is also categorized as Streaming Media. We want to exit the ruleset as soon as we detect the user is accessing YouTube.
    2. Bandwidth Control: Streaming Media - This rule is second to act as a catch all, higher prioritized exceptions (like YouTube) should be placed above this rule.
    3. Bandwidth Control: Social Media - This rule is last because there should not be high likelihood of overlap with the rules above it.

     

     

    Note for Testing

    When testing new rules, it's always best to use a test proxy, or scope the rules to apply to only yourself. If you do not have a test proxy available, just add criteria to the ruleset which causes the rules to only apply to your or test users (i.e. Client.IP is in range list [List of test Clients]).

     

     

     

    Placement of Rules

    For optimal functioning of the Bandwidth Control rules it is best to apply Bandwidth as early as possible. Apply Bandwidth Control too late or applying it incorrectly will likely yield erroneous results. In our example we are only controlling the bandwidth between the MWG and the internet (as this is the most desired example).

     

    To apply Bandwidth Control as early as possible, rules should be placed towards the top of your rules. See screenshot below for example placement.

     

     

    Correct/Incorrect Uses

    Examples of correct uses:

      • For Uploads (FromClient|ToServer) or Downloads (ToClient|FromServer), in Request Cycle based on: Application Name, URL Category, URL, User-Agent, Client IP, Username, Usergroup, or anything which the MWG knows about in the request cycle.

    Examples of incorrect uses:

      • For Downloads (FromServer), in Response Cycle based on Ensured Media Type (different from File Extension). To determine the Media Type MWG must have the file (or at least part of it). MWG cannot apply Bandwidth Control based on information it doesnt have.

     

     

    Troubleshooting

    The underlying mechanism used for Bandwidth Control is tc (traffic control). Various commands can be run from the console to visual the structure of your classes and also determine if the class is being matched correctly. Primarily though rule tracing should be used to understand if the rule matched correctly.

     

    Verify configuration

    less /etc/sysconfig/bandwidth
    

     

    Visualize classes

    tc -g class show dev ifb0
    
    # Example output:
    +---(1:1) htb rate 10Gbit ceil 10Gbit burst 0b cburst 0b
         +---(1:2) htb rate 1Kbit ceil 20Mbit burst 1600b cburst 1600b
              +---(1:3) htb rate 5Mbit ceil 10Mbit burst 1600b cburst 1600b
              |    +---(1:4) htb prio 7 rate 3500Kbit ceil 10Mbit burst 1599b cburst 1600b
              |    +---(1:5) htb prio 7 rate 1500Kbit ceil 5Mbit burst 1599b cburst 1600b
              |
              +---(1:6) htb prio 1 rate 2500Kbit ceil 5Mbit burst 1600b cburst 1600b
    

     

    Monitor class usage (real-time)

    watch -d tc -s -d class show dev ifb0
    
    # Example output:
    class htb 1:1 root rate 10Gbit ceil 10Gbit linklayer ethernet burst 0b/1 mpu 0b overhead 0b cburst 0b/1 mpu 0b overhead 0b level 7
     Sent 74723335 bytes 51203 pkt (dropped 0, overlimits 0 requeues 0)
     rate 0bit 0pps backlog 0b 0p requeues 0
     lended: 5495 borrowed: 0 giants: 0
     tokens: 14 ctokens: 14
    
    
    class htb 1:2 parent 1:1 rate 1Kbit ceil 20Mbit linklayer ethernet burst 1600b/1 mpu 0b overhead 0b cburst 1600b/1 mpu 0b overhead 0b level 6
     Sent 74723335 bytes 51203 pkt (dropped 0, overlimits 0 requeues 0)
     rate 0bit 0pps backlog 0b 0p requeues 0
     lended: 0 borrowed: 5495 giants: 0
     tokens: -85914057 ctokens: 9587
    
    
    class htb 1:3 parent 1:2 rate 5Mbit ceil 10Mbit linklayer ethernet burst 1600b/1 mpu 0b overhead 0b cburst 1600b/1 mpu 0b overhead 0b level 5
     Sent 74723335 bytes 51203 pkt (dropped 0, overlimits 0 requeues 0)
     rate 0bit 0pps backlog 0b 0p requeues 0
     lended: 28577 borrowed: 5495 giants: 0
     tokens: 38350 ctokens: 19175
    
    
    class htb 1:4 parent 1:3 prio 7 quantum 43750 rate 3500Kbit ceil 10Mbit linklayer ethernet burst 1599b/1 mpu 0b overhead 0b cburst 1600b/1 mpu 0b overhead 0b level 0
     Sent 16378282 bytes 11888 pkt (dropped 151, overlimits 0 requeues 0)
     rate 0bit 0pps backlog 0b 0p requeues 0
     lended: 4858 borrowed: 7030 giants: 0
     tokens: 54782 ctokens: 19175
    
    
    class htb 1:5 parent 1:3 prio 7 quantum 18750 rate 1500Kbit ceil 5Mbit linklayer ethernet burst 1599b/1 mpu 0b overhead 0b cburst 1600b/1 mpu 0b overhead 0b level 0
     Sent 58345053 bytes 39315 pkt (dropped 506, overlimits 0 requeues 0)
     rate 0bit 0pps backlog 0b 0p requeues 0
     lended: 12273 borrowed: 27042 giants: 0
     tokens: 68494 ctokens: 20550
    
    
    class htb 1:6 parent 1:2 prio 1 quantum 31250 rate 2500Kbit ceil 5Mbit linklayer ethernet burst 1600b/1 mpu 0b overhead 0b cburst 1600b/1 mpu 0b overhead 0b level 0
     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
     rate 0bit 0pps backlog 0b 0p requeues 0
     lended: 0 borrowed: 0 giants: 0
     tokens: 80000 ctokens: 40000
    

     

     

    Conclusion

    By the end of this best practice you should have tested the waters with Bandwidth Control. If you have any use cases you would like more information on, please drop a comment below.