cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Understanding Proxy HA

Overview

The idea of Proxy HA on Web Gateway is to introduce failover and load balancing without the need for external load balancers. This setup is only recommended for smaller deployments or locations/offices within a larger deployment where an external load balancer is not feasible.

 

Three components make up the Proxy HA functionality:

  • A shared IP address (VIP) that the end user clients communicate with. The node that holds this VIP at any given time is considered the “active director”
  • A kernel module on the mwg that intercepts and routes traffic to scanning nodes (MFEND). Traffic that is not intercepted by MFEND will be handled by the OS as it would be without Proxy HA (for example SSH, UI traffic and so on)
  • Scanning nodes that perform the “normal” MWG filtering

 

 

Set up

You should first join your appliances together in Central Management and then configure Proxy HA.  Note that Central Management and Proxy HA are completely independent of one another, so the priority settings for Central Management have nothing to do with the priority settings for Proxy HA. In fact, you can create a cluster of appliances in Proxy HA without having them in Central Management, but then you run the risk of the rules differing between each appliance because they are not synchronized. We always recommend to have the appliances of a Proxy HA cluster also joined in Central Management.

 

Settings

To set up Proxy HA, In the Webgateway UI go to:

Configuration > Proxies (HTTP(S), FTP, ICAP and IM)

Under ‘Network Setup’ select Proxy HA

 

Port Redirects

    • Select “http” as the default protocol
    • The “Original destination ports” must contain the proxy port the end users are pointed to.
    • The “Destination proxy port” must contain the proxy port the mwg service is using as its proxy port.
    • Port redirects need to be ports that are actually listening. There cannot be a port listed in the port redirects area that is disabled in either the HTTP or FTP proxy port definition list. See the 'mwg-mon' section below for further details.

 

In other words, if your users’ browsers are pointed directly to Proxy port 9090 and the mwg proxy port is also 9090, then we are going to redirect 9090 (original destination port) to 9090 (Destination proxy port) --see screenshot below---

 

9090 redirect only.png

 

 

Your proxy listener port must be unbound 0.0.0.0, binding the port will cause things to ProxyHA to stop functioning.

 

          HTTP port definition list.png

 

 

 

 

Director Priority

For example, 99 on the 1st mwg appliance and 89 on the 2nd.  In Proxy HA one of the nodes will always be the director = the one with the highest priority. It is not possible to have two directors in an active/active way.

 

Management IP

Local IP address of the node (do NOT configure a virtual IP address here). This IP address is used to auto discover the scanning nodes. All nodes have to be on the same subnet to be auto discovered.

 

Virtual IPs

This is the shared IP for the Proxy HA cluster which needs to be the same on all nodes in Proxy HA. Point your users’ browsers to the VIP.

 

Virtual router ID

Default is 51 and this also needs to be the same on all nodes.

This router ID is used by the VRRP health checks. If you have other devices on your network already using VRRP with router ID 51, make sure to change the router ID on your web gateways to something that is not used anywhere else on the network.

 

VRRP interface

eth0 is default. Only if you are using multiple interfaces or you are NOT using eth0 should you change this value.

 

 

How it works – More Details

 

Virtual IP (VIP)

The VIP is an alias IP on the interface of the active director. The active director (the one that owns the VIP) takes ALL traffic from the clients and distributes this traffic to the scanning nodes (all MWGs involved on the same subnet). An active director node is also an active scanning node.

 

To verify which node “owns” the VIP, establish an SSH session with each node in Proxy HA and run the ‘ip addr show’ command. This will show you the active director and the nodes on standby:

check who has VIP.png

 

Failover

VRRP is the protocol used for the health check amongst the pool of participating HA nodes and this is limited to a single subnet. The active director will send one multicast packet to 224.0.0.18 every second.  If there is no multicast packet seen within 3 – 4 seconds from the active director, then failover occurs and the node with the second highest priority will become the active director. It informs the other nodes of its “director” or “master” state.

If this node goes down then the node with the next highest priority becomes director. Once the original Director comes back online (higher priority than the currently active director), it will resume its “master” state and take over the VIP once again.

Gratuitous ARPs are used to update the ARP tables of participating clients and routers. So every time the VIP changes ownership (failover occurs) the new director node sends a gratuitous ARP, the ARP tables are updated, and subsequent TCP/IP packets will reach the new director.

 

    • VRRP errors and info about master/backup status can be found in:

 

/var/log/messages

 

         Also during normal operation, you will see the active director send "protocol 253" packets to broadcast destination 255.255.255.255. Each scanning node will send "protocol 253" unicast packets to the IP of the active director.

 

Load Balancing

The Director node is receiving traffic from the clients and redirects it to scanning nodes on the kernel level using a built-in load sharing algorithm which takes into account resource usage and active number of connections. So if one scanning node is overloaded, the other will get more traffic to compensate. Generally load balancing is source-IP sticky, meaning the same client should reach the same scanning node. Normally, the active director is also an active scanning node.

 

mwg-mon

    • A health-check process that monitors the Proxy HA redirect ports you defined to ensure that they are available to accept incoming traffic.
    • if a redirect port is not listening, then mwg-mon detects this as a failure and load balancing will cease.
    • monitors all of the port redirects as a whole and not individually. See example below.

 

Example:

 

Customer configures a successful Proxy HA set-up, which includes:

 

Proxy Listeners defined: 9090, 2121

Port Redirects: 9090, 2121

 

Later on, they disable the FTP proxy port 2121 listener while it's still listed as a Proxy HA port redirect.

 

Proxy Listeners defined: 9090

Port Redirects: 9090, 2121

 

mwg-mon detects that 2121 is no longer listening and writes an error-message to /var/log/messages

 

Sep 28 05:53:01 PROXY1 mwg-mon: current state: failure

 

At this point, the Proxy HA set-up is broken as MWG will no longer forward any traffic to this MWG. This includes the port 9090 traffic that is still up and listening.

 

In the event that you're unable to determine what port is responsible for the error message, run this command:

 

mwg-mon -v -c

 

 

Common scenarios

Reasons why one box is getting all the traffic:

  • No “Port redirects” configured on the director node. If there is no port redirect the network driver on the director node will not redirect the traffic, but handle it locally.
  • All traffic is coming from the same source IP because there is a downstream proxy or a NATing device in place. Proxy HA load balancing is source IP sticky.
  • The director does not know about other scanning nodes, you will need to review the HA configuration – use mfend, perhaps they are on different subnets?

 

Which machine blocked the request?

To tell which machine handled/blocked a request, put the System.HostName property on the block page.

 

I would like to test something with one specific MWG in my cluster. Can I do that?

Yes! When HA is set up on the Webgateways and one has to test a particular Webgateway, you have to create a new Proxy port - for example 9091 on THAT Webgateway.

Then, in your Browser settings point the browser to <IP address of MWG>:9091

 

Why is this necessary?

Because in Proxy HA we are forwarding (for example) ports 9090. Even if you point a browser directly to 9090, it will still be forwarded / handled via HA, meaning the director node will receive it and will either process that request itself or pass it to the scanning node, based on the mfend statistics. By creating an additional proxy port that is NOT forwarded, we can be sure that the individual Web Gateway to which the browser is pointed will process the test request.

 

What if I have want two independent Proxy HA Groups?

This is possible but requires a change on the CLI to the /etc/sysconfig/mfend file. The default load balancing ID (MFEND_LBID) is 51, so a unique LBID must be given to separate the proxy HA groups. Append the following line to the configuration file after the AUTO GENERATE CONFIG: MFEND_LBID='XX' where XX is the unique load balancing ID. For context here is the source discussion:

 

 

Troubleshooting

  • Search /var/log/messages for VRRP errors and master/backup status
    • Director node = Jupiter 10.10.74.70

    • Backup director = Nibiru 10.10.74.86

    • VIP = 10.10.74.74

 

1.   From /var/log/messages on Nibiru (Backup node): takes VIP when Director (Jupiter) goes down....

nibiru Keepalived_vrrp: VRRP_Instance({) Transition to MASTER STATE

nibiru Keepalived_vrrp: VRRP_Instance({) Entering MASTER STATE

nibiru Keepalived_vrrp: VRRP_Instance({) setting protocol VIPs.

nibiru Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74

nibiru mfend: set lbnetwork 1

nibiru mfend: state=master previous=backup

nibiru kernel: TX Adding IPV4 Address 10.10.74.74

nibiru Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74

nibiru kernel: TX Scanning partner has failed 10.10.74.70

 

2.   From /var/log/messages on Jupiter (original Director): comes back online, takes back the VIP.....

 

Jupiter Keepalived_vrrp: VRRP_Instance({) forcing a new MASTER election

Jupiter Keepalived_vrrp: VRRP_Instance({) Transition to MASTER STATE

Jupiter Keepalived_vrrp: VRRP_Instance({) Entering MASTER STATE

Jupiter Keepalived_vrrp: VRRP_Instance({) setting protocol VIPs.

Jupiter Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74

Jupiter mfend: set lbnetwork 1

Jupiter mfend: state=master previous=backup

Jupiter Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74

Jupiter kernel: TX Adding IPV4 Address 10.10.74.74

 

 

3.   From /var/log/messages on Nibiru: shows transition from the acting director to back up as a result of step two

 

nibiru Keepalived_vrrp: VRRP_Instance({) Received higher prio advert

nibiru Keepalived_vrrp: VRRP_Instance({) Entering BACKUP STATE

nibiru Keepalived_vrrp: VRRP_Instance({) removing protocol VIPs.

nibiru mfend: set lbnetwork 0

nibiru mfend: state=backup previous=master

nibiru kernel: TX Removing IPV4 Address 10.10.74.74

 

  • VIP check on command line using ‘ip addr show’ command (see screenshots above)

 

  • ‘mfend-lb -s’ ……command can also be used to see the current HA status.

 

mfend-lb -s

mfendScreenshots2.png

If you have 3 or more Web Gateways in ProxyHA together, the "mfend-lb -s" output on the active director will show itself and all scanning nodes. When run on a scanning node, the command output will show the active director and only the scanning node itself (not any of the other scanning nodes).

Labels (1)
Comments

Could you elaborate on the "only recommended for smaller deployments" statements? What is a smaller deployment in your terms? How many users? And why only for smaller deployments? Are there any problems with this setup?

Thanks

We use Proxy Ha with 5 proxies (about 20000 concurrent users...) without any problems. I don't think this is smaller deployment.

Since there is only one master distributing the traffic to the other nodes there is limit defined by the ethernet connerction. (We use 1GBit)

I think up to 7 or 8 Proxies HA will scale. For more proxies I  might use external LB or try to seperate the proxies in 2 or 3 different HA envirnement. This will work within the same broadcast domain.

We use the last setup. 4 Proxy HA with 2,2,5 and 5 proxies. We separate the traffic using the pac-file.

Frank

Hi Frank,

interesting, thank you.

With one master there is only one machine handling inbound user connections and outbound traffic, right? It just pushes all scanning jobs to the other nodes?

Thanks!

Hi Frank,

I'm glad that is working for you, however it is not recommended by development to go more 5 per director. It is correct that all of the traffic is proxied by one appliance (although scanning is split) and so you can run into multiple issues running all traffic through the single device.

Smaller deployments generally means hundreds and at most a thousand or two users. At anything over 5000 you should strongly consider getting a load balancer or changing the deployment. Physical load balancers balance the traffic more efficiently and can fail over and perform proxy checks far more intelligently. It's just better overall.

Regards,

--CN

HI!

This setup was suggested by MC. We run this in 3 different countries with up to 5 proxies without any problems.

Frank

I guess I am not taking any risks. Since we have loadbalancers, we will just use them.

We also went the way of loadbalancers, but after looking at our Internet logs, it looks like all traffic is now coming from the loadbalancers, and the original source IP/user name is no longer avaialble. We will be using the proxy HA and moving off of the load balancers next week.

That's just a question on how you configure your loadbalancer. In your case it has probably been configured to do NAT. You can put any loadbalancer in transparent mode, where it acts like a router. That way you will see the original IPs hitting your web gateways.

You can also configure the load balancer to include the X-Forward-For header which MWG will then use as the client ip.

Curious, how are you performing authentication then if not on the Web Gateway? Is authentication occurring on the downstream proxy or loadbalancer? If so, you should be able to configure that device to also send up the X-Authenticated-User header that contains the username.

Yes, You definately should be able to use any load balancer but you need to know which mode:

Layer 7: SNAT - Source NAT only.. typicaly employed by load balancers in Layer 7 i.e. Reverse Proxy mode - is NOT source IP transparent unless you combine it with a 2 arm transparent proxy mode (which is messy).

Layer 4: Full NAT mode (DNAT & SNAT combined) is source IP transparent BUT is a pain because it requires an internal and external subnet (the MWGs would be on the external subnet).

Layer 4: Full transparent mode - All traffic is simply thrown from the load balancer to the MWGs as is without any change - MWGs must also be in transparent mode

But the one that reall works well and is easy (ish) to configure is:

Layer 4: DR (Direct Routing mode) F5 calls this n-path

Layer 4 DR mode load balancing enables you to use the MWGs in proxy mode but with full source IP transparency, the MWGs can't tell the difference. Bandwidth is almost unlimited as well as each MWG could havave its own gateway.

Loadbalancer.org have written a guide to load balancing McAfee Web Gateway.  You should be able to get it working on most vendors load balancers using the same technique.

Out of interest what do people do about combining the security logs from multiple MWGs in a cluster?

Malcolm Turnbull wrote:


                       

Out of interest what do people do about combining the security logs from multiple MWGs in a cluster?                   

When using multiple MWGs, the practice is to use the WebReporter or Content Security Reporter.  These combine the logs into one database and report based on the aggregate data.  Nitro also has some tools available, but I am not familiar with them.

Malcolm, if you don't want to deploy reporting servers, you could just use syslog. There is a great article here in the community on how to enable forwarding of the access logs via syslog. Another option is to pull the logs via scp or something alike and then combine the logs and sort them (Unix sort is your friend here).

One comment on VRirtual Router ID

Virtual router ID

Default is 51 and this also needs to be the same on all nodes.

This router ID is used by the VRRP health checks. If you have other devices on your network already using VRRP with router ID 51, make sure to change the router ID on your web gateways to something that is not used anywhere else on the network.

in case of having more CM on one shared network it is not enought to have uniq IDs for each CM Cluster. You have also to change th MFEND_LBID value in /etc/sysconfig/mfend (CLI access)

You have to add at the end the following line:


MFEND_LBID='<id>'

For easier adminstration we use the same value as used for virtual router ID.

Otherwise you will have unexpectable CM results. (Slow performance, not working...)

Frank

Frank,

What version of MWG were you having to edit the /etc/sysconfig/mfend file?

I have an issue where I need to change the VRRP ID after the fact, and it doesn't seem to matter what I change it to in the GUI, even after a reboot the mfend-lb -s still shows conflicts and issues in the /var/log/messages file.


Jeff

We use this starting with 7.2. Currently we use 7.3 and in our test lab 7.4.

Hi all,

does anyone know what happens if there are different Appliance Models in a HA-Cluster? This means e.g. if there is a WBG-5500-C Appliance and a WBG-4500-C in a HA-Cluster is the different perfomance considered?

Any Infos?

Cheers

If i am configuring 10 web gateways in a HA mode, Do we need to configure 10 different port for each appliance?

Example: 192.168.1.10: 9090

               192.168.2.10:8080

               192.168.4.12:9091

                 192.168.10.10:9092.

Hi,

no, you dont. You can access the whole cluster using one port.
You can, if necessary, configure multiple ports. For load balancing you have to add the ports which should be forwarded.

Any port listed here is afterwards load balanced by HA-Cluster.

Cheers

So i can configure my locations in these way:

India Central manager and HA : 192.168.1.10: 9090

               Austrlia: 192.168.2.10:9090

               US: 192.168.4.12:9090

                Denmark:192.168.10.10:9090

and i need to add all the other location scanner in the HA group of india, am i right?

Hi ,

you just have to separate Central Management and HA Cluster. This are completely different functions with completely different ports.

In your environment.

1) You can manage all your appliances (virtual/physical) in one central management. Just take care about the TCP latency between the locations. Central Management uses it´s own port which can also be changed if needed.

2) HA Cluster is a completely different thing. Port 9090 is the proxy port for client connections.

Some questions for your design. What should happen with the client requests if the proxy e.g. in Denmark fails? Should the clients be redirected to another location?

Cheers

Hi ,

Thanks for the answer, Yes, if the Denmark location proxy fails, the Denmark users should be redirected to another proxy.

Basically I am looking for a design with below requirements:

  • 10 locations with central management, like if i update policies in central manager, policies should be synced to all the other locations.
  • If one location proxy fails, i want the users should be redirected to the another proxy.

Hi ,

so, if a user from Australia connects to a proxy in US is okay?

Please note, if you are using a HA-Cluster and the Master appliance is located in US, any traffic will always go through this appliance.

The HA-cluster Master holds the virtual IP. Any endpoint connects to this IP.

So this is also okay for you?

If you are using MCP i would configure the whole solution in this way.

  • One HA-cluster in every location
  • All proxy systems are in one central management
  • Define an DNS entry for any HA-Cluster

With MCP it is easy, because you can configure multiple Proxies for the endpoint. If one HA-Cluster is completely down, MCP will connect to the next configured proxy (can be configured based on response time or as an static list).

If you do not use MCP you might use a PAC file configuring a Proxy array.

Cheers

Hi Troja ,

Thank you very much for your quick reply!

I am okay with " if a user from Australia connects to a proxy in US", because as i am using a central management proxy any way all my proxy policies will be same.

I have a small query here:

  • If i use HA-cluster, director node will receive the traffic from the clients and redirects it to scanning nodes on the kernel level.
  • If I use central management only policies will be synced and second priority node will became master, if master goes down.

My requirement: i want all my policies should be in sync and if any local proxy goes down i want that traffic to be taken care, so want i should do?

Policy sync i can achieve by configuring nodes in Central management, but what happens if my local proxy goes down, how i can take of the user traffic.

kindly suggest and help me.


Regards,

PRASANTH.

Hi ,

you are confuding me a little bit. Central Management is NOT a HA-Cluster. 🙂

There can be proxies configured as a single Proxy or HA-Cluster, but they all can be configured in one central Management.

Perhaps this helps, the screenshots are in German, but this does not matter. Note, in the examples shown below any appliance has the complete policy.

Option 1:

  • one virtual appliance for management (could also be removed and the proxies can be managed e.g. by proxy 1)
  • two physical appliances in normal proxy mode (No HA, no load balancing, no failover)

1.png

Option 2:

The same as option 1, but proxy1 and proxy2 are acting as a HA-Cluster. Load Sharing and failover is possible.

2.png

Option 3:

  • Proxy1 and proxy2 are responsible for the VIP.
  • The traffic is load balanced over all three appliances.
  • If Proxy1 and Proxy2 fails the VIP will be transfered the the Scanning node. (This means the Scanning node will be promoted to a HA-Master)

If there is no outage, the scanning node will never get the VIP assigned.

All appliances are again in one central Management.

IF proxy 1 is the HA-Master and holds the VIP, any client is always connecting to this appliances. Afterwards the kernel module on the appliance will do the load balancing stuff. Also the return package to the client will always be delivered from proxy1.

3.png

Option 4:

  • There are 5 Appliances in one central Management.
  • There are two completely separated HA Clusters. If proxy1 and proxy2 are not available, the clients will not be able to connect to the internet any more.

4.png

Hope this helps.

Cheers

Hi Troja,

Thank you very much for such a detailed explanation. Than you. now i am clear.

ceers..

Hi Thorsten,

I might just have to go ahead and borrow what you just wrote there because its very well laid out.

Best Regards,

Jon

Hi Jon,

feel free! 🙂

Cheers,

Thorsten

Hi Team,

I got a doubt, if we take WBG-5500-C, it has 4 x 10/100/1000 MB RJ-45 Ethernet ports, can i use  2 NIC ports for two different networks.  is it possible. PF the sample diagram:

Two Leg Architecture.jpg

Regards,

PRASANTH.

Hello Prasanth,


I believe what you are asking about is NIC bonding, which is supported by MWG starting in version 7.5.2. See p.422 of the product guide:

McAfee KnowledgeBase - Web Gateway 7.5.2 Product Guide

Hi Michael Grider ,

Thanks for the reply.

My doubt is about using the 2 NICs fro two different networks.

like: eth0>>>192.168.10.0

      eth1>>10.10.10.0

If you are only using 2 NICs on MWG (your diagram shows 4 on each MWG), then it is easy to connect it to 2 different networks if you understand the basics of routing. MWG only supports 1 default gateway (unlike Windows PCs which allow 1 per NIC). Using your example:

      eth0>>192.168.10.0 /24

      eth1>>10.10.10.0 /24

- You will typically want your default gateway to point towards the Internet, for example: 10.10.10.1 out eth1

- If you have clients only reachable out eth0 that are not directly connected to the same subnet as eth0, you will also need to add static routes (eg.: 192.168.22.0/24 via 192.168.10.1 out eth0)

Hi Michael Grider,

Thanks for the reply!

I have used the above diagram as a reference of MWG in HA. May look in the below diagram:

Two Leg Architecture without HA.jpg

As per the above diagram :

I have to networks:

1.)Normal local traffic

2.)DMZ traffic

I want to filter the both the network traffic, so as per the diagram if i configure:

eth0>>192.168.10.0 /24 DMZ traffic

eth1>>10.10.10.0 /24 Local traffic

MWG MGMT ip 10.10.10.2

Will i be able to filter the traffic of both the networks?

My Question is simple, is it possible to filter the traffic of two different networks using one MWG by assigning two different ip addresses to the interfaces. IF possible how?

Kindly help

Yes, you can do this. If you are having trouble getting it working, I recommend opening a ticket with MWG Tech Support or starting a new discussion on the MWG forum rather than here in the comments section of this article.

Hi Team,

Can I user a proxy HA and interface bonding together?

can it work??

2017-10-31 下午 11-48-25.jpg

2017-10-31 下午 11-48-52.jpg

but the network in not stable

VIP:192.168.1.40

MWG1 192.168.1.41

MWG2 192.168.1.42

2017-10-31 下午 11-54-16.jpg

2017-10-31 下午 11-51-27.jpg

Alan

Yes you can,

we already did such a configuration sucsessfully at customers.

Cheers

Hello Everyone.

I have a question about VRRP.

What if I have to handle 10Gbps traffic?

"The document says: When a 10G network interface is installed on an appliance that is configured as the node director, the Max throughput for this node is increased. However, the scaling limit of the kernel-mode MLOS

Driver must not be exhausted or exceeded."

Do I understand that traffic above 1Gbps is not properly handled by Director node?

I want to use this vm machine?

Hi Requ,

That is correct, when using ProxyHA, the McAfee Network Driver (MFEND) is used, and it has a limitation of 1G.

Other modes like Layer 2 might be viable instead or if you have a physical load balancer. Using L2 or a LB will mean that another device is handling the failover and load balancing whereas with proxyHA, its handled by the MWG.

Best Regards,

Jon

Hi,

Regarding port redirects... Is it so, that it can handle only HTTP(S), FTP and ICAP protocols? Asking because I'd like to balance also SOCKS, but when I configured it as HTTP protocol and port 1080 and tried telnet to VIP and 1080 nothing happened. Also in pcap it looks like the proxy(.37) did get the request to it's VIP(.7) and port 1080 but then didn't know what to do with it.

socks_vip_dump.JPG

Hi Slizka,

SOCKS isnt officially supported for ProxyHA as things like UDP over SOCKS and other things I cant think of wont work (because they require dynamic ports).

From your capture it looks like MWG got the SYN packet, but the director didnt respond to it, or it forwarded it to another device in the pool.

However if the client is only talking to mwg over 1080, it should work. Are you sure you have everything else configured? Like enabling SOCKS on the other device? Otherwise, if MWG was in a different mode before ProxyHA (namely WCCP) you need a reboot.

Best Regards,

Jon

Hi Jon,

Yes, all is configured in same way as the HTTP and FTP proxy, so basically it should work(also the proxy was rebooted). At the same time I was capturing packets also on second node in cluster, but no packet did reach it.

Any idea if this might be implemented later on? I guess I'll just drop this for now as not possible to implement and in case customer would really need it then I'll suggest to connect to proxy IP instead of VIP.

Contributors
Version history
Revision #:
2 of 3
Last update:
‎03-20-2018 11:09 AM
Updated by: