The idea of Proxy HA on Web Gateway is to introduce failover and load balancing without the need for external load balancers. This setup is only recommended for smaller deployments or locations/offices within a larger deployment where an external load balancer is not feasible.Before deploying the Proxy HA setup we recommend to go through Knowledge base articles for Hardware Supported Platforms KB84154 , recommended Vmware platforms KB85908 and recommended Hyper-V platforms KB89366 to check the compatibility with virtual environment and your windows platform.
Three components make up the Proxy HA functionality:
You should first join your appliances together in Central Management and then configure Proxy HA. Note that Central Management and Proxy HA are completely independent of one another, so the priority settings for Central Management have nothing to do with the priority settings for Proxy HA. In fact, you can create a cluster of appliances in Proxy HA without having them in Central Management, but then you run the risk of the rules differing between each appliance because they are not synchronized. We always recommend to have the appliances of a Proxy HA cluster also joined in Central Management.
To set up Proxy HA, In the Webgateway UI go to:
Configuration > Proxies (HTTP(S), FTP, ICAP and IM)
Under ‘Network Setup’ select Proxy HA
In other words, if your users’ browsers are pointed directly to Proxy port 9090 and the mwg proxy port is also 9090, then we are going to redirect 9090 (original destination port) to 9090 (Destination proxy port) --see screenshot below---
Your proxy listener port must be unbound 0.0.0.0, binding the port will cause things to ProxyHA to stop functioning.
For example, 99 on the 1st mwg appliance and 89 on the 2nd. In Proxy HA one of the nodes will always be the director = the one with the highest priority. It is not possible to have two directors in an active/active way.
Local IP address of the node (do NOT configure a virtual IP address here). This IP address is used to auto discover the scanning nodes. All nodes have to be on the same subnet to be auto discovered.
This is the shared IP for the Proxy HA cluster which needs to be the same on all nodes in Proxy HA. Point your users’ browsers to the VIP.
Default is 51 and this also needs to be the same on all nodes.
This router ID is used by the VRRP health checks. If you have other devices on your network already using VRRP with router ID 51, make sure to change the router ID on your web gateways to something that is not used anywhere else on the network.
eth0 is default. Only if you are using multiple interfaces or you are NOT using eth0 should you change this value.
The VIP is an alias IP on the interface of the active director. The active director (the one that owns the VIP) takes ALL traffic from the clients and distributes this traffic to the scanning nodes (all MWGs involved on the same subnet). An active director node is also an active scanning node.
To verify which node “owns” the VIP, establish an SSH session with each node in Proxy HA and run the ‘ip addr show’ command. This will show you the active director and the nodes on standby:
VRRP is the protocol used for the health check amongst the pool of participating HA nodes and this is limited to a single subnet. The active director will send one multicast packet to 224.0.0.18 every second. If there is no multicast packet seen within 3 – 4 seconds from the active director, then failover occurs and the node with the second highest priority will become the active director. It informs the other nodes of its “director” or “master” state.
If this node goes down then the node with the next highest priority becomes director. Once the original Director comes back online (higher priority than the currently active director), it will resume its “master” state and take over the VIP once again.
Gratuitous ARPs are used to update the ARP tables of participating clients and routers. So every time the VIP changes ownership (failover occurs) the new director node sends a gratuitous ARP, the ARP tables are updated, and subsequent TCP/IP packets will reach the new director.
/var/log/messages
Also during normal operation, you will see the active director send "protocol 253" packets to broadcast destination 255.255.255.255. Each scanning node will send "protocol 253" unicast packets to the IP of the active director.
The Director node is receiving traffic from the clients and redirects it to scanning nodes on the kernel level using a built-in load sharing algorithm which takes into account resource usage and active number of connections. So if one scanning node is overloaded, the other will get more traffic to compensate. Generally load balancing is source-IP sticky, meaning the same client should reach the same scanning node. Normally, the active director is also an active scanning node.
mwg-mon
Example:
Customer configures a successful Proxy HA set-up, which includes:
Proxy Listeners defined: 9090, 2121
Port Redirects: 9090, 2121
Later on, they disable the FTP proxy port 2121 listener while it's still listed as a Proxy HA port redirect.
Proxy Listeners defined: 9090
Port Redirects: 9090, 2121
mwg-mon detects that 2121 is no longer listening and writes an error-message to /var/log/messages
Sep 28 05:53:01 PROXY1 mwg-mon: current state: failure
At this point, the Proxy HA set-up is broken as MWG will no longer forward any traffic to this MWG. This includes the port 9090 traffic that is still up and listening.
In the event that you're unable to determine what port is responsible for the error message, run this command:
mwg-mon -v -c
Example output : Working Proxy HA set-up.All defined ports are must be reachable
Reasons why one box is getting all the traffic:
Which machine blocked the request?
To tell which machine handled/blocked a request, put the System.HostName property on the block page.
I would like to test something with one specific MWG in my cluster. Can I do that?
Yes! When HA is set up on the Webgateways and one has to test a particular Webgateway, you have to create a new Proxy port - for example 9091 on THAT Webgateway.
Then, in your Browser settings point the browser to <IP address of MWG>:9091
Why is this necessary?
Because in Proxy HA we are forwarding (for example) ports 9090. Even if you point a browser directly to 9090, it will still be forwarded / handled via HA, meaning the director node will receive it and will either process that request itself or pass it to the scanning node, based on the mfend statistics. By creating an additional proxy port that is NOT forwarded, we can be sure that the individual Web Gateway to which the browser is pointed will process the test request.
What if I have want two independent Proxy HA Groups?
This is possible but requires a change on the CLI to the /etc/sysconfig/mfend file. The default load balancing ID (MFEND_LBID) is 51, so a unique LBID must be given to separate the proxy HA groups. Append the following line to the configuration file after the AUTO GENERATE CONFIG: MFEND_LBID='XX' where XX is the unique load balancing ID. For context here is the source discussion:
Director node = Jupiter 10.10.74.70
Backup director = Nibiru 10.10.74.86
VIP = 10.10.74.74
1. From /var/log/messages on Nibiru (Backup node): takes VIP when Director (Jupiter) goes down....
nibiru Keepalived_vrrp: VRRP_Instance({) Transition to MASTER STATE
nibiru Keepalived_vrrp: VRRP_Instance({) Entering MASTER STATE
nibiru Keepalived_vrrp: VRRP_Instance({) setting protocol VIPs.
nibiru Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74
nibiru mfend: set lbnetwork 1
nibiru mfend: state=master previous=backup
nibiru kernel: TX Adding IPV4 Address 10.10.74.74
nibiru Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74
nibiru kernel: TX Scanning partner has failed 10.10.74.70
2. From /var/log/messages on Jupiter (original Director): comes back online, takes back the VIP.....
Jupiter Keepalived_vrrp: VRRP_Instance({) forcing a new MASTER election
Jupiter Keepalived_vrrp: VRRP_Instance({) Transition to MASTER STATE
Jupiter Keepalived_vrrp: VRRP_Instance({) Entering MASTER STATE
Jupiter Keepalived_vrrp: VRRP_Instance({) setting protocol VIPs.
Jupiter Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74
Jupiter mfend: set lbnetwork 1
Jupiter mfend: state=master previous=backup
Jupiter Keepalived_vrrp: VRRP_Instance({) Sending gratuitous ARPs on eth0 for 10.10.74.74
Jupiter kernel: TX Adding IPV4 Address 10.10.74.74
3. From /var/log/messages on Nibiru: shows transition from the acting director to back up as a result of step two
nibiru Keepalived_vrrp: VRRP_Instance({) Received higher prio advert
nibiru Keepalived_vrrp: VRRP_Instance({) Entering BACKUP STATE
nibiru Keepalived_vrrp: VRRP_Instance({) removing protocol VIPs.
nibiru mfend: set lbnetwork 0
nibiru mfend: state=backup previous=master
nibiru kernel: TX Removing IPV4 Address 10.10.74.74
mfend-lb -s
If you have 3 or more Web Gateways in ProxyHA together, the "mfend-lb -s" output on the active director will show itself and all scanning nodes. When run on a scanning node, the command output will show the active director and only the scanning node itself (not any of the other scanning nodes).
Could you elaborate on the "only recommended for smaller deployments" statements? What is a smaller deployment in your terms? How many users? And why only for smaller deployments? Are there any problems with this setup?
Thanks
We use Proxy Ha with 5 proxies (about 20000 concurrent users...) without any problems. I don't think this is smaller deployment.
Since there is only one master distributing the traffic to the other nodes there is limit defined by the ethernet connerction. (We use 1GBit)
I think up to 7 or 8 Proxies HA will scale. For more proxies I might use external LB or try to seperate the proxies in 2 or 3 different HA envirnement. This will work within the same broadcast domain.
We use the last setup. 4 Proxy HA with 2,2,5 and 5 proxies. We separate the traffic using the pac-file.
Frank
Hi Frank,
interesting, thank you.
With one master there is only one machine handling inbound user connections and outbound traffic, right? It just pushes all scanning jobs to the other nodes?
Thanks!
Hi Frank,
I'm glad that is working for you, however it is not recommended by development to go more 5 per director. It is correct that all of the traffic is proxied by one appliance (although scanning is split) and so you can run into multiple issues running all traffic through the single device.
Smaller deployments generally means hundreds and at most a thousand or two users. At anything over 5000 you should strongly consider getting a load balancer or changing the deployment. Physical load balancers balance the traffic more efficiently and can fail over and perform proxy checks far more intelligently. It's just better overall.
Regards,
--CN
HI!
This setup was suggested by MC. We run this in 3 different countries with up to 5 proxies without any problems.
Frank
I guess I am not taking any risks. Since we have loadbalancers, we will just use them.
We also went the way of loadbalancers, but after looking at our Internet logs, it looks like all traffic is now coming from the loadbalancers, and the original source IP/user name is no longer avaialble. We will be using the proxy HA and moving off of the load balancers next week.
That's just a question on how you configure your loadbalancer. In your case it has probably been configured to do NAT. You can put any loadbalancer in transparent mode, where it acts like a router. That way you will see the original IPs hitting your web gateways.
You can also configure the load balancer to include the X-Forward-For header which MWG will then use as the client ip.
Curious, how are you performing authentication then if not on the Web Gateway? Is authentication occurring on the downstream proxy or loadbalancer? If so, you should be able to configure that device to also send up the X-Authenticated-User header that contains the username.
Yes, You definately should be able to use any load balancer but you need to know which mode:
Layer 7: SNAT - Source NAT only.. typicaly employed by load balancers in Layer 7 i.e. Reverse Proxy mode - is NOT source IP transparent unless you combine it with a 2 arm transparent proxy mode (which is messy).
Layer 4: Full NAT mode (DNAT & SNAT combined) is source IP transparent BUT is a pain because it requires an internal and external subnet (the MWGs would be on the external subnet).
Layer 4: Full transparent mode - All traffic is simply thrown from the load balancer to the MWGs as is without any change - MWGs must also be in transparent mode
But the one that reall works well and is easy (ish) to configure is:
Layer 4: DR (Direct Routing mode) F5 calls this n-path
Layer 4 DR mode load balancing enables you to use the MWGs in proxy mode but with full source IP transparency, the MWGs can't tell the difference. Bandwidth is almost unlimited as well as each MWG could havave its own gateway.
Loadbalancer.org have written a guide to load balancing McAfee Web Gateway. You should be able to get it working on most vendors load balancers using the same technique.
Out of interest what do people do about combining the security logs from multiple MWGs in a cluster?
Malcolm Turnbull wrote:
Out of interest what do people do about combining the security logs from multiple MWGs in a cluster?
When using multiple MWGs, the practice is to use the WebReporter or Content Security Reporter. These combine the logs into one database and report based on the aggregate data. Nitro also has some tools available, but I am not familiar with them.
Malcolm, if you don't want to deploy reporting servers, you could just use syslog. There is a great article here in the community on how to enable forwarding of the access logs via syslog. Another option is to pull the logs via scp or something alike and then combine the logs and sort them (Unix sort is your friend here).
One comment on VRirtual Router ID
Default is 51 and this also needs to be the same on all nodes.
This router ID is used by the VRRP health checks. If you have other devices on your network already using VRRP with router ID 51, make sure to change the router ID on your web gateways to something that is not used anywhere else on the network.
in case of having more CM on one shared network it is not enought to have uniq IDs for each CM Cluster. You have also to change th MFEND_LBID value in /etc/sysconfig/mfend (CLI access)
You have to add at the end the following line:
MFEND_LBID='<id>'
For easier adminstration we use the same value as used for virtual router ID.
Otherwise you will have unexpectable CM results. (Slow performance, not working...)
Frank
Frank,
What version of MWG were you having to edit the /etc/sysconfig/mfend file?
I have an issue where I need to change the VRRP ID after the fact, and it doesn't seem to matter what I change it to in the GUI, even after a reboot the mfend-lb -s still shows conflicts and issues in the /var/log/messages file.
Jeff
We use this starting with 7.2. Currently we use 7.3 and in our test lab 7.4.
Hi all,
does anyone know what happens if there are different Appliance Models in a HA-Cluster? This means e.g. if there is a WBG-5500-C Appliance and a WBG-4500-C in a HA-Cluster is the different perfomance considered?
Any Infos?
Cheers
If i am configuring 10 web gateways in a HA mode, Do we need to configure 10 different port for each appliance?
Example: 192.168.1.10: 9090
192.168.2.10:8080
192.168.4.12:9091
192.168.10.10:9092.
Hi,
no, you dont. You can access the whole cluster using one port.
You can, if necessary, configure multiple ports. For load balancing you have to add the ports which should be forwarded.
Any port listed here is afterwards load balanced by HA-Cluster.
Cheers
So i can configure my locations in these way:
India Central manager and HA : 192.168.1.10: 9090
Austrlia: 192.168.2.10:9090
US: 192.168.4.12:9090
Denmark:192.168.10.10:9090
and i need to add all the other location scanner in the HA group of india, am i right?
you just have to separate Central Management and HA Cluster. This are completely different functions with completely different ports.
In your environment.
1) You can manage all your appliances (virtual/physical) in one central management. Just take care about the TCP latency between the locations. Central Management uses it´s own port which can also be changed if needed.
2) HA Cluster is a completely different thing. Port 9090 is the proxy port for client connections.
Some questions for your design. What should happen with the client requests if the proxy e.g. in Denmark fails? Should the clients be redirected to another location?
Cheers
Thanks for the answer, Yes, if the Denmark location proxy fails, the Denmark users should be redirected to another proxy.
Basically I am looking for a design with below requirements:
so, if a user from Australia connects to a proxy in US is okay?
Please note, if you are using a HA-Cluster and the Master appliance is located in US, any traffic will always go through this appliance.
The HA-cluster Master holds the virtual IP. Any endpoint connects to this IP.
So this is also okay for you?
If you are using MCP i would configure the whole solution in this way.
With MCP it is easy, because you can configure multiple Proxies for the endpoint. If one HA-Cluster is completely down, MCP will connect to the next configured proxy (can be configured based on response time or as an static list).
If you do not use MCP you might use a PAC file configuring a Proxy array.
Cheers
Hi Troja ,
Thank you very much for your quick reply!
I am okay with " if a user from Australia connects to a proxy in US", because as i am using a central management proxy any way all my proxy policies will be same.
I have a small query here:
My requirement: i want all my policies should be in sync and if any local proxy goes down i want that traffic to be taken care, so want i should do?
Policy sync i can achieve by configuring nodes in Central management, but what happens if my local proxy goes down, how i can take of the user traffic.
kindly suggest and help me.
Regards,
PRASANTH.
you are confuding me a little bit. Central Management is NOT a HA-Cluster. 🙂
There can be proxies configured as a single Proxy or HA-Cluster, but they all can be configured in one central Management.
Perhaps this helps, the screenshots are in German, but this does not matter. Note, in the examples shown below any appliance has the complete policy.
Option 1:
Option 2:
The same as option 1, but proxy1 and proxy2 are acting as a HA-Cluster. Load Sharing and failover is possible.
Option 3:
If there is no outage, the scanning node will never get the VIP assigned.
All appliances are again in one central Management.
IF proxy 1 is the HA-Master and holds the VIP, any client is always connecting to this appliances. Afterwards the kernel module on the appliance will do the load balancing stuff. Also the return package to the client will always be delivered from proxy1.
Option 4:
Hope this helps.
Cheers
Hi Thorsten,
I might just have to go ahead and borrow what you just wrote there because its very well laid out.
Best Regards,
Jon
Hi Jon,
feel free! 🙂
Cheers,
Thorsten
Hi Team,
I got a doubt, if we take WBG-5500-C, it has 4 x 10/100/1000 MB RJ-45 Ethernet ports, can i use 2 NIC ports for two different networks. is it possible. PF the sample diagram:
Regards,
PRASANTH.
Hello Prasanth,
I believe what you are asking about is NIC bonding, which is supported by MWG starting in version 7.5.2. See p.422 of the product guide:
Hi Michael Grider ,
Thanks for the reply.
My doubt is about using the 2 NICs fro two different networks.
like: eth0>>>192.168.10.0
eth1>>10.10.10.0
If you are only using 2 NICs on MWG (your diagram shows 4 on each MWG), then it is easy to connect it to 2 different networks if you understand the basics of routing. MWG only supports 1 default gateway (unlike Windows PCs which allow 1 per NIC). Using your example:
eth0>>192.168.10.0 /24
eth1>>10.10.10.0 /24
- You will typically want your default gateway to point towards the Internet, for example: 10.10.10.1 out eth1
- If you have clients only reachable out eth0 that are not directly connected to the same subnet as eth0, you will also need to add static routes (eg.: 192.168.22.0/24 via 192.168.10.1 out eth0)
Hi Michael Grider,
Thanks for the reply!
I have used the above diagram as a reference of MWG in HA. May look in the below diagram:
As per the above diagram :
I have to networks:
1.)Normal local traffic
2.)DMZ traffic
I want to filter the both the network traffic, so as per the diagram if i configure:
eth0>>192.168.10.0 /24 DMZ traffic
eth1>>10.10.10.0 /24 Local traffic
MWG MGMT ip 10.10.10.2
Will i be able to filter the traffic of both the networks?
My Question is simple, is it possible to filter the traffic of two different networks using one MWG by assigning two different ip addresses to the interfaces. IF possible how?
Kindly help
Yes, you can do this. If you are having trouble getting it working, I recommend opening a ticket with MWG Tech Support or starting a new discussion on the MWG forum rather than here in the comments section of this article.
Hi Team,
Can I user a proxy HA and interface bonding together?
can it work??
but the network in not stable
VIP:192.168.1.40
MWG1 192.168.1.41
MWG2 192.168.1.42
Alan
Yes you can,
we already did such a configuration sucsessfully at customers.
Cheers
Hello Everyone.
I have a question about VRRP.
What if I have to handle 10Gbps traffic?
"The document says: When a 10G network interface is installed on an appliance that is configured as the node director, the Max throughput for this node is increased. However, the scaling limit of the kernel-mode MLOS
Driver must not be exhausted or exceeded."
Do I understand that traffic above 1Gbps is not properly handled by Director node?
I want to use this vm machine?
Hi Requ,
That is correct, when using ProxyHA, the McAfee Network Driver (MFEND) is used, and it has a limitation of 1G.
Other modes like Layer 2 might be viable instead or if you have a physical load balancer. Using L2 or a LB will mean that another device is handling the failover and load balancing whereas with proxyHA, its handled by the MWG.
Best Regards,
Jon
Hi,
Regarding port redirects... Is it so, that it can handle only HTTP(S), FTP and ICAP protocols? Asking because I'd like to balance also SOCKS, but when I configured it as HTTP protocol and port 1080 and tried telnet to VIP and 1080 nothing happened. Also in pcap it looks like the proxy(.37) did get the request to it's VIP(.7) and port 1080 but then didn't know what to do with it.
Hi Slizka,
SOCKS isnt officially supported for ProxyHA as things like UDP over SOCKS and other things I cant think of wont work (because they require dynamic ports).
From your capture it looks like MWG got the SYN packet, but the director didnt respond to it, or it forwarded it to another device in the pool.
However if the client is only talking to mwg over 1080, it should work. Are you sure you have everything else configured? Like enabling SOCKS on the other device? Otherwise, if MWG was in a different mode before ProxyHA (namely WCCP) you need a reboot.
Best Regards,
Jon
Hi Jon,
Yes, all is configured in same way as the HTTP and FTP proxy, so basically it should work(also the proxy was rebooted). At the same time I was capturing packets also on second node in cluster, but no packet did reach it.
Any idea if this might be implemented later on? I guess I'll just drop this for now as not possible to implement and in case customer would really need it then I'll suggest to connect to proxy IP instead of VIP.
Corporate Headquarters
6220 America Center Drive
San Jose, CA 95002 USA