I have been working with Sidewinders off and on for 15years. From back in the Borderware years, to SecureZone, and Sidewinder versions 6 and 7. Version 7 is solid and is working fine for us, but our hardware is EOL so we decided to get new appliances with version 8 about 9 months ago. In the last 9 months I have fought with version 8, trying all the new (to us) features, SmartFilter, Logon Collector, Active Directory integration, Web Reporter, Global Threat Intelligence. I have all the bugs and problems ironed out except one.
I like what they’ve done in Version 8 with Applications – it should be very powerful, but there is a serious problem in how the http proxy determines what application the traffic matches. This can be seen on busy websites like newspapers with many ads and statistics tracking embedded html. When the http proxy (or application) doesn’t match the traffic, something is hanging or timing out. Multiply this by dozens of ads and other embedded html links and these pages are unusable. Attached is an example of traffic getting denied because the Application is <Unknown TCP>. Normally when this is not working, the browser will just sit with a blank white screen for a few minutes, spinning, “waiting for whatever.com…” and then the page may or may not appear, maybe partially appear (pictures broken) or maybe the html stylesheet will not load and you get just a text representation of the page. Stranger yet, I can usually replicate it pretty quickly, but sometimes it will work fine for several hours. The problem always comes back though, and it's not just one website.
If I disable GTI and App Defenses, SmartFilter, and anti-virus, it works a little better but still not good enough to put in a production environment at my company. I’ve worked with support on this for hours, it is escalated to Engineering, but they don’t have a root cause. So the question is, is anyone using Version 8 in production with SmartFilter, GTI, Anti-Virus, etc. all enabled? I’ve been patching when new patches come out and the problem remains. I’m thinking about going back to version 7 so I’m looking for any advice or discussion on this.
Thanks for reading this far!
Message was edited by: danob on 8/9/12 4:35:34 PM CDT
My product experience is almost idenitcal to yours - I too started just over 15 years ago with Borderware and have worked throught the different Secure Computing/McAftee products (SecureZone, Firewall for NT! and joining the Sidewinder ranks at v5).
I can't offer any specific help to your situation, but wanted to let you know you are not alone in your frustrations. Despite working with it for the better part of 18 months and enjoying much of the new functionality version 8 offers, I have never become completely comfortable with this version. However, despite the stirling efforts of Sam Liedl and Matt Tuma in the Firewall Enterprise Community, I have always felt that there are situations where v8 suddenly starts to behave in the way you are describing and no amount of support (official or through this community) has revealed the reason behind it.
I am a reseller engineer and I therefore have exposure to multiple installations in different environments. Since 8.2, the product seems to have become far more stable, but there are still occasions where it simply doesn't behave in the way you would expect it to. The only observation I have is that it seems more likely to occur when the network environment is busy. I have a number of installations which seem to be bahiving impeccably, yet one or two which have been a nightmare. In one particular case a customer with an S2008 (which is more than large enough for the number of users) has seeminly teetered on the brink of collapse within days of installing it. When testing the rules out of hours with 4 or 5 users, everything worked as expected, but from the day it went live with the general population, services which had passed testing with flying colours seemly stopped working.
The main symptom? Exactly what you are seeing - tons of <Unknown TCP> entries in the audit.
Matt & Sam have gone a long way to explain what this means (because in order to match traffic to a specific signature, the Firewall must first allow a couple of packets to pass through), but I have never been able to get to the bottom of the real reason *why* it causes so much trouble with specific installations. The only gut feeling I have is that it always seems to be more acutely evident in 'busy' network environments. But, gut feelings aren't particularly useful when there isn't neccesarily and specific evidence to give to the support guys.
What I have found is if you need to pass protocols through the Firewall where needing to know exactly what is going on isn't quite so important, resort to using a custom application entry using the port(s) in question and the Firewall doesn't have to work quite so hard and other services then also seem to benefit.
At the end of the day, I still think there's a point where this version seems to reach a tipping point, and when it does so the audit becomes dominated with <Unknown TCP> entries. I just wish I could tell you why.
I am experiencing similar issues with application discovery. Recently, perhaps just in the past few weeks, we started experiencing very common "blank page" issues with very common sites -- like Google and Amazon -- while using SSL. Since Google has defaulted to using SSL for searches, this means that we saw this issue repeatedly. After a few page reloads we could get it to load, but often the FW would just reset the connection and issue an audit about traffic being denied by ACL. Since I have a rule specifically allowing the "SSL/TLS (HTTPS)" application I couldn't figure out why the traffic wouldn't match.
Like PhilM, I discovered that by creating a custom application for TCP/443 I was able to resolve this issue, albeit at the expense of application discovery over that port.
I'm not sure how application discovery can be very successful by observing an SSL stream. We don't use the SSL decryption features at this point so the passthrough traffic payloads are pretty-much meaningless to the FW.
I"ve also been working with Sidewinders for quite some time (not quite as far back you other guys--started on them right after 6 first arrived). I've also had these problems with version 8 and our current, main cluster protecting our online application has been an ongoing headache for us. Many of our problems stem from issues we've had with active/active HA, which we're trying to switch from; however, I also see tons of those errors you posted. For us, it doesn't seem to be causing any major issues. It seems to only happen with SSL access to our system. Packet captures show that when it happens, the client PC closes the SSL communication very early, but then immediately starts the 3-way handshake over again and it works. All this happens in far less than a second, so the user doesn't notice, which is why I haven't dedicated much time to correct it being swamped like I tend to be. I believe the problem goes away with a generic SSL app, but I'm concerned about a loss of security not having the proper HTTP app defenses active.
So does your custom application have a parent of TCP/UDP or HTTP? Setting the parent to HTTP still gives me similar behavior. Am I correct that an application with a parent of TCP/80 or TCP/443 will not run any Application Defenses (URL Filtering and Anti-Virus Scanning) or GTI Host Reputation?
Thanks for the responses.
GTI should work across the board - it is applied on a per rule basis, so it shouldn't matter whether the rule is passing ICMP, HTTP or a user-defined service/application.
URL Filtering and AV are specific to those application defenses that contain these settings (URL filtering = HTTP, AV = HTTP, FTP and Sendmail, if I recall?).
In my case the custom application has a parent of TCP/UDP. I'm sure this uses a generic proxy which will not provide the same level of granular protection as the http proxy, but unfortunately it's the only way I've come up with to stop the issues I listed in the previous post in this thread.
My generic apps also have parents of TCP/UDP. I have further verification with packet captures that our port 443 <Unknown TCP> errors are not any fault of the firewall, at least in every case I've looked at.
We are also experiencing similar problems on several 8.2.0 or 8.2.1 version with different hardware appliances. It may be related to the Application Defense beacuse, when we rewrite the Application Defense for the rule related, the problem disappears.
Sometimes the new Application Defense is identical to the old one!!!.