Dealing with User-Agent strings can be a pain, but they're the only way to manage users who install browsers they shouldn't.
Of course, the ever spoof-able User-Agent string is not a strong security control, but you end up defending it for the sake of managing users. That is, you are going to have people using browsers they shouldn't if you don't filter the User-Agent string (endpoint controls aside). So, when someone wants you to allow a product with an empty or missing User-Agent string, you have to find something else to filter on. Often, you find yourself with possibility of allowing way too much—creating gianormous holes in your security controls.
I'm finding more and more products that are getting blocked for doing a CONNECT without setting a User-Agent (related: Inconsistent User-Agent Usage on CONNECT/POST for TLS), and I'm almost to the point of allowing this to destinations that have a good reputation. Yes, we can check for that dynamically; but for now, I'm manually reviewing these before I white-list them (and I've added the check for reputation to that white list, in case any are ever compromised).
Consider the possibility that some oddball malware would do a CONNECT and then transfer data inside the CONNECT without doing GET's and POST's, as if that would be a meaningfully stealthy malware feature (and would that get past SSL inspection or break things). And, such oddball malware would need servers that don't care about GET's and POST's. Yet, such a thing would likely know to spoof a User-Agent on the CONNECT, which means all this concern about User-Agent strings doesn't likely apply to this case. For less unconventional malware, we still get to test the User-Agent string on the GET's and POST's, and we can hope any such malware would mess up the User-Agent spoofing with something that we block. This is the scenario I use to make sure I've got my considerations straight for the issue of a missing User-Agent on a CONNECT. Again, filtering User-Agent strings is not a strong security control, but I want to be sure that I don't set my paranoia level too low.
Perhaps, there will eventually be something along the lines of “browser signing” (with private keys and such). That way, malware wouldn't be able to spoof approved browsers (at least, without the private key), and all that malware that depends upon being able to phone home will be dead. But, I'm not holding my breath for this; it's just a fantasy.
So today, I spent a couple hours working with a user to do rule traces for the installation and use of Microsoft Visual Studio 2017. Oh sure; they have a page that says what it is you need to white-list (Install Visual Studio behind a firewall or proxy server | Microsoft Docs). But like most such pages, it's woefully vague and wants you to open things up way too much. They and other vendors just don't seem to understand that if we opened things as loosely as every vendor states, we'd end up tearing huge holes in our security controls. Just look at how many Microsoft and non-Microsoft destinations they want to white-list, not to mention that the list is incomplete. And, most of them we don't actually block—it's the messed up User-Agent strings that ended up requiring all that rule tracing.
Like any other IDE environment, it's a collection of many tools. I found several different User-Agent strings—including one that resembles that of IE 6 (which we block, of course). But, most of these tools don't set the User-Agent on CONNECT—though the major browsers do. I also found some that don't set any User-Agent string at all. I couldn't just white-list those; I had to write a custom rule for that hot mess. On top of all this, I found a feature in Visual Studio 2017 that gets blocked for not being able to authenticate to the proxy, and I'm not white-listing that one.
Remember: while those destinations are not blocked by destination, we still need to be able to prevent users from using tools that aren't approved. If we white-list an empty User-Agent string, then we white list all the web clients that are too lazy to set one. That's just too big a hole—even if this is not the strongest security control.
We can fantasize about browsers and web clients that all do things right, like set a proper User-Agent string on every request—including a CONNECT (and browser signing would be really, really nice), but you know that's never going to happen. Unavoidably, you will be pressured to remove your filtering at some level for badly written and poorly tested software. We need to think long and hard to ensure that we're getting the best controls we can create, and this is one that has been a thorn in my side for some time.
If anyone else has any creative thoughts on managing browsers and dealing with apps that don't properly set a User-Agent string—including for a CONNECT, please chime in.