When Internet Explorer connects to an HTTPS site, I can see that the User-Agent is present and can be tested against in a rule on a CONNECT. However, I've noticed that this is not true for some command-line tools.
And today, I've found one like this—and was flabbergasted to find that the rule trace shows that a User-Agent was logged—but it wasn't available to the rule (repeatedly over several rule traces).
Adding to this quirk, I see that when generating the log entry the Command.Name is “POST”—but under "Top Properties", Command.Name is “CONNECT”. While that seems a minor bug (no silly jokes about it being a feature), it makes me wonder if it relates to the way POST's are not supposed to be HTTPS without being preceded by a “CONNECT” ().
So, I did some more traces to check my sanity, A rule trace for a “CONNECT” followed by a “GET” to https://duckduckgo.com/lite from Internet Explorer works correctly, in that it can use the User-Agent on the “CONNECT”. In another rule trace, for a “CONNECT” followed by a “POST” to StartPage by Ixquick Search Engine from Internet Explorer works correctly, in that it also can use the User-Agent on the “CONNECT”.
So, what's missing that would affect certain command-line tools but not Internet Explorer (and presumably other major browsers)?
To fill in some details, the command-line tool I worked with today is the PowerShell command, “New-PSSession”, and it's User-Agent is “Microsoft WinRM Client”. And, it clearly does a “POST” to HTTPS without preceding it with a “CONNECT”, which is itself a problem (article link above). But, does this also explain why it's treated differently in MWG?
It's really up to the client or tool to include the User-Agent with the request (HTTP or HTTPS). Command line tools like cURL by default include one by default, but one could force it to be removed.
I'm wondering if the User-Agent couldnt be displayed in the rule trace because of a funky character. Do you see it in the access logs?
Just a reminder, you are the only person I've seen talk about lack of CONNECT before an HTTPS request is made.
Whats the PSSession commands you used? It could just be emulating telnet, whereby Powershell has no idea about the application level, and just opens a tcp session to the destination (in this case a proxy), then just sends whatever data you gave it, like a HTTPS POST request. I'd guess the powershell equivalent of cURL (Invoke-WebRequest) would be application aware and include a CONNECT with the HTTPS POST.
I wrote the above, but then given that PSSession included the user-agent, that implies it's application aware... do you have a tcpdump of that?
Well, I may be the only one finding this stuff, but we have a lot of different clients in use, like the AWS script that uses Perl LWP—and I get to do gobs of tracing.
I'm including some images from one of the rule traces.
Criteria for rule checking the allowed User-Agent strings (showing a blank User-Agent value):
Criteria for a rule set finding that User-Agent is missing (Header-Request.Exists(“User-Agent”) showing false):
Top properties display showing User-Agent as “Microsoft WinRM Client”:
Event extracting User-Agent for a log entry (Header.Request.Get(“User-Agent”) shows a value of “Microsoft WinRM Client”):
Top properties showing Command.Name as “CONNECT”:
Criteria for a rule checking Command.Name with value of “CONNECT”:
Event adding Command.Name to log entry showing a value of “POST”
Note also that only request and logging cycle are appear in this single trace (because it was blocked as no User-Agent string).
The trace is for a user's workstation. We'll do a packet trace for it tomorrow.
Another thing that has just sunk in is that there is no logging cycle for a CONNECT that is blocked. At least it doesn't appear in a rule trace. Are we not logging blocked CONNECT requests? My logging people aren't going to want to hear that.
The weird thing is, I'm noticing this for requests blocked due to the absence of a User-Agent string. Packet traces and rule traces show that there is no User-Agent header. Yet, the User-Agent string does appear in the block page. There must be some stuff going on with block pages that I don't think I've seen any evidence to explain this. One packet trace I have has eight CONNECT requests (all blocked) and one GET (allowed), and the GET is the only packet with a User-Agent string. How does the block page get it?
All transactions are logged even CONNECT requests, it might show in a separate rule trace if there is a 5xx error (I would expect this for what you've described).
Would need the packet trace and rule traces to comment on your observations. Perhaps the GET includes the User-Agent, but the CONNECT doesnt. That would explain why the block page shows it (block page is rendered after the CONNECT).
Unfortunately, I'm not going to be able to collect more traces at this point. The product, Burp (portswigger.net, which is a proxy for penetration testing), has an option for how User-Agent strings are handled; and once they clicked it, the problem went away, and they're getting on with their work.
I had taken both packet and rule traces, but not at the same time. I can see that Burp was removing the User-Agent string for CONNECT requests. But, I wanted to verify that there was no other request inside the tunnel that allowed the proxy to capture the User-Agent string and attribute it to the CONNECT. I'll have to consider installing the product myself.