It's really up to the client or tool to include the User-Agent with the request (HTTP or HTTPS). Command line tools like cURL by default include one by default, but one could force it to be removed.
I'm wondering if the User-Agent couldnt be displayed in the rule trace because of a funky character. Do you see it in the access logs?
Just a reminder, you are the only person I've seen talk about lack of CONNECT before an HTTPS request is made.
Whats the PSSession commands you used? It could just be emulating telnet, whereby Powershell has no idea about the application level, and just opens a tcp session to the destination (in this case a proxy), then just sends whatever data you gave it, like a HTTPS POST request. I'd guess the powershell equivalent of cURL (Invoke-WebRequest) would be application aware and include a CONNECT with the HTTPS POST.
I wrote the above, but then given that PSSession included the user-agent, that implies it's application aware... do you have a tcpdump of that?
Well, I may be the only one finding this stuff, but we have a lot of different clients in use, like the AWS script that uses Perl LWP—and I get to do gobs of tracing.
I'm including some images from one of the rule traces.
Criteria for rule checking the allowed User-Agent strings (showing a blank User-Agent value):
Criteria for a rule set finding that User-Agent is missing (Header-Request.Exists(“User-Agent”) showing false):
Top properties display showing User-Agent as “Microsoft WinRM Client”:
Event extracting User-Agent for a log entry (Header.Request.Get(“User-Agent”) shows a value of “Microsoft WinRM Client”):
Top properties showing Command.Name as “CONNECT”:
Criteria for a rule checking Command.Name with value of “CONNECT”:
Event adding Command.Name to log entry showing a value of “POST”
Note also that only request and logging cycle are appear in this single trace (because it was blocked as no User-Agent string).
The trace is for a user's workstation. We'll do a packet trace for it tomorrow.
Another thing that has just sunk in is that there is no logging cycle for a CONNECT that is blocked. At least it doesn't appear in a rule trace. Are we not logging blocked CONNECT requests? My logging people aren't going to want to hear that.
The weird thing is, I'm noticing this for requests blocked due to the absence of a User-Agent string. Packet traces and rule traces show that there is no User-Agent header. Yet, the User-Agent string does appear in the block page. There must be some stuff going on with block pages that I don't think I've seen any evidence to explain this. One packet trace I have has eight CONNECT requests (all blocked) and one GET (allowed), and the GET is the only packet with a User-Agent string. How does the block page get it?
All transactions are logged even CONNECT requests, it might show in a separate rule trace if there is a 5xx error (I would expect this for what you've described).
Would need the packet trace and rule traces to comment on your observations. Perhaps the GET includes the User-Agent, but the CONNECT doesnt. That would explain why the block page shows it (block page is rendered after the CONNECT).
Unfortunately, I'm not going to be able to collect more traces at this point. The product, Burp (portswigger.net, which is a proxy for penetration testing), has an option for how User-Agent strings are handled; and once they clicked it, the problem went away, and they're getting on with their work.
I had taken both packet and rule traces, but not at the same time. I can see that Burp was removing the User-Agent string for CONNECT requests. But, I wanted to verify that there was no other request inside the tunnel that allowed the proxy to capture the User-Agent string and attribute it to the CONNECT. I'll have to consider installing the product myself.