Just to provide (for what seems to be) a solution for this issue - After doing some more analysis and examining the flow of packets between user machines and our website, we started to troubleshoot with the policies by making a new one only containing firewall rules. We also created a python script that would make calls to all the affected pdfs to examine whether a 200 would return or timeout (We did this as we were unsure if the browsers were hitting cache for the files).
What we later found was that two different profiles utilizing different advanced malware policies. Both of these policies had PDFs enabled for GTI, Blacklist, ATD, etc. Disabling all of these under PDF seemed to allow the script to return 200 quickly for each PDF request, where as enabling even one or a set of a few would cause timeouts to occur for some but not all. Ultimately the temporary solution was to disable PDFs on all the offending policies as to not compromise availibility for our users and customers. The ticket has been updated on the support side - perhaps this is a bug of some sort?
What is the load like on the sensor?
If the load is minimal...there should be little to no slow down in packet processing on the sensor. If you are pushing it to the limits, then any additional inspection feature will cause a slow down.
That was another thing that was mentioned in the ticket as a possible problem. We weren't sure whether when the PDFs were being requested if it would spike the sensors load to max causing a delay as the packets were being analyzed. The sensor memory loads are sitting steadily at about 98% while the CPU and Throughput are roughly 10-15%.
Dang, memory at 98%...that is insane on the NS9x00s especially with the cpu/traffic load being so low. Sounds like an internal resource exhaustion bug to me.
Out of curiosity how long has the sensor been Up and running?
What about HTTP Response Traffic Scanning settings?
If you are serving 'content' like pdf files to external files, all your HTTP response traffic will be scanned, which can really push CPU utilization up. If the malware policy is enabled to scan in/outbound, then all your files captured on the HTTP Response Traffic flows will be sent to the antimalware engines as per policies.
Just a thought
Download the new ePolicy Orchestrator (ePO) Support Center Extension which simplifies ePO management and provides support resources directly in the console. Learn more about ePO Support Center