Progress Indication Explained

Version 2

    What is it?

    While McAfee Web Gateway performs anti-virus/anti-malware scanning on downloaded files, Progress Indication sends data back to the client computer to keep the connection alive as well as to provide visual progress information to the user. Progress Indication assures that the user and web browser know that Web Gateway received the request and is processing the download.

     

    Why should I use it?

    Progress Indication should be used for two major reasons:

    1. to prevent TCP connections from timing-out between the Web Gateway and the client computer while the Web Gateway scans large files for viruses and malware
    2. to provide users with visual feedback in their web browser, indicating that their download has started and scanning is in progress (this function is only provided by Progress Pages)

     

    Types of Progress Indication

    Progress Page

    Progress Pages are particularly important when the Web Gateway is configured to do anti-virus/anti-malware scanning. The scanning of large or "dense" files can take significant periods of time. A "dense" file might be relatively small in size, but contain many sub-files that are archives or executables such as: .dll, .exe, .zip, .rar, .jar, .iso, .xap. All of which must be individually unarchived and scanned.

     

    A file that is large and dense, such as a 200MB ZIP file containing software developer tools, can take 30+ minutes to be fully scanned by the anti-virus/anti-malware engine. If progress indication is not provided to the user with Progress Pages, most users become impatient, or worse, believe there is a network or server problem. The impatient user will relaunch their download multiple times, causing the anti-virus/anti-malware engine to scan redundant copies concurrently, adding additional load to the Web Gateway.

     

    Progress Pages give the user visual feedback, indicating that their file is being downloaded and anti-virus/anti-malware scanned, reducing helpdesk calls.

    01_progress_page_downloaded_24.1MB.jpg

     

     

     

    Data Trickling

    Data Trickling can work for any client device or software that downloads data. It is an alternative to Progress Pages, but does not provide any particular visual progress indication beyond that provided by the web browser or client software itself. Data Trickling, from the point of view of an end-user, will make downloads appear very slow. Therefore, we recommend Progress Pages always be enabled and Data Trickling enabled as a fallback for clients that cannot use Progress Pages (this is exactly how the default rule sets are configured).

     

    Data Trickling effectively throttles the download connection speed between the Web Gateway and the client, sending only a tiny amount of data that prevents the TCP connection from timing out. When the Gateway finishes scanning the file for viruses and malware (finding none), the remaining portion of the file is sent to the client at full network speed.

     

    Because estimations of download time are provided by the web browser, and it is not aware of what the Web Gateway is doing, estimated download times will initially look extremely long to a user. The image below shows download progress indication provided by Internet Explorer v8 while Data Trickling is occurring.

     

    trickling_started_02.jpg

     

     

    FTP Upload Timeout Prevention

    When the Web Gateway is used as an FTP proxy, FTP Upload Timeout Prevention enables the Web Gateway to send FTP status messages to FTP clients that prevent the TCP connection from timing-out. This gives Web Gateway time to scan uploaded files for viruses and malware. The image below shows progress indication that was sent by the Web Gateway to Filezilla FTP client software.

    overview_filezilla_progress-finished.jpg

     

    How to Get it

    Recommendations

    We recommend that you import copies of Progress Indication rules from the Rule Set Library, keeping them all enabled and using their default configurations unless you have been advised to do otherwise by McAfee Technical Support or you are certain your environment requires changes. Default settings work well for most customers.

     

    Importing from the Rule Set Library

    If you do not already have these rules as part of your Policy, you can import them from the Rule Set Library. Under "Policies" on the "Rule Sets" tab, click "Add" and then choose "Rule Set from Library...". Find the Rule Set named "Progress Indication". In some version of McAfee Web Gateway, it will be nested inside Rule Set "Common Rules".

    import.jpg

     

     

     

    Detailed Information

    Progress Page

    When can it be used?

    Progress Pages only work for web browsers that announce their compatibility with Mozilla in their HTTP User-Agent header. This includes: Internet Explorer, FireFox, Chrome and Safari (but not Opera).

    What is Mozilla?

    In the early days of graphic web browsers, Mozilla was the project code name of Netscape Navigator during its development. Netscape was an early innovator, introducing new and advanced features for the day. Websites checked the HTTP User-Agent header. If it was a Mozilla web browser, advanced content would be served; if not, basic content. Eventually Netscape's competitors implemented similar features, enabling their browsers to display the same content. However, not all websites updated their User-Agent string check. As a result, competitors such as Microsoft added "Mozilla compatible" to the User-Agent sent by their browsers. This continues today.

     

    How do I know if my browser is Mozilla compatible?

    It is easy to figure out if your web browser is Mozilla compatible by doing a quick packet capture. Start Wireshark, browser to a website, then in the capture, find an HTTP request. In the example below, you can see that the web browser is Microsoft Internet Explorer v8 and is Mozilla v4 compatible.

    user-agent.jpg

     

    HTTP User-Agent Header Samples

    Mozilla Compatible:

    Internet Explorer v8

    User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

     

    FireFox v23

    User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:23.0) Gecko/20100101 Firefox/23.0

     

    Chrome v29

    User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36

     

    Safari v5

    User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2

     

    Not Mozilla Compatible:

    Opera v9

    User-Agent: Opera/9.80 (Windows NT 5.2) Presto/2.12.388 Version/12.15

     

    How does it work?

        1. The client opens a TCP connection to the Web Gateway (if not already open) and requests a file to download.
        2. The Web Gateway begins to download the file. If it is not text or HTML and takes the Gateway more than 5 seconds to download and process the file, the Web Gateway will redirect the client to a Progress Page with an HTTP status of "307 Temporarily Moved". The redirection will be to the same URL Host as the original request, but the subdirectory will be changed to a unique value similar in format to: "/mwg-internal/de5fs23hu73ds/progress?id=29qNbsp9oR".
        3. The client then opens a new TCP connection to the Web Gateway and requests the URL to which it was redirected.
        4. Because the requested URL contains subdirectory "mwg-internal", the Web Gateway knows the requested file(s) are local and stored on the Web Gateway. The Web Gateway itself serves a Progress Page to the client, rather than querying the webserver represented by the URL Host of the request.
        5. Every 5 seconds the client will request a progress update from the Web Gateway.
        6. The Web Gateway's progress update response contains five values, in a format similar to:

      204.5 MB;204.5 MB;100;1;4512

            • 1st value = amount of data downloaded so far, from web server to web gateway

            • 2nd value = total size of file being downloaded

            • 3rd value = percentage of download complete (0-100)

            • 4th value = is AV scanning complete? (0=no; 1=yes)

            • 5th value = cumulative elapsed time during anti-virus/anti-malware scanning

       

       

      What does the user experience?

      When a user downloads a non-text/html file that takes more than five seconds to be processed by the Web Gateway, the Web Gateway will redirect the client to a Progress Page, which will display dynamic progress indication while the download is occurring. The Progress Page shows the amount of data downloaded, total size of the file being downloaded and a progress bar.

      01_progress_page_downloaded_24.1MB.jpg

       

       

      Once the Gateway has finished downloading the requested file, it will begin anti-virus/anti-malware scanning. During this phase, the Progress Page counts the elapsed seconds during scanning. There is also an animated progress bar.

      02_progress_page_scanning_206s.jpg

       

       

      Once scanning is complete, the Progress Page presents the user with a link. When clicked, the file is downloaded from the Web Gateway to the client.

      03_progress_page_complete.jpg

       

       

       

      When the download link is clicked, the web browser presents the user with a File Download dialog box.

      04_progress_page_save.jpg

       

      What settings can be modified?

      We recommend that you use default settings as found in the Rule Set Library, unless you have been advised to do otherwise by McAfee Technical Support or you are certain your environment requires changes. The defaults work well for most users.

       

      However, there are a few settings that can be configured if you drill-down into the Enable Progress Page rule's Event settings container.

       

      Items in the "Templates" section allow you choose and edit the appearance of the Progress Page's HTML. Boxes allow you to choose the collection which contains the templates you want to use, as well as the specific templates for each of the three stages of a Progress Page: the Progress Bar Page, Download Finished Page and Download Canceled Page.

       

      The lower section contains three time values:

       

      "Delay for redirects to Progress Page" in seconds: after a client requests a file, if the Web Gateway needs more than this many seconds to perform its filtering and scanning duties, the Web Gateway will redirect the client to a Progress Page in place of the file.

       

      "File availability time before download" in minutes: after the Web Gateway has finished downloading and scanning a file, it will be available on the Web Gateway for this amount of time until a user downloads a copy.

       

       

      "File availability time after download" in minutes: after a user downloads a copy of the file, it will remain available on the Web Gateway for this amount of time for additional downloads.

      progress_page_settings.jpg

       

      What does a packet capture look like?

      In the packet capture images below, you will see the following devices communicating:

          • client computer = 10.10.80.1
          • McAfee Web Gateway = 10.10.80.57
          • web server = 10.10.80.200

       

      The client computer requests a web page from the Web Gateway. The Gateway will redirect the client to the Progress Page with a "307 Moved Temporarily" HTTP status. The client will then open a new TCP connection to the Web Gateway and request the location to which it was redirected.

      05_progress_page_packetcapture_redirected.jpg

       

       

      The client downloads the Progress Page (HTML, CSS, JPEGs, PNGs, etc.). Every 5 seconds the client will then request a progress update to which the Web Gateway will respond. The progress update in the image below indicates that the Web Gateway is downloading the requested file.

      06_packetcapture_progress_page_starting.jpg

       

       

      Below, the Web Gateway has downloaded the file and begun anti-virus/anti-malware scanning.

      07_packetcapture_progress_page_scanning.jpg

       

       

      The Web Gateway has completed anti-virus/anti-malware scanning. The web browser now presents the user with a link from which the requested file can be downloaded .

      08_packetcapture_progress_page_finished.jpg

       

       

       

      Data Trickling

      When can it be used?

      Data Trickling can be used for any client that downloads data.

       

      How does it work?

      When Data Trickling is enabled, the Web Gateway sends tiny pieces of the requested download file back to the client. This keeps the connection alive while the Web Gateway downloads the whole file and scans it for viruses and malware. By default, the Gateway sends an initial chunk of data that is 4,096 bytes and there after sends 1 byte for every 1,000 it downloads. The data moves at such a slow rate that if the Web Gateway does detect a virus or malware after the file is downloaded, such a small amount of the file has been passed onto the client that no harm should be possible. This is because the client will not have received enough of the malicious data for it to be executed.

       

      What does the user experience?

      When Data Trickling is enabled, downloading a file seems like a typical file download that is not going through a proxy: the web browser's download manager appears, manages and displays information. The drawback of this, however, is that the estimated time displayed will seem outrageously long to a user, because the Web Gateway only sends tiny amounts of data while anti-virus/anti-malware scanning is occurring.

       

      In the image below, you can see that Internet Explorer is estimating 58 hours to download a 204MB file across an Ethernet LAN.

       

      trickling_started_02.jpg


      What settings can be modified?

      We recommend that you use default settings as found in the Rule Set Library, unless you have been advised to do otherwise by McAfee Technical Support or you are certain your environment requires changes. The defaults work well for most users.

       

      However, there are two settings that can be configured if you drill-down into the Enable Data Trickling rule's Event settings container:

       

      "Size of first chunk" in bytes: when data trickling begins, the first chunk of downloaded and scanned data sent by the Web Gateway to the client will be this size.

       

      "Forwarding rate" per thousand, in bytes: after the first chunk, the Web Gateway will send this many bytes to the client for every 1,000 bytes downloaded by the Web Gateway, keeping the client/Web Gateway TCP connection alive.

      data_trickling_settings.jpg

       

       

      What does a packet capture look like?

      In the packet capture image below, you will see the following devices communicating:

          • client computer = 10.10.80.1
          • McAfee Web Gateway = 10.10.80.56
          • web server = 10.10.80.200

       

      A packet capture will look like any other file downloaded without a proxy. The difference will be that packets contain only a small amount of data while the Web Gateway is doing anti-virus/anti-malware scanning.

       

      In the image below of a packet capture, after the initial bytes are sent, the download settles into a consistant rate of 8 bytes per packet from the Web Gateway to the client:

      data_trickling__packet_capture.jpg

       

      FTP Upload Timeout Prevention

      When can it be used?

      FTP Upload Timeout Prevention is only used for FTP uploads via McAfee Web Gateway's FTP Proxy Port (default listening port number is 2121). Settings for the FTP proxy, including enabling it, are configured on the same page as the HTTP Proxy Port. However, their settings are independent of one another and the FTP Proxy Port is disabled by default. See the image below for help finding the FTP Proxy Port settings.

      ftp_proxy_port.jpg

       

      To use the Web Gateway's FTP Proxy Port, you will also need to use a stand-alone FTP client. You can use a program like Filezilla. (FYI: a web browser will not work for uploading data via FTP.)

       

      The image below shows a basic configuration inside Filezilla that allows traffic to FTP server 10.10.80.200 to be sent through McAfee Web Gateway's FTP Proxy Port at 10.10.80.57 (NOTE: in this example, the Web Gateway does not require proxy authentication for FTP traffic).

      filezilla_settings.jpg

       

      How does it work?

      When FTP Upload Timeout Prevention is enabled, the Web Gateway will send a progress indication every 5 seconds while performing anti-virus/anti-malware scanning. The progress indication message is sent as an FTP 226 status code, which generically means: "requested file action successful". Specifically, the McAfee Web Gateway sends a message that reads: "226-data processing in progress".

       

      What does the user experience?

      Every 5 seconds, while the Web Gateway is performing anti-virus/anti-malware scanning, the user will see the message: "226-data processing in progress" inside their FTP client program. After uploading a file, the user will see that 100% of the file has uploaded, but the file will not yet be visible on the FTP server. This is because the McAfee Web Gateway is acting as an FTP proxy. The file indeed was 100% uploaded, but to the Web Gateway, not the FTP server. Thus the file is not visible on the server. The Web Gateway will send "226-data processing in progress" messages to the client every 5 seconds until anti-virus/anti-malware scanning is complete. If no virus or malware was found, then the Web Gateway will upload the file to the FTP server. Once uploaded, it will become visible on the FTP server and a message will be sent to the user's FTP client program indicating success, such as: "File transfer successful".

       

      The image below shows that the file "small_archive.zip" has been uploaded to the Web Gateway and anti-virus/anti-malware scanning as begun:

      filezilla_progress-started.jpg

       

       

      The following imag shows that the Web Gateway has finished anti-virus/anti-malware scanning and the file has been uploaded onto the FTP server:

      filezilla_progress-finished.jpg

       

      What settings can be modified?

      FTP Upload Timeout Prevention does not have user-modifiable settings.

       

      What does a packet capture look like?

      In the packet capture images below, you will see the following devices communicating:

          • client computer = 10.10.80.1
          • McAfee Web Gateway = 10.10.80.57
          • FTP server = 10.10.80.200

       

      If you create a packet capture on the Web Gateway or the client PC, you can clearly see the progress indication sent by the Web Gateway while it does anti-virus/anti-malware scanning: "226-data processing in progress":

      packetcapture_FTPprogress-indication.jpg

       

       

       

      Common Problems

      OMG! - Data Trickling Will Take Forever!

      No, actually it will not. From the point of view of a user sitting at a web browser,  at first it will look like the download is estimated to take hours. In reality it will finish much quicker. See information above to better understand what occurs when Data Trickling is in use.

      How do I diagnose it?

      If a user complains of slow downloads, ask them what exactly they are experiencing. If their web browser is estimating a huge time value for a download, check your Web Gateway's Progress Indication settings. It is possible that Data Trickling is enabled but not Progress Pages.

      How do I fix it?

      If you want better information passed to users than is provided by Data Trickling, consider enabling Progress Pages if they are not.

       

       

      Progress Page not Found Due to External Load Balancing Issues

      If you are using load balancing devices in front of a group of Web Gateways, sometimes a user will receive a Progress Page, but before it completes downloading and scanning, the user will receive a "Page Not Found" error. This is because your load balancer's "stickiness" value is not long enough: it chose to send a client's later request for a Progress Page update to a Web Gateway that did not create the Progress Page. A Progress Page created for a user exists only on the Web Gateway that created it. Therefore, a client's request must arrive at that same Web Gateway until downloading and scanning is complete.

      How do I diagnose it?

       

      The easiest way to diagnose this issue is running simultaneous packet captures on all Web Gateways through which the client might send traffic. When the client receives the "Page Not Found" error from a Web Gateway, stop the captures and inspect. You will see requests arrive at one Web Gateway every five seconds, then all of a sudden, arrive on another Web Gateway which will immediately send an HTTP 404 Page Not Found error.

      How do I fix it?

      To avoid this problem, you will need to increase the "stickiness" value on your load balancer for client source IP addresses.

       

      Progress Page Not Displayed, Even When Enabled

      Even when properly enabled, sometimes Progress Pages do not start when expected. This most often happens if your web browser is not announcing "Mozilla Compatibility" in the HTTP User-Agent header (such as when using Opera). Or, this could happen if you have changed the rule from defaults settings.

      How do I diagnose it?

      Do a packet capture and inspect the HTTP User-Agent header sent by the client's web browser. If it contains "Mozilla", it should receive a Progress Page. Also inspect your Progress Indication rules. Are Progress Pages enabled? Have the criteria been changed?

      How do I fix it?

      If your web browser does not announce Mozilla compatibility, this cannot be changed. However, some customers have had luck adding criteria to the Progress Page rule so it matches "Opera" or other criteria. Also, if your rules have already been modified and you want to return to defaults, you can import a fresh copy from the Rule Set Library. Compare the fresh copy to yours. Cut and paste rules as needed. Delete the unused newly-imported rules before saving changes.

       

      Web Page Parts Are Missing, Embedded Content Not Loaded

      Sometimes Progress Pages will be triggered for content embedded on an HTML web page, such as Silverlight content. Under normal operation, after a Progress Page is activated, once downloading and scanning is complete a user must click the download link in order for the client computer to receive the file. If a user does not see the link or cannot click it, the content is not received.

       

      Dynamic Silverlight content on an HTML page is often contained in a XAP file, which is an archive in ZIP format. Within a XAP file there are many sub-files including executables. If downloading and scanning takes more than five seconds, which it often does for XAPs, a Progress Page will be triggered. However, the user will never see this Progress Page: generally speaking, the web browser will only display an HTML Progress Page in place of an HTML page. It will not display an HTML Progress Page in place of an embedded file such as XAP, JPEG or CSS that is part of an HTML page.

       

      More specifically, a Progress Page will be displayed to a user in place of a file directly requested by the user. This typically means the user clicked a link to view a JPEG, an HTML page or to download a file. These are "primary" files requested directly by the user. A Progress Page, on the other hand, will not be displayed in place of a "secondary" file requested by the web browser. Secondary files are typically used to create primary files. For example, if a user types the address of an HTML page and hits enter, the web browser will load the HTML file which is a "primary" file. Then, the web browser will automatically request and load all files referenced by that HTML file, such as pictures or CSS which are used to render the HTML page as described by the HTML file. These are "secondary" files.

      How do I diagnose it?

      A packet capture on the client computer or Web Gateway will allow you to see this issue. The web browser will appear to be empty or missing content (no Web Gateway Progress or Block Page will be seen by the user). However, in the packet capture, you will be able to see the Web Gateway redirect the client computer to a Progress Page (for more information about packet captures containing Progress Pages, see above).

      How do I fix it?

      A whitelist entry will fix this problem, bypassing the file from Progress Pages and/or anti-virus/anti-malware scanning. You could add it to the Global Whitelist, which will avoid both. Or, you could create a new whitelist within the Progress Indication, Progress Page or Gateway Anti-Malware rule sets. Be sure you make the correct type of whitelist entry, which depends on the criteria used for matching: https://community.mcafee.com/docs/DOC-4514