cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Understanding FTP over HTTP

What Is it?

FTP over HTTP is the name used to refer to a specific type of HTTP traffic between a web browser and an explicitly configured proxy. For the most part, it is like any other HTTP request. The distinguishing factor is that the requested resource resides on an FTP server rather than an HTTP server. Correspondingly, this means an FTP over HTTP request will contain a URL prefixed with "ftp://" instead of "http://" and the HTTP "Host" header value will include port 21 (instead of no port number at all, in which case port 80 is assumed).

 

An example of an initial FTP over HTTP request from a client to the Web Gateway:

GET ftp://speedtest.tele2.net/ HTTP/1.1

User-Agent: Mozilla/4.0 (compatible; MSIE 8.0)
Host: speedtest.tele2.net:21
Proxy-Connection: Keep-Alive

 

When the Web Gateway receives this type of request, it recognizes that the requested resource resides on an FTP server (due to the presence of "ftp://" and port 21). The Web Gateway then uses native FTP to retrieve the file from the remote FTP server, on behalf of the client. FTP response traffic is then "translated" by the Gateway into HTTP before passing it back to the client. Sometimes this requires the Gateway to generate an HTML page so the client can display the results in the web browser. This translation is done for data such as an FTP directory listing:

 ftp dictionary.png

 


How Does Web Gateway Handle It?

The Web Gateway handles FTP over HTTP just like it would any other HTTP communication with the client computer. This means the traffic should be sent to the Web Gateway's HTTP proxy port (default = 9090). For FTP over HTTP, there is no need to enable, configure or use the Web Gateway's FTP proxy port (default = 2121), which is only used for native FTP protocol sent by a client to the Web Gateway. When the Web Gateway receives FTP over HTTP communication, it then does native FTP between itself and the FTP server, sending commands just like a native FTP client.

 


Why/When Would I Use It?

FTP over HTTP has advantages and disadvantages. It can be used to retrieve files from an FTP server using a web browser rather than setting up and configuring an FTP client such as Filezilla. As explained above, this also means you do not have to configure or enable the Web Gateway's FTP proxy port. The major disadvantage of FTP over HTTP is that it does not allow users to upload files. This requires the use of native FTP, an FTP client program on the client computer and the Web Gateway's FTP proxy port, if you choose to send the traffic through the Gateway. Another disadvantage of FTP over HTTP is that different web browsers have different behaviors, quirks and bugs. More discussion of these below.

 


What Does it Look Like?

Below are two images of the same packet capture with different filters applied. The capture was taken on the Web Gateway while an FTP over HTTP session was passing through it. You will see the following devices communicating:

 

Client computer:  10.10.80.1

Web Gateway:      10.10.80.55   (with HTTP Proxy Port 9090)

FTP Server:       10.10.80.200

 

The first image is of the connection between the client computer and the Web Gateway. You can see that Wireshark interprets this as HTTP traffic, which it is:

 http traffic.png

 

The second image shows communication between the Web Gateway and the FTP server. The Gateway recognized that the requested resource resides on an FTP server, thus uses FTP protocol to retrieve data from the FTP server. Here, active FTP was used, thus there are two TCP connections between the Gateway and the FTP server. The "control channel" is used for sending commands back and forth between the Gateway and FTP server (highlighted in pink). The "data" channel is used to send files from the FTP server to the Web Gateway (highlighted in purple):

highlighted in purple.jpg


Configurable Options on the Web Gateway

There are two items on the Web Gateway that can be configured in relation to the usage of FTP over HTTP:

 

Default FTP Username and Password

If the Gateway receives an FTP over HTTP request that does not contain FTP user credentials, the Gateway will send its pre-configured anonymous user credentials to the FTP server when attempting to login to the FTP server. You can change this username and password pair if you like. The settings can be found here:

 

Configuration > Appliances > [CHOOSE_APPLIANCE] > Proxies (HTTP(S), FTP, ICAP and IM) >> section "HTTP Proxy"

 choose appliance.png

Web Gateway Log-In Page

What is it?

This feature adds a JavaScript to the Gateway's "401 - Authentication Required" block page. When received by a web browser, it offers the user the option of inputting a username and password. The JavaScript then re-sends the request via FTP over HTTP and includes the credentials.

 credintials.jpg

How do I get it?

If you have done a fresh installation of McAfee Web Gateway v7.2 or newer, then you should already have the Log-In Page. If you upgraded from a version prior to v7.2, you probably do not have it. If you do not have the log-in page, you can add it yourself. To do so, find the template for the "Authentication Required" block page and replace its code with the contents of the file "MWG_ftp_login_page.txt" attached at the bottom of this article:

 

      • To find the template go to: Policy > Settings > Actions > Block, select "(Default)"
      • In the right window-pane, click "Edit..." next to the "Collection" drop-down box.
      • The "Template Editor" will appear. Find "Authentication Required" and expand the tree until you find the "html" file. Select it.
      • In the right window-pane, the "HTML Editor", replace all the contents with the contents of the attached file: "MWG_ftp_login_page.txt"

  mwgftp.png


Overview of Common Web Browser Issues

Currently, Firefox is the only web browser we have tested that does not require special attention from the user when doing FTP over HTTP. There are four major issues in other browsers, present in different combinations depending on the browser:

 

    1. Some browsers require you take extra steps to log into an FTP server when a username and password other than "anonymous" is required. There are three ways to work around this:
          - Prepend credentials to the URL hostname like this: ftp://UsernameSmiley Tongueassword@ftpserver.com
          - Use the Gateway's Log-In Page to collect user credentials (see information above).
          - Some browsers will prompt you for credentials if anonymous FTP is not allowed.
    2. Some browsers mis-encode FTP usernames and passwords containing special characters, rendering them useless, causing log-in to fail. There is no work around for this issue. These browsers are not recommended for FTP over HTTP when credentials are required.
    3. Some web browsers cannot do proxy authentication for FTP over HTTP traffic. Instead of sending the user's domain credentials in response to the Gateway's "407 - Proxy Authentication Required" message, the browser simply displays the HTML page that was sent with the 407 code. To work around this issue, you cannot enforce proxy authentication for FTP over HTTP traffic from these browsers. To bypass these browsers from proxy authentication, write a rule that recognizes the user-agent and protocol (see below for more detail).
    4. Some browsers do not respect system proxy settings when protocol "ftp://" is entered in their address bar . Instead of sending an FTP over HTTP request to the proxy, they send native FTP traffic directly to the FTP server. There is no work around for this issue. These browsers are not recommended for FTP over HTTP traffic.


Known Issues with Specific Browsers

 

Internet Explorer

    1. To log into non-anonymous FTP servers, the user must send their credentials via the Gateway's Log-In page (described above) or add their username and password to the URL in this format: ftp://USERNAMESmiley TongueASSWORD@servername.com
    2. Internet Explorer cannot log into FTP servers if the username or password contain special characters, especially the “@” symbol, when doing FTP over HTTP. These characters are mis-encoded when sent by the browser. Therefore, the credentials are invalid when received and used by the Web Gateway and FTP server.

 

 

Firefox

No known issues doing FTP over HTTP.

 

 

Chrome

    1. Chrome cannot access FTP servers that require credentials, other than the anonymous user, when doing FTP over HTTP. It does not send credentials as part of the URL, even if you add them in the address bar, or by any other method. Therefore, non-anonymous FTP over HTTP access is not possible.
    2. Chrome cannot do proxy authentication for FTP over HTTP traffic.  Instead, it displays the Block Page that is sent by Web Gateway with the 407 HTTP status code. To send Chrome FTP over HTTP traffic through the Web Gateway, it must be exempted from proxy authentication. You could do so with a rule like this:

                         Header.Get("User-Agent") matches *Chrome* AND URL.Protocol equals "ftp" --> Stop Rule Set

 chrome.jpg

Opera

Opera cannot do proxy authentication for FTP over HTTP traffic. It prompts the user for credentials when it receives the 407 HTTP status code from the Web Gateway, but once they are inputted, the browser displays the Block Page sent by Web Gateway with the 407 HTTP status code rather than retrieving FTP data from the Gateway. To send Opera FTP over HTTP traffic through the Web Gateway, it must be exempted from proxy authentication. You could do so with a rule like this:

 

     Header.Get("User-Agent") matches *Opera* AND URL.Protocol equals "ftp" --> Stop Rule Set

 

FTPoverHTTP_NoAuth-Opera_crop2.jpeg

 

Safari

Safari cannot do FTP over HTTP. It does not respect system proxy settings when the protocol in the address bar is set to "ftp://". Instead, it sends native FTP traffic directly to the FTP server, bypassing the configured proxy.

Labels (1)
Attachments
Comments

I've been using this for a while and the only problem so far is if youse use a reserved character in the password, it blows up.  So for instance username@domain.com - the @ will blow up the form if you stick that in.

Hello,

probably this is a matter of correctly masking the input as mentioned here:

Some browsers mis-encode FTP usernames and passwords containing special characters, rendering them useless, causing log-in to fail. There is no work around for this issue. These browsers are not recommended for FTP over HTTP when credentials are required.

Probably it is not an MWG issue. Using "@" in username/password for FTP is sometimes tricky... as long as you have control about the credentials (internal systems) I recommend to avoid it... makes your life easier :-)

best,

Andre

We found a solution for this problem.

here is the Rule we use for this

FTPoverHTTP
Enabled
Applies to Requests: True / Responses: True / Embedded Objects: True
1: URL.Protocol equals "FTP"
EnabledRuleActionEventsComments
EnabledExtract FTP password
Always
ContinueSet User-Defined.FTP.Password = ""
Set User-Defined.FTP.Password = String.SubStringBetween(String.SubStringBetween(URL.Raw,":","@"),":","")
Set User-Defined.FTP.Password = String.URLDecode(User-Defined.FTP.Password)

EnabledReplace EncodedPassword with DecodedPassword
1: String.IsEmpty(User-Defined.FTP.Password) equals false
ContinueSet URL.Raw = String.ReplaceAllMatches(URL.Raw,regex(:[^:]+@),String.Concat(":",String.Concat(User-Defined.FTP.Password,"@")))

Apparently Chrome is no longer sending the User-Agent for FTP over HTTP, so the work-around rule under known issues for Chrome isn't valid anymore.

Update

It's not elegant, but I managed to get around this by modifying the rule:

     Header.Get("User-Agent") equals "" AND URL.Protocol equals "ftp" --> Stop Rule Set

     chrome.JPG



Another way is to edit the redirect function in MWG add a JavaScript replace, i tested this and got it to work with a password containing "@"

FTP.PNG

In case it is not apparent from above each browser deals with FTP over HTTP differently when not going through a proxy, so although it is possible to mimic a specific browser and version, for the presentation, it would be difficult to set up rulesets and block pages that make the proxied FTP over HTTP user experience similar or identical to the native experience for multiple browsers. When a browser that supports a request of the form ftp://ftp.mcafee.com gets the request, it uses FTP protocol to get the data, and then renders it HTML form to the user through the browser window. With an explicit proxy in place, the communication between the browser and the proxy is HTTP. The proxy is performing the data retrieval via FTP and sending it back to the browser over HTTP. In this case, the proxy is creating the HTML for the user presentation, not the browser. The upside of this is that the user will get the same experience/presentation (provided the noted browser unique exceptions are addressed) regardless of the browser used, so long as the browser can perform FTP over HTTP through a proxy.

Hello,

With a McAfee Web Gateway, I'm trying to go on this website : ftp://ftp.ca.com/pub/ca_itam/ca_sam/catalogs/ and I have the FTP authentication from the McAfee :

The problem is that the CA website doesn't require authentication when I try from an external connection (like my smartphone)

I tried to :

- add this website to an whitelist without authentication

- connect with anonymous login like  ftp://anonymous:anonymous@ftp.ca.com/pub/ca_itam/ca_sam/catalogs/  

But without succeed.... Do you have a suggest to bypass this authentication ?

Best regards.

Hello,

by default MWG uses "anonymous" as the user and "anon@localhost" as the eMail address to log in. This works fine for many FTP sites. The site you found performs a syntax check on the eMail address sent as the password and does not accept it:

220 ftp.ca.com NcFTPd Server (licensed copy) ready.

USER anonymous

331 Guest login ok, send your complete e-mail address as password.

PASS anon@localhost

530-You must supply a valid email address as your password.

530-For example, "mike@nirvana.ncftp.com" is okay.

530 Login failed.

So MWG prompts you for a password. Set a valid eMail address in the MWG config:

It will allow access to that site without a prompt afterwards.

Best,

Andre

Hello Andre,

I don't really understand your manipulation for this problem. The URL website is normally open for all people.

Why do you want to put an email address for the password in the configuration ? I tried your test, but not working.

Best regards.

Ludovic.

Hello,

yes the web site is open for all people. To be open for all people an "anonymous" login is performed, which means:

Username: anonymous

Password: an eMail address

The "anonymous" user is required because the FTP protocol needs a username and a password. Usually an FTP server will accept any value as the password, if the username is anonymous. The server you mentioned here requires you to provide a valid eMail address for the anonymous user. If you go to the site without the Web Gateway just using your browser the browser will automatically send the username and the password for you, for example Firefox:

220 ftp.ca.com NcFTPd Server (licensed copy) ready.

USER anonymous

331 Guest login ok, send your complete e-mail address as password.

PASS mozilla@example.com

As you can see it sends "mozilla@example.com" as the password.

MWG by default sends "anon@localhost" as I have indicated in my previous post. This is not accepted by the server, so you have to adjust the eMail address MWG sends as the password. I have tried the default setting which gave me the same error that you described - after I changed the value to a valid eMail address I was able to connect without the logon prompt.

What value have you configured?

Best,

Andre

Hello,

Here an example that I put but without success... (I still have the FTP authentication).

CaptureMcAfeeFTP.PNG

I think that the CA website doesn't use credentials (username/password) for their connection. Did you succeed to access to the ressources of the URL : ftp://ftp.ca.com/pub/ca_itam/ca_sam/catalogs ?

Best regards.

Hello,

I have changed my MWG to use the same eMail address that you configured in the screenshot above. I can access the site without the prompt and I can also access the ressource you have mentioned.

Did you clear the browsers cache and restarted the browser? If so there is probably another problem that prevents you from accessing. If possible you could take a packet capture on MWG to find out how MWG tries to connect and to verify if this is successful. If you can take a capture you can send it to me and I can take a look. Alternatively you may want to file a ticket with support so that we can collect some data and verify the connection with the same settings you are using.

Best,

Andre

Hello,

I cleared the browser cache. Despite of that, I still have the authentication FTP from McAfee.
I will contact the support. Thanks.

Hello Everyone !

My problem is solved. The method with a valid email was the good method !

Many Thanks.

Ludovic Dk.

I would love to see some additional information for the FTP over HTTPS configuration when using the Client Proxy Agent.  I can't seem to find those details on any forums.

So would I, but unfortunately that is not techincally possible without help from browser settings. You cannot transparently proxy FTP over HTTP or HTTPS. The browser would need to be configured to use a proxy for FTP on a port that is intercepted by MCP.

Contributors
Version history
Revision #:
3 of 3
Last update:
‎03-20-2018 11:27 AM
Updated by: