cancel
Showing results for 
Search instead for 
Did you mean: 

how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Dear expert, 

we have a web application with Web Gateway in the front. The Web application accepts end user file upload. On large file upload, the file is broken into small parts in the client side and then each small parts are uploaded into server and once all the parts are uploaded, then server side would assembly all the small parts back to the original large file.   Just wonder how Web Gateway would be able to do the AV scan for the large file here. What particular we need to do for large file upload to allow Web Gateway to be able to recoganize the whole file thus do the proper AV scan?  I would think that it is common practice to slice the file for upload, and here is what we are using at client side: HTML5 Slicing a file

 

Thanks very much ahead. 

1 Solution

Accepted Solutions
McAfee Employee jscholte
McAfee Employee
Report Inappropriate Content
Message 8 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Attached is a filescanner ruleset, here's an example cURL command you could use to replicate sending files to it from the CLI:

curl -F "myfile=@filename.doc" http://mwg:9090/filescanner

filename.doc is your file that you want sent to the mwg, and mwg is the actual MWG on its proxy port.

Let me know if this helps,

Jon

10 Replies
McAfee Employee johma
McAfee Employee
Report Inappropriate Content
Message 2 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

HI, 

I dont work with Web Gateway and maybe someone who does will be able to assist further, but from a malware scanning point of view, the complete file will need to tbe scanned. Some methodologies will be followed depending on what "large" is. 

As a rule, the whole file will need to be received and re-compiled for the file to be completely scanned. If probaby as Web Gateway uses a stream, it will wait until the whole file has been received before a trigger to scan is made for the downloaded file.

Where very large files are sent then to the most part, header and footer is scanned and a determination made with regards to what else would be required to complete the stream or file scan. 

 




Was my reply helpful?


If this information was helpful in any way, or answered your question, will you please select "Accept as Solution" in my reply, or give kudos as appropriate, so together we can help other members?

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Thanks for information about AV scan.  The file split is a common practice for large file duriing upload. Here are two pages about Amazon S3 file upload amazon-s3-multipart-uploadsplitting-and-uploading-extremely-large-10-gb-files-to-amazon-s3

It looks like that Web Gateway needs to  understand how the large file is splitted at client side so it could assemble or maybe it would be able to do so as long as client puts proper identification at the HTTP upload request per Mcafee Web Gateway's specification . It would be great if anyone from Web Gateway team could shed some light on this. 

Highlighted
McAfee Employee jscholte
McAfee Employee
Report Inappropriate Content
Message 4 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Hi There,

Jon here, I do work with Web Gateway.

Whats been mentioned already is exactly the problem that would be encountered by most services attempting to scan partial upload. We'd need the entire file in order to dissect it properly and understand the intent. If I only have bytes 40-900 of a zip file, I wont be able to unpack it properly.

The same problem exists for partial downloads (206 http response), but MWG is able to detect this and handle it as needed. For downloads, the client will send a "Range" header to indicate what bytes the client would like to receive (i.e. 40-900). If this header remains untouched, then the server will respond witha  206 response code, and just those bytes. If this header is stripped (MWG does by default) then the server will send a 200 and the full file, enabling the MWG to download and scan the entire file.

So for uploads it would be a matter of forcing the client to sent the complete file (if there is a mechanism) or implementing a process on the backend that assembles the file for Web Gateway so it see's it as a single transaction rather than multiple.

Aside from this something would need to be implemented on the Web Gateway to buffer and group together multiple transactions so that it can determine intent for the file upload as a whole. Perhaps this could be based on the multipart form identifier and buffering would only be allowed for X ammount of time (just dreaming possibilities).

At present a "buffering and grouping" like this does not exist, so it may be better to explore the possibility of analysis of files after they have completed the upload and are fully assembled.

Best Regards,

Jon

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Thanks Jon for the detailed information.  

 

>>So for uploads it would be a matter of forcing the client to sent the complete file (if there is a mechanism) or implementing a process on the backend that assembles the file for Web Gateway so it see's it as a single transaction rather than multiple.

We could implement something at our client code side based on Web Gateway response to trigger a complete file upload although this would defeat the purpose of splitting large files and then uploading small parts for performance reason.  You do need to do some new development as well which seems not feasible at this point. 

>>At present a "buffering and grouping" like this does not exist, so it may be better to explore the possibility of analysis of files after they have completed the upload and are fully assembled.

Just wonder what kind of approach we could do here. Do you have a ICAP API endpoint available or any other API which would allow our application to integrate with Mcafee Web Gateway?

McAfee Employee johma
McAfee Employee
Report Inappropriate Content
Message 6 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Q: Just wonder what kind of approach we could do here. Do you have a ICAP API endpoint available or any other API which would allow our application to integrate with Mcafee Web Gateway?

I will again defer to Jon/Web Gateway Tech, another solution may be that the file is transmitted / saved to an FTP server or other "holding" server where the file can be scanned by installed software local to that server before being released to the client or storage service. 

 

We also have VirusScan for storage plugin (VSES) that loads on top of VirusScan Enterprise, that you can install. This will enable you to scan NetApp, ICAP etc as a load balanced/scaleable soloution if you dont find a suitable option via WebGateway. 




Was my reply helpful?


If this information was helpful in any way, or answered your question, will you please select "Accept as Solution" in my reply, or give kudos as appropriate, so together we can help other members?
McAfee Employee jscholte
McAfee Employee
Report Inappropriate Content
Message 7 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

ICAP in a loose way is an API -- there is a structured way to send data to an ICAP server and there are various responses you can expect to be returned. Its also has an RFC that can be referenced as well as third party tools that can be leveraged to make the integration easier.

HOWEVER, with MWG you can simplify things so you dont need to use ICAP.

In the MWG rules you could effectivley turn MWG into an HTTP file scanner (there's a ruleset for this). So you'd have your application download or upload the assembled file through MWG.

For example, you send a POST at MWG with a file, and based on your rules, MWG will interpret the request as an "internal request of sorts".

I've attached a piece of PHP code, its a class that could be used in conjunction with c-icap.

Can only post one file per post, will post a filescanner ruleset that can be used as well.

 

McAfee Employee jscholte
McAfee Employee
Report Inappropriate Content
Message 8 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Attached is a filescanner ruleset, here's an example cURL command you could use to replicate sending files to it from the CLI:

curl -F "myfile=@filename.doc" http://mwg:9090/filescanner

filename.doc is your file that you want sent to the mwg, and mwg is the actual MWG on its proxy port.

Let me know if this helps,

Jon

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

Thanks very much you both for the great help here. 

Hello Jon,  looks like the simple way is to use the http file scan.  We would like to avoid ICAP if possible per our past experience. 

curl -F "myfile=@filename.doc" http://mwg:9090/filescanner

Do you have any document listing out a little more detailed information about different types of response code based on scan results?

 

McAfee Employee jscholte
McAfee Employee
Report Inappropriate Content
Message 10 of 11

Re: how to scan large file uploaded by breaking into multiple small parts

Jump to solution

You would define the responses. This could be a header or something else.

The file scanner ruleset I attached is a good example of this, otherwise I might have another one for ICAP, I'll dig that up.

More McAfee Tools to Help You
  • Subscription Service Notification (SNS)
  • How-to: Endpoint Removal Tool
  • Support: Endpoint Security
  • eSupport: Policy Orchestrator
  • Community Help Hub

      New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

    • Find Forum FAQs
    • Learn How to Earn Badges
    • Ask for Help
    Go to Community Help

    Join the Community

      Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

    • Get helpful solutions from McAfee experts.
    • Stay connected to product conversations that matter to you.
    • Participate in product groups led by McAfee employees.
    Join the Community
    Join the Community