8 Replies Latest reply on Nov 6, 2012 6:15 PM by btlyric

    Upload vs. Download

    btlyric

      We were instructed that if the cycle = Request/Embedded Object, then that equals an upload.

       

      I've been doing some testing and my testing doesn't back up that assertion.

       

      One example:

       

      POST request that results in a file downloaded from remote site

       

      Has anyone else dug into this area?

        • 1. Re: Upload vs. Download
          pbrickey

          There's two parts to the traffic in your scenario - the actual POST (request) from the client and the response from the server that results in the file downloaded.

           

          When MWG process your request, the POST, through the rule engine that is the REQUEST cycle.

           

          When MWG process the response frorm the server with the file that is downloaded that is the RESPONSE cycle.

           

          If it is a zip file or other archive that MWG can extract/decompress other files it will send those files through the rule engine in the Embedded Object cycle.

           

          Hope that helps,

          Patrick

          • 2. Re: Upload vs. Download
            btlyric

            I was probably unclear in my original port. I will try to clarify.

             

            Based on input from Professional Services, I have a rule set with top level criteria of Request and Embedded Objects. The MWG GUI itself says HTTP(S)/FTP uploads if you hover over Requests (and IM) and says HTTP(S)/FTP downloads if you hover over Requests. But I digress.

             

            My rule set then has additional rules...

             

            Bypass Monitoring

            - various criteria for bypassing the monitoring -- known destinations, etc.

             

            Skip Empty Requests

             

            Reset Properties

            - in this rule set, various properties are set to false or null values

             

            Set Specific Properties (criteria Cycle.Name equals EmbeddedObject)

            - in this rule set, I set various values such as Body.Filename, Body.Size, MediaType, Body.NumberofChildren, etc.  to User-Defined Properties

             

            I then have a series of rules that look for things like Body.IsCorrupted equals true, Body.IsEncrypted equals true, Body.Size > X bytes, etc.

             

            This rule set triggers not only on uploads, but also on downloads.

             

            So okay, it's triggering on the Embedded Object cycle, but if that's the case, how can I distinguish between an upload vs. a download?

             

            I can't use BytesFromClient vs. BytesFromServer because that's only valid during the Logging cycle.

             

            So my question is how can I differentiate between something that's being uploaded vs. something that's being downloaded given that Request cycle + Command.Name != GET doesn't do the trick?

            • 3. Re: Upload vs. Download

              Try seeing if Cycle.TopName works for you.

              • 4. Re: Upload vs. Download
                btlyric

                Shouldn't

                 

                Applies to: Requests (and IM) be the same as Cycle.TopName = Request when it's invoked for a rule set?

                 

                If I log the Cycle.TopName and the Cycle.Name during that phase, it shows that Cycle.TopName is "Request"

                 

                Thanks!

                • 5. Re: Upload vs. Download

                  Is that what you want?

                   

                  The way i understand it it is the Top.CycleName is like the root cycle that calls the embedded cycles.

                  So on an upload, i get this as a file is getting unzipped in the embedded cycles:

                   

                  "multipart/form-data" "Request/Request" "-"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|NameLessFile"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|NameLessFile"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|mingwm10.dll"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|Win32DiskImager.exe"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|QtGui4.dll"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|QtCore4.dll"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|GPL-2"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|LGPL-2.1"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|README.txt"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|libstdc++-6.dll"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|win32diskimager-RELEASE-0.3-r27-binary.zip|libgcc_s_dw2-1.dll"
                  "multipart/form-data" "Request/EmbeddedObject" "NameLessFile|NameLessFile"


                  And on a download I get this:

                   

                  "application/x-zip-compressed" "Response/Response" "WinFormHtmlEditor.zip"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/de-DE.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/en-AU.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/en-CA.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/en-GB.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/en-US.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/es-ES.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/es-MX.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/fr-FR.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/it-IT.dic"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Learn about the structure of a dictionary.html"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Learn how to make Custom Dictionary.html"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Licenses/BSD.txt"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Licenses/GNU-GPL.txt"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Licenses/LGPL.txt"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Licenses/Microsoft Public License.htm"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/Licenses/_Simply Licensing Explained.html"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Dictionary files for spell checker/_Read Me.txt"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Main Control DLL/Microsoft.mshtml.dll"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Main Control DLL/WinFormHtmlEditor.dll"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Main Control DLL/_instruction.txt"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Sample Projects/Custom Context Menu/C # Sample/Form1.cs"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Sample Projects/Custom Context Menu/C # Sample/Form1.Designer.cs"
                  "application/x-zip-compressed" "Response/EmbeddedObject" "WinFormHtmlEditor.zip|Sample Projects/Custom Context Menu/C # Sample/Form1.resx"

                  <snip>

                  • 6. Re: Upload vs. Download
                    btlyric

                    Cycle.TopName in the overall criteria seems to do the trick. Does that mean that Applies to: is for the Cycle.Name? Or?

                     

                    Also, what properties are you using to generate the portion of the logged line that shows the main file name and then the embedded file names?

                    • 7. Re: Upload vs. Download

                      That log was generated from this rule I put in the root of my default MediaType Filtering  rules. I just pasted the last few fields into the last message post.

                       

                      Media Type Filtering
                      [Rules to block media types during upload and download for user group "internet_strict".]
                      Enabled
                      Applies to Requests: True / Responses: True / Embedded Objects: True
                      Always
                      EnabledRuleActionEventsComments
                      Enabled

                      Log Cycles

                      Always

                      ContinueSet User-Defined.logLine =
                           DateTime.ToWebReporterString +
                           " "" +
                           String.ReplaceIfEquals(Authentication.UserName,"","-") +
                           "" " +
                           String.ReplaceIfEquals(IP.ToString(Client.IP),"","-") +
                           " "" +
                           String.ReplaceIfEquals(List.OfString.ToString(DNS.Lookup.Reverse(Client.IP)),"" ,"-") +
                           "" " +
                           String.ReplaceIfEquals(IP.ToString(URL.Destination.IP),"","-") +
                           " " +
                           String.ReplaceIfEquals(Number.ToString(Response.StatusCode),"","-") +
                           " "" +
                           String.ReplaceIfEquals(List.OfMediaType.ToString(MediaType.EnsuredTypes),"","-" ) +
                           "" "" +
                           String.ReplaceIfEquals(Cycle.TopName,"","-") +
                           "/" +
                           String.ReplaceIfEquals(Cycle.Name,"","-") +
                           "" "" +
                           String.ReplaceIfEquals(Body.FullFileName,"","-") +
                           "" "
                      FileSystemLogging.WriteLogEntry(User-Defined.logLine)<POST.log>

                       

                       

                      • 8. Re: Upload vs. Download
                        btlyric

                        Body.FullFileName. Bingo. Thanks!