5 Replies Latest reply: Jul 17, 2014 9:15 AM by Regis RSS

    mtalk.google.com logging causing full disk?


      Google Chrome has recently added Google Cloud Messaging for Chrome. They've also tied Chrome Sync in to the same service. Unfortunately, it doesn't exactly play well in corporate environments for the following reasons:


      1. It connects on a non-standard HTTPS port of 5228
      2. It doesn't support proxy authentication.


      There was recently a bug that caused Chrome to continue to retry endlessly if it could not connect to 'mtalk.google.com' on port 5228. If MWG is not configured to allow that port, or if the firewall blocks the MWG from connecting to it, or if authentication is enforced, Chrome will send 700+ requests per second in attempt to establish a connection. This will cause your access.logs on the Web Gateway to grow very fast and in turn your Web Reporter Server / DB could fill very quickly as those logs are pushed over from the proxy.


      It seems Chrome version 35 has partially fixed this. In this new version, Chrome will not retry contiuously. However, it still doesn't support proxy authentication. Therefore, if you notice this behavior, the following may be required:


      1. Identify the source offendors and upgrade to Chrome version 35. It seems that it is usually only a handful of offending workstation IPs at a time.
      2. Ensure that 5228 is added to your ports allowed to connect list. This is typically found in the following places:
        1. SSL Scanner > Common Rules > Restrict destination port to Allowed CONNECT ports
        2. Common Rules > Restrict CONNECT Ports (new installs 7.4+)
      3. whitelist 'mtalk.google.com' from Authentication
      4. Ensure MWG is able to connect to port 5228 through the firewall.


      Now may also be a good time to setup monitoring alerts for your Web Gateway's disk space: https://community.mcafee.com/docs/DOC-4824


      This behavior is documented on Chrome's bug tracker: https://code.google.com/p/chromium/issues/detail?id=373181





        • 1. Re: mtalk.google.com logging causing full disk?

          Appreciate the write up. We recently started dealing with this and based on your response it looks like we are following a similar approach. Appreciate the feedback.

          • 2. Re: mtalk.google.com logging causing full disk?

            Correction. Chrome version 35 did not fix it. Version 36 will contain the fix.

            • 3. Re: mtalk.google.com logging causing full disk?



              There is an additional side effect that should be noted: large report server server.log files.. If the CONNECT on port 5228 is not allowed it could be blocked by the rules pointed out in #2 above. If that is the case, the block reason will be 103. Unfortunately, Web Reporter and CSR do not understand block reason 103 so they will print an error message in the server.log that states as much. This isn't normally a cause for concern but it is in this case due to the sheer amount of requests Chrome can generate in a short amount of time. This will result in very large server.log files on the reporting server for as long as the behavior is occurring.


              Here is an example access.log entry (block reason is the last column):


              [08/Jul/2014:15:32:37 +1000] "" 400"CONNECT mtalk.google.com:5228 HTTP/1.1" "" "-""" 3092 "" "" "103"


              Here is an example of the message as seen in a CSR server.log:


              2014-07-06 05:42:39,984 WARN  [com.mcafee.mesa.logparsing.parsers.util.ParsingUtils]Unmatched block reason number: 103.



              • 4. Re: mtalk.google.com logging causing full disk?

                I have concerns with opening non-standard ports and I don’t think enterprise IT should rollover and do Google’s bidding when the source of the issue is Google.  Organizations should push back and get google to fix the code and architect it’s features so that it can function within normal enterprise constraints.


                I think this thread had been mentioned (https://code.google.com/p/chromium/issues/detail?id=373181), individuals managing networks and security should chime in there so that google may respond to the issue and learn to co-exist in corporate networks.


                My .02


                • 5. Re: mtalk.google.com logging causing full disk?

                  I see that 36 is now out.    Has anyone seen confirmation that it does in fact fix this issue?  I saw on the code.google.com thread that it's alleged to, but then again they told us that with 35.



                  It'll be so nice to have just one access_denied.log per day again.