1 2 Previous Next 15 Replies Latest reply on Feb 29, 2012 10:29 AM by srini2411

    Need help in analyzing the tcpdump out from WebGateway

      Hi Guys,


      I have a specific site which has a flash embedded in it, but for some odd reason my gateway doesnt load the flash content. But when I whitelist the site, it works fine.

      I captured traffic for the same via Tcpdump , but I am having difficulties in analyzing the output as I am unable to understand the reason for issue from the output.

      Herewith attaching the same, highly appreciate your valuable suggestions.




        • 1. Re: Need help in analyzing the tcpdump out from WebGateway

          Sorry forgot to mention the site that was beinga accessed.


          • 2. Re: Need help in analyzing the tcpdump out from WebGateway



            it doesn´t look like there is outgoing communitcation in the dump. I can see Client <-> MWG, but no MWG <-> Server communitcation. All I can see is the client requested the flash content, but no response came back. It would be interesting to see how MWG asks the server to deliver the content, and what happens.


            Can you try to capture that traffic as well? Most likely you did not specify -i any on the command line - in this case only eth0 will be recorded.




            • 3. Re: Need help in analyzing the tcpdump out from WebGateway

              Hi Andre,


              Yeah my bad I was filtering the traffic based on client's IP, hence the communication between server and destination is missing. Just being too concerned about the the size of the dump files which always turnout to be too big.


              The command I ran was : tcpdump-npi any -s0 -C 300 host "Client IP" -Z root  -w "filename"


              is there a way I can filter only the required traffic via dump? generic dump ends up capturing all the unwanted traffic as well.


              • 4. Re: Need help in analyzing the tcpdump out from WebGateway

                Hi Asabban,


                Herewith attaching the tcp with destination trafic as well, Please have a look a advise .


                The command I used for capturing is as below

                tcpdump -npi any -s0 -c 300 host  "client IP" or host "destination IP" -Z root -w "filename"


                Message was edited by: srini2411 on 2/26/12 11:29:16 PM CST
                • 5. Re: Need help in analyzing the tcpdump out from WebGateway

                  Hi Srini,


                  the server seems to have a problem. In frame 158 the client asks MWG to fetch the file. In frame 159 MWG asks the server to deliver the file. In frame 160 the server says "OK, here it comes" and does not send anything.


                  Since MWG does not receive data from the network it can´t forward the data to the client.


                  The interesting part is now that you mentioned it works with a whitelist entry. The only reason for this I can think of is that the server does not like the header we sent, and the application running on the server behaves weird. Do you think you can provide a capture which shows the same data but with a whitelist entry in place and the movie working?


                  Then we could compare the headers. But from what I see it looks like the server is not sending us any data, which is never a good thing :-)




                  • 6. Re: Need help in analyzing the tcpdump out from WebGateway

                    Hi Andre,


                    Herewith attaching the capture while whitelisting was enabled.


                    Also can you pls tell me how do you differentiate the logs as frame 158, 159 etc...so that I can also look accordingly and understand .





                    Message was edited by: srini2411 on 2/27/12 2:35:38 AM CST
                    • 7. Re: Need help in analyzing the tcpdump out from WebGateway

                      Hi Srini,


                      when you whitelist the URL, the request contains a header:


                      Accept-Encoding: gzip, deflate


                      This tells the server what encodings for the content are excepted. If there is no whitelist entry, this header is missing. Without the header, the server is stuck and does not send content.


                      When I compare the requests I notice that the whitelisted request contains more headers compared to the one not whitelisted. Are there any rules in place which delete headers from the request? Especially the "Accept-Encoding" header? I believe the default "HTML Filtering" rule set does this as well... you should make sure the requests to this server do not hit the rule sets removing the headers.


                      Finding out where the traffic is from the dumps is not too complicated. First off all you should tell wireshark to decode your proxy port (9092 I believe) as HTTP. Once you do that, all proxy traffic immediatly becomes very reable. Now you just compare, the communication which includes port 9092 is proxy traffic, the traffic which inclused port 80 is the traffic that goes to the server. Thats all :-)




                      • 8. Re: Need help in analyzing the tcpdump out from WebGateway

                        Hi Andre,


                        That was spot on Mate !!! . I had a rule which was removing header (Accept-encoding) and when I tested disabling that specific rule , Bingo it worked fine .


                        I also went through the dumps and  like you had said I did find the header Accept-Encoding: gzip, deflate missing in the request sent by the MWG to the destination server whereas it is present in the request sent by the client to MWG.  whereas when whitelisted thats not the case.


                        Now I will have to understand why this is happening with this specific site alone? May be its because of the flash player (flip express) embedded in the site?


                        Now that will be quite interesting to know the cause because that specific header rule is in place for all other traffic as well but its impacting only this specific rule.


                        Thanks for your valuable time on this Andre, much appreciated . Now this motivates me to look at the other issues  and learn something new . Thanks again.

                        • 9. Re: Need help in analyzing the tcpdump out from WebGateway

                          Hi Srini,


                          I do not think the issue is related to the flash player. I believe this is a problem on the server side. If the header is missing, the server is not sending back a single byte of data. The client or flash player do not even know that their request was modified on the way to the server. If the server would reply with content, the video would be shown perfectly fine.


                          What I find very often especially with Java Apps or other non-standard software running on web servers is, that they are not very robust in case a request does not look like a "lab" request. In many similar examples I have seen during the past it looked like the developers of custom client or server components perform some tests with an Internet Explorer and maybe a Firefox, and if all works they are happy.


                          Some of them do not even seem to know that there may be a proxy in the loop, I have seen servers which completely fail when you add a proxy header to the connection.


                          From what I saw so far I assume the server has a bug. If there is no Accept-Encoding header in the request, the request is still perfectly fine and should comply to the HTTP RFCs so the server should reply something. Even an error message that says "I am sorry but you have to send an Accept-Encoding header" would be a fine response, but sending nothing and keep the connection open yet unused does not look like valid/wanted behaviour.


                          I know that MWG has some issues from time to time, but many many problems I have seen in the past were related to other products - actually MWG makes them visible ;-)


                          I think there is no need to care too much about this specific site. It seems their server is not sending valid responses to a valid request, but you may want to contact the site administrator and let them know. Generally removing the header should be fine. Certainly if the server does not send responses any longer when the header is missing, MWG cannot show any content :-)




                          1 2 Previous Next