cancel
Showing results for 
Search instead for 
Did you mean: 
watarimono
Level 9

Bad Gateway Proxy did not receive a valid response in time

We have a problem going to one particular website http://www.yaesu.com/ but we we receive the "Bad Gateway Proxy did not receive a valid response in time".  We know this is a valid site as we can access from different networks.  I can even whitelist the site and or my IP address and we still get the same results.

I've tried extending the timeout values but that did not make a difference.  We have an IPS outside the wall but we verified that is not stopping this traffic.

I do watch the traffic and I do see the traffic egressing the firewall. 

I can also do a TCPDUMP for the destination IP and see it registering.  I'll try to attach a pcap if possible but below is a sample from the command line.

******************************************************************************

17:21:29.932591 IP (tos 0x0, ttl 127, id 53410, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x5ce6 (correct), 127544031:127544031(0) ack 2266798793 win 65535 17:21:29.932950 IP (tos 0x0, ttl 127, id 53411, offset 0, flags [DF], proto TCP (6), length 358) 172.28.0.194.2777 > 71.139.254.252.80: P 127544031:127544349(318) ack 2266798793 win 65535 17:22:33.204907 IP (tos 0x0, ttl 127, id 57251, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x5215 (correct), 127544349:127544349(0) ack 2266801244 win 65535 17:22:33.210418 IP (tos 0x0, ttl 127, id 57252, offset 0, flags [DF], proto TCP (6), length 439) 172.28.0.194.2777 > 71.139.254.252.80: P 127544349:127544748(399) ack 2266801244 win 65535 17:22:33.211351 IP (tos 0x0, ttl 127, id 57254, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x4b3b (correct), 127544748:127544748(0) ack 2266802599 win 65535 17:22:33.211704 IP (tos 0x0, ttl 127, id 57255, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x4163 (correct), 127544748:127544748(0) ack 2266805119 win 65535 17:22:33.213155 IP (tos 0x0, ttl 127, id 57257, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x378b (correct), 127544748:127544748(0) ack 2266807639 win 65535 17:22:33.213166 IP (tos 0x0, ttl 127, id 57258, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x30e0 (correct), 127544748:127544748(0) ack 2266809346 win 65535 17:22:33.222436 IP (tos 0x0, ttl 127, id 57259, offset 0, flags [DF], proto TCP (6), length 445) 172.28.0.194.2777 > 71.139.254.252.80: P 127544748:127545153(405) ack 2266809346 win 65535 17:22:33.223252 IP (tos 0x0, ttl 127, id 57261, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x2a00 (correct), 127545153:127545153(0) ack 2266810701 win 65535 17:22:33.223798 IP (tos 0x0, ttl 127, id 57262, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x2484 (correct), 127545153:127545153(0) ack 2266812105 win 65535 17:22:33.228797 IP (tos 0x0, ttl 127, id 57263, offset 0, flags [DF], proto TCP (6), length 446) 172.28.0.194.2777 > 71.139.254.252.80: P 127545153:127545559(406) ack 2266812105 win 65535 17:22:33.231364 IP (tos 0x0, ttl 127, id 57265, offset 0, flags [DF], proto TCP (6), length 48) 172.28.0.194.2793 > 71.139.254.252.80: S, cksum 0x992c (correct), 3547207753:3547207753(0) win 65535 17:22:33.233987 IP (tos 0x0, ttl 127, id 57268, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x1da2 (correct), 127545559:127545559(0) ack 2266813461 win 65535 17:22:33.234005 IP (tos 0x0, ttl 127, id 57269, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x13ca (correct), 127545559:127545559(0) ack 2266815981 win 65535 17:22:33.234017 IP (tos 0x0, ttl 127, id 57270, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x09f2 (correct), 127545559:127545559(0) ack 2266818501 win 65535 17:22:33.234027 IP (tos 0x0, ttl 127, id 57271, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x001a (correct), 127545559:127545559(0) ack 2266821021 win 65535 17:22:33.234032 IP (tos 0x0, ttl 127, id 57272, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2793 > 71.139.254.252.80: ., cksum 0x1d0b (correct), 3547207754:3547207754(0) ack 3580220171 win 65535 17:22:33.236010 IP (tos 0x0, ttl 127, id 57273, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x001a (correct), 127545559:127545559(0) ack 2266823541 win 63015 17:22:33.236024 IP (tos 0x0, ttl 127, id 57274, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x001a (correct), 127545559:127545559(0) ack 2266826061 win 60495 17:22:33.236030 IP (tos 0x0, ttl 127, id 57275, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x001a (correct), 127545559:127545559(0) ack 2266828581 win 57975 17:22:33.236541 IP (tos 0x0, ttl 127, id 57277, offset 0, flags [DF], proto TCP (6), length 448) 172.28.0.194.2793 > 71.139.254.252.80: P 3547207754:3547208162(408) ack 3580220171 win 65535 17:22:33.236978 IP (tos 0x0, ttl 127, id 57279, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xf169 (correct), 127545559:127545559(0) ack 2266829173 win 61143 17:22:33.237173 IP (tos 0x0, ttl 127, id 57280, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2793 > 71.139.254.252.80: ., cksum 0x1b73 (correct), 3547208162:3547208162(0) ack 3580220998 win 64708 17:22:33.239056 IP (tos 0x0, ttl 127, id 57281, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xe041 (correct), 127545559:127545559(0) ack 2266829173 win 65535 17:22:33.247127 IP (tos 0x0, ttl 127, id 57283, offset 0, flags [DF], proto TCP (6), length 446) 172.28.0.194.2777 > 71.139.254.252.80: P 127545559:127545965(406) ack 2266829173 win 65535 17:22:33.248009 IP (tos 0x0, ttl 127, id 57285, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xd95f (correct), 127545965:127545965(0) ack 2266830529 win 65535 17:22:33.248270 IP (tos 0x0, ttl 127, id 57286, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xcf87 (correct), 127545965:127545965(0) ack 2266833049 win 65535 17:22:33.248294 IP (tos 0x0, ttl 127, id 57287, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xc5af (correct), 127545965:127545965(0) ack 2266835569 win 65535 17:22:33.248615 IP (tos 0x0, ttl 127, id 57288, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xbbd7 (correct), 127545965:127545965(0) ack 2266838089 win 65535 17:22:33.248636 IP (tos 0x0, ttl 127, id 57289, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xbbd7 (correct), 127545965:127545965(0) ack 2266840609 win 63015 17:22:33.248892 IP (tos 0x0, ttl 127, id 57290, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xbbd7 (correct), 127545965:127545965(0) ack 2266843129 win 60495 17:22:33.249240 IP (tos 0x0, ttl 127, id 57291, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xbbd7 (correct), 127545965:127545965(0) ack 2266845649 win 57975 17:22:33.249263 IP (tos 0x0, ttl 127, id 57292, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xbbd7 (correct), 127545965:127545965(0) ack 2266848169 win 55455 17:22:33.249731 IP (tos 0x0, ttl 127, id 57293, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0xbbd7 (correct), 127545965:127545965(0) ack 2266850689 win 52935 17:22:33.249972 IP (tos 0x0, ttl 127, id 57294, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x85b3 (correct), 127545965:127545965(0) ack 2266851949 win 65535 17:22:33.250019 IP (tos 0x0, ttl 127, id 57295, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x85b3 (correct), 127545965:127545965(0) ack 2266854469 win 63015 17:22:33.250045 IP (tos 0x0, ttl 127, id 57296, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x85b3 (correct), 127545965:127545965(0) ack 2266856861 win 60623 17:22:33.250353 IP (tos 0x0, ttl 127, id 57297, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: ., cksum 0x7703 (correct), 127545965:127545965(0) ack 2266856861 win 64383 17:23:33.826365 IP (tos 0x0, ttl 127, id 60917, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2793 > 71.139.254.252.80: R, cksum 0x1834 (correct), 3547208162:3547208162(0) ack 3580220998 win 0 17:23:33.826446 IP (tos 0x0, ttl 127, id 60918, offset 0, flags [DF], proto TCP (6), length 40) 172.28.0.194.2777 > 71.139.254.252.80: R, cksum 0x727f (correct), 127545965:127545965(0) ack 2266856861 win 0

Anyone have any ideas?

Thanks!

-Wat

0 Kudos
10 Replies
asabban
Level 17

Re: Bad Gateway Proxy did not receive a valid response in time

Hello,

can you attach the pcap?

Best,

Andre

0 Kudos
watarimono
Level 9

Re: Bad Gateway Proxy did not receive a valid response in time

Thanks Andre, Yea I did but it was initially over 100M so I had to narrow the capture down.  I used the command  -s0 host 71.139.254.252 and I ran it twice. There are two captures attached.  The website ip is the 71.169.254.252 and I was attempting it from the ip of 172.28.0.194.

Thanks for your  help!

-Wat

0 Kudos
asabban
Level 17

Re: Bad Gateway Proxy did not receive a valid response in time

Hello,

from what I can see in the dump some parts of the response get lost. MWG sends the GET  request for the URL once the tcp connection was established. The server replies with a 200 OK and sends headers. Now we wait for further data because the data is not complete. Instead of receiving more data (the beginning of the requested web site) we receive some data, but it seems there is a large chunk missing.

We send an answer to the server again to tell him what parts we have received and that we still miss something, but data does not seem to be returned to MWG. From what I can see in the dump some fragments do not arrive at MWG.

I have tried to access the site on my end through a lab MWG and I confirm the site seems to be up and working as expected. Do you have any chance to capture the traffic on one of the next devices to find out where it gets lost?

Best,

Andre

0 Kudos
watarimono
Level 9

Re: Bad Gateway Proxy did not receive a valid response in time

Thanks man,

Our IPS guys are telling me they don't have anything being stopped from that IP but maybe it's coming back as something different.  I'm not sure.  We are still troubleshooting on our end.  I will update once we have some new info.

Thanks again,

0 Kudos
watarimono
Level 9

Re: Bad Gateway Proxy did not receive a valid response in time

Ok, this just in from the IPS guy.  It's a capture while going to the website.

The host machine is 172.28.0.194 and it's going to the destination of 71.139.254.252 .

The webgateway address is 172.28.10.16.

This is a very small file so I'm hoping something can be derived from it.

Thanks again,

0 Kudos
asabban
Level 17

Re: Bad Gateway Proxy did not receive a valid response in time

Hello,

I see a source IP of 75.112.151.67. Is that the IPS device maybe? I can see that the devices establishes a TCP connection to the remote side. It also sends the GET request and obtains the header. Additionally I can see parts of the HTML source coming (which have not arrived on MWG according to the previous capture). However something seems to go wrong, because the device the capture was made on receives traffic, but does not tell the remote site that the traffic arrived. The remote site tries to resend the same amount of data (the first kilobytes of the website), but gives up after 60 seconds and resets the connection.

What I can see is that between the HTTP header and body there is not only two newlines (which is usually the case), but a lot of newlines are added between header and body. I cannot tell if this is legit, but maybe the IPS device does not like this kind of HTTP response? Maybe it tries to keep HTTP traffic RFC conform and does not accept this kind of response.

However I am not 100%ly sure, but thats what I would say from looking at the dump. Maybe it helps you to move forward.

Best,

Andre

0 Kudos
watarimono
Level 9

Re: Bad Gateway Proxy did not receive a valid response in time

Thanks again Andre,

Yea, I was looking at some past posts about non-RFC compliant traffic and I wonder if it's the IPS or the Gateway that is stopping it.

We are continuing to troubleshoot but thank you so much for looking at those captures.

When we confirm and answer I'll post.

Best of luck to you,

-Wat

0 Kudos
singhsanjeev
Level 7

Re: Bad Gateway Proxy did not receive a valid response in time

Hi,

Have you got any solution for the same, because we are also face same problem

Thanks & Regards

Sanjeev Singh

0 Kudos
watarimono
Level 9

Re: Bad Gateway Proxy did not receive a valid response in time

Hello Sanjeev,

No we actually haven't been able to resolve that particular issue.  We believe it has something to do with the IPS as Andre eluded to in his post but we haven't been able to confirm.

0 Kudos