without taking a deeper look I am pretty sure that this is related to some kind of HTTP Live ticker. We have seen this for several URLs in the past, this technique was mainly used for Stock Tickers in the past, but for the World Cup it also seems to be useful. What happens is in general that the Client opens a TCP connetion to the server, which is left open with no data going through. Wenn something happens and the server wants to update the client, it uses the existing conenction to push data from the server to the client.
So in the past things like that worked like doing a META refresh every couple of seconds or so, but because today a few seconds delay is not acceptable and everything needs to be a Live Push Stream some services make use of this technology.
The problem here is the filtering itself, so if you put these URLs to the ICAP Bypass they work fine, as the "dead" connections will be detected by Webwasher and closed. With filtering enabled the ICAP filter thread waits for data even if the Client connection has been closed. I have not yet fully understood the issue, but from what I got so far is that there is no explicit reference between the Client connection and the Server connection, so when we perform ICAP filtering we do not correctly recognize if the Client has closed his session, and the server session remains active until it is closed from the Server side. The server may keep these connections opened for a long time (or even forever) and we will continue gathering the data in memory for scanning, which may lead to overload conditions etc.
MultiProcess will take care of this, but on SingleProcess machines this problem may become visible by Webwasher going into Overload protection.
I recommend to put the identified links
to the ICAP Bypass. I will check these links. We have some automatic mechanisms coming with the AV Updates that helps to avoid these problems. We have already seen p*.flashsports.co.uk and p*.ergebnisselive.de causing similar problems, and I bet the problem is the same. I will make sure they will be analyzed and handled correctly as well.
I have confirmed the URLs host the same service, but in a different language. I have forwarded the hosts causing the trouble to development, waiting for a feedback now.
yes I made the investigation with p*.flashsports.co.uk and p*.ergebnisselive.de. Both are causing exacly the same problem. Of course I already added the ICAP bypass, but I looking for a more complete workaround as this can happen for a lot of URL's.
I tried to solve the issue with the new 6.8.7 "rate limite" feature by limiting the number of concurrent connection. This work as expected, but as it count ALL concurrent connections and not only the one coming from a specific source or destination, we'll block all traffic for the specified group.
Can you tell me more about other workaround or when and what we can do with the AV pattern update.
there is not yet a full solution as it is not yet known if the problem can be solved with the 6.x architecture. We can work-around the problem by tweaking the AV updates on our side, which has helped a lot in the past.
At the moment the problem can only be really solved by switching to MWG7, as the architecture is different and allows to prevent this issue from occuring.