while working for a customer who has the requirement to have Streaming Media working (especially demand for Web Radios) I have spent some time to check how good MWG7.0.x deals with this kind of data. Unfortunately out of the box the handling of Streaming Media did not work really well for me, so I have checked a couple of Radio stations for "common denominators" and found out some things I would like to share.
Please understand although I am working for McAfee this article is solely community based, which means that neither myself or my company will take any responsibility in regards to the below details. My main intention is to "teach" something, not to provide an "import and forget" solution!
The customer in this case wants to allow his users to access Streaming Media, especially Web Radio. Of course we do not want to whitelist any single Radio station someone probably complains about, so I tried to find some rules which will work for a handful of "known" Radio stations and will probably also match unknown Radio stations, decreasing or eliminating the need to manually add Radio stations once someone finds a new "favorite" Radio station.
It is pretty important to understand what needs to be done to allow Streaming Media. Basically MWG operates in a way that a requested object is downloaded, filtered and afterwards delivered to the Client. Basically we download a file completely before filters such as Archive openers or AV will be applied, because this data cannot be filtered while it "runs through" the proxy.
Obviously it is not possible for Streaming Media to completely download the audio/video content, filter it and pass it to the Client. At least in this scenario we want Web Radios to behave in a similar way than without MWG, e.g. I click the Play button, it takes a couple of seconds for buffering and then I can hear/see my desired content. Therefore it is required to skip some of the filters MWG applied, especially AV/AntiMalware needs to be skipped and also I do not want to apply Data Trickling or Progress Pages.
This Rule Set uses several critera to find out if a data stream is related to Streaming Media and calls a "Stop Cycle" for those requests, to prevent filters from being applied. The Rule Set tries to find a good mix of "keep security as tight as possible" and "make Streams work by bypassing filters". Of course any kind of bypass rule like this allows something to slip through MWG. Please make sure this is what you want.
This may not be the best solutions and as far as I know we are working on a better Streaming identification, but I wanted to share this for anyone who may be interested.
Rule Set Description:
I will describe the Rule Set quickly. Lets have a look first at the "primary" Rule that I use for calling a "Stop Cycle" action:
The Rule has 5 criteria and allows to be matched when two combinations of parameters become true:
1.) Alternative, criteria A, B and C must be fulfilled:
- First of all we do check the "Content-Type" that has been announced by the remote Web Server. We have to trust that a server hosting Streaming Media tells us the truth about the data it is sending, since we won´t double check with the Media Type Filter capabilities of MWG. I have found out that running Media Type Filters on the Streaming Data may actually break the Stream. This criteria is comparable to the "Be less restrictive..." setting that was available in MWG 6.x to allow Streaming Media. We match the "Content-Type" sent by the remote site against a list of trusted Media Types, which is included in the Rule Set and explained later. This list can be extended/reduced as you like. Note: Please note that this is a list of the type "Wildcard".
- Because I do not want to only rely on what the remote Web Server told me I will add some extra security here. I check the category of the URL that has been requested against the TS database. The destination host must be in one of the categories listed in a list I have configured. I have prefilled the list with "Streaming Media", "Content Server", Internet Radio/TV" and "Internet Services". When testing I have found those categories to be used for the servers usually. You can of course modify the list.
- For the little extra security I am not happy when the above is true, but I also check if McAfee/Trusted Source confirms "Minimal Risk" as the reputation for the remote server.
Only if ALL of the above is true, the Rule will fire. This matches for a lot of Streaming Media sites I have found during testing.
2.) Alternative, criteria D and E must be fulfilled:
- Again I am checking the "Content-Type" announced by the remote Web Server. This is the same criteria as "A".
- In this example I do not check the Category of the remote site, but check the User-Agent of the Client. If it matches "NSPlayer/*" it is expected that a Windows Media Player is talking to us and trying to get access to Streaming Media.
I have designed this second alternative since for Streams relying on Streaming Media the content is hosted pretty often on servers which are not accessed with a URL, but by their IP address. Some of them may not be categorized accordingly. Windows Media Player sends us the User-Agent "NSPlayer/*" which I used to identify it. The good thing is that Windows Media Player only sends "NSPlayer/*" when it tries to call the related Audio/Video data. There is a rudimentary browsing feature embedded into the Player, but when doing so "Internet Explorer" is sent as the User-Agent, so normal filtering will apply. This criteria pretty much matches the "Allow User-Agent..." settings that can be found in MWG 6.x.
Note: If you have enough permissions on your workstation you may be able to fake the User-Agent header. This will allow you to gain some more permissions, although you can only download data which has a matching "Content Type" as defined in criteria D. if you are not using Windows Media Player and/or want to allow other Players, you will need to get the correct "User-Agent" header which these players use for identification, and add them to the Rule.
Once the Rule triggers it applied a "Stop Cycle" action, causing the processing of Rules to stop immediatly. Depending on where you have placed the Rule Set, things like AV/Anti Malware, Openers, Authentication etc. can be skipped. I recommend to place the Rule Set somewhere after global Allow/Block lists. If you call other filters before this Rule Set matches, this may lead to Streaming Media becoming unavailable, due to those other filters trying to download the Audo/Video files to disk for filtering.
I am referring to two lists within the Rule Set.
1.) Streaming Media "from Header" Wildcard List: This list contains Media Types that we want to allow for Streaming Media. It is basically a copy of the Default lists for Streaming Media, but is a list of type "Wildcard", because I use "MediaType.FromHeader", which takes a Wildcard parameter, instead of a pre-defined Media Type. The list is basically a Plaintext list. You can use Wildcards, such as "audio/*" or "video/*", depending on your requirements.
2.) Streaming Media Categories: This list contains categories which we accept for Streaming Media. I have prefille it with some values I thought to "make sense". It may need to be modified.
Of course it would have been too simple if the Rule Set would allow any kind of Streams, right? :-)
There are some Streams which won´t play:
These Streams won´t play in MWG 7.0.x, as the Server does not send a valid HTTP response, but a Header not matching HTTP RFCs. MWG will send a "502 Bad Gateway" instead of the Stream. You can identify those kinds of Streams by looking into access.logs, which will show you the 502 as a response code. Since this is not a 100% proof you can run a packet capture and have a look at the communication between MWG and the remote Web Server. You will see something like this:
GET /stream.mp3 HTTP/1.1
ICY 200 OK
As explained an error template will be forwarded to the Client. In MWG 7.0.x these streams won´t work, but the current Beta version MWG 7.1 (available for Beta customers, released version will follow) allows this data to flow through. With 7.1 Beta I had pretty good experience that those Streams will work.
RTMP type streams
RTMP seems to be a pretty new, proprietary protocol for Streaming developed by Adobe. The Flash Player on the Web Site actually does not necessarily respect your proxy settings, but will try to get access to the server directly on tcp port 1935. As long as your Firewall allows outgoing connections on this port, the streams will play fine. If port 1935 is blocked, some kind of fall back to a tunneled HTTP connection is implemented, but I have found that most likely the remote Servers won´t accept those connection attempts, and the streams will fail to play.
This kind of Stream is not easy to detect, but you will notice that your Client PC will try to get out to the Internet on port 1935 when you click on "Play". Also when the HTTP fall back applies, you will see data with a Content-Type of "application/x-fcs" going through MWG, most likely you will see POST requests sent to a URL containing "/fcs/ident" or "/open/1". I have seen those requests failing pretty often, the root cause is still unknown, but it seems that the remote Server is not accepting those requests, since the issue also occur if I pick up the request sent by the Client application, and issue it directly to the remote Server on port 80.
Currently I am not aware of a good way to get those streams working.
--- This is recommended for testing purposes only ---
There is an Open-Source RTMP proxy available (see http://rtmpdump.mplayerhq.hu/). For those Linux fans who like the challenge I have tried to set this up:
- Pick a Linux box (I have used my Ubuntu Jaunty server)
- Get the sources from the above mentioned URL
- Run "apt-get install libssl-dev" to install OpenSSL libraries
- Run "ldconfig"
- Extract the sources and "cd" into the extracted folder
- Run "make SYS=posix"
- Run ./rtmpsuck
The proxy will automatically start listening on all interfaces on Port 1935. You can now transparently redirect your Clients request to port 1935 to the proxy. Do this on a Router you cross on your way to the Internet or locally on a Linux box by setting an iptables Rule:
iptables -t nat -I OUTPUT -p tcp --dport 1935 -j DNAT --to 192.168.0.1:1935
If your Client now access a Stream on 1935 it will be redirected to the rtmpsuck proxy running on 192.168.0.1 port 1935. This worked like a charme.
Note: I am not sured if this is something you want to use for your enterprise. I did these tests to better understand what is happening and for learning/demonstration purposes ONLY.
--- This was recommended for testing purposes only ---
Currently MWG7.0.x does not provide on-the-box coverage for RTMP. A FMR is filed, we should be able to handle those streams in the future.
If you are still reading I hope you were able to extract some interesting information in regards to streaming. If you are following the above mentioned hints and are not struggling those Streaming Media types listed in the "ISSUES" section, you should cover a huge variety of available Streams. My tests are related to German Web Radios only, maybe there are regional differences.
If you find Streams not working, I would be really interested if you can provide me with the URL since I can probably improve my Rule Set. In this case please sent me a PM, I will have a look ASAP.
Yes, thank you for taking the time to write this. I had the same issue about a week about with a streaming site. Combination of the file being scanned, lack of progress indicator was causing to hang. I whitelisted the backend site in question, but I really wanted to figure out what was going on. Turned out it was pretty similar to what you pointed above.
in some cases streaming is not working even anything is whitelisted. Often the streams are not supporting chunked encoding. When you have this problem the streams sounds choppy.
Just enable a workaround not using chunked encoding.