2 Replies Latest reply on Jul 13, 2017 12:38 PM by johnaldridge

    A Shortened URL Category and Ratings Through a Link Expander API


      Oh, those shortened URL's.  We just out-and-out block them.  Easy enough—except that we have to do it according to the known services.


      But, it's so easy to create a shortening service.  Do we really have to go searching for them periodically to update our lists.


      Should trustedsource.org have a category for shortened URL's?  (Question 1)


      I've heard our analysts complain that shortened URL's aren't categorized and risk scored according to the final destination.  And, I was pondering how this might be improved.


      It seems that I could leverage a Link Expander API, like http://www.linkexpander.com/api.  The rule set would be just a little more complicated than that of CheckURL.  The shortened URL would be expanded through the API; and if it passes muster through the risk rating and categorization service, would be redirected to the long URL—perhaps with coaching for moderate risk ratings and/or somewhat questionable categories.


      But, we'd still be detecting the shortened URL's by known shortening domains.  It would be nice to have a category for this—and, correct me if I've missed it, but I don't see one.


      So, it seems a rule set is possible even with the current limitations of trustedsource.org.


      However, given that I hear analysts complaining that ratings on trustedsource.org don't reflect the ratings of the final destination, would it be worth a feature request directed at trustedsource.org: a feature request for trustedsource.org to do it's own link expansion and rate shortened URL's accordingly? (Question 2)


      Of course, we could just keep blocking them; and for now, we will.


      However, it would be nice if there was a category just for shortened URL's (Question 1), though it would be a killer feature if trustedsource.org also did its own link expansion (Question 2)—eliminating the temptation to write a rule set to handle that may or may not be trustworthy enough to handle this.


      (Note that I did search the forum for discussions along this line, and the one question that comes close, shorter URLs like ow.ly, which received no responses.)

        • 1. Re: A Shortened URL Category and Ratings Through a Link Expander API



          as the shortened URL only redirect the browser to a new location, WebGateway will check both URLs against the configured rules. So it's likely that the shortened URL is not listed, as there can be hundreds of shortened URLs pointing to one URL, and it's enough to block/categorize the final URL.




          • 2. Re: A Shortened URL Category and Ratings Through a Link Expander API

            That's a good point.  However, the decision to block all shortened URL's is on the basis that we want stricter filtering for the final destinations on shortened URL's.  That is, we would want only the most trusted destinations to be allowed by way of a shortened (obfuscated) URL, and definitely not unverified destinations.


            Note that a user is welcome to manually expand a shortened URL.  Hopefully, they'll know better than to go to getinfectedhere.ru.


            What our policy means is that if we allowed shortened URL's we would want to scrutinize the shortened URL at the time of the request for the shortened URL.


            Note that doing this with a referrer would not satisfy what our policy makers regard as healthy security paranoia. Back to the point about a training page: imagine if a user was prompted with: "That URL resolves to getinfectedhere.ru, which is unverified and located in Russia.  Are you really sure you want to go there?"  (Of course, blocking by geo-location is also possible, but there's plenty of CDN content for U.S service that is hosted outside the U.S.)  So, this is really a discussion about improving capabilities.  And, it seems appropriate to compare the advantages of what is currently possible to what could be done beyond what is currently available.