(In)security in URL shortening and redirection services

(In)security in URL shortening and redirection servicesAnthony BettiniMcAfee Labs6 May 2011 – CARO 2011 Workshop

2

• What is a URL shortening and redirection provider?– (Shortening vs. Redirection)

• Who are the popular providers?• Who uses a shortening provider and why?• Hypotheses• Case study: Deep-dive of a popular shortening provider• Hardening the Net• For malware researchers, why does all this (app-level security) matter?• Future research areas• Conclusions• Questions

Table of ContentsAnalysis of a URL shortening and redirection provider

6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague

3

• Technically, they are two separate services, although generally all providers combine them into one seamless service via APIs

• Shortening concatenates a string• Redirection does an HTTP 3xx redirect• Shortening example:

– We take this string:• http://www.gadgetguy.com.au/small-kitchen-appliances-toaster-kettle-coffee-mac

hine-blender-juicer-channel7-sunrise-australia-42.html

– We are returned this:• http://bit.ly/2Ngz9Q

What are we talking about?URL shortening and redirection what?


http://www.gadgetguy.com.au/small-kitchen-appliances-toaster-kettle-coffee-machine-blender-juicer-channel7-sunrise-australia-42.html



http://bit.ly/2Ngz9Q

http://bit.ly/2Ngz9Q

4

What are we talking about?Redirection as compared to shortening


• Redirection example:– We make an HTTP request for this, via an HTTP connection to bit.ly:80:

• GET /2Ngz9Q HTTP/1.1• (with other appropriate HTTP headers)

– Our client gets an HTTP 3xx code and makes a connection to:• www.gadgetguy.com.au:80

– While making a seamless request for:• /small-kitchen-appliances-toaster-kettle-coffee-machine-blender-juicer-channel7-

sunrise-australia-42.html

• Throughout the remainder of the presentation, we’ll just say shortening, when we really mean shortening & redirection (we do this to save space!)

http://www.gadgetguy.com.au:80/




5

• Most widely used providers:– bit.ly (bitly – startup, widely used)– goo.gl (Google)– fb.me (Facebook)– t.co (Twitter)

• There are many other less popular ones…

Who are the players in the game?Crowded space with big names


6

• Users of various Twitter clients are having their URLs auto-shortened to stay within the 140 character limit

• Facebook and other pages are doing similar things, for similar reasons• Corporate users are increasingly using shortening providers to convert long

internal URLs (such as direct links to docs buried on an internal Sharepoint) to to ease sending a link to a coworker

• Evidence of SPAM and malware authors leveraging shortening providers to mask URLs from being immediately read as suspect by users (making programmatic URL checking/filtering more important)

Who is shortening URLs and why?Only consumers, right?


7

• Complexity of the systems would increase over time as competition in this space increased

• Developers building URL shortening and redirection applications had limited exposure to real world security attacks; same for QA

• This limited understanding would result in mistakes, from a defense, safety, and security standpoint

• Testing for a wide variety of imperfections, across all providers, would have surprising security findings

• Summary: It would be easy to find INFOSEC problems

HypothesesThese came before the testing, really


8

• Examined one of the major URL shortening and redirection providers for security implementation limitations (weaknesses)

• Many provide APIs that require API keys• Reading the API documentation gives a good understanding of the system• API keys exist to:

– Track behavior– Rate limit requests– Restrict functionality

• We skipped the API key and wrote our own HTTP client– Problem solved

Case study: Analysis of a major shortening serviceOverview


9

Case study: Analysis of a major shortening serviceSome of the tests we ran, at a high-level


• Example tests we ran (basic unit testing, primarily for security):– Shortening of invalid URLs– Space efficiency when shortening strings with meaningless repetitions– Sequence predictability for keys in the dictionary– Shortening of self-referential URLs– Quick note on dictionary sizing– CAPTCHA requirements– Updating the source URL– Attempting to obtain competitive intelligence from providers on marketing campaigns

10

Case study: Analysis of a major shortening serviceThe findings…


• TEST NAME: Shorten invalid URL (invalid based on TLD)• EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should

result in an INVALID_REQUEST• RESULT:

– Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF– Result for BAD_URL: www.microsoft.ny -> http://svc/a0ewwA– #fail

11



• TEST NAME: Shorten what obviously isn’t a URL• EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should


– Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF– Result for BAD_URL: junk -> INVALID_REQUEST– #success

12



• TEST NAME: The same URLs should have the same result• EXPECTATION: GOOD_URL should be properly shortened, SAME_URLS

should receive the same result• RESULT (additional related findings on next slide):

– Result for GOOD_URL[0]: www.bing.com -> http://svc/9kkMVF– Result for SAME_URLS[0][0]: wWw.bing.com -> http://svc/b43j5I– #fail– Result for SAME_URLS[0][1]: http://www.bing.com -> http://svc/9kkMVF– #success

13



• RESULT (continued):– Result for GOOD_URL[1]: www.google.com/advanced_search ->

http://svc/aRxMmM– Result for SAME_URLS[1][0]: www.google.com////advanced_search ->

INVALID_REQUEST– #fail– Result for SAME_URLS[1][1]: www.google.com///advanced_search ->

http://svc/alc49z– #fail– Result for SAME_URLS[1][2]: www.google.com//advanced_search ->

http://svc/dpTc57– #fail

14



• TEST NAME: Shorten the shortener’s URL (self-reference)• EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should


– Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF– Result for BAD_URL: http://svc/9kkMVF -> INVALID_REQUEST– #success

15



• Quick note on dictionary/key space sizing, which we’ll discuss later• Provider A uses a default key space size, excluding custom URLs (we’ll get to

this later), of: 62^6 ([A-Z][a-z][0-9]^[fixed number of characters]) [56 billion]• Provider B uses a default key space size of: 36^6 [2 billion]

16



• TEST NAME: Quick check on predictability of keys in the dictionary• EXPECTATION: Keys should at least appear pseudo-random at first glance• RESULT:

– Result for www.d9204faa-d91b-4b85-8503-6d83a6b49842.ny -> http://svc/bd9Huo– Result for www.806b675b-da10-4791-b801-e7c78577c9b9.ny -> http://svc/cMuTnB– Result for www.559b37d2-505d-4155-8153-c84507decfde.ny -> http://svc/d8k7Zx– Result for www.7b21b10c-9380-497c-b933-a51ad56b66e2.ny -> http://svc/bbzXBM– Result for www.e0c42320-16be-4b3d-a627-b5cc7a2eae05.ny -> http://svc/cG2EpA– Result for www.0df5718c-7738-45bc-a1ac-a8c67b0433f4.ny -> http://svc/aGp6P8– Result for www.be2cb653-072b-44d6-a280-2f060a2e7e02.ny -> http://svc/dbIyo2– #success

17



• TEST NAME: Determine if programmatic scraping is occurring• EXPECTATION: People are brute force scanning URL shortening providers for

URLs• RESULT:

– Result for www.d9204faa-d91b-4b85-8503-6d83a6b49842.ny -> http://svc/bd9Huo– Result for www.806b675b-da10-4791-b801-e7c78577c9b9.ny -> http://svc/cMuTnB– Result for www.559b37d2-505d-4155-8153-c84507decfde.ny -> http://svc/d8k7Zx– Did analytics checks on the results; 0 clicks– #success

• However, if the scraping was for http://svc/123456+, the inverse lookup scraping wouldn’t be detectable in this manner (other than by the provider)

18



• TEST NAME: Account registration valid email and CAPTCHA• EXPECTATION: We estimated that registered users of the system have “more

rights” and that CAPTCHA was a requirement upon registration (to make it harder to programmatically generate accounts)

• RESULT:– Confirmed if you register, you have additional privileges– #success– No registration information is validated; fake email address permitted– #fail– No CAPTCHA required to create an account– #fail

19



• TEST NAME: Source URL replacement (i.e. the rickrolling test)• EXPECTATION: We estimated that registered users of the system may have

some means to replace a source URL• RESULT:

– Result for GOOD_URL: link.confusion.com/assumed_clean -> http://svc/fB3xbu– (Requested to “Edit” the link)– http://svc/fB3xbu now points to link.confusion.com/assumed_dirty– #fail

• An attacker could get Pointer A to become popular, where Pointer A redirects to Site A; then the attacker edit Pointer A to redirect to Site B, while keeping the integrity of Pointer A, thus compromising user’s who viewed the page after the change

20



• TEST NAME: Two providers friends that won’t play nicely together• EXPECTATION: We estimated that registered users of the system may have

some means to replace a source URL• EXAMPLE TEST:

– Shorten URL http://blah.invalid.url.com/blah on Provider A and get http://svc/dAf32X– Shorten URL http://svc/dAf32X on Provider B and get http://svc2/Adf392z– Edit http://svc/dAf32X on Provider A and redirect it to http://svc2/Adf392z– What happens?

• Left as a reader exercise…

21



• HACK IS WACK– “Upload your 2-min or less anti-cybercrime rap video”– Really clever and interesting marketing campaign http://www.hackiswack.com/– Went to bitly and did an inverse lookup (shortened an already shortened URL)– Was provided with: http://bit.ly/bOJg8r– Made an HTTP request for: http://bit.ly/bOJg8r+– Was handed all the analytics on the campaign’s “success”

http://www.hackiswack.com/

http://www.hackiswack.com/

http://bit.ly/bOJg8r




22

Future research areasWhy dictionary size matters


• Have established that creating a bunch of keys in the dictionary:– Can be programmatically done and doesn’t require valid URLs– There are a lot of way to hand the provider an invalid URL and get a key– Doesn’t require a valid email– Doesn’t require CAPTCHA

• Seems a lot like the traditional buffer overflow problem• When the dictionary fills up, what happens?

– Do old links get expired? If so, can they be tracked and then re-used?– Do links get overwritten? If so, can we first determine the most popular links? Can

we predict which will get overwritten first?• How fast can a botnet fill up a keyed dictionary with between 2-56 billion keys?• Answer: TBD

23

Hardening the NetMaking the interweb a safer place


• As we learned yesterday in Igor’s keynote, we’re not going to find problems where we’re not looking

• The same way we proactively scrape such providers as IRC, Usenet, and web sites for malware, we should be scraping the shortening providers

• While web security scraping systems and web site reputation systems have had to deal with site redirection, the mass-use of shortening and redirection services presents new challenges to these systems

24

Why does application security matter?URL shortening security design’s applicability for malware researchers


• New systems to scrape for malware• New user experience (at a consumer level), where URLs are intentionally

obfuscated and redirected, leading to a higher importance level for web site warning systems (ex. SiteAdvisor)

• Their increasing usage, presents new opportunities as well, to scan URLs prior to user’s even browsing to the page

• New challenges as well, such as the dynamic nature of shortened URLs pointing to new locations

25

• New technology brings new risks• Game theory can be used to present a nice backdrop to INFOSEC

– As new systems become prevalent, which were built limited awareness of attacks, those systems will become attacked

• The security of URL shortening and redirection providers will improve with time– But whether they improve their security, faster than their complexity increases, is

TBD!• Programmatic scraping of shortening providers is likely in the best interest of

anti-malware vendors, for the purpose of finding suspect behavior (the same way older systems, like IRC and Usenet are scraped)

• In short, exercise caution

ConclusionsAs with any new and widely used technology, exercise caution


26

• New technologies bring new challenges; if you need assistance, don’t hesitate to ask for help

• For additional questions and follow up, I can be reached at:– [email protected]

Q&AQuestions?


mailto:[email protected]

(In)security in URL shortening and redirection services

Documents

Transcript of (In)security in URL shortening and redirection services