THE GOOD BAD - Akamai · • Java checkers • CAPTCHAs • Rate controls • Validation on...
1
Statistics based on attack campaigns mitigated by Akamai Poorly-coded bots can harm site performance and increase the load on web servers PERFORMANCE LOAD Fraud Mobile apps d d Statistics Financial statements Metasearch e e WHAT THEY DO: GOOD BOTS & BAD BOTS Data mashups Statistic Statistic Portfolio analysis Competitive intelligence Location tracking Hotel rates & airfares www.akamai.com/stateoftheinternet-security 1. Volume of requests 2. Type of scraped content 3. User agent info ways to identify client reputation for bots, spiders and scrapers 3 Highly desired, low aggression • Helps users • Respects robots.txt • Not too many requests Solution • Offer an API • Dedicated data feed Undesired, highly aggressive • Poor error handling • GET and POST floods • Thousands of requests/second Solution • Blocklists • Rate controls • Tar pits • Spider traps • Protect login pages Highly desired, high aggression • Access to emerging markets • Baidu-bots • Poor request throttling Solution • Rate controls with high thresshold • User-prioritization application Low desirability, low aggression • Scrape content for reuse • Fraud & counterfeiting • Headless browsers • Scraping-as-a-service Solution • Java checkers • CAPTCHAs • Rate controls • Validation on sensitive pages • Validation for suspicious IPs DESIRABILITY AGGRESSIVENESS HIGH HIGH LOW LOW Website Copier Baidu-bot Googlebot Location Scraper Product page scraper Metasearch scraper BOTS, SPIDERS AND SCRAPERS: & THE GOOD THE BAD THE UGLY
Transcript of THE GOOD BAD - Akamai · • Java checkers • CAPTCHAs • Rate controls • Validation on...
Statistics based on attack campaigns mitigated by Akamai
Poorly-coded bots can harm site performance and increase the load on web serversPE
RFO
RMA
NCE
LOAD
FraudMobile apps
Fraudapps
FraudStatistics
Financial statements
MetasearchMobileMetasearchMobile
WHAT THEY DO:WHAT THEY DO:
GOOD BOTS & BAD BOTS
Data mashupsStatistics
Data mashupsStatisticsPortfolio analysis
Data mashupsPortfolio analysis
Data mashups
Competitive intelligence
Location trackingHotel rates & airfares
www.akamai.com/stateoftheinternet-security
1. Volume of requests2. Type of scraped content3. User agent info
ways toidentify client reputation for bots, spiders and scrapers3
Highly desired, low aggression• Helps users• Respects robots.txt• Not too many requestsSolution• Offer an API• Dedicated data feed
Undesired, highly aggressive• Poor error handling• GET and POST floods• Thousands of requests/secondSolution• Blocklists• Rate controls• Tar pits• Spider traps• Protect login pages
Highly desired, high aggression• Access to emerging markets• Baidu-bots• Poor request throttlingSolution• Rate controls with high thresshold• User-prioritization application
Low desirability, low aggression• Scrape content for reuse• Fraud & counterfeiting• Headless browsers• Scraping-as-a-serviceSolution• Java checkers• CAPTCHAs• Rate controls• Validation on sensitive pages• Validation for suspicious IPs
DESIRABILITY
AG
GRE
SSIV
ENES
S
HIGH
HIGH
LOW
LOW
Website Copier
Baidu-bot
Googlebot
LocationScraper
Product pagescraper
Metasearchscraper
BOTS, SPIDERS AND SCRAPERS:
&&&THE
GOODGOODTHE
BAD
&&& THEUGLY
