How to Find & Fix Crawl Optimisation Issues - #BrightonSEO
28
@badams Crawl Optimisation Barry Adams
-
Upload
barry-adams -
Category
Internet
-
view
11.625 -
download
0
Transcript of How to Find & Fix Crawl Optimisation Issues - #BrightonSEO
@badams
Crawl Optimisation
Barry Adams
@badams
Barry Adams
• Doing SEO since 1998• Founder of Polemic Digital• Senior Editor at State of Digital
@badams
Crawl Optimisation
@badams
What is Crawl Optimisation?
Ensuring search engine spiders waste as little time as possible crawling the right URLs on your site.
@badams
Why is Crawl Optimisation important?
If you waste crawl budget, the right pages are unlikely to be crawled & indexed.
@badams
Crawl Budget
@badams
Google’s Crawl Sources
• Site crawl• XML Sitemaps• Inbound links
• DNS records• Domain registrations• Browsing data
@badams
Identifying Crawl Waste
@badams
Identifying Crawl Waste
@badams
Identifying Crawl Waste
@badams
Crawl Waste
• Bogus URLs in XML Sitemap
@badams
Optimise XML Sitemaps• Ensure your sitemap contains final URLs only
• Minimise 301-redirects or other non-200 status codes
• Use multiple sitemaps to identify crawl waste in GSC
@badams
Crawl Waste
• Paginated Listings• Faceted Navigation
http://website.com/jewellery/?page=2&cat=5&color=silver&style=glass&collection=autumnsort=a&…
@badams
Optimise Paginated Listings
• List more items on a single page
• Implement rel=prev/next pagination meta tags
• Block sorting parameters in robots.txt Disallow: /*?sort=*
@badams
Optimise Faceted Navigation
• Decide which facets have SEO value Build static pages for these
• All other facets: robots.txt disallow ‘rel=nofollow’ on facet links
@badams
Crawl Waste
• Internal Site Search Results
@badams
Block Internal Site Search Pages
• Block in robots.txt
User-agent: *Disallow: /SearchResults.aspxDisallow: /*query=*Disallow: /*s=*
@badams
Crawl Waste
• Internal redirects
@badams
Minimise Internal Redirects
• Find redirects with Screaming Frog• Internal links should all be 200 OK• Flat site structure
@badams
Crawl Waste
• Canonicalised Pages
@badams
Use Canonicals Wisely• “rel=canonical” is primarily for index issues
It is not a fix for crawl waste
Search engines need to see the canonical tag before they can act on it
Ergo, pages need to be crawled before rel=canonical has any effect
Ditto with meta noindex tags
@badams
DON’T use Canonicals for…
• Faceted navigation
• Pagination & sorting
• Site Search pages
@badams
OK to use Canonicals for…
• Separate mobile URLs
• Session-specific URL parameters
• Content syndication
• Unavoidable content duplication
@badams
Crawl Waste
• Slow loading pages
@badams
Optimise Load Speed• Time to First Byte• Lightweight pages• Caching• Compression
@badams
Crawl OptimisationSummarised
• Don’t let search enginesdo the hard work
• Tools at your disposal; DeepCrawl Google Search Console Screaming Frog SEO Crawler WebPageTest.org
• Solutions;– XML Sitemaps– robots.txt– rel=nofollow– rel=prev / rel=next– Load speed
@badams
The End Goal
