Diagnosing Crawl Issues Using SEO

Crawling

• Crawling means the visit of Google Bot (spiders) to track your page.

• In SEO words, crawling means following your links and crawling your website.

• When bots come to your website it follows other linked page and crawls your site.

• Crawling is done by search engine crawlers.

• A search engine crawler is an automated script that browses the WWW to provide up to date data to the search engine

www.yourseopick.com

Indexing

• After crawling, the search engine processes each of the page it crawls.

• It compiles massive index of all the words it sees and their location on each page.

• This content is then stored, with the information and organized by the search engines.

• Storing and sorting this informations requires space and both Yahoo and Google have over a million server each.

www.yourseopick.com

Crawl Log Example

www.yourseopick.com

Crawl Issue Example

www.yourseopick.comwww.yourseopick.com

Diagnosis - Webmaster ToolYou’ll see three possible types of crawl errors.

HTTP errors – These aren’t common, and they usually occur in small numbers. This happens when Google gets a server-side response that doesn’t yield an “ok” or “found” status.

Not Found –In succeeding crawls, Googlebot will revisit these pages and compare them with what it has in its cache. When it doesn’t find the pages that it wants to revisit, it lists the URLs as “404 Not Found.”

• Restricted by Robots.txt – Your robots.txt file is a part of your site where you tell search engine spiders which pages in your site they can crawl and which ones are off- limits.

www.yourseopick.comwww.yourseopick.com

Not Found URLs from old pages that have been replaced

by new ones – 301 redirect them to the URL of the new page that replaced them.

HTTP errors – Ask your developer to diagnose what type of HTTP error it is, since there’s a lot of these.

Private/Secure Pages restricted by robots.txt – No action

Public pages restricted by robots.txt – Edit robots.txt file and remove the URL from the list. -

www.yourseopick.com

www.yourseopick.com

Crawl Efficiency Checklist

• Ensure page load time aren't slow as to reduce the number of pages crawled.

• Ensure server is responsive.• Return a 303 for unchanged content.• Use a compression.• Return a 404 for not found content.• Ensure each page has at least one link.• Avoid infinite redirects and redirect loops.• Ensure most important pages are linked from homepage.• Reasonable crawl delay setting (if used at all).• Reasonable use of Google Webmaster tool crawl setting.

www.yourseopick.com

Who we Are ? YourSEOPick is an initiative by Smartinfosys.net, a brand owned by

WebConnect Private Limited. We are committed to provide the best SEO services to all businesses to make the most out of their websites.

Our goal is to help our clients maximize their return on investment by generating high quality traffic from the major search engines.

Unlike other SEO companies that are just focused on hiking ranks on Google, YourSEOPick is focused on leads and sales.

All type of SEO services can be found under one roof YourSeoPick.

www.yourseopick.com

Our Services Local SEO : Optimize your website for local searches. Get found locally

and maximize your earning through people in your targeted geographical Areas.

Mobile SEO : Optimize your website for mobile searches and get maximum exposure from more than 60% of total Internet traffic that comes from mobile device visitors.

Comprehensive SEO audit : A website is audited for various on-page and off page factors that affect visibility of a website on various search engines.

www.yourseopick.com

Reach Us

E-mail: support@yourseopick.com

URL: www.yourseopick.com

Address : 401/604/801, Meridian Tower, Nr. Apple Hospital, Udhna Darwaja,Surat. Gujarat (INDIA) - 395002

Contact: +91-9426157202

www.yourseopick.com