Bing Webmaster Tools Crawling Basics Webinar

Crawling Basics

35,000 feet…

What we’ll touch on today• Sitemaps• URL Submissions• Rich Media• Down-level

experiences• URL Structures• Navigation• Robots.txt• Popularity• Redirects &

Canonical• Mobile

Your data, always fully provided.

Where to Start?


Websites would do better in general if they followed these simple steps: • Register with the Bing Webmaster Tools.• Make your site easy to crawl (no crazy JavaScript, no

convoluted redirection scheme).• Have quality content.

When they register to the webmaster tools, they put a seed in our system, which will develop organically if their site is easy to crawl.

And if they have quality content (i.e. not a scrape of Amazon results with referral links, as we’ve seen recently from a webmaster who contacted support because his “website” wasn’t indexed), they will be selected and ranked accordingly.

Sitemaps

• XML• OK to zip these• Limited to ~50K entries per file• Maintain to keep them clean• Remove 404 and redirects URLs

• HTML• Great for humans and crawlers• Does not replace an XML file• Perfect for smaller sites

• TXT• Keep the list clean and current

• RSS as a sitemap• Great way to manage sharing content with us


URL Submissions

• Be sure your content is excellent

• Bypasses usual crawl controls and hits the index directly

• Sinks or swims based on searchers interacting with it – this is why quality matters so much

• Limited to deter spammers – 50/month at 10/day

• If it fails here, it’ll still get indexed via crawling and have a chance to rank


Rich Media Issues

• Don’t bury links inside rich media

• If we can’t crawl it, we can’t index it

• Crawlers don’t have rich media enabled

• Avoid flash-based sites which reside on a single URL

• Optimize any technology you do use to manage PLT issues


Down Level Experience

• What do people see when they have rich media turned off?

• Your down level experience should contain all the same content & links

• Avoid cloaking – crawlers should see the same content humans see


Cloaking could mean you get the

URL Structure

• Clean and keyword rich if you can

• Avoid excessive parameters

• Avoid excessive length

• Limit the depth of layers

• Content should be easy and quick to reach


Navigation

• Be sure to link to the content you want indexed

• Do UX testing to see what works for your visitors

• Be logical - not everything needs to be up front

• Don’t offer too many choices

• Keep your depth shallow


Robots.txt

• Most small sites can get by with us just crawling

• Larger sites should use this tool, but be careful

• Recently saw 50% of the sites we met with had an incorrect robots.txt

• Powerful, so be careful – easy to block

• Watch your crawl delays and set short times - we prefer 0 as a crawl delay

• Use Webmaster Tools Crawl Control to manage Bingbot


You want me to do WHAT?!

Popularity as a Driver

• Want to get found and indexed faster? Get popular.

• Popularity whether social or web is a signal

• Write engaging content, encourage social sharing and we’ll see it

• Nothing attracts our attention like people legitimately talking you up


Redirects & Rel-Canonical

• Be sure you understand how to use rel-canonical before implementing

• Limit your redirects to 2 hops – more and you can start losing the crawler’s interest

• Too many hops can slow content visualization and lead visitors to think the site is slow

• Do not use rel-canonical in place of a proper 301 redirect


Going mobile?


• Skip m. domains

• m. domains produce unwanted duplication

• Responsive design is your friend

• HTML5 can help

6 major investment areas for SEO

• Crawlability– Xml sitemaps– Navigational structure– Rich media cautions– Graceful degradation– URL structure– Robots.txt

• Site Structure– Internal Links– URL structure and keyword usage– Clean URLs – no extraneous parameters (sessions,

tracking, etc.)– HTML & XML sitemaps– Content hierarchy– Rel-canonical usage to manage duplication issues– Rich media – don’t bury links in Javascript/flash

/silverlight/AJAX– Responsive design for managing mobile & web

• On-Page– Head copy

• Titles – unique, relevant, 60 characters or so long• Descriptions – unique, relevant, grammatically correct,

160 or fewer characters

– Body Copy• H1, H2 and other H tag usage• ALT tag importance & usage• Keyword usage within the content/text • Anchor text – using target keywords to support other

internal pages

• Content– Build based on keyword research– Down-level experience enhances discoverability– Keep out of rich media and images– Produce new content frequently– Make it unique – don’t reuse content from other

sources– Content management – using 301s to reclaim value

from retiring content/pages– <LINK> canonical to help engines understand which

page should be indexed and have value attributed to it– 404 error page management to help cleanse old pages

from search engine indexes

• Links– Plan for incoming & outgoing link generation– Internal & external link management– Content selection – planning where to link to– Link promotion via social spaces – direct traffic & seo

value– Managing anchor text properly– URL structure can help insert keywords where they are

needed

• Social– Be an authority– Build your community– Interact often– Share useful content– Be consistent and useful– Grow facebook, twitter, pinterest, etc.


Tools should enable doing

Found @ Google WMT Found @ Bing WMT

crawl data

crawl rate configure

sitemap submissions

parameter handling

link data for your site

traffic stats

keyword data

messaging

block URLs

fetch as bot

crawl data

crawl rate configure

sitemap submissions

parameter handling

link data for your sitetraffic stats

keyword data

messaging

block URLs

fetch as bot

organic keyword researchrank stats

index stats

index tracker

canonical alerts

SEO reports

SEO analyzer

link explorer


Organic keyword research tool revisions

Select multiple countries for the same languages

True organic data from Bing Search

Research multiple keywords at once


Fetch As Bingbot


Link explorer

Explore internal and external links

Explore links on your site and other sites


SEO reporting

Runs automatically every other week

Scans all URLs for all sites you have verified


SEO analyzer

On demand SEO scanning tool

More than 15 best practices checked


SEO analyzer

1 2 3


Wrapping up

• Content is the reason• Keyword research is a beacon• Quality is your watch word• Authority is your goal• Niche is your starting point• User Experience is your religion

…and use the free tools provided.

Thanks for your time!Duane Forrester

Sr. Product Manager - Bing

@duaneforrester


Bing Webmaster Tools Crawling Basics Webinar

Documents

Transcript of Bing Webmaster Tools Crawling Basics Webinar