Bartosz GóralewiczELEPHATE

Can Google properly crawl and index JavaScript?SEO Experiments - Results and findings@bart_goralewiczhttps://www.linkedin.com/in/goralewicz/

Why is JavaScript SEO such a complex topic?

DATANO

Why is JavaScript SEO such a complex topic?

DATANO

Making JavaScript SEO easy

JS SEO

JS SEO

What is the problem with JavaScript?HTML

JS

What is the problem with JavaScript?HTML

HTML

What is the problem with JavaScript?HTML

HTML

JavaScript

and JavaScript

JavaScript & SEO

50 shades of JavaScript

*more about it later

not as “forgiving” as HTML*

It has all with…

Reverse engineering algorithms is no longer an

option.

Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers. To reflect this improvement, we recently updated our technical Webmaster Guidelines to recommend against disallowing Googlebot from crawling your site's CSS or JS files.

ele.ph/ajax

ele.ph/scotch

Can this backfire?

Hulu.com JavaScript & SEO backfire

-37%

ele.ph/hulu

Hulu case study

Google and JavaScript – difficult relationship

The only solution…

Java Script Experiment

KamilGrymuza

http://jsseo.expert

Content!

OK.

ele.ph/AI

Inspect Element

vs.

View Source

Inspect element

View source

First JavaScript rendering issues

Timeout, GSC bug or real ?

/test/

JavaScript links were not parsed by Googlebot

Google kept us

waiting…

Fail?

Re-fetched all URLs via GSC

/html/test/ (duh!)

/plain-inline/test//jquery-inline/test//jquery-ajax/test//vue/test//react-inline/test/

And… that’s it! /test/

Let’s start with JS frameworks created by

Version 1

Checklist

Check if ”framework/test/” URL was crawled

Search for unique content from framework’s page

Is URL indexed by Google?

Fetch and render test (GSC)

Fetch and render test (GSC)

Is URL indexed by Google?

Search for unique content from framework’s page

Check if ”framework/test/” URL was crawled

V2 Inline

GSCfetch

URLIndexed?ContentIndexed?

TestURL

Inline Inline

GSCfetch

URLIndexed?

ContentIndexed?

TestURL

External Ajax callInline

GSCfetch

URLIndexed?

ContentIndexed?

TestURL

Inline External

GSCfetch

URLIndexed?ContentIndexed?

TestURL

Inline External

Inline vs. External JS makes a huge difference

Not all JavaScript frameworks are crawled and indexed in the same way

Mindblowing findings

JS generated links aren’t always crawled

ele.ph/angularU

Also…

Regardless of what your developers say, Angular JS 2 always has to be server rendered…

according toAngular JS 2 community.

If you search for any competitive keyword terms, it’s always going to be server rendered sites. And the reason is because although Google does index client-siderendered HTML, it’s not perfect yet and other search engines don’t do it as well. So if you care about SEO, you still need to have server-rendered content.

Jeff Whelpley

Angular U conference, June 22-25, 2015, Hyatt Regency, San Francisco Airport“Angular 2 Server Rendering”

ele.ph/angularU

?How about Search Engines other than

63,4%

22,8%

11,7%

1,3% 0,8%

Google Bing Yahoo! Ask AOL

Search Engine market share

Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/

83,5%

11,1%3,8% 0,8% 0,6%

Google Bing Yahoo! MSN DuckDuckGo

Search Engine market share

Source: https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/

ele.ph/beyondgoogle

ele.ph/jsforum

Tell me why I can’t…

Clear your mind of

questions

*not the actual conversation

GSCfetch

URLIndexed?

ContentIndexed?

TestURL

External Ajax callInline

http://jsseo.expert/plain-external/test/

http://jsseo.expert/react-external/test/

http://jsseo.expert/jquery-external/test/

we found this URL from a link ca 3 weeks ago, but we haven't crawled it

we picked this one up ~19 days ago, and similar to above, our systems just didn't feel like crawling it yet

picked up ~21 days ago, also not crawled yet.

Thanks, John!

You’re the best,

Bartosz!*

*he didn’t say that

John Mueller’s feedback got me thinking…

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sitamet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuadafames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusceposuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna erosquis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proinpharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinarultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbitristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverraimperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummypede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectusmalesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus etnetus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitorcongue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusceest. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Loremipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit ametcommodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames acturpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere,magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna erosquis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proinpharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinarultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbitristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverraimperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummypede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectusmalesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus etnetus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitorcongue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusceest. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci.

JavaScript vs. crawler budgetSome JavaScript frameworks are indexable, but there is a fine print…

JavaScript

BUT…

HTML

Do you already know where I am going with this?

KamilGrymuza

Crawler budget experiment (early beta)

We repeated this experiment a few times and the results were always the same.

JavaScript, even when indexable,

can kill your crawler budget…

JS is less effective than

HTML

JavaScript is not bad for SEO, if

done right**he said exactly that

Ilya Grigorik

Understanding technology behind

ele.ph/ilya

ele.ph/chrome41

ele.ph/chrome41

Download Chrome 41 here:

Using Chrome 41for debugging JS indexing

http://jsseo.expert/angular2/in Chrome 41

http://jsseo.expert/angular2/in up-to-date Chrome

JavaScript SEO isn’t only affecting technical

At the same times those links count just as HTML ones.

Most link indexes(e.g. Majestic, MOZ etc.)

can’t crawl JavaScript generated backlinks.

Also…...there is more JavaScript generated links out therethan I would imagine

(I’ll tell you more about it in a second)

Problems?Manual penalties

Tracking mentions

Google Penguin

Link audits

Link building

Crawling JavaScript is expensive…

To execute JS for every page at our scale [would] require 10,000-15,000 servers, and we believe our customers are not ready to pay for that yet”

Dmitry Gerasimenko, CEO of Ahrefs

~ July 2017

No hope?

Ahrefs guys changed their mind after reading my article…

ele.ph/js-ahrefs

ele.ph/js-ahrefs

…We accepted the challenge and crawled this same website with our JS crawler, which rendered all of the pages that were there.

ele.ph/js-ahrefs

ele.ph/js-ahrefs

ele.ph/js-ahrefs

So earlier thisyear Dmitry (our Founder & CEO) decided that Ahrefs has to be the first among its rivals to crawl JavaScript links.Which we achieved with ease last month.

JS

ele.ph/js-ahrefs

JSele.ph/js-ahrefs

78762291574612765419674516754624524615796412743254612794214512161504918805916076096067461350967619097645684795464436557433443675698347856654756434587655436465543787695487879686564565445645321254332156454665567626156552446759887999808798764242233140031324256543426987547367556456544654745876983747876229 43225461279421451216150491880591607609606746135096761909764568479546443655743344367569834785665475643458765543646554378769548787968656456544564532125433215645466556762615655244675988799980879876424223314003132425654342698754736755645654465474587698374787622915746127654196745167546245246157964127432546

So how many JS links are out there?

JS LINKS PER DAY!

Is there a perfect JavaScript SEO scenario?

YES

The setup for that is called Isomorphic JavaScript on React & Universal on Angular. That's what I'd recommend doing in general, if you can.

Isomorphic JavaScript

ele.ph/Iso-John

ele.ph/iso

What if it’s too late?

Prerender.io

userserver

middleware

Prerenderingcrawlers

JS

HTML

JS

??

Experiments continue…

JS

Only Google and Askcan handle JavaScript

Key takeways

JS

Only Google and Askcan handle JavaScript

Not all JavaScript frameworks are indexable

Key takeways

Not all JavaScript frameworks are indexable

Indexing JavaScript is only half the battle

Key takeways

Indexing JavaScript is only half the battle

JavaScript even whenrendered and indexed,can kill crawler budget

Key takeways

JavaScript even whenrendered and indexed,can kill crawler budget

Use Chrome 41to debug JS

Key takeways

Use Chrome 41to debug JS

If too late,use prerendering

Key takeways

If too late,use prerendering

Key takeways

Isomorphic JS=

perfect scenario

Not all JavaScript frameworks are indexable

Indexing JavaScript is only half the battle

JavaScript even whenrendered and indexed,can kill crawler budget

Use Chrome 41to debug JS

JS

Only Google and Askcan handle JavaScript

If too late,use prerendering

Key takeways

Isomorphic JS=

perfect scenario

@bart_goralewicz

/in/goralewicz/

Thanks! :*