Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings
132
Bartosz Góralewicz ELEPHATE Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings @bart_goralewicz https://www.linkedin.com/in/goralewicz/
Transcript of Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings
Bartosz GóralewiczELEPHATE
Can Google properly crawl and index JavaScript?SEO Experiments - Results and findings@bart_goralewiczhttps://www.linkedin.com/in/goralewicz/
Why is JavaScript SEO such a complex topic?
DATANO
Why is JavaScript SEO such a complex topic?
DATANO
Making JavaScript SEO easy
JS SEO
JS SEO
What is the problem with JavaScript?HTML
JS
What is the problem with JavaScript?HTML
HTML
What is the problem with JavaScript?HTML
HTML
JavaScript
and JavaScript
JavaScript & SEO
50 shades of JavaScript
*more about it later
not as “forgiving” as HTML*
It has all with…
Reverse engineering algorithms is no longer an
option.
Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers. To reflect this improvement, we recently updated our technical Webmaster Guidelines to recommend against disallowing Googlebot from crawling your site's CSS or JS files.
ele.ph/ajax
ele.ph/scotch
Can this backfire?
Hulu.com JavaScript & SEO backfire
-37%
ele.ph/hulu
Hulu case study
Google and JavaScript – difficult relationship
The only solution…
Java Script Experiment
KamilGrymuza
http://jsseo.expert
Content!
OK.
ele.ph/AI
Inspect Element
vs.
View Source
Inspect element
View source
First JavaScript rendering issues
Timeout, GSC bug or real ?
/test/
JavaScript links were not parsed by Googlebot
Google kept us
waiting…
Fail?
Re-fetched all URLs via GSC
/html/test/ (duh!)
/plain-inline/test//jquery-inline/test//jquery-ajax/test//vue/test//react-inline/test/
And… that’s it! /test/
Let’s start with JS frameworks created by
Version 1
Checklist
Check if ”framework/test/” URL was crawled
Search for unique content from framework’s page
Is URL indexed by Google?
Fetch and render test (GSC)
Fetch and render test (GSC)
Is URL indexed by Google?
Search for unique content from framework’s page
Check if ”framework/test/” URL was crawled
V2 Inline
GSCfetch
URLIndexed?ContentIndexed?
TestURL
Inline Inline
GSCfetch
URLIndexed?
ContentIndexed?
TestURL
External Ajax callInline
GSCfetch
URLIndexed?
ContentIndexed?
TestURL
Inline External
GSCfetch
URLIndexed?ContentIndexed?
TestURL
Inline External
Inline vs. External JS makes a huge difference
Not all JavaScript frameworks are crawled and indexed in the same way
Mindblowing findings
JS generated links aren’t always crawled
ele.ph/angularU
Also…
Regardless of what your developers say, Angular JS 2 always has to be server rendered…
according toAngular JS 2 community.
If you search for any competitive keyword terms, it’s always going to be server rendered sites. And the reason is because although Google does index client-siderendered HTML, it’s not perfect yet and other search engines don’t do it as well. So if you care about SEO, you still need to have server-rendered content.
Jeff Whelpley
Angular U conference, June 22-25, 2015, Hyatt Regency, San Francisco Airport“Angular 2 Server Rendering”
ele.ph/angularU
?How about Search Engines other than
63,4%
22,8%
11,7%
1,3% 0,8%
Google Bing Yahoo! Ask AOL
Search Engine market share
Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/
83,5%
11,1%3,8% 0,8% 0,6%
Google Bing Yahoo! MSN DuckDuckGo
Search Engine market share
Source: https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/
ele.ph/beyondgoogle
ele.ph/jsforum
Tell me why I can’t…
Clear your mind of
questions
*not the actual conversation
GSCfetch
URLIndexed?
ContentIndexed?
TestURL
External Ajax callInline
http://jsseo.expert/plain-external/test/
http://jsseo.expert/react-external/test/
http://jsseo.expert/jquery-external/test/
we found this URL from a link ca 3 weeks ago, but we haven't crawled it
we picked this one up ~19 days ago, and similar to above, our systems just didn't feel like crawling it yet
picked up ~21 days ago, also not crawled yet.
Thanks, John!
You’re the best,
Bartosz!*
*he didn’t say that
John Mueller’s feedback got me thinking…
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sitamet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuadafames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusceposuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna erosquis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proinpharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinarultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbitristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverraimperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummypede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectusmalesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus etnetus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitorcongue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusceest. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Loremipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit ametcommodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames acturpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere,magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna erosquis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proinpharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinarultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbitristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverraimperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummypede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectusmalesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus etnetus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitorcongue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusceest. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci.
JavaScript vs. crawler budgetSome JavaScript frameworks are indexable, but there is a fine print…
JavaScript
BUT…
HTML
Do you already know where I am going with this?
KamilGrymuza
Crawler budget experiment (early beta)
We repeated this experiment a few times and the results were always the same.
JavaScript, even when indexable,
can kill your crawler budget…
JS is less effective than
HTML
JavaScript is not bad for SEO, if
done right**he said exactly that
Ilya Grigorik
Understanding technology behind
ele.ph/ilya
ele.ph/chrome41
ele.ph/chrome41
Download Chrome 41 here:
Using Chrome 41for debugging JS indexing
JavaScript SEO isn’t only affecting technical
At the same times those links count just as HTML ones.
Most link indexes(e.g. Majestic, MOZ etc.)
can’t crawl JavaScript generated backlinks.
Also…...there is more JavaScript generated links out therethan I would imagine
(I’ll tell you more about it in a second)
Problems?Manual penalties
Tracking mentions
Google Penguin
Link audits
Link building
Crawling JavaScript is expensive…
To execute JS for every page at our scale [would] require 10,000-15,000 servers, and we believe our customers are not ready to pay for that yet”
Dmitry Gerasimenko, CEO of Ahrefs
~ July 2017
No hope?
Ahrefs guys changed their mind after reading my article…
ele.ph/js-ahrefs
ele.ph/js-ahrefs
…We accepted the challenge and crawled this same website with our JS crawler, which rendered all of the pages that were there.
ele.ph/js-ahrefs
ele.ph/js-ahrefs
ele.ph/js-ahrefs
So earlier thisyear Dmitry (our Founder & CEO) decided that Ahrefs has to be the first among its rivals to crawl JavaScript links.Which we achieved with ease last month.
JS
ele.ph/js-ahrefs
JSele.ph/js-ahrefs
78762291574612765419674516754624524615796412743254612794214512161504918805916076096067461350967619097645684795464436557433443675698347856654756434587655436465543787695487879686564565445645321254332156454665567626156552446759887999808798764242233140031324256543426987547367556456544654745876983747876229 43225461279421451216150491880591607609606746135096761909764568479546443655743344367569834785665475643458765543646554378769548787968656456544564532125433215645466556762615655244675988799980879876424223314003132425654342698754736755645654465474587698374787622915746127654196745167546245246157964127432546
So how many JS links are out there?
JS LINKS PER DAY!
Is there a perfect JavaScript SEO scenario?
YES
The setup for that is called Isomorphic JavaScript on React & Universal on Angular. That's what I'd recommend doing in general, if you can.
Isomorphic JavaScript
ele.ph/Iso-John
ele.ph/iso
What if it’s too late?
Prerender.io
userserver
middleware
Prerenderingcrawlers
JS
HTML
JS
??
Experiments continue…
JS
Only Google and Askcan handle JavaScript
Key takeways
JS
Only Google and Askcan handle JavaScript
Not all JavaScript frameworks are indexable
Key takeways
Not all JavaScript frameworks are indexable
Indexing JavaScript is only half the battle
Key takeways
Indexing JavaScript is only half the battle
JavaScript even whenrendered and indexed,can kill crawler budget
Key takeways
JavaScript even whenrendered and indexed,can kill crawler budget
Use Chrome 41to debug JS
Key takeways
Use Chrome 41to debug JS
If too late,use prerendering
Key takeways
If too late,use prerendering
Key takeways
Isomorphic JS=
perfect scenario
Not all JavaScript frameworks are indexable
Indexing JavaScript is only half the battle
JavaScript even whenrendered and indexed,can kill crawler budget
Use Chrome 41to debug JS
JS
Only Google and Askcan handle JavaScript
If too late,use prerendering
Key takeways
Isomorphic JS=
perfect scenario
@bart_goralewicz
/in/goralewicz/
Thanks! :*
