SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid...

147
Knowing ranking factors won’t be enough How to avoid losing your job to a robot @willcritchlow

Transcript of SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid...

Page 1: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Knowing ranking factors won’t be enough

How to avoid losing your job to a robot

@willcritchlow

Page 2: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

I’m going to tell you about a robot that understands ranking factors

better than any of you

...but before I get to that, let’s look at a bit of history...

Page 3: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The other day I searched:

Page 4: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Unsurprisingly, I got an answer

Page 5: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

But it got me thinking about how, in 2009, the results would have looked more like this.

Page 6: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

In 2009, it would have looked more like this.

With every title containing the keyphrase.

Page 7: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

In 2009, it would have looked more like this.

With every title containing the keyphrase.

Most at the beginning.

Page 8: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

OK. Maybe wikipedia would have been #1.

Page 9: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

We used to have a pretty good understanding of ranking factors

Page 10: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

My mental model for c. 2009 ranking factors had three different modes:

Page 11: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

My mental model for ~2009 ranking factors had three different modes:

One in the hyper-competitive

head

One in the

competitive

mid-tail

...and o

ne in th

e

long-t

ail

Page 12: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

One in the hyper-competitive

head

Page 13: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Tons of perfectly on-topic pages to choose from

One in the hyper-competitive

head

Page 14: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

So pick only perfectly-on-topic pages

One in the hyper-competitive

head

Page 15: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

...and rank by authority (*)

(*) Page authority, but the domain inevitably factors into that calculation. This is why

so many homepages ranked

One in the hyper-competitive

head

Page 16: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

This resulted in a mix of homepages of mid-size sites, and inner pages on huge sites

One in the hyper-competitive

head

Page 17: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

But the general way to move up was through increased authority

One in the hyper-competitive

head

Page 18: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Kind of search result

Pages ranking To move up...

Head Homepages of mid-size sites and inner pages of massive sites. All perfectly-targeted.

Improve authority.

Mid-tail

Long-tail

Page 19: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

One in the hyper-competitive

head

One in the

competitive

mid-tail

Page 20: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Wealth of ROUGHLY on-topic pages to choose from

One in the

competitive

mid-tail

Page 21: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

PERFECTLY on-topic could do well even on a relatively weak site

One in the

competitive

mid-tail

Page 22: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Rank the roughly on-topic pages by

authority x “on-topicness”

One in the

competitive

mid-tail

Page 23: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Move up with better targeting or more authority

One in the

competitive

mid-tail

Page 24: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Kind of search result

Pages ranking To move up...

Head Homepages of mid-size sites and inner pages of massive sites. All perfectly-targeted.

Improve authority.

Mid-tail Perfectly on-topic pages on relatively weak sites plus roughly on-topic on bigger sites.

Improve targeting or authority.

Long-tail

Page 25: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

One in the

competitive

mid-tail

One in the hyper-competitive

head

...and o

ne in th

e

long-t

ail

Page 26: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

In the long-tail, a site of arbitrary weakness could rank if it was the most relevant

...and o

ne in th

e

long-t

ail

Page 27: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Otherwise, massive sites rank with off-topic pages that mention something similar

...and o

ne in th

e

long-t

ail

Page 28: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Generally, move up with better targeting

...and o

ne in th

e

long-t

ail

Page 29: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Kind of search result

Pages ranking To move up...

Head Homepages of mid-size sites and inner pages of massive sites. All perfectly-targeted.

Improve authority.

Mid-tail Perfectly on-topic pages on relatively weak sites plus roughly on-topic on bigger sites.

Improve targeting or authority.

Long-tail Arbitrarily-weak on-topic pages and roughly-targeted deep pages on massive sites.

Improve targeting.

Page 30: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Kind of search result

Pages ranking To move up...

Head Homepages of mid-size sites and inner pages of massive sites. All perfectly-targeted.

Improve authority.

Mid-tail Perfectly on-topic pages on relatively weak sites plus roughly on-topic on bigger sites.

Improve targeting or authority.

Long-tail Arbitrarily-weak on-topic pages and roughly-targeted deep pages on massive sites.

Improve targeting.

So that was

~2009

Page 31: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

It’s not so simple any more.Google is harder to understand these days.

Page 32: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

PageRank(the first algorithm to use the link structure

of the web)

We know how we got to ~2009...

Page 33: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Information retrieval

PageRank

Page 34: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Information retrieval

PageRankOriginal research

Page 35: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Information retrieval

PageRankOriginal research

TWEAKS

...with growing complexity in subsequent years

Page 36: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

When Amit left Google, there was a fascinating thread on Hacker News in discussion of this article

Page 37: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Particularly this comment from a user called Kevin Lacker (@lacker):

Page 38: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

I was thinking about it like it was a math puzzle and if I just thought

really hard it would all make sense.

-- Kevin Lacker (@lacker)

Page 39: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Hey why don't you take the square root?

-- Amit Singhal according to Kevin Lacker (@lacker)

Page 40: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

oh... am I allowed to write code that doesn't make any sense?

-- Kevin Lacker (@lacker)

Page 41: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Multiply by 2 if it helps, add 5, whatever, just make things work and we can make it make sense

later.

-- Amit Singhal according to Kevin Lacker (@lacker)

Page 42: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Why does this make the algorithm so hard to understand?

Page 43: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

High-dimension

Non-linear

Discontinuous

3 big reasons:

Page 44: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

High-dimension

Non-linear

Discontinuous

Page 45: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

High-dimension

Non-linear

Discontinuous

Page 46: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

High-dimension

Non-linear

Discontinuous

Page 47: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

You might know what any one of the levers does, but they can

interact with each other in complex ways

This is what a high-dimensional function looks like

Page 48: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

High-dimension

Non-linear

Discontinuous

Page 49: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

We sell custom cigar humidors. Our custom cigar humidors are handmade. If you’re thinking of buying a custom cigar

humidor, please contact our custom cigar humidor specialists at

[email protected]

What this needs is another mention of [cigar humidors]

Page 50: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

With no mentions of [cigar] or [humidor] this page would be unlikely to rank

And yet you can clearly go too far, and have the effect turn negative.

This is called nonlinearity.

The cigar example is taken directly from Google’s quality guidelines.

Page 51: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

High-dimension

Non-linear

Discontinuous

Page 52: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Discontinuities are steps in the function

Think about so-called “over-optimization” tipping points

Page 53: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s put all this togetherinto a practical example:

Page 54: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Think about category pages:Do you recommend removing “SEO text”?

We’ve tested it, so we know the answer.

Page 55: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

If you said “yes”, congratulations(+3.1% organic sessions in a split-test)

Page 56: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Unless you’re responsible for this siteNo effect / possible negative effect

Page 57: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

No, but I’m still pretty good at this

You’re thinking this to yourself right now.

Page 58: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 59: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 60: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 61: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 62: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 63: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 64: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 65: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 66: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 67: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 68: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 69: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 70: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 71: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 72: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 73: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 74: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

I promised to tell you about a robot that is better than even

experienced SEOs...

Well. It turns out all we needed was a coin to flip. You’re all fired.

Page 75: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 76: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 77: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

It’s only going to get worse under Sundar Pichai

Page 78: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Who knows who this is?(This is the only CC-licensed photo of him on the internet)

Page 79: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

ENHANCEWhat about now?

Page 81: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

...and of course Jeff Dean is doing Jeff Dean things(c.f. Chuck Norris)

Page 82: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Jeff Dean puts his pants on one leg at a time, but if he had more legs,

you would see that his approach is O(log n).

Source: Jeff Dean facts

Page 83: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Once, in early 2002, when the search back-ends went down, Jeff

Dean answered user queries manually for two hours.

Result quality improved markedly during this time

Page 84: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

When Jeff Dean goes on vacation, production services across Google mysteriously stop working within a

few days.

This was reportedly actually true

Page 85: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The original Google Translate was the result of the work of hundreds of engineers over 10 years.

Page 86: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Director of Translate, Macduff Hughes said that it sounded to him as if maybe they could pull off a neural-network-based replacement in three years.

Page 87: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Jeff Dean said “we can do it by the end of the year, if we put our minds to it”.

Page 88: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Hughes: “I’m not going to be the one to say Jeff Dean can’t deliver speed.”

Page 89: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

A month later, the work of a team of 3 engineers was tested against the existing system. The improvement was roughly equivalent to the improvement of the old system over the previous 10 years.

Page 90: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Hughes sent his team an email. All projects on the old system were to be suspended immediately.

[Read the whole story ]

Page 92: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

How to avoid losing your job to a robot

This is what you promised, Will.

Page 93: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s start by understanding

some robot weaknesses

Page 94: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

What’s this?

Page 95: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Ooh. Ooh.

I know this one.

-- robot

Page 96: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

“It’s a leopard. I’m like 99% sure.”

Page 97: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Computers are better than humans at classification, but struggle with adversaries

Read more about this here -- Cheetah, Leopard, Jaguar

Page 98: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Lesson:

We expect adversarial abilities to take a step backwards

They will remain good at classifying bad links but will be likely to fall

prey to weird outcomes in adversarial situations

Page 99: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Example:

Remember Tay, the Microsoft chatbot that Twitter taught to be

racist and sexist in less than a day?

Read more here

Page 100: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

We’re going to see new kinds of bugs

Page 101: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Rules of ML [PDF] outlines engineering lessons from getting ML into production at Google

Page 102: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Example lesson: There will be silent failures

“This is a problem that occurs more for machine learning systems than for other

kinds of systems. Suppose that a particular table that is being joined is no longer

being updated. The machine learning system will adjust, and behavior will

continue to be reasonably good, decaying gradually. Sometimes tables are found

that were months out of date, and a simple refresh improved performance more

than any other launch that quarter! For example, the coverage of a feature may

change due to implementation changes: for example a feature column could be

populated in 90% of the examples, and suddenly drop to 60% of the examples.

Play once had a table that was stale for 6 months, and refreshing the table alone

gave a boost of 2% in install rate. If you track statistics of the data, as well as

manually inspect the data on occassion, you can reduce these kinds of failures.”

Page 103: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Example lesson: There will be silent failures

“This is a problem that occurs more for machine learning systems than for other

kinds of systems. Suppose that a particular table that is being joined is no longer

being updated. The machine learning system will adjust, and behavior will

continue to be reasonably good, decaying gradually. Sometimes tables are found

that were months out of date, and a simple refresh improved performance more

than any other launch that quarter! For example, the coverage of a feature may

change due to implementation changes: for example a feature column could be

populated in 90% of the examples, and suddenly drop to 60% of the examples.

Play once had a table that was stale for 6 months, and refreshing the table alone

gave a boost of 2% in install rate. If you track statistics of the data, as well as

manually inspect the data on occassion, you can reduce these kinds of failures.”

Page 104: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Example lesson: There will be silent failures

“This is a problem that occurs more for machine learning systems than for other

kinds of systems. Suppose that a particular table that is being joined is no longer

being updated. The machine learning system will adjust, and behavior will

continue to be reasonably good, decaying gradually. Sometimes tables are found

that were months out of date, and a simple refresh improved performance more

than any other launch that quarter! For example, the coverage of a feature may

change due to implementation changes: for example a feature column could be

populated in 90% of the examples, and suddenly drop to 60% of the examples.

Play once had a table that was stale for 6 months, and refreshing the table alone

gave a boost of 2% in install rate. If you track statistics of the data, as well as

manually inspect the data on occassion, you can reduce these kinds of failures.”

Page 105: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

That document also has a section on trying to understand what the machines are doing

Page 106: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

But human explainability may not even be possible

Not every concept a neural network uses fits neatly into a concept for

which we have a word. It’s not clear this is a weakness per se, but...

Page 107: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

...this means that engineers won’t always know more than we do

about why a page does or doesn’t rank

The big knowledge gap of the future is data - clickthrough rates,

bounce rates etc.

Page 108: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

As Tom Capper said, engineers’ statements can already be misleading

Page 109: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

...and remember the confounding split-testsIt’s already not always as simple as “feature X is good”

Which all means we may need to be more independent-minded and do more of our own research

Page 110: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

So how do we fight back?

Page 111: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Michael Lewis’ latest book is about Kahneman and Tversky spelling.

It recounts a story about a piece of medical software that existed in the 1960s.

Page 112: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

It was designed to encapsulate how a range of doctors diagnosed stomach cancer from x-rays.

Page 113: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

It proceeded to outperform those same doctors despite only containing their expertise.

Real people have biases, and fool themselves.

Encapsulate your own expert knowledge.

Page 114: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

At Distilled, we use a methodology we call the balanced digital scorecard.

This encapsulates our beliefs about how to build a high-performing business.

Applying it helps avoid our own biases.

Page 115: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Also, while we are talking about books, The Checklist Manifesto is an important part of avoiding the same cognitive biases.

Page 116: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Focus on consulting skills

I’ve written a few things about this (DistilledU module, writing better business documents, using split-tests to consult better).

Use case studies and creativity. Computers are better at diagnosis than cure.

This means: getting things done, convincing organizations, applying general knowledge, learning new things.

Page 117: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

We are going to need to be better than ever at debugging things.

I wrote about debugging skills for non-developers here.

A lot of the story of enterprise consulting is going to be about figuring out why things have gone wrong in the face of sparse or incorrect information from Google.

Page 118: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 119: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Disregard expert surveys

Firstly, there are all the problems outlined in the search result pairs study - both in the ability of experts to understand factors, and in your ability to use the information even if they do.

Secondly, they are broken with another bias called the “law of small numbers” from Lewis’ book.

PS - I say this as a participant in many of them

Me

Page 121: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

This is why we have been investing so much in split-testing

Check out www.distilledodn.com if you haven’t already.

The team will be happy to demo for you.

We’re now serving ~1.5 billion requests / month, and recently published information covering everything from response times to our +£100k / month split test.

Page 122: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

Page 123: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

Page 124: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

Page 125: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

4. We remain relevant by:

a. Using methodologies and checklists to capture human capabilities and

avoid our biases

Page 126: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

4. We remain relevant by:

a. Using methodologies and checklists to capture human capabilities and

avoid our biases

b. Becoming great consultants and change agents

Page 127: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

4. We remain relevant by:

a. Using methodologies and checklists to capture human capabilities and

avoid our biases

b. Becoming great consultants and change agents

c. Debugging the heck out of everything

Page 128: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

4. We remain relevant by:

a. Using methodologies and checklists to capture human capabilities and

avoid our biases

b. Becoming great consultants and change agents

c. Debugging the heck out of everything

d. Avoiding being misled by experts or Google

Page 129: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

4. We remain relevant by:

a. Using methodologies and checklists to capture human capabilities and

avoid our biases

b. Becoming great consultants and change agents

c. Debugging the heck out of everything

d. Avoiding being misled by experts or Google

e. Testing!

Page 130: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Oh, and one more thing

Page 131: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

What about that robot I promised you?

The coin flip wasn’t really it

Page 132: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 133: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 134: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

keras.io

Page 135: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The specifics of DeepRank

Gather and process

training data

We started with a broad range of unbranded keywords from our STAT rank tracking.

For each of the URLs ranking in the top 10, we gathered key metrics about the domain and page - both from direct crawling and various APIs.

We turned this into a set of pairs of URLs {A,B} with their associated keyword, metrics, and their rank ordering.

Page 136: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The specifics of DeepRank

Gather and process

training data

We started with a broad range of unbranded keywords from our STAT rank tracking.

For each of the URLs ranking in the top 10, we gathered key metrics about the domain and page - both from direct crawling and various APIs.

We turned this into a set of pairs of URLs {A,B} with their associated keyword, metrics, and their rank ordering.

Page 137: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The specifics of DeepRank

We have so far trained on just 10 metrics for a relatively small sample (hundreds) of keywords.

Our current version is only a few layers deep with only 10 hidden dimensions.

The current training samples 30 pairs at a time and trains against them for 500 epochs.

Train the model

Gather and process

training data

Page 138: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The specifics of DeepRank

The next task is to get way more metrics for thousands of keywords.

This will enable us to train a much deeper model for much longer without overfitting.

We also have some more hyperparameter tuning to do,

Model

Train the model

Gather and process

training data

Page 139: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

To run the model, we input a pair of pages with their associated metrics.

New input

Page 140: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Model

New input

Page 141: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

We get back a probability of page A outranking page B.

Model

Probability-weighted

predictions

New input

Page 142: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 143: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot
Page 144: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

The goal is a winning combination of human and machine

Human + computer beats computer (for now)

Page 145: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Let’s recap

1. Even in a world of 200+ “classical” ranking factors, humans were bad at

understanding the algorithm

2. Machine learning will make this worse, and is accelerating under Sundar

3. There are things computers remain bad at, and rankings will become more

opaque even to Google engineers

4. We remain relevant by:

a. Using methodologies and checklists to capture human capabilities and

avoid our biases

b. Becoming great consultants and change agents

c. Debugging the heck out of everything

d. Avoiding being misled by experts or Google

e. Testing!

5. Human + robot is the only thing that has a chance of beating the robots

Page 146: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Questions: @willcritchlow

Page 147: SearchLove San Diego 2017 | Will Critchlow | Knowing Ranking Factors Won't Be Enough: How To Avoid Losing Your Job to a Robot

Image credits

● Mobius strip

● Confusion

● Signal box

● Cigar

● Discontinuity

● Confidence

● Burt Totaro

● Sundar Pichai

● John Giannandrea

● Chuck Norris

● Jeff Dean

● Fencing

● Keyboard

● Go

● Robot

● Leopard print sofa

● Leopard

● Bug

● Lego robots

● Iron Man

● San Diego