SearchLove Boston 2016 | Paul Shapiro | How to Automate Your Keyword Research
Transcript of SearchLove Boston 2016 | Paul Shapiro | How to Automate Your Keyword Research
HOW TO AUTOMATE YOUR KEYWORD RESEARCH
Paul Shapiro – Organic Search Director at CatalystSearchLove Boston – May 4th,
2016
@fighto
2
Without Automation
@fighto
3
With Automation
@fighto
WHO AM I?
4
@fighto
5
Tool Data Sources
Visualization
+ +
@fighto
WHAT YOU WILL LEARN
• The problems with the standard keyword research process• How you can save time on keyword research with automation• How to automate keyword research with a powerful, but easy-to-use
tool• How APIs work and how to leverage them• What data sources are useful for keyword research automation• How to use data visualization to make keyword research decisions
6
MANUAL KEYWORD RESEARCH IS
OFTEN BLAND &TIME CONSUMING
7
@fighto
8
@fighto
9
Most Keyword Research Looks Like This
@fighto
10
That’s just a Google
Keyword Planner Export
@fighto
TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH
Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages)
Large (250-499 pages) Extra Large (>500 pages)
0
5
10
15
20
25
30
58
12
21 21
6
10
18
22
28
Hours to Complete Keyword Research by Site Size
Average (Low End) Average (High End)
11
@fighto
12
AIN’T NOBODY GOT TIME FOR MANUAL KEYWORD RESEARCH!
@fighto
SIZE OF THE DATA SET
13
vs.
@fighto
14
We can do better!
WE CAN SAVE TIME WITH AUTOMATION
@fighto
TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH
Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages)
Large (250-499 pages) Extra Large (>500 pages)
0
5
10
15
20
25
30
58
12
21 21
6
10
18
22
28
Hours to Complete Keyword Research by Site Size
Average (Low End) Average (High End)
16
@fighto
17
If we do keyword research once per month, we will save around 24 to 45 full days over a 5-year period using automation.
http://xkcd.com/1205/
WE CAN DO BETTER THAN AN EXCEL SPREADSHEET
@fighto
THAT’S MORE LIKE IT!
19h/t @victorpan
@fighto
HOW TO USE FOR KEYWORD SELECTION / TARGETING
20
This top-right quadrant contains keywords with:
• Low competition• Good growth
Larger bubbles show higher search volumes.
You can alternatively use current rank on the x-axis to signal organic market share like a traditional growth-share matrix.
@fighto
21
That’s much better! actionability.
MY KEYWORD RESEARCH AUTOMATION PROCESS
@fighto
23
https://www.knime.orgIt starts with a tool…
@fighto
WHY KNIME?
• Singular tool for data collection, manipulation, analysis, and reporting
• Easy to use and no development skills necessary
• If you are a developer, it provides faster prototyping
• Large library of built-in “nodes”• Free/Open Source• Run on Windows/Mac/Linux• Infinitely expandable
24
THE BASICS OF KNIME
@fighto
WHAT’S A NODE?
• Nodes are prebuilt, drag and drop modules designed perform a singular task• Nodes are strung together like a chain to accomplish larger, more complex tasks• Nodes can be grouped together into “meta-nodes”, which can be configured in unison
26
@fighto
HOW DO YOU ADD NODES & HOW DO THEY CONNECT?
How do you add nodes to your “workflow”?
How do you string nodes together?
27
@fighto
HOW DO YOU CONFIGURE & RUN NODES?
Configuring Nodes
Running Workflows
28
OR
APIs 101
29
@fighto
EASY & APPLICABLE API: GOOGLE AUTOSUGGEST
http://suggestqueries.google.com/complete/search?output=toolbar&hl=en&q=ninja%20turtles
30
Simple API using “GET Request”
This is a variable that changes
@fighto
31
Output = XML
Parse me!
A KNIME-API EXAMPLE
32
@fighto
STEP 1: LIST OF KEYWORDS
33
@fighto
STEP 2: ENCODE SPACES IN KEYWORD PHRASES
34
Space Character
@fighto
STEP 3: CONCATENATE API URL + KEYWORDS
35
@fighto
STEP 4: DOWNLOAD API OUTPUT
36
@fighto
STEP 6: LOOP THROUGH EACH KEYWORD
37
@fighto
STEP 7: PARSE THE XML OUTPUT FORM THE API
38
@fighto
39
Auto-CompleteSuggestions
@fighto
40
@fighto
FYI: I’M NOT A KNIME SALESMAN
41
DATA INPUTS / SOURCES
@fighto
43# of primary data sources to aggregate
@fighto
44 http://searchwilderness.com/gwmt-data-python/
Google Search Analytics – Historical MySQL Database via Python Script
@fighto
45
AdWords - Search Query Report API or CSV
@fighto
46
SEMRush – API (if <100,000) or Domain vs. Domain:
Competitor Keywords (Advanced Mode)
https://www.semrush.com/info/domain_vs_domain
@fighto
47
GrepWords API – For Google Keyword Planner related query suggestions &
search volumes
http://grepwords.com/
@fighto
48
Google Autosuggest API or Emulation with Selenium
@fighto
49
Rank Checking/Full SERP API - GetSTAT
https://getstat.com/
@fighto
50
MarketMuse – Semantic Keyword Recommendations
https://www.marketmuse.com/
Alternative: http://pshapi.ro/SemanticKWR
@fighto
51
Moz API + Moz Nodes
https://tech.knime.org/book/mmi-labs-nodeshttps://moz.com/products/mozscape/access
@fighto
52
Google Trends – Unofficial API
https://github.com/GeneralMills/pytrendsScript soon to follow: http://pshapi.ro/slopetrends
@fighto
53
More data sources:http://pshapi.ro/kwmetrics
THE DATA-TO-KNIME PROCESS
54
@fighto
Filtering + Data
Manipula-tions
ONE BIG KEYWORD LIST
Seed Keywords- List- GWMT- SEMRush Comp. KWs- SQR Keywords
Keyword Planner Suggestions (via
GrepWords)Google
Autocomplete
Semantic Keyword Recommendations (via MarketMuse)
Google Autocomplete
SEMRush Domain vs.
Domain KeywordsGoogle
Autocomplete
55
@fighto
STRING ‘EM ALL TOGETHER AND THEN…
56
@fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic Competition
Search Volume
Keyword Trends
57
@fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic Competition
• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume
Keyword Trends
58
@fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic Competition
• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume
• Use 2 Years of Google Trends data to calculate slope and determine growing/declining keywords
Keyword Trends
59
@fighto
60
@fighto
61
Reusable
5-Minutes
@fighto
62
You can use this data to filter keywords and reduce massive list... OR, you can leverage data visualization!
VISUALIZING KEYWORD RESEARCH
@fighto
64
We have a few
options…
@fighto
OPTION: EXCEL + BUBBLE CHART
65
Not Recommended
@fighto
OPTION: TABLEAU
66
@fighto
OPTION: PYTHON + BOKEH CHARTS IN KNIME
67
@fighto
THEY’RE INTERACTIVE…
68
@fighto
69
Sample Python Code:http://pshapi.ro/SLBokeh
@fighto
SO WHAT…
• Free up time spend doing mundane, repetitive work and spend more doing impactful, strategic work• Free up staffing resources and utilize them in other ways• Extract and analyze a larger pool of data for greater data pool
70
From this… To this…
@fighto
SUMMARY
• You should rethink how your currently do keyword research.• Keyword research is often bland, time consuming, and not as actionable as it can be.• KNIME provides an easy-to-use interface for automating your keyword research• APIs are easier to use than they sound; Use them• There are great data sources you can leverage via APIs/built-in KNIME nodes• Visualize your keywords using a growth-share matrix like-graph to make keyword usage
decisions• Emojis are awesome
71
@fighto
RESOURCES: DATA SOURCES > KNIME
72
Data Source KNIME Node(s)Google Search Analytics Database Table Connector/ReaderAdWords Search Query Report XLS ReaderSEMRush Domain vs. Domain (or API if less than 100,000 keyword)
XLS/CSV Reader or GET Resource & Read REST Representation
Google Keyword Planner/GrepWords GET Resource & Read REST Representation
Google Autosuggest GET Resource & Read REST Representation (or Selenium nodes)
Rank Checking (via GetSTAT) GET Resource & Read REST Representation
Semantic Keywords/MarketMuse POST ResourceMOZ Moz API Connector/URL Metrics nodesGoogle Trends TBD – Either CSV or Python Script nodes
@fighto
ADDITIONAL RESOURCES• KNIME Version to install: “KNIME Analytics Platform + all free extensions”• To install KNIME Add-ons, Choose Help->Install New Software
• Moz nodes• Should come with Nightly Builds. If not can add
http://update.knime.org/community-contributions/3.1 as a repository.• Add-on Selenium nodes
• http://seleniumnodes.com/• Search Engine Land article addressing additional data inputs: http://pshapi.ro/kwmetrics• Information about Growth-share matrices: https://
en.wikipedia.org/wiki/Growth%E2%80%93share_matrix• Sanple Python + Bokeh script for an interactive keyword research visualization in KNIME:
http://pshapi.ro/SLBokeh
73
@fighto
Questions?
74
THANK YOU!@fightohttp://
searchwilderness.comhttp://catalystdigital.com