Yahoo Query Language: Select * from Internet
-
Upload
drgath -
Category
Technology
-
view
4.823 -
download
1
description
Transcript of Yahoo Query Language: Select * from Internet
![Page 1: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/1.jpg)
select * from internetYahoo Query Language:
Derek Gathright
![Page 2: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/2.jpg)
![Page 3: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/3.jpg)
var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
![Page 4: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/4.jpg)
var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
![Page 5: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/5.jpg)
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
YQLYQL stmt
![Page 6: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/6.jpg)
Why YQL?
Thousands of Web Services and sources that provide valuable data
Require Developers to read documentation and form URLs/queries.
Data is isolated
Needs filtering, combining, tweaking, shaping even after it gets to the developer.
![Page 7: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/7.jpg)
What is YQL?
Cloud web service and SQL-Like Language • Familiar to developers• Synonymous with Data access• Expressive enough to get the right data.
Self describing - show, desc table
Allows you to query, filter, join and update data across any structured data on the web / web services
• And Yahoo’s sherpa cloud storage
Real time engine
![Page 8: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/8.jpg)
YQL Statement Basics
show: lists the supported tables
desc: describes the structure of a table
select: fetches data
insert/update/delete: modify data
use: use an Open Data Table
set: define key-values across Open Data Tables
![Page 9: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/9.jpg)
Insert / Update / Delete
With INSERT / UPDATE / DELETE SQL verbs you can modify data on the internet or cloud
• create tweets, add blog entries, update DBs etc• store tables or pieces of code in sherpa
Some tables with I/U/D• Twitter, Bit.ly shortener• Yahoo profile status, application update stream• Amazon SimpleDB
![Page 11: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/11.jpg)
Get any data:
Open Data Tables
![Page 12: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/12.jpg)
Open Data Tables
• http://github.com/yql/yql-tables/tree/master
• datatables.org mirrors the master and builds a single environment with all tables (faster serving)
• Console with community tables
• Brings the power of the table model to new APIs
• Schema defines mapping between YQL and Endpoint
• Keys can either be query, path or matrix parameters
• Keys can be marked as required
• YQL Compiler validates existence of required keys
![Page 13: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/13.jpg)
YQL
Open Data Table example
![Page 14: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/14.jpg)
Open Data Table example
![Page 15: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/15.jpg)
Open Data Table example
![Page 16: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/16.jpg)
ODT <execute> element
Extends Open Data Tables with server-side code
Javascript server-side scripting• No DOM, events etc, pure JS scripting• E4X support - XML literals, filtering ...
YQL no longer performs a GET request to the table binding URL
YQL provides additional useful global objects• request, response, y.rest, y.include, y.query...
Execute tag must return the data that will be the output of the YQL select
Move business logic of your application to the cloud
![Page 17: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/17.jpg)
What you can do with <execute>
access APIs that require authentication• Netflix OAuth, FlickrAuth, Google AuthSub
join data across services• grab New York Times article tags and generate find associated flickr photos• combine multiple searches into a single result (twitter, web, news and images)
augment data:• city/state to APIs that just return zip code
create APIs from web page• celebrity birthdays scraped from imdb, caltrain
data transformation• convert the result from xml to Google's kml format
![Page 18: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/18.jpg)
hello world
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where a=’hello' and b=’world';</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls><url>http://fake.url/{a}</url></urls> <inputs> <key id='a' type='xs:string' paramType='path' required="true" /> <key id='b' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ // Your javascript goes here. We will run it on our servers response.object = <item> <url>{request.url}</url> <a>{a}</a> <b>{b}</b> </item>; ]]></execute> </select> </bindings></table>
![Page 19: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/19.jpg)
Scraping a page
<select produces="XML" itemPath="deals.deal"> <urls><url>http://deals.yahoo.com</url></urls> <execute><![CDATA[var dealhtml = y.query("select * from html where url='http://deals.yahoo.com' and xpath='//div[contains(@class, \"product-display\")]'").results;var deals = <deals/>;function compress(object) { if (!object) return ""; return object.toString().replace(/\s+/, " ")}for each (var div in dealhtml.div) { var deal = <deal/>; deal.name += <name>{compress(div.h3.a.text())}</name>; deal.link += <link>{div.div[0].a.@href}</link>;… deal.description += <description>{compress(div.p[0].*)}</description>; deal.logo += <logo>{compress(div.div[1].*)}</logo>; deals.deal += deal;}response.object = deals; ]]></execute> </select>
![Page 20: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/20.jpg)
<select itemPath="" produces="XML"><urls><url>http://twitter.com/statuses/show/{id}.xml</url></urls><inputs>
<key id="username" type="xs:string" required="false" paramType="variable"/><key id="password" type="xs:string" required="false" paramType="variable"/><key id="id" type="xs:integer" required="true" paramType="path"/>
</inputs><execute><![CDATA[
var r = null; if (username && password) { y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); r = request.header("Authorization",authheader).get().response; } else { r = request.get().response; } response.object = r;
]]></execute></select>
select * from twitter.status where id="2108869549" and username="xxx" and password="yyy”
Twitter Select
![Page 21: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/21.jpg)
<insert itemPath="" produces="XML"><urls>
<url>http://twitter.com/statuses/update.xml</url></urls><inputs>
<value id="username" type="xs:string" required="true" paramType="variable"/><value id="password" type="xs:string" required="true" paramType="variable"/><value id="status" type="xs:string" required="true" paramType="variable"/>
</inputs><execute><![CDATA[
y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); var content = "status="+status; response.object = request.header("Authorization",authheader).post(content).response;
]]></execute></insert>
insert into twitter.status (status,username,password) values ("new tweet from YQL",”xxx”,”yyy”);
Twitter Update
![Page 22: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/22.jpg)
<delete itemPath="" produces="XML"><urls>
<url>http://twitter.com/statuses/destroy/{id}.xml</url></urls><inputs>
<key id="username" type="xs:string" required="true" paramType="variable"/><key id="password" type="xs:string" required="true" paramType="variable"/><key id="id" type="xs:string" required="true" paramType="path"/>
</inputs><execute><![CDATA[
y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); response.object = request.header("Authorization",authheader).del().response; ]]></execute></delete>
delete from twitter.status where id="2108869549" and username="xxx" and password="yyy”
Twitter Delete
![Page 23: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/23.jpg)
Conclusion
More easily build applications• fewer lines of code• data processing done away from app• consistent and familiar syntax for all data access
![Page 24: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/24.jpg)
Conclusion
Build applications that run faster• service in the cloud does the work
• store tables, environments, js, text files…
• conversion, filtering, parallel requests...
With smaller network footprint• fewer (client) calls
• smaller data amounts
• closer to the data, fatter pipes
![Page 25: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/25.jpg)
Conclusion
Flexible• Supports XML, JSON, JSONP
• Transform documents between formats
• Easy to integrate into any development environment w/ HTTP
![Page 26: Yahoo Query Language: Select * from Internet](https://reader033.fdocuments.net/reader033/viewer/2022061114/5462791caf7959e2278b6077/html5/thumbnails/26.jpg)
Questions?
http://developer.yahoo.com/yql
Derek Gathright
twitter.com/derek
meetup.com/socaljs