Web as a data resource
-
Upload
subramanyan-murali -
Category
Documents
-
view
1.142 -
download
1
description
Transcript of Web as a data resource
`
Get me my Data !Use the web as a data source
2
Hack U @ Georgia Tech - March 8 – 12 2011
Subramanyan MuraliyahooMail Engineer Hacker, Photographer, Traveler @rmsguhan
Data and hacks
4
Putting something together in 24 hours can be a real
rush !5
Need innovative ways to make the data into
information that is useful
7
Any resource on the internet is a source of
data …
8
… that means that there is a lot of data
9
Loads and loads of data and data
services
10
Data that is complex and in many formats
11
Navigating through all the complexity can be hard !#$%&
12
How to solve the data problem ?
13
http://pipes.yahoo.com saves the day!
14
15
Graphical interface to play with data !
WYSIWYG = pipes
16
Pipes is a visual tool, where is my command
line ??
18
Put your hands up for YQL !
19
Common data access language across the
webdeveloper.yahoo.com/y
ql
20
SQL like syntaxStandard and easy to use
21
select {what} from {service} where {condition} in ({sub-query} )
22
Several services can be chained using in ()
23
YQL Console
24
Lots of Integrations and predefined tables
anyone can be part of the interface
25
Fetching data from a resource has never
been simpler
!
26
http://query.yahooapis.com/v1/ public/yql?q={query} &format={format}
27
Irrespective of the source, output can be
standardized …
28
29
… thus giving you a solid & consistent data foundation
Input can be XML, HTML, CSV, micro-
formats, Atom feeds, JSON, RSS
30
Output can be JSON or XML
31
Mix and match YQL and Pipes ?
32
Pipes provides you with processing
capabilities
33
YQL is another feed source
34
Interlinking YQL and Pipes =
fewer lines of code
35
Not limited to Yahoo! sources
select * from google.translate where q in (select text from twitter.search where q="georgia tech") and target='fr';
Fire off multiple queries at once
select * from query.multi where
queries=‘{query1};{query2}’;
Create your own tables with Open
tables
www.datatables.org/
http://www.yqlblog.net/
Go explore !
developer.yahoo.com/yql/console/
Happy Hacking !!
Photos @ http://flickr.com/photos/rmsguhan