Rubyconf2012slides 121104193212-phpapp01

Building Data Driven Products Using Ruby

Ryan Weald, @rwealdRubyConf 2012

1Friday, November 2, 12

Who is this guy?

Why should I listen to him?


Studied Computer Science/Bioinformatics @UCSD

Data Scientist @Sharethrough


http://sharethrough.com

http://sharethrough.com

Sharethrough

Native video advertising platform


Outline1) What is a data driven product?

2) What does the development cycle look like for a data driven product?

3) Where does Ruby fit in the world of “data science?”

4) How can Ruby be improved to stay relevant in the age of “big data?”


Ruby is not your only option


</warning>


What is a Data Driven Product?


Anything that uses data to improve the

bottom line


Visualizing Friendships


How do I actually build something?


Ask the right question

Collect & CleanData

Build Model

Publish ResultsData Driven

Product Cycle


Phase 1

Ask the right question


Only English Required


Personal Example


Data dump % of users on publisher X and Y

What is value of a user on an ad network?

What is the supply of given type of user?

Can we predict it?


Phase 2

Data Collection

& Cleaning


90% of your time


{"body":"bc98f346-21c0-11e2-ae5b-12313d25cc96 10.204.85.116 - [29/Oct/2012:12:04:05 +0000] \"GET /beacons?type=targus&oid=10492&se=acaf5e7d-4fd6-d3d0-275b-e2d8389a0bd1&score=011&zip=50644&age=1946&gender=F HTTP/1.1\" 200 145 \"INTENTIONALLY_BLANK_HTTP_REFERER\" \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3)\" \"-\" \"208.126.63.187\" \"-\"","timestamp":1351512246597,"pri":"INFO","nanos":560039789217711,"host":"ip-10-32-67-127.ec2.internal","fields":{"AckTag":"log.00000019.20121029-120406607+0000.560039799240771.seq","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000eaP\u001D","tailSrcFile":"access.log","rolltag":"log.00000019.20121029-120416979+0000.2874668128387074.seq"}}

{"body":"bc9ce55a-21c0-11e2-9855-12313d25cc96 10.205.1.37 - [29/Oct/2012:12:04:05 +0000] \"GET /impression?oid=10492&pid=192&vid=1613&tid=10c0d102&ddid=8504&v=a176475e&se=5a474210-97a1-aa4c-5927-3221443eb38b&pext=&uid=dae4df96-68f6-4602-95b8-60756f27941d&vph=386&vpw=525&ploc=&pref=&635781436849409700 HTTP/1.1\" 200 145 \"INTENTIONALLY_BLANK_HTTP_REFERER\" \"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)\" \"-\" \"74.128.61.37\" \"-\"","timestamp":1351512246597,"pri":"INFO","nanos":560039789242791,"host":"ip-10-32-67-127.ec2.internal","fields":{"AckTag":"log.00000019.20121029-120406607+0000.560039799240771.seq","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000yæJ","tailSrcFile":"access.log","rolltag":"log.00000019.20121029-120416979+0000.2874668128387074.seq"}}

Start With This


...79,28779,28779,282287,282291,270270,273270,42270,291270,42270,4279,261253,7979,253287,282

...

Output This


How do we get the data?


Where does your data come from?

• Server logs

• Third party APIs

• Web scraping

• Direct user input


Ruby FTW


rest-clientpdf-reader


* TweetStream https://github.com/intridea/tweetstream

Tracking Sandy


https://github.com/intridea/tweetstream

https://github.com/intridea/tweetstream

But my data is “big data”




Remember Me?


























{"body":"bc98f346-21c0-11e2-ae5b-12313d25cc96 10.204.85.116 - [29/Oct/2012:12:04:05 +0000] \"GET /beacons?type=targus&oid=10492&se=acaf5e7d-4fd6-d3d0-275b-e2d8389a0bd1&score=011&zip=50644&age=1946&gender=F HTTP/1.1\" 200 145 \"INTENTIONALLY_BLANK_HTTP_REFERER\" \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3)\" \"-\" \"208.126.63.187\" \"-\"","timestamp":1351512246597,"pri":"INFO","nanos":560039789217711,"host":"ip-10-32-67-127.ec2.internal","fields":{"AckTag":"log.00000019.20121029-120406607+0000.560039799240771.seq","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000eaP\u001D","tailSrcFile":"access.log","rolltag":"log.00000019.20121029-120416979+0000.2874668128387074.seq"}}{"body":"bc98f346-21c0-11e2-ae5b-12313d25cc96 10.204.85.116 - [29/Oct/2012:12:04:05 +0000] \"GET /beacons?type=targus&oid=10492&se=acaf5e7d-4fd6-d3d0-275b-e2d8389a0bd1&score=011&zip=50644&age=1946&gender=F HTTP/1.1\" 200 145 \"INTENTIONALLY_BLANK_HTTP_REFERER\" \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3)\" \"-\" \"208.126.63.187\" \"-\"","timestamp":1351512246597,"pri":"INFO","nanos":560039789217711,"host":"ip-10-32-67-127.ec2.internal","fields":{"AckTag":"log.00000019.20121029-120406607+0000.560039799240771.seq","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000eaP\u001D","tailSrcFile":"access.log","rolltag":"log.00000019.20121029-120416979+0000.2874668128387074.seq"}}
























{"body":"bc98f346-21c0-11e2-ae5b-12313d25cc96 10.204.85.116 - [29/Oct/2012:12:04:05 +0000] \"GET /beacons?type=targus&oid=10492&se=acaf5e7d-4fd6-d3d0-275b-e2d8389a0bd1&score=011&zip=50644&age=1946&gender=F HTTP/1.1\" 200 145 \"INTENTIONALLY_BLANK_HTTP_REFERER\" \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3)\" \"-\" \"208.126.63.187\" \"-\"","timestamp":1351512246597,"pri":"INFO","nanos":560039789217711,"host":"ip-10-32-67-127.ec2.internal","fields":{"AckTag":"log.00000019.20121029-120406607+0000.560039799240771.seq","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000eaP\u001D","tailSrcFile":"access.log","rolltag":"log.00000019.20121029-120416979+0000.2874668128387074.seq"}}{"body":"bc98f346-21c0-11e2-ae5b-12313d25cc96 10.204.85.116 - [29/Oct/2012:12:04:05 +0000] \"GET /beacons?type=targus&oid=10492&se=acaf5e7d-4fd6-d3d0-275b-e2d8389a0bd1&score=011&zip=50644&age=1946&gender=F HTTP/1.1\" 200 145 \"INTENTIONALLY_BLANK_HTTP_REFERER\" \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3)\" \"-\" \"208.126.63.187\" \"-\"","timestamp":1351512246597,"pri":"INFO","nanos":560039789217711,"host":"ip-10-32-67-127.ec2.internal","fields":{"AckTag":"log.00000019.20121029-120406607+0000.560039799240771.seq","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000eaP\u001D","tailSrcFile":"access.log","rolltag":"log.00000019.20121029-120416979+0000.2874668128387074.seq"}}

























Lots and Lots of Logs


Finding Bad Data


Ruby on Hadoop

•Vanilla Hadoop streaming

•Wukong

•cascading.jruby


https://github.com/infochimps-labs/wukong

https://github.com/infochimps-labs/wukong

https://github.com/gmarabout/cascading.jruby

https://github.com/gmarabout/cascading.jruby

JRuby38Friday, November 2, 12

Cascading JRuby

* Based on https://github.com/gmarabout/cascading.jruby/blob/master/samples/cascading.rb


https://github.com/gmarabout/cascading.jruby/blob/master/samples/cascading.rb

https://github.com/gmarabout/cascading.jruby/blob/master/samples/cascading.rb

Ruby is powerful tool for data collection and

cleaning


Phase 3

Statistical Modeling&

Prediction


F(user_id, pub_x) = P(on_pub_y)

F(x1, x2, ... xi) = Yi

Personal Example


But Ruby sucks at statistical computing?!


Most of your time is going to be spent collecting & cleaning


Have you tried running R in production?


Ruby Stats Packages

• Statsample

• SciRuby

• LIBSVM

• RinRuby & RSRuby


https://github.com/clbustos/statsample

https://github.com/clbustos/statsample

http://sciruby.com/

http://sciruby.com/

http://www.csie.ntu.edu.tw/~cjlin/libsvm/%23ruby

http://www.csie.ntu.edu.tw/~cjlin/libsvm/%23ruby

https://sites.google.com/a/ddahl.org/rinruby-users/

https://sites.google.com/a/ddahl.org/rinruby-users/

https://github.com/alexgutteridge/rsruby

https://github.com/alexgutteridge/rsruby

JRuby47Friday, November 2, 12

Phase 4

Publish ResultsA.K.A. show data to users


Web UI or Mobile App


Personal Example


Not all Roses


Ruby Improvements

• Graphing library

• Unified matrix and vector library

• More publishing around Ruby & ML

• Academic buy in


Ruby + Data=

Agile Data Products


We’re Hiringhttp://bit.ly/sharethrough-jobs

Twitter: @rwealdEmail: [email protected]


https://bitly.com/#

https://bitly.com/#

http://twitter.com/rweald

http://twitter.com/rweald

mailto:[email protected]

mailto:[email protected]

Rubyconf2012slides 121104193212-phpapp01

Documents

Transcript of Rubyconf2012slides 121104193212-phpapp01