Spark Seattle meetup - Breaking ETL barrier with Spark Streaming
Spark streamingを使用したtwitter解析によるレコメンドサービス例
46
1 GMO イイイイイイイイイイイ イイイイイイイイイイ イイイ一 2015.8.1 Spark Streaming イイイイイ Twitter イイイイイ イイイイイイイイイイ
-
Upload
junichi-noda -
Category
Engineering
-
view
552 -
download
4
Transcript of Spark streamingを使用したtwitter解析によるレコメンドサービス例
- 1. 1 GMO 2015.8.1 Spark Streaming Twitter
- 2. 1. 2. 3. 4.Spark StreamingTwitter Mikasa- 5.Spark 6.Spark Streaming 7. 8. 2
- 3. 1. 3 Java2EE Perl/Ruby Hadoop Github: https://github.com/AKB428 Twitter: @n428dev Qiita: http://qiita.com/AKB428 Hatena: http://akb428.hatenablog.com/ : http://akibalab.info/ Junichi Noda Larry Wall
- 4. 2. GMO 132 20152 Spark Streaming and GraphX (45) Spark Streaming 4
- 5. 2. Twitter Spark Streaming SparkScalaScala Spark 5
- 6. 3. 6
- 7. 7
- 8. 4. Spark Streaming Twitter Mikasa- https://github.com/AKB428/mikasa_ol https://github.com/AKB428/mikasa_rs 8
- 9. 9
- 10. http://youtu.be/o6xq_RdNv6s 10
- 11. 11
- 12. 12
- 13. Twitter Twitter Streaming API 400 Spark Streaming 13
- 14. 14 kuromoji (CSV)
- 15. 15
- 16. 16
- 17. Spark Streaming Mikasa560 60*2=120) 17
- 18. https://github.com/AKB428/mikasa_ol/blob/maste r/src/main/scala/mikasa.scala https://github.com/AKB428/mikasa_ol/blob/maste r/dictionary/anime_2015_3Q.txt 18
- 19. 19
- 20. 5 (F1)20153 TV http://tv-anime.xyz/5/ (F1) http://tv-anime.biz/5/ (F2) http://lovelive-net.com/5/ (F2) http://akiba-net.com/5/ (F2)TV http://telev.net/5/ F1= F2= 20
- 21. 21 60 (F1)20153 TV http://tv-anime.xyz/60/ (F1) http://tv-anime.biz/60/ (F2) http://lovelive-net.com/60/ (F2) http://akiba-net.com/60/ (F2)TV http://telev.net/60/ F1= F2=
- 22. 22
- 23. https://akiba-souken.com/article/24303/ 23
- 24. 2 (2015/) 24
- 25. 3 2015722 722 25
- 26. Twitter https://twitter.com/Akihabaara_itso https://twitter.com/AuctionMadoka 26
- 27. 27
- 28. 28
- 29. 5! (F1)20152 TV http://tv-anime.xyz/5/ (F1) http://tv-anime.biz/5/ (F2) http://lovelive-net.com/5/ (F2) http://akiba-net.com/5/ (F2)TV http://telev.net/5/ F1= F2= 29
- 30. 30 60 (F1)20152 TV http://tv-anime.xyz/60/ (F1) http://tv-anime.biz/60/ (F2) http://lovelive-net.com/60/ (F2) http://akiba-net.com/60/ (F2)TV http://telev.net/60/ F1= F2=
- 31. Mikasa Twitter Twitter Filter + 31
- 32. Twitter Filter + 32
- 33. MikasaConoHa() .com http://www.onamae.com/ https://www.conoha.jp/ 33
- 34. 4. Spark Streaming TwitterUtilJavaTwitter twitter4j TwitterTwitterAPIAPI IP Twitter 34
- 35. 5. Spark HadoopMapReduce(DAG) HadoopHadoop 2015611 1.4.0 https://spark.apache.org/releases/spark-release-1-4-0.html 1.4.0SparkR(RSpark APIScala, Java, Python 35 SparkR
- 36. 5. Spark SparkDAG 36
- 37. 5. Spark 2015615 IBM3500Spark http://japan.zdnet.com/article/35065964/ 2015 61517 Spark summit 2015 () NTT 2Hadoop CDHHDPSpark Spark1.4UI http://qiita.com/AKB428/items/71900eecdf2350e1e45f 37
- 38. 6. Spark Streaming Spark 13 IP 38
- 39. 6. Spark Streaming ssc.socketTextStream(" localhost", 7777) IP 39
- 40. Mikasa https://gist.github.com/AKB428/c30bc6a979e05fa3a022 TwitterAPIAmazonAPI1 AmazonAPI TwitterAPI10 Mikasa OL https://github.com/AKB428/mikasa_ol Mikasa RS https://github.com/AKB428/mikasa_rs 40
- 41. 7. -1 Scala Scala PlaySBT 41
- 42. 7. -2 Learning Spark Spark 42
- 43. 8. TwitterSpark StreamingSQLDB SQL 43
- 44. GMO Hadoop http://recruit.gmo.jp/engineer/jisedai/ 44
- 45. 45
- 46. 46
