Eric Fan Insight Project Demo
-
Upload
eric-fan -
Category
Data & Analytics
-
view
95 -
download
0
Transcript of Eric Fan Insight Project Demo
SpentAtAGlance
By Eric FanInsight Data Engineering (Silicon Valley 2016)
Help Track Your Expense!
SpentAtAGlance
A budgeting tool (similar to mint.com) that summarizes your current months and recent credit card transaction history
How much have I spent this month?
A quick visualization of your acrued expenses before making your next purchase
SpentAtAGlance
Gather statistical data for the credit card company for marketing strategies
Total/average monthly transactions for all users
Front-End
For card holderhttp://spentglance.onlinehttp://ec2-52-41-11-131.us-west-2.compute.amazonaws.com
For card companyhttp://spentglance.online/card_companyhttp://ec2-52-41-11-131.us-west-2.compute.amazonaws.com/card_company
https://youtu.be/miDFxvX_4eQ
Data Pipeline
Producer.py
Camus
{"date":string,day_of_week: string,time: string,transaction_type:stringname:string,"amount":string}
50 GB of data
Data Pipeline Cost
Kafka1 x m4.large(spot)$0.04
Spark/HDFS4 x m4.xlarge(1 on-demand,3 spot)$0.239+3 * $0.06=$0.419
Cassandra4 x m4.large(spot)$0.06 x4=$0.24
Flask1 x m4.large(spot) 5000 w/s for the 4 nodes configuration
[1] T. Rabl et al. (2012), Solving Big Data Challenges for Enterprise Application Performance Management