AWS at Mendeley (London, September 27th 2011)
-
Upload
dan-harvey -
Category
Technology
-
view
1.772 -
download
5
description
Transcript of AWS at Mendeley (London, September 27th 2011)
![Page 1: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/1.jpg)
Dan HarveySystems Architect
Revolutionising the world of research
with Amazon Web Services
![Page 2: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/2.jpg)
Mendeley helps researchers work smarter
![Page 3: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/3.jpg)
Mendeley makes science morecollaborative and transparent
Sync between computersand peers around the world
![Page 4: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/4.jpg)
1 million researchers...
Uploaded 130 million citations
40 million unique
17 million papers
16 TB of data
![Page 5: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/5.jpg)
Document Storage
• Backed on to S3
• Previews for 16TB of PDFs?
![Page 6: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/6.jpg)
PDF Previews
ElasticBeanstalk S3
Process Queue
Load PDFs
Store PNGServe
via Cloud Front
Render to PDF
![Page 7: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/7.jpg)
System Overview
EMREC2
S3
EBS
RDS
VPC
![Page 8: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/8.jpg)
Article Search
• Based on Solr (open source search)• 40GB index• Variable usage
Day of the month
H-pRequests
![Page 9: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/9.jpg)
Solr Layout
SolrSlave Solr
SlaveSolr
Slave
EBS
ElasticLoad Balancer
SolrMaster
SearchQueries
Inside VPC
Outside VPC
![Page 10: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/10.jpg)
• Machine learning on Hadoop
• Personalised article recommendations• Collaborative filtering based
• Running on Elastic Map Reduce
![Page 11: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/11.jpg)
Summary
• Not all or nothing
• Focus on your problemnot “Undifferentiated heavy lifting”
- Werner Vogels
• Learn the building blocks AWS provide
![Page 12: AWS at Mendeley (London, September 27th 2011)](https://reader035.fdocuments.net/reader035/viewer/2022062419/558c2363d8b42a932e8b46bd/html5/thumbnails/12.jpg)
Enjoy what you’ve seen?
We’re hiring!
Senior Java Engineers
chat to me afteror e-mail/tweet