RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019...
Transcript of RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019...
![Page 1: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/1.jpg)
Robert Kisteleki | CAIDA AIMS 2019
RIPE Atlas infrastructure &
Measurement Results Sharing
![Page 2: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/2.jpg)
Part IRIPE Atlas changes
![Page 3: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/3.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• About 10.500 probes are up
• Of which 440 are anchors
• Of which 80 are VMs
• Close to 20M measurements defined
• About 20k running concurrently
• Collecting 7000-8000 data points / sec (~600M/day)
RIPE Atlas current status
!3
![Page 4: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/4.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• Use more client-side rendering relying more on APIs
• Strongly considering changing queues from RabbitMQ to Kafka
• Switching measurement metadata store from MySQL to ElasticSearch
• Switching main database from MySQL to PostgreSQL
• Python2 -> Python3 migration, other framework upgrades
RIPE Atlas current work items
!4
![Page 5: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/5.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• Pilot ran between May-October 2018
• Service is official since November 2018
• High interest since launch time
• Includes nodes in AWS
• Talking to other cloud providers too
• ASN holder vs VM host needs attention
• VMs are virtually indistinguishable from HW anchors
RIPE Atlas anchor VMs
!5
![Page 6: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/6.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• Probes are still hardware based (excl. anchor VMs)
• There’s demand for an installable software package
• Would allow the next level of growth
• We’re evaluating this at the moment
• It has important consequences to operations
• Now to encourage growth while encouraging spread?
RIPE Atlas software probes
!6
![Page 7: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/7.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• We’re at version 4 of the probes (NanoPi based)
• The distributed nature brings some unique challenges
• So does the volume of data we’re handling
Other bits and blobs
!7
![Page 8: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/8.jpg)
Part IIMeasurement Results Sharing
![Page 9: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/9.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• All public measurement results are available in bulk via FTP
• Much easier/faster to download large amounts
• Measurement metadata (via API) helps with indexing
• Download API is still the main path for individual msms
• Result streaming is still available and supported
• Fully real-time
• Has different availability characteristics
Downloadable results
!9
![Page 10: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/10.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• Pushing data to BigQuery too
• Has unique advantages
• See a more detailed presentation by Stephen
RIPE Atlas data in the cloud
!10
![Page 11: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/11.jpg)
Robert Kisteleki | CAIDA AIMS 2019
• “Can we lower the barrier for users (researchers or otherwise) to use multiple measurement systems without having to relearn or reimplement anything?”
• Different systems use dirrerent APIs and data formats
• Can we improve on this?
• Is this an incentive or a technology question?
Measurement infrastructure collaboration
!11
![Page 12: RIPE Atlas infrastructure Measurement Results Sharing · Robert Kisteleki | CAIDA AIMS 2019 •About 10.500 probes are up •Of which 440 are anchors •Of which 80 are VMs •Close](https://reader033.fdocuments.net/reader033/viewer/2022060310/5f0a95487e708231d42c5875/html5/thumbnails/12.jpg)
Questions
Robert Kisteleki | CAIDA AIMS 2019 !12