How (and why!) we built Packet
-
Upload
bob-sokol -
Category
Technology
-
view
93 -
download
1
Transcript of How (and why!) we built Packet
![Page 1: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/1.jpg)
![Page 2: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/2.jpg)
OUR PASSION:
BUILD A BETTER INTERNET.
![Page 3: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/3.jpg)
A Premium Bare Metal Cloud Built for Automation
● Fully dedicated servers, no co-tenancy● Billed by the hour & available in minutes via API
● No hypervisor, no virtualization● Backed by our global, IPv6 network
![Page 4: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/4.jpg)
Wait, no virtualization at all?
Nope.(unless you bring your own)
Which, when building a cloud platform, presents some unique challenges...
![Page 5: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/5.jpg)
Public clouds (AWS, Digital Ocean, GCE) all rely on a hypervisor or containers to automate the provisioning of virtual machines to end users.
THE CHALLENGE
In order to automate “Layer 0” for developers, we we had to build core infrastructure components from the ground up.
![Page 6: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/6.jpg)
But first we tried to do it with OpenStack.
THE FAILURE
( our now infamous blog post about this available here: https://www.packet.net/blog/how-we-failed-at-openstack/ )
Learning from the limitations in OpenStack, we decided to start over with an api-driven microservices framework strategy. This meant developing purpose-built services for each requirement.
![Page 7: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/7.jpg)
Micro services for each component of the physical datacenter.
PACKET PLATFORM
➢ Built in golang and Ruby
➢ Everything in Docker and CoreOS
➢ Orchestrated and managed with Rancher
➢ Testing with shippable, image builds with quay.io
![Page 8: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/8.jpg)
Micro services for each component of the physical datacenter.
PACKET PLATFORM
Power and Boot Control
PB&J
EC2 Style Metadata
Kant
iPXE Server & Imaging
Tinkerbell
Netflow Agg & Analysis
Soren
Multi-Tenant IPAM
Magnum IP
Physical Switch SDN
Narwhal
Device, Project, Billing, Token Management
Client Portal
Internal and External Services
APISerial Console
Out of Band Access
S.O.S.
Image Building
Casper
![Page 9: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/9.jpg)
Let’s review a selection of the more interesting microservices...
![Page 10: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/10.jpg)
KANTEC2 Compatible Metadata Service
![Page 11: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/11.jpg)
Security challenge: IP based authentication, no hypervisor, our layer 3 acl solution
KANTEC2 Compatible Metadata Service
EC2 has a problematic url structure / IP address (eg: http://169.254.169.254/latest/meta-data/ )
Available at https://metadata.packet.net from all hosts
![Page 12: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/12.jpg)
TinkerbelliPXE Server & Imaging
pxe server… tinkerbell … get it? :D
![Page 13: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/13.jpg)
API driven DHCP and iPXE server
TinkerbelliPXE Server & Imaging
Handles delivery and selection of both end user operating system images, and also our deprovision images
Replacement for cobbler, provides massive stability and speed improvements as well as better diagnostic data and visibility in every stage of provisioning
![Page 14: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/14.jpg)
NarwhalPhysical Switch / Router “SDN”
apparently there aren’t any entertaining public domain pictures or gifs of narwhals. apologies.
![Page 15: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/15.jpg)
Overcomes one of the biggest limitations of OpenStack for us, which is very vlan oriented
Configures the network ACLs and handles our elastic IP addressing, batches thousands of operations an hour
NarwhalPhysical Switch / Router “SDN”
Built to be “pluggable” to accommodate other vendors in the future, like Cisco and Arista
![Page 16: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/16.jpg)
SorenNetflow Aggregation & Analysis
can you tell we have some philosophy majors on the team yet?
![Page 17: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/17.jpg)
Ingests all of our network flow data and tags it
Allows us to do things like destination based bandwidth billing and tiered bandwidth
SorenNetflow Aggregation & Analysis
Gives us the ability to expose much more network intelligence to end users, such at top talkers and last mile reachability
![Page 18: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/18.jpg)
PB&JPower and Boot Control (IPMI / DRAC)
this is our favorite result in google image search for “fucking ipmi”
![Page 19: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/19.jpg)
Abstracts much of the ugliness of IPMI and DRAC from our API
Gives us access to health and other hardware monitoring as well
PB&JPower and Boot Control (IPMI / DRAC)
Roadmap to support redfish and potentially open source the library in the future
![Page 20: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/20.jpg)
In summary, where we are now...
![Page 21: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/21.jpg)
NO OVERLAYSFrom network to host, we provide elastic control without adding network or hardware abstraction.
MICROSERVICE BASEDShippable, immutable independent services that are easy to use, scale and troubleshoot.
REPRODUCIBLE & DEPENDABLE99% of our monthly installs succeed with an average provision time of ~8 minutes.
MODERN & SCALABLE
![Page 22: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/22.jpg)
PLATFORM HIGHLIGHTSLaunched in August 2015. No Co-Tenancy. No Hypervisor. Full Automation.
![Page 23: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/23.jpg)
CURRENT CONFIGS
We Also Offer High Performance Block Storage, Elastic IP Addressing and Self-Service BGP.
TYPE 0
TINY BUT MIGHTY
TYPE 1
FLEXIBLE WORKHORSE
TYPE 2
THE VIRTUALIZOR
TYPE 2A
ARM ‘NHAMMER
$0.05/hour
$0.40/hour
$1.25/hour
$0.50/hour
4 Physical Cores @ 2.4 GHz
(1 × Atom C2550)8 GB of RAM
1 x 120 GB SSD
4 Physical Cores @ 3.4 GHz
(1 × E3-1240 v3)32 GB of RAM
2 x 240 GB SSD
24 Physical Cores @ 2.2 GHz
(2 x E5-2650 v4)256 GB of RAM 6 x 480 GB SSD
96 Physical Cores @ 2.0 GHz
(2 × ThunderX)128 GB of RAM
1 x 320 GB M.2 SSD
TYPE 3
HIGH I/O MONSTER
$1.75/hour
16 Physical Cores @ 2.6 GHz
(2 × E5-2640 v3)128 GB of RAM
1 x 2.4 TB NVME
![Page 24: How (and why!) we built Packet](https://reader033.fdocuments.net/reader033/viewer/2022042907/5886f8591a28ab4e3a8b4ebb/html5/thumbnails/24.jpg)
INTEGRATIONSWe offer developers a consistent infrastructure experience that
integrates with leading platforms and dev tools - but with the firepower and flexibility of bare metal.