Post on 02-May-2022
Layers
Yelp’s MissionConnecting people with great
local businesses.
(Abstraction) Layers(related: separation of concerns)
Abstractions?
"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." - Edsger Dijkstra
All non-trivial abstractions, to some degree, are leaky. - Joel Spolsky
Any problem in computer science can be solved with another layer of indirection. But that usually will create another problem - David Wheeler
Danger: I don’t know as much as Rich Hickey
Identify the Author’s Purpose
A. Persuade the audience that his/her design recommendations are better than the current status quo
B. Inform others about the history of architecture decisions in computer science
C. Inspire engineers to think outside the box when designing systems
D. Entertain the audience with historic anecdotes
E. None of the above
DevOps
DevOpsEst. 2009
Let’s Talk About Devops Features
Wireframes
Tasks
Code / Unit Tests
Build + Release
Deployment | Monitoring | Alerting
QA | Acceptance Tests
PM
UX
Lead Dev
Devs
QA
Release Engineering
Operations
“What if we got rid of some of the layers?”
?
Let’s Talk About Devops (no-ops?)
Features
Wireframes
Tasks
Code / Unit Tests
QA | Acceptance Tests
Build + Release
Deployment | Monitoring | Alerting
PM
UX
DevsSeparation
OfConcerns
Why?
● Increased “Performance” (deploys per day)● Better ownership of issues● Harder to hire● Massive increase in developer complexity● Large inter-team communication overhead
Filesystems
Let’s talk about filesystems
files
ext4
dm-crypt
Software Raid
Raw Block Devices
LVM
/mnt/stuff/foo
/mnt/stuff
/dev/mapper/luks-17c8...
/dev/mapper/lv0
/dev/md0
/dev/sd{a,b}
“What if we got rid of some of the layers?”
Let’s talk about zfs
files
zfs
(zraid)(encryption)(checksums)
(pools)(vdevs)
Raw Block Devices
/mnt/stuff/foo
/mnt/stuff
/dev/sd{a,b}
SeparationOf
Concerns
Now What?
● Better error detection/recovery● We can deduplicate things● More efficient snapshots / Repair● All new tooling is not interoperable
New Concept: “Sympathetic Abstraction”
● “Opposite” of “Leaky”?● https://mechanical-sympathy.blogspot.com/
~ $ ssh -vv leb1.xkyle.com -- trueOpenSSH_7.7p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017debug1: Reading configuration data /home/kyle/.ssh/configdebug1: /home/kyle/.ssh/config line 1: Applying options for leb1.xkyle.comdebug1: Reading configuration data /etc/ssh/ssh_configdebug1: /etc/ssh/ssh_config line 19: Applying options for *debug2: resolving "leb1.xkyle.com" port 4242debug2: ssh_connect_direct: needpriv 0debug1: Connecting to leb1.xkyle.com [2607:8b00:0:96::d0eb:ec0a] port 4242....debug1: Entering interactive session.debug1: pledge: networkdebug2: channel_input_open_confirmation: channel 0: callback startdebug2: fd 3 setting TCP_NODELAYdebug2: client_session2_setup: id 0debug1: Sending environment.debug1: Sending env LANG = en_US.UTF-8debug2: channel 0: request env confirm 0debug1: Sending command: true...Transferred: sent 2696, received 2516 bytes, in 0.1 secondsBytes per second: sent 23959.5, received 22359.8debug1: Exit status 0
https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html?hl=commitlog_sync
commit_failure_policy
(Default: stop) Policy for commit disk failures:
● die● Shut down gossip and Thrift and kill the JVM, so the node can be replaced.● stop● Shut down gossip and Thrift, leaving the node effectively dead, available for inspection using JMX.● stop_commit● Shut down the commit log, letting writes collect but continuing to service reads (as in pre-2.0.5 Cassandra).● ignore● Ignore fatal errors and let the batches fail.
disk_optimization_strategy
(Default: ssd) The strategy for optimizing disk reads. Possible values: ssd or spinning.
disk_failure_policy
(Default: stop) Sets how Cassandra responds to disk failure. Recommend settings: stop or best_effort. Valid values:
● die● Shut down gossip and Thrift and kill the JVM for any file system errors or single SSTable errors, so the node can be
replaced.● stop_paranoid
Kyle’s Abstraction Law:
“Over time, layers tend toward becoming more sympathetic (or merged) in the pursuit of performance.”
HTTP(S)
OSI Layer Model History
● Published in 1984 under ISO7498 (35 years ago)● Mostly by Charles Bachman from his experience with ARPANET, etc● Written with the purpose of giving protocol designers a Map to follow
Layer 6: Presentation
Layer 5: Session
Layer 4: Transport
Layer 2: Data link
Layer 1: Physical
Layer 3: Network
SSL
N/A (Right? HTTP is “stateless”)
TCP (Transmission Control Protocol)
IP (Internet Protocol)
802.11 (Wi-Fi?)
802.11 (Wi-Fi?)
Layer 7: ApplicationHTTP (Hypertext Transfer Protocol)HTTPS in the OSI Network Model
https://en.wikipedia.org/wiki/Internet_protocol_suite#Comparison_of_TCP/IP_and_OSI_layering
The IETF protocol development effort is not concerned with strict layering. Some of its protocols may not fit cleanly into the OSI model, although RFCs sometimes refer to it and often use the old OSI layer numbers.
The IETF has repeatedly stated that Internet protocol and architecture development is not intended to be OSI-compliant. RFC 3439, addressing Internet architecture, contains a section entitled: "Layering Considered Harmful".
RFC 1122, Internet STD 3
(1989)
Cisco Academy[28]
Kurose,[29] Forouzan[30]
Comer,[31] Kozierok[32]
Stallings[33] Arpanet Reference Model (RFC 871)
OSI model
Four layers Four layers Five layers Four+one layers Five layers Three layers Seven layers
"Internet model" "Internet model" "Five-layer Internet model" or "TCP/IP protocol suite"
"TCP/IP 5-layer reference model"
"TCP/IP model" "Arpanet reference model"
OSI model
Application Application Application Application Application Application/Process Application
Presentation
Session
Transport Transport Transport Transport Host-to-host or transport
Host-to-host Transport
Internet Internetwork Network Internet Internet Network
Link Network interface Data link Data link (Network interface)
Network access Network interface Data link
Physical (Hardware) Physical Physical
Layer 6: Presentation
Layer 5: Session
Layer 4: Transport
Layer 2: Data link
Layer 1: Physical
Layer 3: Network
SSL
N/A (Right? HTTP is “stateless”)
TCP (Transmission Control Protocol)
IP (Internet Protocol)
802.11 (Wi-Fi?)
802.11 (Wi-Fi?)
Layer 7: ApplicationHTTP (Hypertext Transfer Protocol)HTTP in the OSI Network Model
“What if we got rid of some of the layers?”
Transport Layer (OSI 4)
Link Layer (OSI 1,2)
Internet Layer (OSI 3)
UDP (User Datagram Protocol)
IP (Internet Protocol)
802.11 (Wi-Fi)
Application Layer (OSI 5,6,7)
EncryptionSessions
MultiplexingError Correction
QUIC(Quick UDP Internet Connections)
QUIC in the IP Model
SeparationOf
Concerns
Now What?
● Roaming sessions● 0-RRT Handshakes● Better throughput● Better loss recovery● You can’t debug via tcpdump anymore!● Only a few companies has dared to ship this!● Very poor tooling / ecosystems around it!● Must have faith in QUIC-crypto!
(Micro) Services
Rails App
Model View
Controller
Users
Mysql
“What if we ADDED some more layers?”
Users
API Gateway
Users Service SearchTweetsAds Moments
Ads Mysql User
MysqlSearch
Infra MomentsSQLTweets
NoSQL
Followers
Graph DB
Timeline
Now What?
● No more atomic cross-cutting changes● Simplified deploy process? (Simple for who?)● Polyglot stack (easy to hire for)● No more unified tooling● You scale out components (scale to millions of rps)● You scale out development (1000 engineers)
k8s
Layer 3: Application Code
Layer 1: IaaS VM
Layer 0: IaaS Provider Bare Metal
Layer 2: OS
Java Jar
Ubuntu x86_64
c5.xlarge
N/A
Traditional Cloud
Deployment
“What if we ADDED EVEN MORE layers?”
Layer 4: Pod
Layer 3: Container Orchestration
Layer 1: IaaS VM
Layer 0: IaaS Provider Bare Metal
Layer 2: OS
pod-foo
kublet
Ubuntu x86_64
c5.xlarge
N/A
Layers of a k8s Infra
Layer 5: Containerdocker
Layer 6: Processjava -jar
Layer 5: k8s federation
Layer 4: k8s cluster
Layer 3: Deployment / ReplicaSet
Layer 1: Docker Container
Layer 0: Process
Layer 2: Pod
Prod k8s federation
uswest1-prod
Frontend replicaset
myapp-pod
FROM apache2
httpd
K8s Layers
“What if we got rid of some of the layers?”
AWS Metal Instances
Layer 0/1: IaaS Provider Bare MetalHardware
Future Layers
Layer 0/1: IaaS Provider Bare MetalHardware
Future Layers Extreme Example -
Unikernels?
Layer 2-7: Unikernel(No OS, just a directly booting
application)MirageOS, etc
Layer 0/1: IaaS Provider Bare MetalHardware
Future Layers - AWS Lambda / FaaS?
Layer 2-5: Amazon / k8s StuffSame Stuff As Before
Layer 7: Your ProcessYour Code
Layer 6: ContainerFirecracker / Docker
What? We Are Back To Where We Started?
● Where is the mechanical sympathy?● What happened to Layer-Smashing, Kyle?● Three Reasons why we are moving this direction...
Layer 0/1: IaaS Provider Bare MetalHardware
Reason 1:You have a layer for just the stuff you care about
Layer 2-5: Amazon / k8s StuffSame Stuff As Before
Layer 7: Your ProcessYour Code
Layer 6: ContainerFirecracker / Docker
k8s pods
Apache
API Gateway + FaaS Controller(Request Routing)(Compute Dispatch)
Reason 2: Sympathetic routing and dispatchOr “Functions as a Service (Serverless)” Layer Smashing
Autoscaling Group(Compute Dispatch)
Apache
Elastic Load Balancer(Request Routing)
Apache
Reason 3:Sympathetic Compute Scheduling Layers(k8s operators)
Autoscaling Group
MySQL
MySQL
?
K8s MySQL Operator
MySQL
MySQL
I know what I
must do
Conclusions?
1. “Sympathetic Layers” are the best layers!
2. k8s operators and FaaS as re-slicing where the layers are (for the greater good!)
3. You should watch “Simple Made Easy” before changing your mindhttps://www.infoq.com/presentations/Simple-Made-Easy
Hold for (shameless) applause,
Then
Questions?
SEPARATION OF
CONCERNS
IS OVERRATED
Attribution● OSI Layers - http://www.gargasz.info/osi-model-how-internet-works/● Simple Made Easy - The Art of Unix Programming (2003) - Eric S. Raymond
X
Linux Kernel
GPU
X Server
X Client (GUI App)
X11 Protocol over Socket /tmp/.X11-unix/X0
Ioctl via libdrm/dev/dri/card0
3d Game
Ioctl via libdrm/dev/dri/card0
Window Manager
“Mechanism, not policy”
“What if we got rid of some of the layers?”
Now What?
● Lower latency GUI Interactions● NO TEARING