unba.seFramework for distributed computing and collaboration

Daniel NormanCTO, güdTECHunba.se contributorTwitter: @DreamingInCode

Michael MacFaddenCTO, Convergence Labsunba.se contributorTwitter: @MMacFadden

BackgroundThe Problem

For decades, we’ve suffered from various myths:

● Networks can be made reliable

● A single arbiter of state makes a consistency-model “strong”

● Objective state exists

● Objective simultaneity exists

These convenient beliefs were “close enough” for a while, but not much longer.

BackgroundThe Problem

Distance between you and the stuff you probably care about:

Much much less

Distance between you and the arbiter of linearization:

Usually thousands of Km

(Stuff you care about)

Collaborating with Bob & Alice down the hall

YouMessages/docs you authoredMessages/docs you readYour wristwatchYour IOT devicesNext door neighbors

The linearization you must visitBackh

oes

Congestion

Light travel time

Alligators

Net non-neutrality

BGP Screwups

Power outagesState Hacking

Spanning tree errors

Tripped-over cables

Cat on serverSubmarine cable break

Settlement-free

peering disputes

State Surveillance

Packet corruption

Lie-FiUnder Provisioned hardware

Cheap electrolytic capacitors

Thousands of Km of:

DNS errors

Saturated cell-tower backhaul

RF reflections

Interference from microwave oven

Cosmic Rays

Disgruntled employees

DDOS AttacksRat-chewed cables

Corrosion Late internet payments

Gravitational time-dilation

Doppler effect

F*#KING BOINGO

Failed B-side PowerCore router problem

F*#KING TMOBILE

F*#KING NETGEAR

Cabinet switch failure

Misconfigured health check

NAT Misconfiguration

Load balancer failure

NTP failures

BackgroundThe Problem

We’ve tried lots of things

● Multi-mastering

● Geo-sharding

● Eventual consistency

● Simulated Simultaneity ( global wallclock )

One way or another, these all violate the user’s expectation

BackgroundWhat do humans expect?

I set my glass on the table ≻ It’s there when I pick it up.

Edge computing IRL.Local, coordination-free consistency.No need to visit Ashburn.

BackgroundExamples of The Problem

● Total internet outage○ Bob and Alice prevented from collaborating, even if they’re just down the hall

● LieFi○ Elevator door closes, messaging UI hangs, unable to determine message state

● Replication failure○ IO/network failure causes data loss – If you’re lucky, failover state machine works promptly,

decisively and correctly. You’re not that lucky

● Application state gets “weird”○ Eventual consistency models lead to non-causal outcomes, and unhappy/confused users.

BackgroundThis problem isn’t going away

● Globalization, Trade, Collaboration are increasing

● System complexity is becoming unmanageable

● Local bandwidth requirements will scale exponentially, backbone will not

● Safety / Mission critical uses are proliferating, outages will get worse

● Distributed systems challenges becoming increasingly problematic on-chip

unba.se design ideologyIt’s time for another approach

● State is an illusion, Sort of...

● Computation and storage must occur near the edge

● A priori resource planning is undesirable

● Durability is non-boolean

● Distributed causal consistency – possible, efficient, desirable

unba.se design ideologyMotivating Suppositions

unba.se motivating suppositionsState is an illusion, sort of

● State is a projection of observed events○ Absolutely no such thing as shared state, ever.

● It’s an abstraction we like○ Not entirely unlike “solid matter” vs coulomb repulsion

● Some abstractions we need not question in CS○ Example: “why don’t I fall through the floor?”

● Collaborative systems however, DO warrant deeper reflection on “State”○ ALL multi-user systems are collaborative systems○ The user expects causal fidelity beyond which can be provided serializable systems.

Philosophy Corner:

What if there was no reality, only causality?

unba.se motivating suppositionsLocation, Location, Location

● Humans expect devices to behave like physical objects○ Physical proximity is literally the only way to meet their causal expectations○ How proximal is “proximal”?

● Travel to arbiter of linearization is an unacceptable default assumption○ Physical reality does not have centralized arbiters of truth, and neither should we

● Data should be localized by origin and utility, not by shard○ Why would you want to manually geo-shard?○ Or worse, shard by some non-spatial criteria

● A priori shard/resource planning is expensive, and you’ll do it wrong○ Humans are expensive, terrible at system orchestration, capacity planning○ Automated A priority planning is marginally better○ A squeezed balloon does not “plan” the expansion on the other side

unba.se motivating suppositionsDurability is not a boolean

● Every piece of data has a nonzero probability of loss

● Most database systems merely tend to experience data loss all at once○ Usually through corruption○ Sometimes through common mode failure

● We care more about some data than others

● So let’s set a specific target durability for each piece of data

We prefer to plan how often to lose data rather than engage in self-deception about its permanence.

unba.se core conceptsCore Concepts Overview

● Immutable Data Structure

● Gravity and pressure

● Sparse vector clocks

● Commutative index merging

● Selective hearing

● Infectious knowledge consistency model

unba.se core conceptsImmutable Data Structure

unba.se core conceptsImmutable Data Structure

unba.se core conceptsImmutable Data Structure

unba.se core conceptsState Projection

unba.se core conceptsState Projection

unba.se core conceptsInfectious Knowledge

unba.se core conceptsGravity and Pressure

unba.se core conceptsBeacons / Sparse Vector Clocks

unba.se core conceptsProbabilistic Merging

unba.se core conceptsCommutative Index Merging

With this design we may:

● Align system performance with user’s causal expectations

● Increase availability (for data I care about)

● Reduce system complexity

● Save time and money on system operations / staff

unba.se core conceptsConclusion

Thank You!

Daniel NormanCTO, güdTECHunba.se contributorTwitter: @DreamingInCode

Michael MacFaddenCTO, Convergence Labsunba.se contributorTwitter: @MMacFadden

Bonus slides

● Consistency○ Is against human causal expectation, not serializability○ Serializability usually just a hack for causality

● Availability○ Objectively no such thing as availability○ Is only ever declared according to a human observational standard

● Partitions○ Objectively no such thing as a partition○ Only exist according to an observational standard

With sufficiently tortured definitions, one can prove a ham sandwich.

See Martin Kleppmann’s excellent paper: A critique of the CAP theorem – https://arxiv.org/abs/1509.05393

Obligatory SlideProblems with CAP Theorem