Freedom, Responsibility, and Dependencies

JON SCHNEIDERSenior Software Engineer – Build Tools at Netflix@jon_k_schneider

JFrog SwampUP – May 2015

Responsible people thrive on freedomand are worthy of freedom

Why do most companies curtail freedom and become bureaucratic as they grow?

Desire for Bigger Positive Impact Creates Growth

Growth

Growth Increases Complexity

Complexity

Growth Also Often Shrinks

Talent Density

% High Performance Employees

Growth + Complexity

Chaos Emerges

% High Performance Employees

Complexity

CHAOS

Process Emerges to Stop the Chaos

Process

Process-focus Drives More Talent Out

% High Performance Employees

Process

Then the market shifts…

The Key: Increase Talent Density Faster Than Complexity Grows

% High Performance Employees

Business Complexity

The Binary Dependency Choice

[1] Much like Scala itself, the SBT logo has gone through at least 3 iterations

[1]

Freedom begets responsibility?

My freedom

The responsibility costof my freedom to othersin my organization

“The correct amount of constraint enables creativity more than total freedom does”

Freedom begets responsibility?

My freedom

The responsibility costof my freedom to othersin my organization

Our role is to

curtail this effect

“The correct amount of constraint enables creativity more than total freedom does”

Build Tools Focus: Nebula

REDUCEBOILERPLATE

ENRICHARTIFACTMETADATA

IDENTIFY PROJECT OWNERS

COLLECT BUILD METRICS

Our gaze has shifted to dependency management

Upstream vs. DownstreamDependencies

Astrid Downstreams

Version

ocp~1

b~5

d~7

b~6

c~5

c~6

ocp~2

version=1

version=1 version=2

version=2

Skew

ocp~1

b~1 b~2 b~3 b~4

c~1 c~2

skew=0

skew=0

skew=0skew=1

skew=3

Adoption Rate

lam

bda =

1

lam

bda =

4

greater lambda = faster adoption

Degrees of Separation

ocp~1

b~1

d~1

c~1

degrees=1

degrees=1

degrees=2

Astrid Upstreams

Dependency Insight: Astrid

Release Notes

Dependency

Resolution

Upstream

Downstream

Event Store

Read Models

Version conflict resolution is the hardest problem we face

The Conflict Resolution Problem

app

guava:14

guava:14

guava:14

unit tests prove the accuracy of these systems relative to guava 14

The Conflict Resolution Problem

app

guava:16

guava:14

upgrade to a later version

guava:14

The Conflict Resolution Problem

app

guava:16

guava:16

causes evictions

guava:14

guava:16

guava:14

The Conflict Resolution Problem

app

guava:16

guava:16

these code paths need to be tested for correctness

guava:14

guava:16

guava:14

Consequences are starting to get serious…

3 Classes of Solutions

Stability Avoidance Modularization

non-intrusive intrusive

few many

scope of problems solved

intrusiveness

earlier recent

recency as a solution at Netflix

Stability solutions are stopgap measures that postpone resolving conflict

Nebula Dependency Locking is our main stability solution

Dependency Locking

1

2

3

./gradlew generateLock saveLock

./gradlew test

./gradlew commitLock

if tests pass,

Dependency Locking

{ "com.google.guava:guava": { "locked": "14.0", "requested": "14.0" }, "org.foo:foo": { "locked": "1.0", "requested": "1.0" }}

Extra artifact produced that is later used to add forces on dependencies

3 Classes of Solutions

Stability Avoidance Modularization

non-intrusive intrusive

few many

intrusiveness

earlier recent

recency as a solution at Netflix

scope of problems solved

What if everybody built more frequently?

3 Classes of Solutions

Stability Avoidance Modularization

non-intrusive intrusive

few many

intrusiveness

earlier recent

recency as a solution at Netflix

scope of problems solved

Modularization

Just-in-Time Shading

REVERSE PROXY

Revisiting Conflict Resolution

app

lib1 lib2

guava:16

guava:14

guava:14Unshaded

Shaded

lib3

guava:14

Public API Methodsapp

A

guava:16

guava:14 Unshaded

Shaded

public Multiset<String> wordCount(String word) public int totalSales(RangeSet<Integer> hourRange)

Hard if A has API methods such as

Thanks!(We are hiring)