8/14/2019 Joost Network Architecture

1/29

mailto:Colm.MacCarthaigh@joost.commailto:Colm.MacCarthaigh@joost.com

8/14/2019 Joost Network Architecture

2/29

8/14/2019 Joost Network Architecture

3/29

8/14/2019 Joost Network Architecture

4/29

Joost Network Architecture

Ground rules

No firewalls No hardware load-balancers

High availability (this is TV)

Lots of bandwidth (this is TV)

Ethernet only

Rapidly provisionable

Business requirements

Cost-effective

4 main service topologies ...

8/14/2019 Joost Network Architecture

5/29

Joost P2P Magic

Video Streaming

8/14/2019 Joost Network Architecture

6/29

Joost P2P Magic

Joost servers are the original seedersof all content

Joost servers also handle the long-

tail (which is still pretty long)

Joost servers top-up the DSLbandwidth gap

8/14/2019 Joost Network Architecture

7/29

Joost P2P Magic

A long-tail server cluster is 11 servers 1 control server

10 long-tail servers

2x Cisco 3560s (crypto image) HSRP

EBGP

802.1q

OSPF (for anycast services)

8/14/2019 Joost Network Architecture

8/29

Joost P2P Magic

Each cluster gets 1Gbit/sec of IP transit

/26 from transit provider PA space were not sensitive to re-numbering

avoids prefix-filtering and dampening

Each cluster is an island with noconnectivity back to Joost HQ apart froman IP tunnel for server lights-outmanagement

Preference is to add sites when scaling

Decreases average latencies

8/14/2019 Joost Network Architecture

9/29

Joost P2P Magic

8/14/2019 Joost Network Architecture

10/29

Joost P2P Magic

Pros Highly cost-effective for 1Gbit/sec of

usable, resilient, bandwidth

No multi-gig network scalability

issues, just add another cluster Rapidly provisionable, almost

anywhere

Cons May make future peerings pointless

3560s cant hold a full table, or doNetflow

8/14/2019 Joost Network Architecture

11/29

Joost P2P Magic

UDP-based (Port 33333)

Client will perform STUN, ICE inprogress.

TCP congestion control would kill anyvideo. Buffering usually significantlyincreases bandwidth usage.

Packets are generally 1k in size.

8/14/2019 Joost Network Architecture

12/29

Joost P2P Magic

Client first contacts super-node, whichhandles control traffic only, and directclients to peers. Peers are re-negotiated frequently.

Each video stream comes frommultiple peers, with FEC to handle livepeer loss.

1 hour of video

8/14/2019 Joost Network Architecture

13/29

8/14/2019 Joost Network Architecture

14/29

Joost P2P Magic

p2p code is prefix aware now, willprefer peers in same /24, /16 etc ...

adding real AS-level awareness is in

progress.

latency-based decisions aresomething to watch out for.

8/14/2019 Joost Network Architecture

15/29

Joost P2P Magic

30ms50ms

15msAB

C

What if asupernodeat C has tocoordinate

A and B?

This makesus highly

sensitive tolatency.

8/14/2019 Joost Network Architecture

16/29

HTTPS

Backend services (search ...)

8/14/2019 Joost Network Architecture

17/29

Joost Backend Services

Accessed via HTTPS

Provided using Apache Lucene, Hadoop,and many internally-developed services

Each IP is a wack-a-mole virtual-ip

Geographic fail-over provided by DNS

8/14/2019 Joost Network Architecture

18/29

HTTP, SMTP, DNS

General Internet Services

8/14/2019 Joost Network Architecture

19/29

Joost General Services

Joost.com website and e-mail

Content Owner Website (COW)

Provided by resilient servers withwackamole

Geographic fail-over provided by DNS

8/14/2019 Joost Network Architecture

20/29

Joost Internal Services

Recursive DNS

Syslog

Some authentication services

All provisioned via IP anycast, at each siteinclude long-tail cluster sites (the

addresses are internally reachable only).

8/14/2019 Joost Network Architecture

21/29

Super-secretTranscoding

DAV, SFTP, FASP,FEDEX

Content Ingestion

8/14/2019 Joost Network Architecture

22/29

Content Ingestion

Content has to get to Leiden

Currently investigating various networkmedia-delivery options for this

Transcoded in Leiden, and then sent toLuxembourg, and onward to all long-tailserver sites

Content-owner website for meta-data

8/14/2019 Joost Network Architecture

23/29

Leiden

AS5400

AS5400

All routing is viaLuxembourg

8/14/2019 Joost Network Architecture

24/29

Joost Benelux

Main Joost location is the Joost BeneluxNetwork

Hosts Joost Leiden office, Luxembourg

Datacentre, Primary Long-tail serversite, Primary back-end site

89.251.0.0/20 (deaggregates during

some outages)

AS42072

8/14/2019 Joost Network Architecture

25/29

8/14/2019 Joost Network Architecture

26/29

Joost Benelux

Routers are Cisco 7301 Can do a Gig with 1k packets, justabout

Full netflow support, anonymisedvia CryptoPan, for detailed analysisof p2p network performance

Using OSPF as IGP

8/14/2019 Joost Network Architecture

27/29

Joost Network Management

RT and JIRA for tickets

RANCID-SVN for config management

NAGIOS and syslog for monitoring

MRTG and Cricket for graphing

SSH only

8/14/2019 Joost Network Architecture

28/29

So, what can we do for ISPs?

Were willing to peer, but is there muchpoint? Only portions of the long-tail arepeerable.

In-ISP Long-tail servers?

Were L3 and up, not much we can do aboutthe last-mile

Any promising revenue share models?

8/14/2019 Joost Network Architecture

29/29

Any Questions?