2nd SESERV workshop - Internet Market Failures: Technological Causes and Solutions - Bob Briscoe

Internet Market Failures:Technological Causes and SolutionsBob BriscoeChief Researcher, BT

Jan 2012

This work was partly funded by Trilogy, a research project supported by the European Community www.trilogy-project.org

http://www.trilogy-project.org/

© British Telecommunications plc

Traffic Growth• to meet conservative traffic growth forecasts with conservative cost improvements

• traffic growth: fixed 35% pa, mobile 74% pa• cost efficiency improvements: 15% pa

• EU fixed & mobile operators would need €30B CapEx 2011–2014* • in addition to €73B CapEx expected using 2006–2010 trend [ATKearney]

• expected revenues only maintain the CapEx trend, not the extra• most traffic growth only commodity (over-the-top) services

*of course, the problem continues to get worse beyond 2014

alternative outcomes?• return on capital employed falls for all

operators (12% 9%)• capital leaves telecoms sector

• traffic management somehow identifies 50% least valuable videos• vendors admit deep packet

inspection cannot do this

• surely the market will sort this out...?

Cumulative €9.8Bn

Estimated Capex required to fund incremental capacityfor European fixed Internet networks


Surely the market will sort this out?

• won’t the price per byte rise to suppress demand? • no, faith in the market is misplaced for the Internet

• the problem is buried under two levels of confusion1. the wrong metric

2. the right metric is only selectively visible


#1. the wrong metric

• volume in bytes doesn’t represent cost of usage• raising the price per byte will suppress bytes, not cost

the right metric• marginal* cost of usage = contribution to congestion

• bytes are only half the story• need bytes weighted by instantaneous congestion level

• represents contribution to the cost others experience (degraded QoE)• equivalently, cost of capacity upgrade to alleviate the congestion

* plus subscription element to cover fixed costs


measuring contribution to congestion

• user’s contribution to congestion= bytes weighted by congestion level= bytes dropped (or marked)= ‘congestion-volume’

• as simple to measure as volume

bit-rate

time

congestion

time

10GB

0.01% congestion

1MB

1% congestion

1MB

300MB100MB

3MB

1%

0.01%


#2. right metric is only selectively visible

• in the Internet architecture...

• end hosts detect congestion as gaps in sequence spaces• not necessarily visible by network operator

• sequence numbers in transport layer

• potentially encrypted payload of network layer

• gaps may be filled by packets on other paths

• network nodes know what they drop locally

• but a traffic management node • cannot see what other nodes have dropped,

especially in other networks

• cannot collect up each customer’s contribution to congestion

• network operator cannot control what it cannot see


menu

• problems with cost metrics for a shared network

• relation to economic theory of market failure

• technical solution

• multiple types of market failure

• summary


8

Sharing: Lest we Forget

• IP, Ethernet, MPLS are all designed to share a network

• sharing is central to all developments in consumer access*• because largest element of costs is geographical dispersion

• for dedicated consumer access, utilisation as speed • average utilisation only 0.5% for a 40M dedicated access• was 1.25% for 4M access; still higher in dial-up days

• cost efficiency is driving sharing closer to the end-user• that’s why

ADSL VDSL, passive optical networks, cable, WiFi, cellular

Inte

rne

t to

po

log

y vi

sua

liza

tion

pro

du

ced

by

Wa

lrus

(C

ourt

esy

of

Yo

ung

Hyu

n,

CA

IDA

)

* and obviously core, campus & enterprise networks too

http://www.caida.org/


what economic theory tells us

• shared Internet bandwidth: ‘use-up-able’ and non-excludable• a common good

• free-riding typically reduces the incentive to supply

• common goods tend to be under-supplied and over-consumed

• common goods suffer from what is called a market failure• when revenue does not rise to fund extra ~€30B of capacity in Europe

• the market will not “sort this out”

prevent non-contributors benefiting?

excludable non-excludable

use by A prevents simultaneous use

by B?

rivalrous(‘use-up-able’)

private goodsfood, clothing, toys, furniture, cars

common goodsfish, hunting game, water

non-rivalrous(irreducible)

club goodssatellite television

public goodsnational defense, free-to-air TV, air, published info

free-riding problems

tragedy of the commons

problems


Market failure

“when a free marketdoes not allocate goods and services efficiently”

• non-excludability is one of the causes of market failure

price supply

demand

shared capacity


the wrong metricmarket failure as a consequence

• operators suppress bytes, not cost• using volume limits or charges

• punishes cost sensitive protocols, eg:• BitTorrent uTP (micro transport protocol)

IETF LEDBAT (Low Extra Delay Background Transport)*• EQS Equitable Quality Streaming [Crabtree09]

– e.g. EQS can halve capacity needed for video streaming– but ‘fair’ queuing stops it working – ironic as EQS is fairer

• operators cannot limit congestion when necessary

• capacity is under-supplied and over-consumed

price supply

demand

shared capacity

* uTP & LEDBAT are actually designed to only yield to self-congestion, because operators do not reward yielding to others


Potential Consequences of Market Failure

• congestion and poor quality• “under-supplied and over-consumed”

• random slaughter• operator can’t target costly usage, so it picks on traffic it likes

least until cost reduced enough• infrastructure no longer an engine for innovation

• vertical integration• Internet service subsidised by value-added services• However, can never compete with over-the-top services while

also subsidising their bandwidth• resort to government subsidy and control

• or someone averts the market failure...

price supply

demand

shared capacity


(Just) Change the Internet Protocol

• the challenge

• shift IP to the private goods model

• keep traditional freedom to share everything

excludable non-excludable

rivalrous(‘use-up-able’)

private goods common goods IP

non-rivalrous(irreducible)

club goods public goods


14

IETF Congestion Exposure (ConEx)basic signals and functional units

transportsender

transportreceiver

congestednetwork element

DATA

ACKS

congestion feedback

ConEx markingsin the IP layerIETF working-group in progress

congestionloss/marks

new with ConEx

existing


15

Congestion Exposure (ConEx)functions built on basic signals

transportsender

transportreceiver


policy auditDATA

ACKS

congestion feedback

ConEx markingsin the IP layer


new with ConEx

existing


policy • only throttles traffic when your contribution to congestion anywhere in the Internet exceeds your allowance

• incentive to avoid congestion

• only throttles traffic when your contribution to congestion anywhere in the Internet exceeds your allowance

• incentive to avoid congestion

ConEx-based policy function exampleflat-fee congestion-policing

bulkcongestion

policer

Internet

0.3%congestion

0%

0.1%

2 Mb/s0.3Mb/s6 Mb/s

Acceptable Use Policy

contribution to congestion allowance: 1GB/month

@ €15/month

Allows ~70GB per day of data in typical conditions

ConEx makes many policy actions possible • downgrade class of service• re-route• report to management system, e.g. for charging• or just ignore


Market Failure – not one, but many

1. non-excludability2. information asymmetry

A. consumer-providerB. provider-provider

3. externalities A. (-) congestionB. (+) network effects

4. transaction costs5. market power6. switching costs

the Internet suffers from them all…


information asymmetry

• Classic example: info about quality of used cars [Akerlof71]

• seller knows which cars are ‘lemons’ (poor quality)• buyer doesn’t, but knows chance of getting a lemon• buyer suppresses average willingness to pay in case • demand falls, supply falls, market failure• Nobel Prize in Economics, 2001

a) between network consumer & provider• TCP/IP exhibits a weird reversal of the same problem• cost/quality more visible to end-host than network• same outcome: market failure [Briscoe08]

b) between networks• sending net takes out sub-contract with receiving net to deliver packets• but quality of delivery is invisible to the principle contractor [Constan01]


information symmetry solutiona) between consumer & provider

• light usage can go much faster• hardly affects completion time of

heavy usage

• if operator can see congestionit can reward those who avoid it• equitable quality streaming (EQS) and

protocols like uTP, LEDBAT & BITSbit-rate

time

bit-rate

time

1. TCP

4. DPI + FQ

ConEx +weightedsharing

bit-rate

time

2. FQ fair queuing

bit-rate

time

3. volume cap

bit-rate

time

p2p


information asymmetry problemsb) between networks

• NetFlix vs Comcast

• gov.fr vs google.com


information symmetry solutionb) between networks

transportsender

transportreceiver

congestednetworkelement

policycontrolbased onmetering

blackminus

red

congestionmarking

ConExmarking


1. non-excludability

2. information asymmetryA. consumer-providerB. provider-provider


4. transaction costs

5. market power

6. switching costs



market power

Net Neutrality

• a multi-faceted issue• primarily an issue with the lower layer (access network) market• that has surfaced at the Internet layer in the US• (see spare slide)

focusing on Internet Layer issues…• there are concerns around who controls quality of service

• ConEx has beeen used in the US debate • to show that QoS per se must not be outlawed• because it can be user-controlled

* so far


1. non-excludability

2. information asymmetryA. consumer-providerB. provider-provider


4. transaction costs

5. market power

6. switching costs


multipath TCP (MPTCP)…another subject


Take-away Messages

• 7 types of market failure• Internet technology suffers from them all

• causes and solutions are technological – at the Internet layer

• On-track to add ConEx (and MPTCP) to IP• addressed 6 / 7 types of market failure• without losing feel of classic Internet• flat fee & freedom to share, anything anywhere

• Simple – imagine an industry that just works by simple economics• without all the strife and government intervention we are so used to• without all the complex capacity allocation & QoS mechanisms


more info

• Bob Briscoe, “Practical Microeconomics and Internet Resource Sharing Protocols”Trilogy Summer School lecture: Videos available from www.trilogy-project.org

• IETF ConEx: tools.ietf.org/wg/conex/

• Resources related to ConeX and re-feedback: bobbriscoe.net/projects/refb/

• [ATKearney] "A Viable Future Model for the Internet," Technical Report, A.T.Kearney (2010)

• [Akerlof71] Akerlof, G., "The Market for `Lemons': Quality, Uncertainty and Market Mechanisms," Quarterly Journal of Economics 84:488--500 (August 1970)

• [Constan01] Constantiou, I.D. & Courcoubetis, C.A., "Information Asymmetry Models in the Internet Connectivity Market," In: Proc. 4th Internet Economics Workshop (May 2001)

• [Briscoe08] Briscoe, B., Moncaster, T. & Burness, L., "Problem Statement: Transport Protocols Don't Have To Do Fairness," Internet Engineering Task Force Internet Draft draft-briscoe-tsvwg-relax-fairness-01 (July 2008) (Work in progress)

• [Crabtree09] B. Crabtree, M. Nilsson, P. Mulroy and S. Appleby “Equitable quality video streaming” Computer Communications and Networking Conference, Las Vegas, (January 2009)

http://www.trilogy-project.org/

http://tools.ietf.org/wg/conex/

http://tools.ietf.org/wg/conex/

http://bobbriscoe.net/projects/refb/

http://bobbriscoe.net/projects/refb/

Internet Market Failures:Technological Causes and Solutions

Q&A


28

Congestion Exposure (ConEx)basic signals and functional units

transportsender

transportreceiver


ECN

loss

SACK

ECE

Re-Echo-ECN

Re-Echo-Loss

DATA

ACKS

congestion feedback

ConEx markingsin the IP layer



market power

Net Neutrality

• Well-known that access networks are a natural monopoly

• US regulatory failure to remedy at lowest layer• resulted in issue resurfacing at the IP layer• monopoly/duopoly access provider favouring its own services

• EU: government interventions at lower layer • has averted need to intervene and distort the Internet layer*

• net neutrality is a solution at the Internet layer for a problem at the lower layer (access link)

* so far


example integrated end-to-end business model

• focused on revealing variable costs

• capacity fees can still reflect value flows

• kick-starts new e2e value flows(another talk)• otherwise advertising

is limit to growth

• note: an example• you choose how to use a cost metric• not me

NA NBND

R1S1

£$¥

€

£ $

downstreamcongestion

usagecharges

flat charges

€

¥

¥

2nd SESERV workshop - Internet Market Failures: Technological Causes and Solutions - Bob Briscoe

Technology

Transcript of 2nd SESERV workshop - Internet Market Failures: Technological Causes and Solutions - Bob Briscoe