Wanadoo Portals

Home Communication Services

A long-lived home-grown Linux-based low-A long-lived home-grown Linux-based low-

cost leading mass-market servicecost leading mass-market service

Pierre AUBERT & Eric OLIVERI Pierre AUBERT & Eric OLIVERI

FranceTelecom/Home Communication Services/Broadband & InternetFranceTelecom/Home Communication Services/Broadband & Internet

Wanadoo PortalsWanadoo Portals


We talk about mass market We talk about mass market net net

& computer engineering& computer engineering the #1 in France

a simple web page

the Big Picture :

pure* Open source – Linux – PC hardware

some thoughts about the past and the future

works well, costs little delivers more from less


The #1 portals in FranceThe #1 portals in France wanadoo.fr – voila.fr – MMGs – IM/IRC…

65% reach – 13 M unique visitors

80 M pages to 4.2 M customers daily

Every day’s peaks : 150 000 simultaneous users 1 Gb/s Internet content 5 000 new users/mn 100 000 web pages/mn transport conn/s, forks/s, SELECTs/s…




a simple web page

the Big Picture :





A simple web page …A simple web page …


……, but …, but …Mathopd

Apache/C++

Apache/Oracle Solaris/Linux

Apache/PHP/mySQL

IIS/ASP/SQLsvr

TUX

LVSembedded firewalls




a simple web page

the Big Picture :




* well, mostly


Not only a web serverNot only a web server Front office

Network* – Load Balancers* – Firewalls – DNS Web servers : Apache/Mathopd/TUX, mostly PHP Databases : mySQL, postgreSQL

Back office Publishing – Content management Audience collection Databases**

Supervision, backups, etc.

* sometimes open source/Linux/PC hardware** most often non-open source/non-Linux/non-PC hardware


DNS

databasescontent

management

audiencecollection

supervision

backups

webfarms

load-balcers

+ firewalls+ routers

redirect

N x FRONTSERVICES

databasesother servicesIMAP, HTTP…

VPN

BACKSERVICES


3 ways to deliver Web service Apache for dynamic content

Mostly PHP, some C++, some SSI 20 M pages for 2.5 M users daily 12 servers : Bi-CPU Intel, type 1 – 60 Mb/s

Mathopd for static content Maximize browser cache effect 2000 HTTP/second 6 servers : 1-CPU Intel, type 0 – 25 Mb/s

TUX for redirection service Built-in Linux kernel 2500 HTTP/second – 2 Mb/s 3 servers (1 is enough) : 1-CPU Intel, type -1 ;–)


Some other services Cookies (identity, tracking…)

1250 HTTP/sec 4 servers : 1-CPU Intel type 0 – 10 Mb/s

Databases : authentication 1000 reads/mn – 150 simultaneous connections 1 server 1-CPU 1G RAM, type 1

Databases : search engine crawler 500 M docs – 4 Tb – 100k writes/mn 12 servers 2-CPU 5x72Gb each, type 1

WAN-LAN DNS : 4 servers 1-CPU Intel type 0 – Zebra (BGP/OSPF) Firewall + load-balance appliance : 1-CPU Intel, type -1

Tricks : lingerd, nscd, multicast… tuning : 20 000 sockets per IRC server


Supervision etc. 60+ parameters per server on each of 1000 servers

Net-SNMP, big brother/big sister, Perl… ad hoc monitoring (~ 30%) : 1 h to develop and deploy 20 servers 1-CPU Intel, type 1

At-the-fly graph generation : 50 000 parameters 300+ samples/s – 3 servers Bi-CPU Intel, type 1 RRD-Tool + SNMP managers (Perl) + Apache Env – Network – from HW&CPU to application internals

Backups, storage Amanda’s self-planning, no routine restore 1/10th FTE 2 servers P2-400MHz 2,4To (2€/GO/y)– cheaper than tape : (6€/GO/y)

backs up daily 300 servers




a simple web page

the Big Picture :




* well, mostly


Lessons from the past Choose what fits best

1996 : from FreeBSD to Solaris 2003 : from Solaris to Linux voila.fr launched 1998 : Apache/PHP/Linux

Service first system administrators are front-line

diagnose and fix fast (or alternate service/systems) end-user satisfaction = systems health

accounting transparency (CAPEX & OPEX per server/day)


Lessons from the past PC hardware is a key factor

Failures are facts not problems « Small is beautiful » : cost of redundancy is 1/N Competitive mass market sourcing

PC operating system is hw-independent Reuse – assemble components

Open source OS is a key factor Best use of hardware resources Flexible kernels – Excellent network support

Application level Open source platforms – Internet applications Interoperability is in DNA


Lessons from the past Design – skills

Not a sequential « product » processExpert staff allow short & long term savings

EfficiencyLow-range (90%) = 3 to 5 k€/year

High-end PC servers cost 5+ times moreMid-range Unix servers cost 20+ times more

Numerous distributed small systemsscale at front officekeep back office + infra stabledistribute bandwidth, IO buffers, axis…




a simple web page

the Big Picture :




* well, mostly


Costs little Some figures

1 system administrator for 40 servers1 low-range server = 3 to 5 k€/year OPEXPer unique visitor : 1 €/yearPer 1k web pages : 0.5 €

margin cost = 2 c€ OPEX, 3 c€ CAPEX

Cost breakdown – OPEX

15% shell & core15% hardware15% bandwidth 5% maintenance/support50% man power


Costs little = « works » well

Home page Solaris12 Sun 4-CPU : 6 dynamic + 6 static content400 k€/year OPEX (amort. included)Additional CAPEX : 10 € per 1k daily pages

Home page Linux18 PC 1&2-CPU : 12 dynamic + 6 static content100 k€/year OPEX (amort. included)Additional CAPEX : 3 € per 1k daily pages works well


Works well

Voila.fr ranked* #1, Wanadoo.fr ranked* #3(*) Google’s home page always #1… but that’s not a real portal home page

Both fastest real portal home pages

Unavailability mostly due to a sub-service :freeware expensive : fees + maintce + poor supportOpen source proprietary code : can’t fix

This sub-service delivers less from more;–)


Next Steps

u Convince top management that Open-Source solutions can replace proprietary one!

u Most agree on web front office (1-tier)u Most disagree on 2nd tier: Jonas,Jboss / Websphere, 9iASu All disagree on DB: MySQL, PostgreSQL / Oracle, DB2, SQLs

u Carefully select an appropriated database for your needsu Deploy Oracle on PC + Linux+ attached disks (Instead of Sun + Solaris + SAN or NAS :)


Thank youThank you

Some Open Source Oriented Question ?Some Open Source Oriented Question ?

[email protected]

[email protected]

Wanadoo Portals

Documents

Transcript of Wanadoo Portals