Joachim Vanheuverzwijn aka Zoa ([email protected]) Davy Van De Moere ([email protected]) Asterisk...

Joachim Vanheuverzwijn aka Zoa ([email protected])Davy Van De Moere ([email protected])

Asterisk PerformanceAsterisk Performance building your system for performance building your system for performance

and scalabilityand scalability

Asterisk PerformanceAsterisk Performance

Performance tests Boosting performance Benchmarks * Loadbalance * Hardware, BIOS & kernel

“building your system for performance and scalability”

22 september 2004 – Asterisk Performance by Joachim Vanheuverzwijn & Davy Van De Moere - Slide 2

Performance TestsPerformance Tests

Objective Different parameters Tools Used Tests

Performance TestsObjectiveDifferent ParametersTools UsedTests


ObjectiveObjective

Generate a lot of calls, see what happens to: load, cpu% and memory.

Measure quality of the calls: Monitor using 2 phones (MOS-score) Use app_milliwatt RTP timestamps

Find a correlation between quality and load, cpu% and memory.


Performance Tests Objective Different

Parameters Tools Used Tests

Call Quality - app_milliwattCall Quality - app_milliwatt

Sends constant 1000Hz tone at 0dbm Save the stream with openphone /

app_record. Analysing with spectrum Laboratory or

audacity spectrum analyser



Audacity Spectrum AnalyserAudacity Spectrum Analyser



Call-Quality : timestampsCall-Quality : timestamps

Some packet decoders / callgenerators / softphones provide statistics for jitter and delay.

E.g.: Openphone

Touchstone suite

Ethereal

Does not seem to be good enough to measure the quality.



OpenphoneOpenphone



Ethereal – RTP statisticsEthereal – RTP statistics



Voip Quality tests: resultsVoip Quality tests: results

CPU load has nothing to do with it Soft- and IP-phones are insufficient for

testing (jitter buffer) Idle cpu% seems useful for testing

don't go under 50%



Different ParametersDifferent Parameters

Protocols (SIP / H323 / IAX2) Codecs (u/A-law, ilbc, speex, g729,…) Applications (meetme, AGI, ...) Zaptel hardware (te410p, t100p, ...) +

echo cancellers PC hardware (intel, via, amd, ...) Kernels (2.4, 2.6) IAX2 Jitter buffer Gcc / icc



Tools usedTools used

Open Source callgenerators:- SIPP- H323ac- Ohphone- CallGen323- Sipsack- Sipstone



SIPPSIPP

Can send and receive simultaneous calls. Signalling traffic generator Can only mirror RTP streams (echo), not

send RTP itself. -> can handle all codecs! SIP only. http://sipp.sourceforge.net



SIPPSIPP



h323ach323ac

http://www.peternixon.net/h323ac.shtml Not a real callgenerator, just a console

softphone capable of sending prerecorded RTP streams. (not answering calls)

- Alaw or ulaw only ?



ohphoneohphone

Console h323 softphone Able to auto answer/auto dial Support for a lot of codecs.

g723.1 gsm g711-ulaw/alaw g728 g729 speex & ilbc can be added very easilyhttp://www.openh323.org



ohphoneohphone



CallGen323 CallGen323

Can make and answer calls Support for multiple codecs Can play and ENCODE RTP streams. H323 only Alaw onlyhttp://callgen323.sourceforge.net/



SipsakSipsak

More a stresstest tool than a call generator

Not the purpose of our testshttp://sipsak.berlios.de/



SipsakSipsak



SipStoneSipStone

http://www.sipstone.org/



Tools used Tools used

Closed Source Call generators: Winsip / Win323 NetIQ 323Sim …



One King to rule them allOne King to rule them all

Asterisk: does all protocols does most of the codecs is opensource is scriptable just works


bkwbkw


TestsTests

1. Protocol vs CPU2. Codec vs CPU3. Protocol vs Bandwidth4. Codec vs Bandwidth5. Application Tests6. Zaptel Tests7. Round Up



Protocol vs. CPU - test01Protocol vs. CPU - test01

Max amount of calls, no RTP sent or received.

Both ends:Exten => s,1,Answer();Exten => s,2,Wait(1000);

1) * (ALAW) -> Sip -> * (ALAW) 2) * (ALAW) -> H323 -> * (ALAW) 3) * (ALAW) -> IAX2 -> * (ALAW)4) * (ALAW) -> Trunked IAX2 -> * (ALAW)

Performance TestsObjectiveDifferent ParametersTools UsedTests – Prot vs CPU


Test01: ResultsTest01: Results

Untweaked os:NOT ENOUGH OPEN FILES!

IAX2: max 494 channels SIP: max 246 channels H323: max 141 channels



OS tweaksOS tweaks

echo "65535" > /proc/sys/fs/file-max (add it also to /etc/sysctl.conf)

/usr/include/bits/types.h /usr/include/linux/posix_types.h: #define __FD_SETSIZE 1024

-> /usr/src/asterisk/include/…

*http://bbcr.uwaterloo.ca/~brecht/servers/openfiles.html



Test01: Results – tweaked ulimitTest01: Results – tweaked ulimit

0

10

20

30

40

50

60

70

80

90

100

1 92 183 274 365 456 547 638 729 820 911

sip

iax2

h323



Test 1: ConclusionsTest 1: Conclusions

IAX2 uses 4 times less open files than h323 and two times less than SIP.

There is a hard limit of +/- 984 connections for sip and iax2, 490 for h323.



Encoding Test:Encoding Test:

alaw * -> SIP -> * (codec X)

(no audio is sent in the other direction)

We have to repeat this for all codecs



Decoding testDecoding test

(codec X) -> protocol Y ->* (ulaw) (no audio is sent in the other direction)

Exten => s,1,record(‘/dev/null/blabla.raw’)



Encoding vs decodingEncoding vs decoding

The purpose was to test both encoding and decoding seperately, but * might always decode even if it doesn’t need the stream decodes.

In practice both are used together, we decided to test them together.



Encoding + Decoding test (test02)Encoding + Decoding test (test02)

SIP(codec X) ↔ * (ulaw)

Sending server:Exten => 1,1,echo;

Accepting (tested) server:Exten => 1,1,Playfile(test.pcm)

5 seconds between 2 calls.



Encoding – decoding : MovieEncoding – decoding : Movie

Belgian Movie Day



Battle of the TitansBattle of the Titans

ROUND 1: Presented by twisted



Test02: Protocol ResultsTest02: Protocol Results

protocol comparison

0

20

40

60

80

100

1 19 37 55 73 91 109 127 145 163 181 199

# encodings

idle

CP

U %

iax2 / g729A

sip / g729A

sip / alaw

iax2 / alaw

h323 / g729

h323 / alaw

iax2 trunking / g729

iax2 trunking / alaw



Test02: Protocol ResultsTest02: Protocol Results

protocol comparison

0

20

40

60

80

100

1 3 5 7 9 11 13

# encodings

idle

CP

U %

iax2 / g729A

sip / g729A

h323 / g729

iax2 trunking /g729a



Round up: CPU vs Protocol Round up: CPU vs Protocol

Protocol influence on CPU is minimal using an CPU intense codec.

When low cpu codec is used, SIP/H323 is 50% ‘faster’ than IAX2.

IAX2 trunking is more than twice as fast as non trunking IAX2.



We have a winnerWe have a winner

Trunked IAX2 wherever possible SIP is your alternative H323 IS EVIL!



Some thoughts – On protocol testsSome thoughts – On protocol tests

Most codecs are floating point, 64 bit processors are faster with floating point.“Floating point and SIMD operations (SSE, SSE2) profit from 64-bit processing. A 64-bit processor can natively calculate the important 64-bit floating point format ("double precision" - precise up to 15 decimal places) and is therefore faster - this is the main reason why 64-bit processors take the lead in the

floating point benchmarks.”* Asterisk “show translations” table gets build

while starting asterisk, results are variable and incorrect for smp systems.

*http://www6.tomshardware.com/cpu/20030422/opteron-15.html



Test 2: All codec resultsTest 2: All codec results


codec vs idle cpu%

0

20

40

60

80

100

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88

# encodings

idle

CP

U %

g729A

iLBC

speex

g726

lpc10

gsm

alaw

none

Performance TestsObjectiveDifferent ParametersTools UsedTests – Codec vs CPU

Test 2: detail high cpu codecTest 2: detail high cpu codec


codec vs idle cpu%

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

# encodings

idle

CP

U %

g729A

iLBC

speex

g726

lpc10

gsm


IAX2 vs SIP / h323 RTPIAX2 vs SIP / h323 RTP

IAX2 is not using seperate ports for RTP

Advantage: - NAT

- QOS

Disadvantage: - (QOS)

- No signalling control



Codec Bandwidth – IAX2Codec Bandwidth – IAX2

g711: n.(150.0)kbps / n.100pps GSM: n.(50.78)kbps / n.100pps lpc10: n.(29.88)kbps / n.90pps g726: n.(87.50)kbps / n.100pps g729A: n.(40,62)kbps / n.100pps iLBC: n.(44.84)kbps / n.70pps Speex: n.(55.47)kbps / n.100pps



Codec Bandwidth – trunked IAX2 Codec Bandwidth – trunked IAX2

g711: n.2+14,84 + (n.2.8.1000.rate/1024)+(64*n) kbit/s

GSM: n.2+14,84 + (n.2.8.1000.rate/1024)+(13*n) kbit/s

lpc10: n.2+14,84 + (n.2.8.1000.rate/1024)+(2,5*n) kbit/s

g726: n.2+14,84 + (n.2.8.1000.rate/1024)+(16*n) kbit/s

g729A: n.2+14,84 + (n.2.8.1000.rate/1024)+(8*n) kbit/s

iLBC: n.2+14,84 + (n.2.8.1000.rate/1024)+(9*n) kbit/s

g723.1: n.2+14,84 + (n.2.8.1000.rate/1024)+(6,1*n) kbit/s

Speex: n.2+14,84 + (n.2.8.1000.rate/1024)+(?*n) kbit/s

* http://www.convergence.com.pk/iax2/trunked.html



Codec Bandwidth – trunked IAX2 Codec Bandwidth – trunked IAX2

iax2 bandwidth usage - Trunked

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193

# calls

kbit

/s

alaw

g729

gsm

ilbc

g726

lpc10

speex



Codec Bandwidth – SIPCodec Bandwidth – SIP

g711: n.(156.2)kbps / n.100pps GSM: n.(156.2)kbps / n.100pps Lpc10: Not compatible g726: n.(93.75)kbps / n.100pps g729A: n.(46,88)kbps / n.100pps iLBC: n.(42-49)kbps / n.60-70pps Speex: n.(61.72)kbps / n.100pps



Codec BW: h323Codec BW: h323

g711: I gave up GSM: I couldn't take it anymore Lpc10: Gave up, I did g726: Didn't feel well g729A: n.(46.88)kbps / n.100pps iLBC: There is no god Speex: My laptop learned to fly



Codec ChoiceCodec Choice

Choose sitting on that couch watching mind-numbing, spirit-crushing game shows, stuffing fucking junk food into your mouth. Choose rotting away at the end of it all, pissing your last in a miserable home, nothing more than an embarrassment to the selfish, fucked up brats you spawned to replace yourselves. Choose your future. Choose life... Choose GSM.



Codec Choices: ResultsCodec Choices: Results

Choose g711, GSM or g729. (based on bandwidth and your bankaccount.)

All other codecs have either bad quality or are too cpu expensive.



Protocol RecommendationsProtocol Recommendations

Use :

- SIP on your LAN network.

- IAX2 for Inter asterisk (expensive) internet connections

- IAX2 when using NAT.

- H323 IS EVIL!



Codec Choice:Codec Choice:

Compatibility uLAW or ALAW CPU Bandwidth



Performance tricksPerformance tricks

Extensions.conf optimizations

- avoid unneeded transcodings (playback / moh etc)

- avoid perl / php agi’s

- res_perl

- res_sqlite

- compress extensions.conf


Avoid transcodingAvoid transcoding

make sure your sound files exist in every format. (sox & app_record are your friends).

try not to use mp3 for music on hold



Optimize your dialplan Make sure no swap memory is used Use nice(r) when running other applications Don’t use hyperthreading on 2.4 kernels +

use hyperthreading on 2.6 kernels.



Avoid swapping Get plenty of memory

Avoid mysql on the same server



Nice –n 19 “application” E.g. when doing compression of .wav

to .mp3 with the hangup extension. Make sure no updatedb or similar scripts are

in your cron server. E100 uses less cpu than via rhine. No firewall or traffic shaper on the * machine AVOID ALL DISK ACTIVITY !


Perl / php agi’sPerl / php agi’s

Although very nice and very helpfull, its not recommended to use it when doing a lot of simultaneous calls.

Perl gets loaded every time it gets called.

Try to use C programs, res_perl or extension logic instead.



Embedded SQL server.

Although the extensions file is quite fast, when you have thousands of extensions, a reload might kill your box.

SQLite (anthm) should be a lot faster.http://bugs.digium.com/bug_view_page.php?bug_id=0002384


Quality TricksQuality Tricks

Don’t use wget or ftp or anything else using up max bandwidth without the traffic being shaped. (Even if its taken another route, it might hog your networkcard).

Avoid big data movements on your harddisk due to backups etc.

Don’t go over 50% cpu usage.


Kernel 2.4 vs 2.6Kernel 2.4 vs 2.6

0

10

20

30

40

50

60

70

80

90

100

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 191

iLBC - 2.6 single

iLBC - 2.6 smp HT

iLBC - 2.4 SMP


Kernel 2.4 – 2.6 ?Kernel 2.4 – 2.6 ?

I think kernel 2.6 is faster, and has already proven its stability. (zaptel is stable, ztdummy doesnt even need usb).

If you need HT, go for 2.6


HT or No HT: Problems testingHT or No HT: Problems testing

all three servers with HT capabilities we did test on, were dual xeons.

If I disable HT in the bios, only 1 CPU is found instead of 2.

Kernel 2.4 does not allow disabling HT Kernel 2.6 setting doesn't deactivate it,

just uses a different scheduler for it.


HT or no HTHT or no HT

We tested: Dual xeon on 2.4 / 2.6 with ht disabled / (1

cpu) / enabled in bios (1|4 cpu's) Dual xeon with smp disabled in 2.4 and 2.6

(1 cpu)


SMP ?SMP ?

A dual Xeon will always be less fast than 2 single Xeons, but will cost more.

Therefore I recommend to use a single PIV or Xeon unless doing cpu intense codec translations for a 4 span pri board.


Kernel vs Arch vs SMP vs HTKernel vs Arch vs SMP vs HT

Personal opinion: use single or dual Xeons (no quad boards) use kernel 2.6 use HyperThreading on 1 cpu, disable on 2

cpu's use preemptive kernel don't use 64 bit yet


Load BalancingLoad Balancing

SIP -> SER IAX2 -> native bridging / transfers H323 -> gnugk


Asterisk Load Balancing - SIPAsterisk Load Balancing - SIP

[SIP to SIP] SER

[SIP to POTS] SER dns round robin

[POTS to SIP] SER


Asterisk Load Balancing – iax2Asterisk Load Balancing – iax2

[IAX2 to IAX2] Where is that guy registered ??? Use one/several registration only server(s) On the registration server, know on what other server the

other guy is registered. (enforce it) [IAX2 to POTS]

Use another IAX2 server to distribute the calls over your te410p farm. Think random, think app_random.

[POTS to IAX2] Where is that guy registered ??? Use one/several registration only server(s) (enforce it)


Asterisk Load Balancing – h323Asterisk Load Balancing – h323

Think again !


What we didn’t test (yet)What we didn’t test (yet)

Registration rates Jitter buffer Call setup rates Zaptel + echo cancelling AGI Large extensions.conf reloading 64bit vs 32bit Memory / HD Speed Opteron vs Xeon vs Pentium IV


Suggestions for the futureSuggestions for the future

Inter Asterisk management interface.

(to allow easier clustering) Extensions.conf reloading: load the

extensions in a new memory block, then change pointers

Iax2 optimization


DISCLAIMERDISCLAIMER

Don't try this at home. These stunts are performed by professionals, under the strict supervision of a h323 psychiatrist.

No Animals were harmed while testing

Be sure to check out:‘Using Asterisk in a Large Scale Carrier Environment’ [email protected]


Joachim Vanheuverzwijn aka Zoa ([email protected]) Davy Van De Moere ([email protected]) Asterisk...

Documents

Transcript of Joachim Vanheuverzwijn aka Zoa ([email protected]) Davy Van De Moere ([email protected]) Asterisk...