1

Advancing Metrics on the Standards TrackEquivalence Procedure & Lab Resultsdraft-morton-ippm-advance-metrics-02

Al Morton Nov 2010ACK: Len Ciavattone for NetProbe

2

Outline Continue Definition-centric metric

advancement Procedure to determine Threshold of

Equivalence Key substitution for Interoperability

Possible Report Template with Procedures Supports the Protocol Action Request

Open call for participation in Lab tests

3

Definition-Centric Process (revised) ,---. / \ ( Start ) \ / Implementations `-+-' +-------+ | /| 1 `. +---+----+ / +-------+ `.-----------+ ,-------. | RFC | / |Check for | ,' was RFC `. YES | | / |Equivalence..... clause x -------+ | |/ +-------+ |under | `. clear? ,' | | Metric \.....| 2 ....relevant | `---+---' +----+---+ | Metric |\ +-------+ |identical | No | |Report | | Metric | \ |network | +--+----+ |results+| | ... | \ |conditions | |Modify | |Advance | | | \ +-------+ | | |Spec +----+RFC | +--------+ \| n |.'+-----------+ +-------+ |request?| +-------+ +--------+

4

Key Points (the sub-points) Start with an RFC

Focus on a specific clause Run test(s) with Implementations

Test plan is customized to a specific clause Evaluate Measurements & Compare

Expected measurement results are Clear Obvious place to take action if text is found to be

ambiguous Final state is Report Dev. for Protocol Action Req.

5

Setting the Equivalence Threshold Proposal – part of the allowable error (or

variability) is both test configuration and implementation dependent Maximum_Error = Same_Implementation_Error + Systematic_Error Only Systematic_Error need be agreed a priori Systematic_Error should be set such that unexpected

implementation differences are detectable <<= KEY At least, the variability in results from the Same

Implementation should be allowable.

6

Section 3.4 One-way Delay, ADK Sample Metric (Lab) 1. Configure a path with X ms one-way constant delay. 2. Measure a sample of one-way delay singletons with

2 or more implementations, using identical options. 3. Measure a sample of one-way delay singletons with

additional instances of the *same* implementations, using identical options, noting that connectivity differences MUST be the

same as for the *cross* implementation testing.

7

Test Set-up (6 mcast streams from MS-1 to all others, OW meas at MS-2)

Network Emulator

NetProbe MS-1

NetProbe MS-2

NetProbe MS-3

NetProbe MS-4

eth2 10.201.0.254

eth3 10.202.0.254

eth4 10.203.0.254

eth5 10.204.0.254

negotiated 100baseTx-FD

10.201.0.1

10.202.0.1

10.203.0.1

10.204.0.1

ManagementLAN

Test LANs

8

Section 3.4 One-way Delay, ADK Sample Metric (Lab) --- Results

k- sample Anderson Darling Test for batch equivalence (ADK < ADC for same population)ADK 76.109 77.332 77.438 77.438 77.319ADC (a = 0.05) 2.479 2.479 2.479 2.479 2.479ADC (a = 0.025) 3.050 3.050 3.050 3.050 3.050ADC (a = 0.01) 3.830 3.830 3.830 3.830 3.830Same Population ?(a=0.025) NO NO NO NO NOModified CV DataADK Same Population ?(a=0.025)

Agate Statistical Analysis Program

9

Section 3.4 One-way Delay, ADK Sample Metric (Lab) --- Results

0

1

2

3

4

100000 102000 104000 106000 1080000

1

2

3

4

100000 102000 104000 106000 108000

10

One-way Delay, ADK Sample Metric (Lab) --- Normalized Results

k- sample Anderson Darling Test for batch equivalenceADK 2.248ADC (a = 0.05) 2.479ADC (a = 0.025) 3.050ADC (a = 0.01) 3.830Same Population ?(a=0.025) YESModified CV DataADK Same Population ?(a=0.025)

0

1

2

3

4

0 200 400 600 800 1000

11

Notes on Network Emulator Correlated Loss Generation Many emulators use this relationship:Corr_value = Last_value * corr_coeff + New_value * (1-corr_coeff) Works adequately for Delay Tests with NIST Net indicate some shortcomings

Possible that this function was lost in revisions Also Possible that our installation has this shortcoming

Investigating these possibilities and alternative, yet simple correlation functions

12

3.2: First-bit to Last-bit, NetProbe

1400 byte payload 32 byte payload Delay for each mode (one mode) Delay Diff Expected Diff microseconds microseconds microseconds microseconds

1001621 1000356 1265 1094.4 1002735 1000356 2379 1094.4

TxDelay(us)

1000500

1001000

1001500

1002000

1002500

1003000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58

13

Update on the Bi-modal Delay Additional investigation appears to conclude that the modal

behavior is related to interrupt-to-frame arrival settings of the specific interface board.

Various options appear to be configurable, but only when the interface driver is compiled as a module.

Also, the board/driver does not support the "coalesce" options of ethtool.

Until we can rebuild the Linux machine with this and other planned modifications, confirmation will have to wait.

14

Summary Key clauses of RFC 2679 evaluated in Lab Proposal for Equivalence Threshold and

Procedure Network Emulator Testing – Loss Correlation One Implementation: NetProbe

Other volunteers? P/O This draft could become basis for the

Protocol Action Request for RFC 2679 draft-morton-ippm-advance-metrics-02

What other RFCs should we target first?

15

NetProbe 5.8.5 Runs on Solaris (and Linux, occasionally) Pre-dates *WAMP, functionally similar Software-based packet generator Provides performance measurements

including Loss, Delay, PDV, Reordering, Duplication, burst loss, etc. in post-processing on stored packet records

16

Report Template – Loss Threshold 3.1. One-way Delay, Loss threshold, RFC

2679 (procedure) 3.1.1. NetProbe Lab results for Loss Threshold 3.1.2. XXX Lab Results for Loss Threshold

<your compliant implementation here> 3.1.3. Conclusions on Lab Results for Loss

Threshold

17

Section 3.1 – Loss Threshold See Section 3.5 of [RFC2679], 3rd bullet point and also

Section 3.8.2 of [RFC2679]. 1. configure a path with 1 sec one-way constant delay 2. measure (average) one-way delay with 2 or more

implementations, using identical waiting time thresholds for loss set at 2 seconds

3. configure the path with 3 sec one-way delay (or change the delay while test is in progress, measurements in step 2)

4. repeat measurements 5. observe that the increase measured in step 4 caused all

packets to be declared lost, and that all packets that arrive successfully in step 2 are assigned a valid one-way delay.

Results: 22 of 38 packets were declared lost (post-process).

18

Section 3.2: First-bit to Last-bitSee Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330].

1. configure a path with 1000 ms one-way constant delay, and ideally including a low-speed link (10-baseT, FD)

2. measure (average) one-way delay with 2 or more implementations, using identical options and equal size small packets (e.g., 32 octet IP payload)

3. maintain the same path with 1000 ms one-way delay 4. measure (average) one-way delay with 2 or more implementations,

using identical options and equal size large packets (e.g., 1400 octet IP payload)

5. observe that the increase measured in steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.

19

3.2: First-bit to Last-bit, NetProbe

Bi-modal delay behavior in the Network Emulator for UDP payload 96 octets & up Same for Poisson, Periodic, 1 sec, 10ms spacing Not evident in tests on the Management LAN

Delay increased with larger payload, but more than expected (170 us for low mode)

Suggestions to investigate further are welcomeTests on Management LAN, 100baseTx-FD 1400 byte payload 32 byte payload Ave Delay Ave Delay Diff Expected Diff microseconds microseconds microseconds microseconds

443.37 143.57 299.8 109.44 (190.36)

20

Other Examples One-way Delay, RFC 2679

This test is intended to evaluate measurements in sections 3 and 4 of [RFC2679].

Average delays before/after 2 second increase

Average pre-increase delay, microseconds 1000276.6 Average post 2s additional, microseconds 3000282.6 Difference (should be ~= Y = 2s) 2000006

Error Calibration, RFC 2679 This is a simple check to determine if an implementation reports the error

calibration as required in Section 4.8 of [RFC2679].