Debugging Floating-Point Math in Racket · Finite Approximation • Actual flonum fields are...
Transcript of Debugging Floating-Point Math in Racket · Finite Approximation • Actual flonum fields are...
Debugging Floating-PointDebugging Floating-PointDebugging Floating-Point
Math in RacketMath in RacketMath in Racket
Neil Toronto
RacketCon 2013
Racket Floating-Point SupportRacket Floating-Point SupportRacket Floating-Point Support
• Fast (JIT-ed) and compliant (IEEE 754 and C99)
1111111111111111
Racket Floating-Point SupportRacket Floating-Point SupportRacket Floating-Point Support
• Fast (JIT-ed) and compliant (IEEE 754 and C99)
• More flonum (i.e. 64-bit float) functions:
math/special-functions: gamma, beta, psi,zeta, erf, etc.
math/distributions: Gamma, Normal, etc.
1111111111111111
Racket Floating-Point SupportRacket Floating-Point SupportRacket Floating-Point Support
• Fast (JIT-ed) and compliant (IEEE 754 and C99)
• More flonum (i.e. 64-bit float) functions:
math/special-functions: gamma, beta, psi,zeta, erf, etc.
math/distributions: Gamma, Normal, etc.
• Other floating-point modules:
racket/extflonum: basic 80-bit operations
math/bigfloat: arbitrary-precision floats
1111111111111111
Racket Floating-Point SupportRacket Floating-Point SupportRacket Floating-Point Support
• Fast (JIT-ed) and compliant (IEEE 754 and C99)
• More flonum (i.e. 64-bit float) functions:
math/special-functions: gamma, beta, psi,zeta, erf, etc.
math/distributions: Gamma, Normal, etc.
• Other floating-point modules:
racket/extflonum: basic 80-bit operations
math/bigfloat: arbitrary-precision floats
• math/flonum: a bunch of weird things like fl,flnext, +max.0, flonum->ordinal, fllog1p,flsqrt1pm1, flcospix
1111111111111111
You Could Have Invented Floating-PointYou Could Have Invented Floating-PointYou Could Have Invented Floating-Point
Need to represent or ...
2222222222222222
You Could Have Invented Floating-PointYou Could Have Invented Floating-PointYou Could Have Invented Floating-Point
Need to represent or ...
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
2222222222222222
You Could Have Invented Floating-PointYou Could Have Invented Floating-PointYou Could Have Invented Floating-Point
Need to represent or ...
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
(: float->real (float -> Real))(define (float->real x) (match-define (float s n m) x) (* s n (expt 2 m)))
2222222222222222
You Could Have Invented Floating-PointYou Could Have Invented Floating-PointYou Could Have Invented Floating-Point
Need to represent or ...
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
(: float->real (float -> Real))(define (float->real x) (match-define (float s n m) x) (* s n (expt 2 m)))
> (float->real (float -1 10 0))-10
2222222222222222
You Could Have Invented Floating-PointYou Could Have Invented Floating-PointYou Could Have Invented Floating-Point
Need to represent or ...
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
(: float->real (float -> Real))(define (float->real x) (match-define (float s n m) x) (* s n (expt 2 m)))
> (float->real (float -1 10 0))-10
> (float->real (float -1 10 3))-80
2222222222222222
You Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point Multiplication
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
3333333333333333
You Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point Multiplication
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
3333333333333333
You Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point Multiplication
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
(: float* (float float -> float))(define (float* x1 x2) (match-define (float s1 n1 m1) x1) (match-define (float s2 n2 m2) x2) (float (* s1 s2) (* n1 n2) (+ m1 m2)))
3333333333333333
You Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point MultiplicationYou Could Have Invented Floating-Point Multiplication
(struct: float ([sign : (U -1 1)][sig : Natural][exp : Integer]))
(: float* (float float -> float))(define (float* x1 x2) (match-define (float s1 n1 m1) x1) (match-define (float s2 n2 m2) x2) (float (* s1 s2) (* n1 n2) (+ m1 m2)))
> (float->real (float* (float -1 10 0)(float -1 10 3)))
800
3333333333333333
Finite ApproximationFinite ApproximationFinite Approximation
• Actual flonum fields are fixed-size, requiring
Rounding least significant bit after operations
Representations for overflow (i.e. +inf.0 and-inf.0) and underflow (i.e. +0.0 and -0.0)
4444444444444444
Finite ApproximationFinite ApproximationFinite Approximation
• Actual flonum fields are fixed-size, requiring
Rounding least significant bit after operations
Representations for overflow (i.e. +inf.0 and-inf.0) and underflow (i.e. +0.0 and -0.0)
• Consequence: a natural well-order over flonums
> (flnext 0.0) ; from math/flonum4.9406564584125e-324
4444444444444444
Finite ApproximationFinite ApproximationFinite Approximation
• Actual flonum fields are fixed-size, requiring
Rounding least significant bit after operations
Representations for overflow (i.e. +inf.0 and-inf.0) and underflow (i.e. +0.0 and -0.0)
• Consequence: a natural well-order over flonums
> (flnext 0.0) ; from math/flonum4.9406564584125e-324
> (list (flonum->ordinal 0.0)(flonum->ordinal +max.0))
'(0 9218868437227405311)
4444444444444444
Finite ApproximationFinite ApproximationFinite Approximation
• Actual flonum fields are fixed-size, requiring
Rounding least significant bit after operations
Representations for overflow (i.e. +inf.0 and-inf.0) and underflow (i.e. +0.0 and -0.0)
• Consequence: a natural well-order over flonums
> (flnext 0.0) ; from math/flonum4.9406564584125e-324
> (list (flonum->ordinal 0.0)(flonum->ordinal +max.0))
'(0 9218868437227405311)
> (flonums-between 0.0 1.0)4607182418800017408 ; 4.6 *quintillion*
4444444444444444
Finite ApproximationFinite ApproximationFinite Approximation
• Actual flonum fields are fixed-size, requiring
Rounding least significant bit after operations
Representations for overflow (i.e. +inf.0 and-inf.0) and underflow (i.e. +0.0 and -0.0)
• Consequence: a natural well-order over flonums
> (flnext 0.0) ; from math/flonum4.9406564584125e-324
> (list (flonum->ordinal 0.0)(flonum->ordinal +max.0))
'(0 9218868437227405311)
> (flonums-between 0.0 1.0)4607182418800017408 ; 4.6 *quintillion*
• Consequence: most flonum functions aren’t exact 4444444444444444
Correctness Means Minimizing ErrorCorrectness Means Minimizing ErrorCorrectness Means Minimizing Error
• A flonum’s unit in last place (ulp) is the distancebetween it and the next flonum
> (flulp #i355/113) ; from math/flonum4.440892098500626e-16
5555555555555555
Correctness Means Minimizing ErrorCorrectness Means Minimizing ErrorCorrectness Means Minimizing Error
• A flonum’s unit in last place (ulp) is the distancebetween it and the next flonum
> (flulp #i355/113) ; from math/flonum4.440892098500626e-16
• Error is distance from the true value, in ulps
> #i355/113 pi3.14159292035398253.141592653589793
5555555555555555
Correctness Means Minimizing ErrorCorrectness Means Minimizing ErrorCorrectness Means Minimizing Error
• A flonum’s unit in last place (ulp) is the distancebetween it and the next flonum
> (flulp #i355/113) ; from math/flonum4.440892098500626e-16
• Error is distance from the true value, in ulps
> #i355/113 pi3.14159292035398253.141592653589793
> (flulp-error #i355/113 pi)600699552.0 ; 600.7 million ulps
5555555555555555
Correctness Means Minimizing ErrorCorrectness Means Minimizing ErrorCorrectness Means Minimizing Error
• A flonum’s unit in last place (ulp) is the distancebetween it and the next flonum
> (flulp #i355/113) ; from math/flonum4.440892098500626e-16
• Error is distance from the true value, in ulps
> #i355/113 pi3.14159292035398253.141592653589793
> (flulp-error #i355/113 pi)600699552.0 ; 600.7 million ulps
• A flonum function is correctly rounded* when itsoutputs’ maximum error is no more than 0.5 ulps
5555555555555555
Correctness Means Minimizing ErrorCorrectness Means Minimizing ErrorCorrectness Means Minimizing Error
• A flonum’s unit in last place (ulp) is the distancebetween it and the next flonum
> (flulp #i355/113) ; from math/flonum4.440892098500626e-16
• Error is distance from the true value, in ulps
> #i355/113 pi3.14159292035398253.141592653589793
> (flulp-error #i355/113 pi)600699552.0 ; 600.7 million ulps
• A flonum function is correctly rounded* when itsoutputs’ maximum error is no more than 0.5 ulps
* assuming inputs are exact; i.e. no guarantees aregiven for arguments with error 5555555555555555
Correctness Example: SubtractionCorrectness Example: SubtractionCorrectness Example: Subtraction
> (plot3d (contour-intervals3d(λ (x y) (let ([x (fl x)] [y (fl y)])
(define z* (- (inexact->exact x)(inexact->exact y)))
(flulp-error (fl- x y) z*)))-1 1 -1 1))
6666666666666666
Correctness Example: SubtractionCorrectness Example: SubtractionCorrectness Example: Subtraction
> (plot3d (contour-intervals3d(λ (x y) (let ([x (fl x)] [y (fl y)])
(define z* (- (inexact->exact x)(inexact->exact y)))
(flulp-error (fl- x y) z*)))-1 1 -1 1))
000000000
.1.1.1.1.1.1.1.1.1
.2.2.2.2.2.2.2.2.2
.3.3.3.3.3.3.3.3.3
.4.4.4.4.4.4.4.4.4
.5.5.5.5.5.5.5.5.5
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y axis
y axis
y axisy axis
y axisy axis
y axis
y axis
y axis
6666666666666666
Correctness Example: LogarithmCorrectness Example: LogarithmCorrectness Example: Logarithm
> (require math/bigfloat) ; default sig. size: 128 bits> (plot (function
(λ (x) (let ([x (fl x)]) (define z* (bigfloat->real (bflog (bf x)))) (flulp-error (fllog x) z*)))
0 10))
7777777777777777
Correctness Example: LogarithmCorrectness Example: LogarithmCorrectness Example: Logarithm
> (require math/bigfloat) ; default sig. size: 128 bits> (plot (function
(λ (x) (let ([x (fl x)]) (define z* (bigfloat->real (bflog (bf x)))) (flulp-error (fllog x) z*)))
0 10))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 222222222 444444444 666666666 888888888 101010101010101010000000000
.1.1.1.1.1.1.1.1.1
.2.2.2.2.2.2.2.2.2
.3.3.3.3.3.3.3.3.3
.4.4.4.4.4.4.4.4.4
7777777777777777
Correctness is NoncompositionalCorrectness is NoncompositionalCorrectness is Noncompositional
> (plot (function(λ (x) (define z* (bigfloat->real (bflog (bf x)))) (flulp-error (fllog (fl x)) z*)))
0 10)
8888888888888888
Correctness is NoncompositionalCorrectness is NoncompositionalCorrectness is Noncompositional
> (plot (function(λ (x) (define z* (bigfloat->real (bflog (bf x)))) (flulp-error (fllog (fl x)) z*)))
0 10)
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 222222222 444444444 666666666 888888888 101010101010101010000000000
505050505050505050
100100100100100100100100100
150150150150150150150150150
200200200200200200200200200
250250250250250250250250250
8888888888888888
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
Implement for
9999999999999999
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
Implement for
• First stab:
(define (geom p u)(fl/ (fllog u) (fllog (fl- 1.0 p))))
9999999999999999
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
Implement for
• First stab:
(define (geom p u)(fl/ (fllog u) (fllog (fl- 1.0 p))))
• Reference implementation:
(define (geom* p u) (let ([p (bf p)] [u (bf u)]) (bigfloat->real
(bf/ (bflog u) (bflog (bf- 1.bf p))))))
9999999999999999
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
• Error plot for geom for p <= 1:
000000000
222222222
444444444
666666666
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
ppppppppp
uuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
10101010101010101010101010101010
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
• Error plot for geom for p <= 0.1:
000000000
505050505050505050
100100100100100100100100100
150150150150150150150150150
.1.1.1.1.1.1.1.1.1
.08.08.08.08.08.08.08.08.08
.06.06.06.06.06.06.06.06.06
.04.04.04.04.04.04.04.04.04
.02.02.02.02.02.02.02.02.02
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
ppppppppp
uuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
10101010101010101010101010101010
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
• Error plot for geom for p <= 1e-05:
000000000
250000250000250000250000250000250000250000250000250000
5×10⁵5×10⁵5×10⁵5×10⁵5×10⁵5×10⁵5×10⁵5×10⁵5×10⁵
750000750000750000750000750000750000750000750000750000
1×10⁶1×10⁶1×10⁶1×10⁶1×10⁶1×10⁶1×10⁶1×10⁶1×10⁶
.00001.00001.00001.00001.00001.00001.00001.00001.00001
8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶
6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶
4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶
2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000ppppppppp
uuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
10101010101010101010101010101010
Debugging: Geometric Inverse CDFDebugging: Geometric Inverse CDFDebugging: Geometric Inverse CDF
• Error plot for geom for p <= 1e-10:
000000000
2×10⁸2×10⁸2×10⁸2×10⁸2×10⁸2×10⁸2×10⁸2×10⁸2×10⁸
4×10⁸4×10⁸4×10⁸4×10⁸4×10⁸4×10⁸4×10⁸4×10⁸4×10⁸
6×10⁸6×10⁸6×10⁸6×10⁸6×10⁸6×10⁸6×10⁸6×10⁸6×10⁸
1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰
8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹
6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹
4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹
2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000ppppppppp
uuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
10101010101010101010101010101010
Argh!Argh!Argh!
Q. Is this normal???
A. Yes. Most straightforward flonum functionimplementations have low error most places andhigh error (often unbounded) in others.
11111111111111111111111111111111
Argh!Argh!Argh!
Q. Is this normal???
A. Yes. Most straightforward flonum functionimplementations have low error most places andhigh error (often unbounded) in others.
The good news:You can usually fix them using just flonum ops.
11111111111111111111111111111111
Argh!Argh!Argh!
Q. Is this normal???
A. Yes. Most straightforward flonum functionimplementations have low error most places andhigh error (often unbounded) in others.
The good news:You can usually fix them using just flonum ops.
Q. How do I fix them???
A. Most functions—not implementations, butfunctions themselves—have ill-conditioned placeswhere they turn low input error into high outputerror. Avoid these badlands.
11111111111111111111111111111111
The Floating-Point Priest Says...The Floating-Point Priest Says...The Floating-Point Priest Says...
“The condition number of afunction is the absolutevalue of the ratio of itsderivative and its value,multiplied by the blah blahblah blah blah blah blahblah blah blah blah blahblah blah blah blah blahblah blah blah blah blah.”
12121212121212121212121212121212
The Badlands: SubtractionThe Badlands: SubtractionThe Badlands: Subtraction
> (plot3d (contour-intervals3d(λ (x y) (define z* (- x y)) (flulp-error (fl- (fl x) (fl y)) z*))-1 1 -1 1))
13131313131313131313131313131313
The Badlands: SubtractionThe Badlands: SubtractionThe Badlands: Subtraction
> (plot3d (contour-intervals3d(λ (x y) (define z* (- x y)) (flulp-error (fl- (fl x) (fl y)) z*))-1 1 -1 1))
000000000
222222222
444444444
666666666
888888888
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y axis
y axis
y axisy axis
y axisy axis
y axis
y axis
y axis
13131313131313131313131313131313
The Badlands: LogarithmThe Badlands: LogarithmThe Badlands: Logarithm
> (plot (function(λ (x) (define z* (bigfloat->real (bflog (bf x)))) (flulp-error (fllog (fl x)) z*)))
0 10)
14141414141414141414141414141414
The Badlands: LogarithmThe Badlands: LogarithmThe Badlands: Logarithm
> (plot (function(λ (x) (define z* (bigfloat->real (bflog (bf x)))) (flulp-error (fllog (fl x)) z*)))
0 10)
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 222222222 444444444 666666666 888888888 101010101010101010000000000
505050505050505050
100100100100100100100100100
150150150150150150150150150
200200200200200200200200200
250250250250250250250250250
14141414141414141414141414141414
The Badlands: DivisionThe Badlands: DivisionThe Badlands: Division
000000000
.25.25.25.25.25.25.25.25.25
.5.5.5.5.5.5.5.5.5
.75.75.75.75.75.75.75.75.75
111111111
1.251.251.251.251.251.251.251.251.25
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y axis
y axis
y axisy axis
y axisy axis
y axis
y axis
y axis
15151515151515151515151515151515
The Badlands: DivisionThe Badlands: DivisionThe Badlands: Division
000000000
.25.25.25.25.25.25.25.25.25
.5.5.5.5.5.5.5.5.5
.75.75.75.75.75.75.75.75.75
111111111
1.251.251.251.251.251.251.251.251.25
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y axis
y axis
y axisy axis
y axisy axis
y axis
y axis
y axis
• No badlands: except for flonum rounding error,division error doesn’t depend on inputs
15151515151515151515151515151515
The Badlands: DivisionThe Badlands: DivisionThe Badlands: Division
000000000
.25.25.25.25.25.25.25.25.25
.5.5.5.5.5.5.5.5.5
.75.75.75.75.75.75.75.75.75
111111111
1.251.251.251.251.251.251.251.251.25
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
111111111
.5.5.5.5.5.5.5.5.5
000000000
-.5-.5-.5-.5-.5-.5-.5-.5-.5
-1-1-1-1-1-1-1-1-1
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y axis
y axis
y axisy axis
y axisy axis
y axis
y axis
y axis
• No badlands: except for flonum rounding error,division error doesn’t depend on inputs
• Multiplication error is the same way
15151515151515151515151515151515
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
• If u is exact, (fllog u) has <= 0.5 ulps error
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
• If u is exact, (fllog u) has <= 0.5 ulps error
• If p is exact, (fl- 1.0 p) has <= 0.5 ulps error
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
• If u is exact, (fllog u) has <= 0.5 ulps error
• If p is exact, (fl- 1.0 p) has <= 0.5 ulps error
• If p is exact and near 0.0...
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
• If u is exact, (fllog u) has <= 0.5 ulps error
• If p is exact, (fl- 1.0 p) has <= 0.5 ulps error
• If p is exact and near 0.0...
Then (fl- 1.0 p) is inexact and near 1.0...
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
• If u is exact, (fllog u) has <= 0.5 ulps error
• If p is exact, (fl- 1.0 p) has <= 0.5 ulps error
• If p is exact and near 0.0...
Then (fl- 1.0 p) is inexact and near 1.0...
So (fllog (fl- 1.0 p)) may have high error
16161616161616161616161616161616
Informal Error AnalysisInformal Error AnalysisInformal Error Analysis
• Recursively reason about the body of geom:
(define (geom p u) (fl/ (fllog u) (fllog (fl- 1.0 p))))
• Can’t do anything about fl/ except make sure itsarguments are as accurate as possible
• If u is exact, (fllog u) has <= 0.5 ulps error
• If p is exact, (fl- 1.0 p) has <= 0.5 ulps error
• If p is exact and near 0.0...
Then (fl- 1.0 p) is inexact and near 1.0...
So (fllog (fl- 1.0 p)) may have high error
• Let’s check math/flonum for another incantation...16161616161616161616161616161616
log(1+x)log(1+x)log(1+x)
• Looks interesting: fllog1p
xxxxxxxxx
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
000000000 2.52.52.52.52.52.52.52.52.5 555555555 7.57.57.57.57.57.57.57.57.5
-2-2-2-2-2-2-2-2-2
000000000
222222222
17171717171717171717171717171717
log(1+x)log(1+x)log(1+x)
• Looks interesting: fllog1p
xxxxxxxxx
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
log
(1+
x)
000000000 2.52.52.52.52.52.52.52.52.5 555555555 7.57.57.57.57.57.57.57.57.5
-2-2-2-2-2-2-2-2-2
000000000
222222222
• We can use it almost directly—mathematically,
17171717171717171717171717171717
Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)
(define (geom p u) (fl/ (fllog u) (fllog1p (- p))))
18181818181818181818181818181818
Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)
(define (geom p u) (fl/ (fllog u) (fllog1p (- p))))
• Error plot for geom for p <= 1:
000000000
.5.5.5.5.5.5.5.5.5
111111111
1.51.51.51.51.51.51.51.51.5
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
pppppppppuuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
18181818181818181818181818181818
Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)
(define (geom p u) (fl/ (fllog u) (fllog1p (- p))))
• Error plot for geom for p <= 0.1:
000000000
.5.5.5.5.5.5.5.5.5
111111111
1.51.51.51.51.51.51.51.51.5
222222222
.1.1.1.1.1.1.1.1.1
.08.08.08.08.08.08.08.08.08
.06.06.06.06.06.06.06.06.06
.04.04.04.04.04.04.04.04.04
.02.02.02.02.02.02.02.02.02
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
pppppppppuuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
18181818181818181818181818181818
Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)
(define (geom p u) (fl/ (fllog u) (fllog1p (- p))))
• Error plot for geom for p <= 1e-05:
000000000
.5.5.5.5.5.5.5.5.5
111111111
1.51.51.51.51.51.51.51.51.5
222222222
.00001.00001.00001.00001.00001.00001.00001.00001.00001
8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶8×10⁻⁶
6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶6×10⁻⁶
4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶4×10⁻⁶
2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶2×10⁻⁶
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000pppppppppuuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
18181818181818181818181818181818
Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)Debugging: Geometric Inverse CDF (Second Stab)
(define (geom p u) (fl/ (fllog u) (fllog1p (- p))))
• Error plot for geom for p <= 1e-10:
000000000
.5.5.5.5.5.5.5.5.5
111111111
1.51.51.51.51.51.51.51.51.5
1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰1×10⁻¹⁰
8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹8×10⁻¹¹
6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹6×10⁻¹¹
4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹4×10⁻¹¹
2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹2×10⁻¹¹
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000pppppppppuuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
18181818181818181818181818181818
Argh It Is Not Perfect!Argh It Is Not Perfect!Argh It Is Not Perfect!
• But < 3 ulps error is very accurate
19191919191919191919191919191919
Argh It Is Not Perfect!Argh It Is Not Perfect!Argh It Is Not Perfect!
• But < 3 ulps error is very accurate
• Does geom have badlands?
19191919191919191919191919191919
Argh It Is Not Perfect!Argh It Is Not Perfect!Argh It Is Not Perfect!
• But < 3 ulps error is very accurate
• Does geom have badlands?
000000000
222222222
444444444
666666666
888888888
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
ppppppppp
uuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
19191919191919191919191919191919
Argh It Is Not Perfect!Argh It Is Not Perfect!Argh It Is Not Perfect!
• But < 3 ulps error is very accurate
• Does geom have badlands?
000000000
222222222
444444444
666666666
888888888
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
111111111
.8.8.8.8.8.8.8.8.8
.6.6.6.6.6.6.6.6.6
.4.4.4.4.4.4.4.4.4
.2.2.2.2.2.2.2.2.2
000000000
ppppppppp
uuuuuuuuu
ulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps errorulps error
• This is a property of the function, so we can’t doanything about it 19191919191919191919191919191919
Debugging SummaryDebugging SummaryDebugging Summary
• Make direct and reference implementations
20202020202020202020202020202020
Debugging SummaryDebugging SummaryDebugging Summary
• Make direct and reference implementations
• Repeat:
Plot error, identify high-error regions
Find badlands inputs, replace computations
20202020202020202020202020202020
Debugging SummaryDebugging SummaryDebugging Summary
• Make direct and reference implementations
• Repeat:
Plot error, identify high-error regions
Find badlands inputs, replace computations
• Avoid:
Subtracting nearby values
Taking logs of values near 1.0
Other badlands (most zero crossings,exponential growth)
20202020202020202020202020202020
Debugging SummaryDebugging SummaryDebugging Summary
• Make direct and reference implementations
• Repeat:
Plot error, identify high-error regions
Find badlands inputs, replace computations
• Avoid:
Subtracting nearby values
Taking logs of values near 1.0
Other badlands (most zero crossings,exponential growth)
• Move multiplication and division outward20202020202020202020202020202020
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• racket/extflonum: 80-bit extended flonums
Requires (extflonum-available?) = #t
64-bit significand, 15-bit exponent
(extfl->exact (extflexp (real->extfl 1/7)))
21212121212121212121212121212121
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• racket/extflonum: 80-bit extended flonums
Requires (extflonum-available?) = #t
64-bit significand, 15-bit exponent
(extfl->exact (extflexp (real->extfl 1/7)))
• double-double flonums: sum of twononoverlapping flonums represent a number
Requires correctly rounded arithmetic
~105-bit significand, 11-bit exponent
(let*-values ([(x2 x1) (fl2 1/7)][(y2 y1) (fl2exp x2 x1)])
(fl2->real y2 y1))
21212121212121212121212121212121
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• math/bigfloat: arbitrary precision floats
Requires MPFR (Racket ships with libraries)
Any size significand, 32-bit exponent
22222222222222222222222222222222
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• math/bigfloat: arbitrary precision floats
Requires MPFR (Racket ships with libraries)
Any size significand, 32-bit exponent
• Exact rationals
22222222222222222222222222222222
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• math/bigfloat: arbitrary precision floats
Requires MPFR (Racket ships with libraries)
Any size significand, 32-bit exponent
• Exact rationals
• AN IMPORTANT DATA POINT:
22222222222222222222222222222222
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• math/bigfloat: arbitrary precision floats
Requires MPFR (Racket ships with libraries)
Any size significand, 32-bit exponent
• Exact rationals
• AN IMPORTANT DATA POINT:
It takes 1074-bit bigfloats to fix geomIt takes 1074-bit bigfloats to fix geomIt takes 1074-bit bigfloats to fix geom
using (bflog (bf- 1.bf p))using (bflog (bf- 1.bf p))using (bflog (bf- 1.bf p))
22222222222222222222222222222222
What If I Need Moar Bits?What If I Need Moar Bits?What If I Need Moar Bits?
• math/bigfloat: arbitrary precision floats
Requires MPFR (Racket ships with libraries)
Any size significand, 32-bit exponent
• Exact rationals
• AN IMPORTANT DATA POINT:
It takes 1074-bit bigfloats to fix geomIt takes 1074-bit bigfloats to fix geomIt takes 1074-bit bigfloats to fix geom
using (bflog (bf- 1.bf p))using (bflog (bf- 1.bf p))using (bflog (bf- 1.bf p))• Conclusion: “moar bits” is not a general solution
22222222222222222222222222222222
The Badlands: Square OotThe Badlands: Square OotThe Badlands: Square Oot
> (plot (function(λ (x) (define z* (bigfloat->real (bfsqrt (bf x)))) (flulp-error (flsqrt (fl x)) z*))0 1e-320))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹2×10⁻³²¹ 4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹4×10⁻³²¹ 6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹6×10⁻³²¹ 8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹8×10⁻³²¹000000000
1×10¹³1×10¹³1×10¹³1×10¹³1×10¹³1×10¹³1×10¹³1×10¹³1×10¹³
2×10¹³2×10¹³2×10¹³2×10¹³2×10¹³2×10¹³2×10¹³2×10¹³2×10¹³
3×10¹³3×10¹³3×10¹³3×10¹³3×10¹³3×10¹³3×10¹³3×10¹³3×10¹³
4×10¹³4×10¹³4×10¹³4×10¹³4×10¹³4×10¹³4×10¹³4×10¹³4×10¹³
5×10¹³5×10¹³5×10¹³5×10¹³5×10¹³5×10¹³5×10¹³5×10¹³5×10¹³
The Badlands: SineThe Badlands: SineThe Badlands: Sine
> (plot (function(λ (x) (define z* (bigfloat->real (bfsin (bf x)))) (flulp-error (flsin (fl x)) z*))0 (* 2 pi)))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 222222222 444444444 666666666000000000
100100100100100100100100100
200200200200200200200200200
300300300300300300300300300
The Badlands: CosineThe Badlands: CosineThe Badlands: Cosine
> (plot (function(λ (x) (define z* (bigfloat->real (bfcos (bf x)))) (flulp-error (flcos (fl x)) z*))0 (* 2 pi)))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 222222222 444444444 666666666000000000
200200200200200200200200200
400400400400400400400400400
600600600600600600600600600
800800800800800800800800800
The Badlands: TangentThe Badlands: TangentThe Badlands: Tangent
> (plot (function(λ (x) (define z* (bigfloat->real (bftan (bf x)))) (flulp-error (fltan (fl x)) z*))0 (* 2 pi)))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
000000000 222222222 444444444 666666666000000000
200200200200200200200200200
400400400400400400400400400
600600600600600600600600600
The Badlands: ArcsineThe Badlands: ArcsineThe Badlands: Arcsine
> (plot (function(λ (x) (define z* (bigfloat->real (bfasin (bf x)))) (flulp-error (flasin (fl x)) z*))-1 1))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
-1-1-1-1-1-1-1-1-1 -.5-.5-.5-.5-.5-.5-.5-.5-.5 000000000 .5.5.5.5.5.5.5.5.5 111111111
.25.25.25.25.25.25.25.25.25
.5.5.5.5.5.5.5.5.5
.75.75.75.75.75.75.75.75.75
111111111
1.251.251.251.251.251.251.251.251.25
The Badlands: ArccosineThe Badlands: ArccosineThe Badlands: Arccosine
> (plot (function(λ (x) (define z* (bigfloat->real (bfacos (bf x)))) (flulp-error (flacos (fl x)) z*))0.8 1))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
.85.85.85.85.85.85.85.85.85 .9.9.9.9.9.9.9.9.9 .95.95.95.95.95.95.95.95.95 111111111000000000
505050505050505050
100100100100100100100100100
150150150150150150150150150
200200200200200200200200200
The Badlands: ArctangentThe Badlands: ArctangentThe Badlands: Arctangent
> (plot (function(λ (x) (define z* (bigfloat->real (bfatan (bf x)))) (flulp-error (flatan (fl x)) z*))-10 10))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
-10-10-10-10-10-10-10-10-10 -5-5-5-5-5-5-5-5-5 000000000 555555555 101010101010101010
.2.2.2.2.2.2.2.2.2
.4.4.4.4.4.4.4.4.4
.6.6.6.6.6.6.6.6.6
The Badlands: ExponentialThe Badlands: ExponentialThe Badlands: Exponential
> (plot (function(λ (x) (define z* (bigfloat->real (bfexp (bf x)))) (flulp-error (flexp (fl x)) z*))-100 100))
x axisx axisx axisx axisx axisx axisx axisx axisx axis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
y a
xis
-100-100-100-100-100-100-100-100-100 -50-50-50-50-50-50-50-50-50 000000000 505050505050505050 100100100100100100100100100
101010101010101010
202020202020202020
303030303030303030
404040404040404040
505050505050505050
The Badlands: Exponential With BaseThe Badlands: Exponential With BaseThe Badlands: Exponential With Base
> (plot3d (contour-intervals3d(λ (x y) (define z* (bigfloat->real
(bfexpt (bf x) (bf y)))) (flulp-error (flexpt (fl x) (fl y)) z*))0 101 -101 101))
000000000
505050505050505050
100100100100100100100100100
150150150150150150150150150
200200200200200200200200200
250250250250250250250250250
100100100100100100100100100
757575757575757575
505050505050505050
252525252525252525
000000000
100100100100100100100100100
505050505050505050
000000000
-50-50-50-50-50-50-50-50-50
-100-100-100-100-100-100-100-100-100x axisx axisx axisx axisx axisx axisx axisx axisx axis
y axis
y axis
y axisy axis
y axisy axis
y axis
y axis
y axis
Condition NumberCondition NumberCondition Number
(: condition ((Flonum -> Flonum)(Flonum -> Flonum)-> (Flonum -> Flonum)))
(define ((condition f df) x) (abs (/ (* x (df x)) (f x)))) (: condition2d ((Flonum Flonum -> Flonum)
(Flonum Flonum -> (Values Flonum Flonum))-> (Flonum Flonum -> Flonum)))
(define ((condition2d f df) x y) (define-values (dx dy) (df x y)) (define z (f x y)) (max (abs (/ (* x dx) z))
(abs (/ (* y dy) z))))