Benford's Law: Tables of Logarithms, Tax Cheats, and The...

History Applications Formalism Benford and Integer Sequences Benford and Recurrence Relations

Benford’s Law: Tables of Logarithms, TaxCheats, and The Leading Digit

Phenomenon

Michelle Manes ([email protected])

USC Women in Math24 April, 2008

1


History

(1881) Simon Newcomb publishes “Note on thefrequency of use of the different digits in naturalnumbers.” The world ignores it.

(1938) Frank Benford (unaware of Newcomb’s work,presumably) publishes “The law of anomalousnumbers.”

2


History

(1881) Simon Newcomb publishes “Note on thefrequency of use of the different digits in naturalnumbers.” The world ignores it.

(1938) Frank Benford (unaware of Newcomb’s work,presumably) publishes “The law of anomalousnumbers.”

3


Statement of Benford’s Law

Newcomb noticed that the early pages of the book oftables of logarithms were much dirtier than the laterpages, so were presumably referenced more often.

He stated the rule this way:

Prob(first significant digit = d) = log10

(1 +

1d

).

4


Statement of Benford’s Law

Newcomb noticed that the early pages of the book oftables of logarithms were much dirtier than the laterpages, so were presumably referenced more often.

He stated the rule this way:

Prob(first significant digit = d) = log10

(1 +

1d

).

5


Benford Base b

DefinitionA sequence of positive numbers {xn} is Benford(base b) if

Prob(first significant digit = d) = logb

(1 +

1d

).

6


Benford’s Law

Base 10 Predictionsdigit probability it occurs as a leading digit

1 30.1%2 17.6%3 12.5%4 9.7%5 7.9%6 6.7%7 5.8%8 5.1%9 4.6%

7


Benford’s Data

8


More Data

Benford’s Law compared with: numbers from the frontpages of newspapers, U.S. county populations, and theDow Jones Industrial Average.

9


Dow Illustrates Benford’s Law

Suppose the Dow Jones average is about $1,000. If theaverage goes up at a rate of about 20% a year, it wouldtake five years to get from 1 to 2 as a first digit.

If we start with a first digit 5, it only requires a 20%increase to get from $5,000 to $6,000, and that isachieved in one year.

When the Dow reaches $9,000, it takes only an 11%increase and just seven months to reach the $10,000mark. This again has first digit 1, so it will take anotherdoubling (and five more years) to get back to first digit 2.

10






11






12


Benford’s Law and Tax Fraud (Nigrini, 1992)

Most people can’t fake data convincingly.

Many states (including California) and the IRS now usefraud-detection software based on Benford’s Law.

13





14





15


True Life Tale

Manager from Arizona State Treasurer wasembezzling funds.

Most amounts were below $100,000 (criticalthreshold for checks that would require morescrutiny).Over 90% of the checks had a first digit 7, 8, or 9.(Trying to get close to the threshold without goingover — artificially changes the data and so breaks fitwith Benford’s law.)

16


True Life Tale

Manager from Arizona State Treasurer wasembezzling funds.Most amounts were below $100,000 (criticalthreshold for checks that would require morescrutiny).

Over 90% of the checks had a first digit 7, 8, or 9.(Trying to get close to the threshold without goingover — artificially changes the data and so breaks fitwith Benford’s law.)

17


True Life Tale

Manager from Arizona State Treasurer wasembezzling funds.Most amounts were below $100,000 (criticalthreshold for checks that would require morescrutiny).Over 90% of the checks had a first digit 7, 8, or 9.(Trying to get close to the threshold without goingover — artificially changes the data and so breaks fitwith Benford’s law.)

18


True Life Tale

19


Problems with “Proofs” of Benford’s Law

Discrete density and summability methods.

Continuous density and summability methods. (Sameproblem.)

Scale invariance.

If there is a reasonable first-digit law, it should bescale-invariant. That is, it shouldn’t matter if themeasurements are in feet or meters, pounds orkilograms, etc.

20



Discrete density and summability methods.Fd = {x ∈ N | first digit of x is d}. No natural density.

That is,

limn→∞

Fd ∩ {1,2, . . . ,n}n

does not exist.


Scale invariance.


21





Scale invariance.


22





Scale invariance.


23





Scale invariance.


24


Hill’s Formulation (1988)

DefinitionFor each integer b > 1, define the mantissa function

Mb : R+ → [1,b)

x 7→ r

where r is the unique number in [1,b) such that x = rbn

for some n ∈ Z.

ExamplesM10(9) = 9 = M100(9).M2(9) = 9/8 = 1.001 (base 2).

25



DefinitionFor each integer b > 1, define the mantissa function

Mb : R+ → [1,b)

x 7→ r

where r is the unique number in [1,b) such that x = rbn

for some n ∈ Z.

ExamplesM10(9) = 9 = M100(9).M2(9) = 9/8 = 1.001 (base 2).

26



DefinitionFor E ⊂ [1,b), let

〈E〉b = M−1b (E) =

⋃n∈Z

bnE ⊂ R+.

DefinitionMb = {〈E〉b | E ⊂ B(1,b)}is the σ-algebra on R+

generated by Mb.

27



DefinitionFor E ⊂ [1,b), let

〈E〉b = M−1b (E) =

⋃n∈Z

bnE ⊂ R+.

DefinitionMb = {〈E〉b | E ⊂ B(1,b)}is the σ-algebra on R+

generated by Mb.

28



DefinitionLet Pb be the probability measure on (R+,Mb) defined by

Pb(〈[1, γ)〉b) = logb γ.

This probability measure:Agrees with Benford’s law.Is the unique scale-invariant probability measure on(R+,Mb).

Proof comes down to uniqueness of Haar measure.

29




Pb(〈[1, γ)〉b) = logb γ.

This probability measure:

Agrees with Benford’s law.Is the unique scale-invariant probability measure on(R+,Mb).


30




Pb(〈[1, γ)〉b) = logb γ.

This probability measure:Agrees with Benford’s law.

Is the unique scale-invariant probability measure on(R+,Mb).


31




Pb(〈[1, γ)〉b) = logb γ.



32




Pb(〈[1, γ)〉b) = logb γ.



33


What types of sequences are Benford?

Real-world data can be a good fit or not, depending onthe type of data. Data that is a good fit is “suitablyrandom” — comes in many different scales, and is a largeand randomly distributed data set, with no artificial orexternal limitations on the range of the numbers.

Some numerical sequences are clearly not Benforddistributed base-10:

1,2,3,4,5,6,7, . . . (uniform distribution)

1,10,100,1000, . . . (first digit is always 1)

34







35







36







37


Some numerical sequences seem to be a good fit

Powers of Two38


Some numerical sequences seem to be a good fit

Fibonacci Numbers39


Logarithms and Benford’s Law

Fundamental EquivalenceData set {xi} is Benford base b if {yi} is equidistributedmod 1, where yi = logb xi .

Proof:x = Mb(x) · bk for some k ∈ Z.First digit of x in base b is d iff d ≤ Mb(x) < d + 1.logb d ≤ y < logb(d + 1), wherey = logb(Mb(x)) = logb x mod 1.If the distribution is uniform (mod 1), then theprobability y is in this range is

logb(d +1)−logb(d) = logb

(d + 1

d

)= logb

(1 +

1d

).

40




Proof:x = Mb(x) · bk for some k ∈ Z.

First digit of x in base b is d iff d ≤ Mb(x) < d + 1.logb d ≤ y < logb(d + 1), wherey = logb(Mb(x)) = logb x mod 1.If the distribution is uniform (mod 1), then theprobability y is in this range is


(d + 1

d

)= logb

(1 +

1d

).

41




Proof:x = Mb(x) · bk for some k ∈ Z.First digit of x in base b is d iff d ≤ Mb(x) < d + 1.

logb d ≤ y < logb(d + 1), wherey = logb(Mb(x)) = logb x mod 1.If the distribution is uniform (mod 1), then theprobability y is in this range is


(d + 1

d

)= logb

(1 +

1d

).

42




Proof:x = Mb(x) · bk for some k ∈ Z.First digit of x in base b is d iff d ≤ Mb(x) < d + 1.logb d ≤ y < logb(d + 1), wherey = logb(Mb(x)) = logb x mod 1.

If the distribution is uniform (mod 1), then theprobability y is in this range is


(d + 1

d

)= logb

(1 +

1d

).

43




Proof:x = Mb(x) · bk for some k ∈ Z.First digit of x in base b is d iff d ≤ Mb(x) < d + 1.logb d ≤ y < logb(d + 1), wherey = logb(Mb(x)) = logb x mod 1.If the distribution is uniform (mod 1), then theprobability y is in this range is


(d + 1

d

)= logb

(1 +

1d

).

44




45




0 1

1 102

log 2 ! log 10

46




Kronecker-Weyl TheoremIf β 6∈ Q then nβ mod 1 is equidistributed.(Thus if logb α 6∈ Q, then αn is Benford.)

47


Powers of 2

TheoremThe sequence {2n} for n ≥ 0 is Benford base b for any bthat is not a rational power of 2.

Proof:Consider the sequence of logarithms {n(logb 2)}.By the Kronecker-Weyl Theorem, this is uniform(mod 1) as long as logb 2 6∈ Q.If b is not a rational power of 2, then the sequence oflogarithms is uniformly distributed (mod 1), so theoriginal sequence is Benford base b.

48


Powers of 2

TheoremThe sequence {2n} for n ≥ 0 is Benford base b for any bthat is not a rational power of 2.

Proof:Consider the sequence of logarithms {n(logb 2)}.By the Kronecker-Weyl Theorem, this is uniform(mod 1) as long as logb 2 6∈ Q.If b is not a rational power of 2, then the sequence oflogarithms is uniformly distributed (mod 1), so theoriginal sequence is Benford base b.

49


Fibonacci Numbers

TheoremThe sequence {Fn} of Fibonacci numbers Benford base bfor almost every b.

Heuristic Argument:Closed form for Fibonacci numbers:

Fn =1√5

[(1 +√

52

)n

−

(1−√

52

)n].∣∣∣(1−

√5

2

)∣∣∣ < 1, so the leading digits are completely

determined by 1√5

(1+√

52

)n.

This sequence will be Benford base-b for any b wherelogb

(1+√

52

)6∈ Q.

50


Fibonacci Numbers



Fn =1√5

[(1 +√

52

)n

−

(1−√

52

)n].

∣∣∣(1−√

52


determined by 1√5

(1+√

52

)n.


(1+√

52

)6∈ Q.

51


Fibonacci Numbers



Fn =1√5

[(1 +√

52

)n

−

(1−√

52

)n].∣∣∣(1−

√5

2


determined by 1√5

(1+√

52

)n.


(1+√

52

)6∈ Q.

52


Fibonacci Numbers



Fn =1√5

[(1 +√

52

)n

−

(1−√

52

)n].∣∣∣(1−

√5

2


determined by 1√5

(1+√

52

)n.


(1+√

52

)6∈ Q.

53


Linear Recurrence Sequences

Consider the sequence {an} given by some initialconditions a0,a1, . . . ,ak−1 and then a recurrence relation

an+k = c1an+k−1 + c2an+k−2 + · · ·+ ckan,

with c1, c2, . . . , ck fixed real numbers.

Find the eigenvalues of the recurrence relation and orderthem so that |λ1| ≥ |λ2| ≥ · · · ≥ |λk |.

There exist number u1,u2, . . . ,uk (which depend on theinitial conditions) so that an = u1λ

n1 + u2λ

n2 + · · ·+ ukλ

nk .

54








n1 + u2λ

n2 + · · ·+ ukλ

nk .

55








n1 + u2λ

n2 + · · ·+ ukλ

nk .

56



TheoremWith a linear recurrence sequence as described, iflogb |λ1| 6∈ Qand the initial conditions are such that u1 6= 0,then the sequence {an} is Benford base b.

Sketch of Proof:

Rewrite the closed form as an = u1λn1

(1 +O

(kuλn

2λn

1

))where u = maxi |ui |+ 1.Some clever algebra using our assumptions to rewritethis as an = u1λ

n1 (1 +O(βn)).

Then yn = logb(an) = n logb λ1 + logb u1 +O(βn).Show in the limit the error term affects a vanishinglysmall portion of the distribution.

57




Sketch of Proof:


(1 +O

(kuλn

2λn

1

))where u = maxi |ui |+ 1.

Some clever algebra using our assumptions to rewritethis as an = u1λ

n1 (1 +O(βn)).


58




Sketch of Proof:


(1 +O

(kuλn

2λn

1


n1 (1 +O(βn)).


59




Sketch of Proof:


(1 +O

(kuλn

2λn

1


n1 (1 +O(βn)).

Then yn = logb(an) = n logb λ1 + logb u1 +O(βn).

Show in the limit the error term affects a vanishinglysmall portion of the distribution.

60




Sketch of Proof:


(1 +O

(kuλn

2λn

1


n1 (1 +O(βn)).


61


Elliptic Divisibility Sequences

DefinitionAn integral divisibility sequence is a sequence of integers{un} satisfying

un | um whenever n | m.

An elliptic divisibility sequence is an integral divisibilitysequence which satisfies the following recurrence relationfor all m ≥ n ≥ 1:

um+num−nu21

= um+1um−1u2n − un+1un−1u2

m.

62


Boring Elliptic Divisibility Sequences

The sequences of integers, where un = n.

The sequence 0,1,−1,0,1,−1, . . ..

The sequence1,3,8,21,55,144,377,987,2584,6765, . . . (this isevery-other Fibonacci number).

63




The sequence 0,1,−1,0,1,−1, . . ..


64




The sequence 0,1,−1,0,1,−1, . . ..


65


Not-So-Boring Elliptic Divisibility Sequences

The sequences which begins0,1,1,−1,1,2,−1,−3,−5,7,−4,−28,29,59,129,−314,−65,1529,−3689,−8209,−16264,833313,113689,−620297,2382785,7869898,7001471,−126742987,−398035821,168705471, . . .(This is sequence A006769 in the On-LineEncyclopedia of Integer Sequences.)

The sequence which begins1,1,−3,11,38,249,−2357,8767,496036,−3769372,−299154043,−12064147359, . . ..

66


Not-So-Boring Elliptic Divisibility Sequences

The sequences which begins0,1,1,−1,1,2,−1,−3,−5,7,−4,−28,29,59,129,−314,−65,1529,−3689,−8209,−16264,833313,113689,−620297,2382785,7869898,7001471,−126742987,−398035821,168705471, . . .(This is sequence A006769 in the On-LineEncyclopedia of Integer Sequences.)

The sequence which begins1,1,−3,11,38,249,−2357,8767,496036,−3769372,−299154043,−12064147359, . . ..

67


Why We Like Elliptic Divisibility Sequences

Special case of Somos sequences, which areinteresting and an active area of research.

Connection to elliptic curves, also an active area ofresearch.

Elliptic curve

E over Qwith rational point

P on E

↔ EDS: denominators

of the sequenceof points {P,2P,3P, . . .}

Applications to elliptic curve cryptography.

68





Elliptic curve


P on E



Applications to elliptic curve cryptography.

69





Elliptic curve


P on E



Applications to elliptic curve cryptography.70


Elliptic Divisibility Sequences are Benford?

71



72



73


Heuristic Argument

It’s well-known that elliptic divisibility sequencessatisfy a growth condition like un ≈ cn2 where theconstant c depends on the arithmetic height of thepoint P and on the curve E .

Weyl’s theorem tells us that {nkα} is uniformdistributed (mod 1) iff α 6∈ Q.

So we should at least be able to conclude that a givenEDS is Benford base b for almost every b.

But: The argument with the big-O error terms isdelicate, and not enough is known in the case of EDS.

74


Heuristic Argument





75


Heuristic Argument





76


Heuristic Argument





77

Benford's Law: Tables of Logarithms, Tax Cheats, and The...

Documents

Transcript of Benford's Law: Tables of Logarithms, Tax Cheats, and The...