When and why data tend to start with 1

Count the leading digit of every US county's population and you will find a 1 about 30% of the time and a 9 about 5% of the time. Many other datasets, from river lengths to share prices, follow the same frequencies. What's going on?

Each tick is a county, at an angle set by the fractional part of log10(population). Bigger counties get longer ticks, and pointing at one shows its integer part. Drag the ring. A turn of t multiplies every population by 10t.

1

Leading digits of 3,144 county populations

The pmf of the leading digit of US county populations, compared with the Benford pmf.

The Benford pmf is a single probability model. It has no parameters and is not estimated from the data.

P(D(X) = d) = log10(1 + 1d),  d = 1, …, 9
2

Multiplicative processes

Many processes, like county populations, are multiplicative, in that each outcome is a product of many random factors.

40 simulated counties, each starting at 1,000 people
3

Characteristic and mantissa

log10 x can be decomposed into an integer (the characteristic) plus a fractional part (the mantissa). D(x) depends on the mantissa alone.

4

The density of the mantissa

Cut the density of log10 X at every integer and add the pieces together on [0, 1). The sum is the density of the mantissa.

5

Mantissa to leading digit

Let M be the mantissa of X, so that D(X) = d exactly when M falls in [log10 d, log10(d + 1)). If M is uniform, each digit's probability is the width of its interval, which is the Benford pmf.

6

Looking for a scale-invariant pmf

The Benford pmf can also be derived from a different starting point. Changing the units of a dataset multiplies every value by a constant c. Units are arbitrary, so a leading digit pmf found in nature should be scale invariant, with D(cX) distributed like D(X). What does multiplying by c do to a leading digit pmf?

M(cX)= log10(cX) mod 1= (log10 X + log10 c) mod 1= (M(X) + log10 c) mod 1

So the density of M(cX) is the density of M(X) shifted right by log10 c, with whatever passes 1 wrapping round to 0. At c = 10 it has gone once all the way round.

after × cat c = 1

Only a uniform density on M is invariant to shifts. Hence the Benford pmf is the only scale-invariant leading digit pmf (Pinkham 1961).

7

Rescaling real data

So a leading digit pmf found in nature has to be Benford. Real data can only approximate it. Multiply each dataset by every c from 1 to 10 and see how far its digits move.

MAD from the Benford pmf for every c from 1 to 10, five datasets
8

Examples of what is and is not approximately Benford

the data on the log10 scale
Kuiper distance of the mantissa from uniform against MAD of the digits from Benford, for all thirteen datasets
9

Supply your own data

or paste numbers

Write t=log⁡10ct = \log_{10} c with 0≤t<10 \le t \lt 1. Let M=M(X)M = M(X) have cdf FF and density ff, and call the density of M(cX)M(cX) gg. Then M(cX)M(cX) is M+tM + t, less 1 once M≥1−tM \ge 1 - t.

For t≤y<1t \le y \lt 1

P(M(cX)≤y)=P(M≤y−t)+P(M≥1−t)=F(y−t)+1−F(1−t)g(y)=ddy P(M(cX)≤y)=f(y−t)\begin{aligned} \mathrm{P}(M(cX) \le y) &= \mathrm{P}(M \le y - t) + \mathrm{P}(M \ge 1 - t)\\ &= F(y - t) + 1 - F(1 - t)\\[6pt] g(y) &= \tfrac{\mathrm{d}}{\mathrm{d}y}\,\mathrm{P}(M(cX) \le y) = f(y - t) \end{aligned}
P(M(cX)≤y)=P(M≤y−t)+P(M≥1−t)=F(y−t)+1−F(1−t)g(y)=ddy P(M(cX)≤y)=f(y−t)\begin{aligned} &\mathrm{P}(M(cX) \le y)\\ &\quad= \mathrm{P}(M \le y - t)\\ &\qquad + \mathrm{P}(M \ge 1 - t)\\ &\quad= F(y - t) + 1 - F(1 - t)\\[6pt] &g(y) = \tfrac{\mathrm{d}}{\mathrm{d}y}\,\mathrm{P}(M(cX) \le y)\\ &\quad= f(y - t) \end{aligned}

For 0≤y<t0 \le y \lt t, only the part that wrapped can land below yy

P(M(cX)≤y)=P(1−t≤M≤1−t+y)=F(y−t+1)−F(1−t)g(y)=ddy P(M(cX)≤y)=f(y−t+1)\begin{aligned} \mathrm{P}(M(cX) \le y) &= \mathrm{P}(1 - t \le M \le 1 - t + y)\\ &= F(y - t + 1) - F(1 - t)\\[6pt] g(y) &= \tfrac{\mathrm{d}}{\mathrm{d}y}\,\mathrm{P}(M(cX) \le y) = f(y - t + 1) \end{aligned}
P(M(cX)≤y)=P(1−t≤M≤1−t+y)=F(y−t+1)−F(1−t)g(y)=ddy P(M(cX)≤y)=f(y−t+1)\begin{aligned} &\mathrm{P}(M(cX) \le y)\\ &\quad= \mathrm{P}(1 - t \le M \le 1 - t + y)\\ &\quad= F(y - t + 1) - F(1 - t)\\[6pt] &g(y) = \tfrac{\mathrm{d}}{\mathrm{d}y}\,\mathrm{P}(M(cX) \le y)\\ &\quad= f(y - t + 1) \end{aligned}

So g(y)=f((y−t) mod 1)g(y) = f\big((y - t) \bmod 1\big), the curve of ff moved right by tt, with the part past 1 coming back in at 0. Each piece is a shift with slope 1, so nothing stretches.

Take MM uniform on [0,1)[0, 1), so P(a≤M<b)=b−a\mathrm{P}(a \le M \lt b) = b - a.

P(D(X)=d)=P(log⁡10d≤M<log⁡10(d+1))=log⁡10(d+1)−log⁡10d=log⁡10 ⁣(1+1d)\begin{aligned} \mathrm{P}(D(X) = d) &= \mathrm{P}\big(\log_{10} d \le M \lt \log_{10}(d+1)\big)\\[8pt] &= \log_{10}(d+1) - \log_{10} d\\[8pt] &= \log_{10}\!\left(1 + \frac{1}{d}\right) \end{aligned}
P(D(X)=d)=P(log⁡10d≤M<log⁡10(d+1))=log⁡10(d+1)−log⁡10d=log⁡10 ⁣(1+1d)\begin{aligned} &\mathrm{P}(D(X) = d)\\ &\quad= \mathrm{P}\big(\log_{10} d \le M \lt \log_{10}(d+1)\big)\\[6pt] &\quad= \log_{10}(d+1) - \log_{10} d\\[6pt] &\quad= \log_{10}\!\left(1 + \frac{1}{d}\right) \end{aligned}

Each pd=log⁡10(1+1/d)p_d = \log_{10}(1 + 1/d) is positive, since 1+1/d>11 + 1/d \gt 1. And they sum to 1.

∑d=19log⁡10 ⁣(1+1d)=∑d=19log⁡10d+1d=log⁡10(21⋅32⋅43⋅54⋅65⋅76⋅87⋅98⋅109)=log⁡10101=1\begin{aligned} \sum_{d=1}^{9}\log_{10}\!\left(1 + \frac{1}{d}\right) &= \sum_{d=1}^{9}\log_{10}\frac{d+1}{d}\\[6pt] &= \log_{10}\left(\frac{\X{2}}{1}\cdot\frac{\X{3}}{\X{2}}\cdot\frac{\X{4}}{\X{3}}\cdot\frac{\X{5}}{\X{4}}\cdot\frac{\X{6}}{\X{5}}\cdot\frac{\X{7}}{\X{6}}\cdot\frac{\X{8}}{\X{7}}\cdot\frac{\X{9}}{\X{8}}\cdot\frac{10}{\X{9}}\right)\\[6pt] &= \log_{10}\frac{10}{1} = 1 \end{aligned}
∑d=19log⁡10 ⁣(1+1d)=∑d=19log⁡10d+1d=log⁡10(21⋅32⋅43⋯98⋅109)=log⁡10101=1\begin{aligned} &\sum_{d=1}^{9}\log_{10}\!\left(1 + \frac{1}{d}\right)\\[2pt] &\quad= \sum_{d=1}^{9}\log_{10}\frac{d+1}{d}\\[6pt] &\quad= \log_{10}\left(\frac{\X{2}}{1}\cdot\frac{\X{3}}{\X{2}}\cdot\frac{\X{4}}{\X{3}}\cdots\frac{\X{9}}{\X{8}}\cdot\frac{10}{\X{9}}\right)\\[6pt] &\quad= \log_{10}\frac{10}{1} = 1 \end{aligned}

A sum of logs is the log of a product, and the product telescopes.

Physical constants, CODATA 2022

The distinct values in SI units, signs dropped, smallest to largest. Constants that share a value share a row. The official table is at NIST.