Exercises and Solutions in Statistical Theory

Appendix: Useful Math Results

1. Summations

(a) Binomial:

n

P

j=0

C

n

j

a

j

b

(n−j)

= (a + b)

n

, where C

n

j

=

n!

j!(n−j)!

.

(b) Geometric:

i.

∞

P

j=0

r

j

=

1

1−r

, |r| < 1.

ii.

∞

P

j=1

r

j

=

r

1−r

, |r| < 1.

iii.

n

P

j=0

r

j

=

1−r

(n+1)

1−r

, −∞ < r < +∞.

(c) N egative Binomial:

∞

P

j=0

C

j+k

k

π

j

= (1 − π)

−(k+1)

, 0 < π < 1, k a positive

integer.

(d) Exponential:

∞

P

j=0

x

j

j!

= e

x

, −∞ < x < +∞.

(e) S ums of Integers:

i.

n

P

i=1

i =

n(n+1)

2

.

ii.

n

P

i=1

i

2

=

n(n+1)(2n+1)

6

.

iii.

n

P

i=1

i

3

=

h

n(n+1)

2

i

2

.

2. Limits

(a) lim

n→∞



1 +

a

n



n

= e

a

, −∞ < a < +∞.

3. Important Calculus-Based Results

(a) L’Hˆopital’s Rule: For differentiable functions f(x) and g(x) and an

“extended” rea l number c (i.e., c ∈ ℜ

1

or c = ±∞), suppose that

lim

x→c

f(x) = lim

x→c

g(x) = 0, or that lim

x→c

f(x) = lim

x→c

g(x) =

±∞. Suppose also that lim

x→c

f

′

(x)/g

′

(x) e xists [in particular, g

′

(x) 6= 0

353

354 APPENDIX: USEFUL MATH RESULTS

near c, except possibly at c]. Then,

lim

x→c

f(x)

g(x)

= lim

x→c

f

′

(x)

g

′

(x)

.

L’Hˆopital’s Rule is also valid for one-sided limits.

(b) Integration by Parts: Let u = f(x) and v = g(x), with differentials du =

f

′

(x)dx and dv = g

′

(x)dx. Then,

Z

u dv = uv −

Z

v du.

(c) Jacobians for One- and Two-Dimensional Change-of-Variable Transfor-

mations: Let X be a scalar variable with support A ⊆ ℜ

1

. Consider a

one-to-one transformation U = g(X) tha t maps A → B ⊆ ℜ

1

. Denote

the inverse of U as X = h(U ). Then, the corresponding o ne -dimensional

Jacobian of the transformation is defined as

J =

d[h(U)]

dU

,

so that

Z

A

f(X)dX =

Z

B

f[h(U)]|J|dU.

Similarly, consider scalar variables X and Y defined on a two-

dimensional set A ⊆ ℜ

2

, and let U = g

1

(X, Y ) and V = g

2

(X, Y ) define

a one-to- one transformation that maps A in the xy-plane to B ⊆ ℜ

2

in

the uv-plane. Define X = h

1

(U, V ) and Y = h

2

(U, V ). The n, the Ja-

cobian of the (two-dimensional) transformation is given by the second-

order determinant

J =













∂h

1

(U,V )

∂U

∂h

1

(U,V )

∂V

∂h

2

(U,V )

∂U

∂h

2

(U,V )

∂V













,

so that

Z Z

A

f(X, Y )dXdY =

Z Z

B

f[h

1

(U, V ), h

2

(U, V )]|J|dU dV.

4. Special Functions

(a) Gamma Function:

i. For any real number t > 0, the Gamma function is defined as

Γ(t) =

Z

∞

0

y

t−1

e

−y

dy.

355

ii. For any real number t > 0, Γ(t + 1) = tΓ(t).

iii. For any positive integer n, Γ(n) = (n − 1)!

iv. Γ(1/2) =

√

π; Γ(3/2) =

√

π/2; Γ(5 /2) = (3

√

π)/4.

(b) Beta Function:

i. For α > 0 and β > 0, the Beta function is defined as

B(α, β) =

Z

1

0

y

α−1

(1 − y)

β−1

dy.

ii. B(α, β) =

Γ(α)Γ(β)

Γ(α+β)

(c) Convex and Concave Functions: A real-valued function f(·) is sa id to be

convex if, for any two points x and y in its domain and any t ∈ [0, 1], we

have

f[tx + (1 − t)y] ≤ tf(x) + (1 − t)f(y).

Likewise, f(·) is said to be concave if

f[tx + (1 − t)y] ≥ tf(x) + (1 − t)f(y).

Also, f(x) is concave on [a, b] if and only if −f(x) is convex on [a, b].

5. Approximations

(a) Stirling’s Approximation:

For n a large non-negative integer, n! ≈

√

2πn



n

e



n

.

(b) Taylor S eries Approximations:

(i) Univariate Taylor Series: If f(x) is a real- valued function of x that

is infinitely differentiable in a neighbor hood of a real number a, then a

Taylor series expansion of f(x) around a is equal to

f(x) =

∞

X

k=0

f

(k)

(a)

k!

(x − a)

k

,

where

f

(k)

(a) =

"

d

k

f(x)

dx

k

#

|x=a

, k = 0, 1, . . . , ∞.

When a = 0, the infinite se ries expa nsion above is called a Maclaurin

series.

As examples, a fi r st-order (or linear) Taylor series approximation to f(x)

around the real number a is equal to

f(x) ≈ f(a) +



df(x)

dx



|x=a

(x − a),

356 APPENDIX: USEFUL MATH RESULTS

and a second-order Taylor series approximation to f(x) around the real

number a is e qual to

f(x) ≈ f(a) +



df(x)

dx



|x=a

(x − a) +

1

2!



d

2

f(x)

dx

2



|x=a

(x − a)

2

.

(ii) Multivariate Taylor series: For p ≥ 2, if f(x

1

, x

2

, . . . , x

p

) is a real-

valued function of x

1

, x

2

, . . . , x

p

that is infinitely differentiable in a neigh-

borhood of (a

1

, a

2

, . . . , a

p

), where a

i

, i = 1, 2, . . . , p, is a real number,

then a multivariate Taylor series expansion of f(x

1

, x

2

, . . . , x

p

) around

(a

1

, a

2

, . . . , a

p

) is equal to

f(x

1

, x

2

, . . . , x

p

) =

∞

X

k

1

=0

∞

X

k

2

=0

···

∞

X

k

p

=0

f

(k

1

+k

2

+···+k

p

)

(a

1

, a

2

, . . . , a

p

)

k

1

!k

2

! ···k

p

!

×

p

Y

i=1

(x

i

− a

i

)

k

i

,

where

f

(k

1

+k

2

+···+k

p

)

(a

1

, a

2

, . . . , a

p

)

=

"

∂

k

1

+k

2

+···+k

p

f(x

1

, x

2

, . . . , x

p

)

∂x

k

1

1

∂x

k

2

2

···∂x

k

p

p

#

|(x

1

,x

2

,...,x

p

)=(a

1

,a

2

,...,a

p

)

.

As examples, when p = 2, a first-order (or linear) multivariate Taylor

series approximation to f(x

1

, x

2

) around (a

1

, a

2

) is equal to

f(x

1

, x

2

) ≈ f (a

1

, a

2

) +

2

X

i=1



∂f(x

1

, x

2

)

∂x

i



|(x

1

,x

2

)=(a

1

,a

2

)

(x

i

− a

i

),

and a second-order multivariate Taylor series approximation to f(x

1

, x

2

)

around (a

1

, a

2

) is equal to

f(x

1

, x

2

) ≈ f(a

1

, a

2

) +

2

X

i=1



∂f(x

1

, x

2

)

∂x

i



|(x

1

,x

2

)=(a

1

,a

2

)

(x

i

− a

i

)

+

1

2!

2

X

i=1



∂

2

f(x

1

, x

2

)

∂x

2

i



|(x

1

,x

2

)=(a

1

,a

2

)

(x

i

− a

i

)

2

+



∂

2

f(x

1

, x

2

)

∂x

1

∂x

2



|(x

1

,x

2

)=(a

1

,a

2

)

(x

1

− a

1

)(x

2

− a

2

).

6. Lagrange Multipliers: The method of Lagrange multipliers provides a