Understanding Log-odds

/personal-practices

Likelihood Function

In [1]:
import matplotlib.pyplot as plt
import numpy as np

We will use a binomial dataset and we have set the p to 0.7 for this example.

In [2]:
TRUE_P = 0.7

We are creating two datasets. Both datasets will be configured so that there is a 0.7 probability of a result being 1 and a 0.3 probability of it being 0, but one will run this experiment 500 times while the other will run it 5,000 times.

In [3]:
data = np.random.binomial(n=1, p=TRUE_P, size=500)
log_likelihood_reason = np.random.binomial(n=1, p=TRUE_P, size=5000)
data[:10]
Out[3]:
array([0, 0, 1, 1, 0, 1, 1, 1, 1, 1])

Raw likelihood function of a Bernoulli trial is:

$$ L(p) = p^x(1-p)^{1-x} $$

If we want to calculate it for the entire dataset, it looks like this:

$$ L(p) = p^{\sum x_i}(1-p)^{n-\sum x_i} $$

In [4]:
def raw_likelihood(p, data):
    ones = np.sum(data == 1)
    zeros = np.sum(data == 0)
    return (p ** ones) * ((1-p) ** zeros)

The likelihood function actually serves to provide information about what probability values are possible when we have data but TRUE_P is not known. In other words, it acts as the reverse of the probability function.

$$ \text{Probability funciton: } P(x \mid \theta) $$

$$ \text{Likelihood funciton: } \mathcal{L}(\theta \mid x) = P(x \mid \theta) $$

And as you can see below, likelihood function gives the maximum output near 0.7 which was our TRUE_P

In [5]:
likelihoods = []
likelihoods_comparison = []
for p in np.linspace(start=0,stop=1,num=20):
    l = raw_likelihood(p,data)
    l_comp = raw_likelihood(p,log_likelihood_reason)
    print(f"If p={p}, Likelihood is:", l)
    likelihoods.append((p,l))
    likelihoods_comparison.append((p,l_comp))
If p=0.0, Likelihood is: 0.0
If p=0.05263157894736842, Likelihood is: 0.0
If p=0.10526315789473684, Likelihood is: 0.0
If p=0.15789473684210525, Likelihood is: 2.116611552147787e-295
If p=0.21052631578947367, Likelihood is: 2.895744737833496e-255
If p=0.2631578947368421, Likelihood is: 2.4732855920997336e-225
If p=0.3157894736842105, Likelihood is: 5.302389127136169e-202
If p=0.3684210526315789, Likelihood is: 2.230322222256273e-183
If p=0.42105263157894735, Likelihood is: 2.292909701437804e-168
If p=0.47368421052631576, Likelihood is: 2.6610183237295226e-156
If p=0.5263157894736842, Likelihood is: 8.762053522778876e-147
If p=0.5789473684210527, Likelihood is: 1.3407511261640346e-139
If p=0.631578947368421, Likelihood is: 1.0906225336645953e-134
If p=0.6842105263157894, Likelihood is: 3.708760562723947e-132
If p=0.7368421052631579, Likelihood is: 2.5243222003169164e-132
If p=0.7894736842105263, Likelihood is: 7.248764044088785e-136
If p=0.8421052631578947, Likelihood is: 3.4783763761933626e-144
If p=0.894736842105263, Likelihood is: 1.4206825902575737e-160
If p=0.9473684210526315, Likelihood is: 9.76486694815781e-196
If p=1.0, Likelihood is: 0.0

We can see that our dataset of 500 trials has the highest likelihood value, around 0.7. However, in our dataset of 5,000 trials, we see a meaningless line. The reason for this is floating-point underflow: when we multiply each p value by the others, we end up with numbers so small that 5,000 of them cause our program to crash.

In [6]:
fig, axes = plt.subplots(1,2, figsize=(12,4))
x,y = zip(*likelihoods)
x_comp,y_comp = zip(*likelihoods_comparison)
axes[0].plot(x,y)
axes[1].plot(x_comp,y_comp)
plt.show()
No description has been provided for this image

Log-likelihood Function

To solve the previous problem, we could have used a library that supports operations with very small numbers. However, the raw likelihood is a function that is difficult to compute even with pen and paper.

The reason for this is that the raw likelihood is a product function:

$$L(\theta) = \prod_{i=1}^N f(x_i \mid \theta)$$

This is exactly where the logarithm comes into play. The fastest way to convert a product into a sum is to take its logarithm. The log-likelihood ensures that our function above transforms into a sum function:

$$\ln L(\theta) = \sum_{i=1}^N \ln f(x_i \mid \theta)$$

In [7]:
def log_likelihood(p,data):
    ones = np.sum(data == 1)
    zeros = np.sum(data == 0)
    return ones * np.log(p) + zeros * np.log(1-p)
In [8]:
log_likelihoods = []
log_likelihoods_comparison = []
for p in np.linspace(start=0,stop=1,num=20):
    l = log_likelihood(p,data)
    l_comp = log_likelihood(p,log_likelihood_reason)
    print(f"If p={p}, Log-likelihood is:", l)
    log_likelihoods.append((p,l))
    log_likelihoods_comparison.append((p,l_comp))
If p=0.0, Log-likelihood is: -inf
If p=0.05263157894736842, Log-likelihood is: -1050.2252129303802
If p=0.10526315789473684, Log-likelihood is: -813.1962394327921
If p=0.15789473684210525, Log-likelihood is: -678.5127859477014
If p=0.21052631578947367, Log-likelihood is: -586.0959563858563
If p=0.2631578947368421, Log-likelihood is: -517.176098457721
If p=0.3157894736842105, Log-likelihood is: -463.4540312871044
If p=0.3684210526315789, Log-likelihood is: -420.5709259485913
If p=0.42105263157894735, Log-likelihood is: -386.0044739999962
If p=0.47368421052631576, Log-likelihood is: -358.2245656290679
If p=0.5263157894736842, Log-likelihood is: -336.30957837224
If p=0.5789473684210527, Log-likelihood is: -319.76609792734104
If p=0.631578947368421, Log-likelihood is: -308.45965379619236
If p=0.6842105263157894, Log-likelihood is: -302.6305345345402
If p=0.7368421052631579, Log-likelihood is: -303.01525968404
If p=0.7894736842105263, Log-likelihood is: -311.17074166953387
If p=0.8421052631578947, Log-likelihood is: -330.32568776479366
If p=0.894736842105263, Log-likelihood is: -368.0624774255676
If p=0.9473684210526315, Log-likelihood is: -449.0278872879778
If p=1.0, Log-likelihood is: -inf
/tmp/ipykernel_2360/1610253491.py:4: RuntimeWarning: divide by zero encountered in log
  return ones * np.log(p) + zeros * np.log(1-p)
In [9]:
fig, axes = plt.subplots(1,2, figsize=(12,4))
x,y = zip(*log_likelihoods)
x_comp,y_comp = zip(*log_likelihoods_comparison)
axes[0].plot(x,y)
axes[1].plot(x_comp,y_comp)
plt.show()
No description has been provided for this image

Even though our numbers differ from those in the likelihood function, and even though our curve is different, we can see that our graph peaks around 0.7.

Fisher Information

So what does the sharpness of the log-likelihood curve tell us? Understanding this will actually explain what Fisher’s information means.

Since our datasets are too far up on the page, let's create two new datasets.

In [10]:
p_grid = np.linspace(0.05, 0.95, 1000)
In [11]:
data_normal = np.random.binomial(n=1, p=0.5, size=50)
data_high_size = np.random.binomial(n=1, p=0.5, size=500)
In [12]:
def compute_log_lik_curve(data, p_grid):
    k = np.count_nonzero(data)
    n = len(data)
    return k * np.log(p_grid) + (n - k) * np.log(1.0 - p_grid)

First, let’s look at the plots we obtain with low n and high n.

In [13]:
ll_curve_normal    = compute_log_lik_curve(data_normal, p_grid)
ll_curve_high_size = compute_log_lik_curve(data_high_size, p_grid)
In [14]:
fig, ax = plt.subplots()
ax.plot(p_grid, ll_curve_normal, label="n=50")
ax.plot(p_grid, ll_curve_high_size, label="n=500")
plt.legend()
plt.show()
No description has been provided for this image

As you can see, as our sample size increases, we get an even sharper curve.

In other words, we see that the larger the dataset we have, the more directly it shapes how confident we can be in the probability value we estimate. As we move further away from 0.5 in our large sample, our log-likelihood value decreases more, but when the sample is small, it decreases slowly, meaning we are actually less confident about the p being 0.5.

Now, again in a binomial dataset like this, how does the curve change if we change p?

In [15]:
p_grid = np.linspace(0, 1, 1000)
In [16]:
data_half = np.random.binomial(n=1, p=0.5, size=100)
data_rigged = np.random.binomial(n=1, p=0.9, size=100)
In [17]:
ll_curve_half    = compute_log_lik_curve(data_half, p_grid)
ll_curve_rigged = compute_log_lik_curve(data_rigged, p_grid)
/tmp/ipykernel_2360/3386238865.py:4: RuntimeWarning: divide by zero encountered in log
  return k * np.log(p_grid) + (n - k) * np.log(1.0 - p_grid)
In [18]:
fig, ax = plt.subplots()
ax.plot(p_grid, ll_curve_half, label="p=0.5")
ax.plot(p_grid, ll_curve_rigged, label="p=0.8")
plt.legend()
plt.show()
No description has been provided for this image

Mathematically, Fisher information is the negative expected value of the second derivative of the log-likelihood function with respect to a parameter, here that parameter is the true p.

And statistically, Fisher information measures how much evidence our data actually carries about an unknown parameter (here it is p).

In other words, Fisher information is a statistic that can express in a single number what the likelihood function is trying to capture.

In [19]:
p1 = np.mean(data_normal)
p2 = np.mean(data_high_size)

info_norm = len(data_normal) / (p1 * (1 - p1))
info_size = len(data_high_size) / (p2 * (1 - p2))

print("Scenario 1 (Sample Size):")
print(f"  n=50:  Fisher Info = {info_norm:.1f}")
print(f"  n=500: Fisher Info = {info_size:.1f}")
Scenario 1 (Sample Size):
  n=50:  Fisher Info = 200.3
  n=500: Fisher Info = 2002.6
In [20]:
p3 = np.mean(data_half)
p4 = np.mean(data_rigged)

info_half = len(data_half) / (p3 * (1 - p3))
info_rigged = len(data_rigged) / (p4 * (1 - p4))

print("\nScenario 2 (Location of p):")
print(f"  p=0.5: Fisher Info = {info_half:.1f}")
print(f"  p=0.9: Fisher Info = {info_rigged:.1f}")
Scenario 2 (Location of p):
  p=0.5: Fisher Info = 400.2
  p=0.9: Fisher Info = 1021.5

The Problem with a Straight Line

We want to predict probability $p$ using a feature $x$ (e.g. hours studied).

If we use a standard linear equation:

$$p = w \cdot x + b$$

It breaks:

  1. It predicts probabilities below 0 and above 1.
  2. It adds the same fixed amount of probability per hour, ignoring the fact that moving $p$ near the edges requires much more evidence.
In [21]:
x = np.linspace(0, 10, 100)
p_line = 0.2 * x - 0.5

plt.plot(x, p_line)
plt.axhline(0)
plt.axhline(1)
plt.show()
No description has been provided for this image

Log-odds

To let a linear equation add up freely, we need a scale that runs from $-\infty$ to $+\infty$.

We integrate the Bernoulli Fisher Information:

$$z = \int \frac{1}{p(1 - p)} \, dp = \ln\left(\frac{p}{1 - p}\right)$$

This is log-odds.

In [22]:
p = np.linspace(0.01, 0.99, 100)
z = np.log(p / (1 - p))

plt.plot(p, z)
plt.show()
No description has been provided for this image

Sigmoid

Now our linear model can safely predict $z$:

$$z = w \cdot x + b$$

To convert this score $z$ back into a valid probability $p$, we take the inverse:

$$p = \frac{1}{1 + e^{-z}}$$

This is the sigmoid function. It takes any number from $-\infty$ to $+\infty$ and squashes it into $(0, 1)$.

In [23]:
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z_values = np.linspace(-6, 6, 100)
plt.plot(z_values, sigmoid(z_values))
plt.show()
No description has been provided for this image

In a future article, we will discuss how all of these can be useful in logistic regression.

>