When is Inverse-Variance Weighting Useful?

/personal-practices

A situation where inverse-variance weighting is useless

Let's import our libraries first.

In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Imagine a temperature sensor that only works occasionally and sends you the data when it's active. We can simulate this situation by having a fake data and only working with it's samples.

In [2]:
data = np.random.normal(50, 100, 1000)
data[:5]
Out[2]:
array([  37.27693311,   15.13259373,    7.46115117,  -50.77290745,
       -270.6140831 ])

Let's also see how the actual data looks like in a KDE plot.

In [3]:
fig, ax = plt.subplots()
sns.kdeplot(data, ax=ax)
plt.show()
No description has been provided for this image
In [4]:
samples = []
for _ in range(20):
    samples.append(np.random.choice(data, 20, replace=False))

samples[:5]
Out[4]:
[array([ -74.98662301,  179.62455323,  224.11065512,   46.56147177,
         -18.94661844, -132.20663336,  -15.28235586, -112.97457116,
          81.13206881,   36.86413283,  160.24334145, -120.37302178,
        -101.21686082,   26.4634801 ,  183.37522426,  195.71379819,
          19.88449303,   37.18591332,   12.47076881,   65.43001358]),
 array([  56.79702411,   44.2865592 ,  145.32558899,   65.56996895,
         173.41508726,  100.44317883, -161.83279981,   35.06114224,
        -140.94216883,   61.78155748,   50.82105654,   93.29778735,
         -23.0767073 ,  -25.71994147,    1.10164306,  117.40508376,
        -119.12100472,   93.48208878,  -74.59389243,  219.08214477]),
 array([222.09474198,  -7.42307391, 129.71596102, 119.91140318,
        128.56112191,  87.2229868 ,  24.30421903,  42.80483025,
         85.95878824, 223.86183504, -41.23215708, 133.25428582,
         12.94644762,  46.1620496 , -45.1695612 , 118.65432811,
         53.98543064,  90.45520425,   9.31113739,  16.77390818]),
 array([-191.10166318,  130.80138794,  191.9284419 ,  -16.64185157,
         247.265694  ,  220.10679527,   -8.97064516,   91.82789801,
         234.63743083, -119.66786406,   19.88449303,  -71.03990917,
         148.3850635 ,   26.67132917,  218.85442457,   59.69259592,
         -14.43402757,  103.8770208 , -153.76071984,  105.78844119]),
 array([ 100.56682631,  -64.59824508, -123.77941753,  127.21319734,
          63.88411722, -117.4214935 ,   35.91868866,  -32.51310577,
         -47.20836955,    7.47059462,  -31.1678416 ,   48.21624288,
           0.24512755,   69.57679444,  166.10058754,  -89.07919175,
          53.25749328,   68.17405875,  173.41508726,  129.32812613])]

And here we have 20 samples from the population data.

In [5]:
for i, s in enumerate(samples, 1):
    print(f"Var of {i}:", np.var(s), f"| Mean of {i}:", np.mean(s))
    print()
Var of 1: 11714.883573891868 | Mean of 1: 34.65366150276499

Var of 2: 9961.811848037496 | Mean of 2: 35.62916983855876

Var of 3: 5427.475608073739 | Mean of 3: 72.6076943439655

Var of 4: 16438.01904652968 | Mean of 4: 61.205216779095224

Var of 5: 7589.277610611684 | Mean of 5: 26.87996385949532

Var of 6: 5190.640450575755 | Mean of 6: -33.6151483632085

Var of 7: 7079.193741171267 | Mean of 7: 70.12837573511774

Var of 8: 4560.790067073571 | Mean of 8: 50.88782482076003

Var of 9: 6972.190813058676 | Mean of 9: 59.940971039769785

Var of 10: 7003.905919452887 | Mean of 10: 88.94777860503591

Var of 11: 8740.14767106515 | Mean of 11: 45.34101462318393

Var of 12: 4642.355580173226 | Mean of 12: 80.23587147512384

Var of 13: 6785.502067856843 | Mean of 13: 16.31232123212671

Var of 14: 9655.111999338962 | Mean of 14: 52.47163092369749

Var of 15: 7836.815893677497 | Mean of 15: 37.91041730520227

Var of 16: 5171.5134994666605 | Mean of 16: 84.12385911874793

Var of 17: 10559.70658949736 | Mean of 17: 43.960035121310064

Var of 18: 10580.842974883 | Mean of 18: 43.15486810017212

Var of 19: 5026.274739745471 | Mean of 19: 66.41926003953594

Var of 20: 5460.737933810502 | Mean of 20: 38.35677856947583

We can show the mean and the variance of these samples in a beautiful pandas data frame. It is also going to make it easier for us to work with parameters of the samples.

In [6]:
data_sum = pd.DataFrame(
    {
        "mean": [np.mean(s) for s in samples],
        "var": [np.var(s) for s in samples],
        "weights": [1 / np.var(s) for s in samples],
    }
)
data_sum
Out[6]:
mean var weights
0 34.653662 11714.883574 0.000085
1 35.629170 9961.811848 0.000100
2 72.607694 5427.475608 0.000184
3 61.205217 16438.019047 0.000061
4 26.879964 7589.277611 0.000132
5 -33.615148 5190.640451 0.000193
6 70.128376 7079.193741 0.000141
7 50.887825 4560.790067 0.000219
8 59.940971 6972.190813 0.000143
9 88.947779 7003.905919 0.000143
10 45.341015 8740.147671 0.000114
11 80.235871 4642.355580 0.000215
12 16.312321 6785.502068 0.000147
13 52.471631 9655.111999 0.000104
14 37.910417 7836.815894 0.000128
15 84.123859 5171.513499 0.000193
16 43.960035 10559.706589 0.000095
17 43.154868 10580.842975 0.000095
18 66.419260 5026.274740 0.000199
19 38.356779 5460.737934 0.000183

Here we can also see a column called weights, it is calculated by taking the reciprocal of variance ($\frac{1}{\sigma^2}$). We are going to use this weights to calculate the weighted mean, which we expect to yield better results.

In [7]:
weighted_mean = np.sum(data_sum["weights"] * data_sum["mean"]) / np.sum(
    data_sum["weights"]
)
In [8]:
actual_mean = np.mean(data)
print("Actual mean:", actual_mean)
print("Weighted mean:", weighted_mean)
print("Grand mean:", np.mean(data_sum["mean"]))
Actual mean: 52.84392615401629
Weighted mean: 49.72403371490683
Grand mean: 48.77757823349655

As you can see in the results, grand mean is giving us a better result compared to weighted mean. This is because we are using the same distribution, or same sensor, for sampling. When we draw samples from the same distribution, since the “uncertainity” of the samples remains the same, there is no point in using the weighted mean.

A situation where inverse-variance weighting is useful

You can think of this actual mean as the actual temperature of a room.

In [9]:
actual_mean = 30

Below, we simulated different sensors with different variances (or in this context, different quailities). But because the temperature of the room cannot change from sensor to sensor, it remains the same (actual_mean).

In [10]:
samples_from_different_dists = []

samples_from_different_dists.append(np.random.normal(actual_mean, 10, 10))
samples_from_different_dists.append(np.random.normal(actual_mean, 40, 10))

# Adding a lot of bad sensors to prove a point :D
samples_from_different_dists.append(np.random.normal(actual_mean, 150, 10))
samples_from_different_dists.append(np.random.normal(actual_mean, 150, 10))

We created the same pandas data frame the see the results clearly. But here, difference between weights are more noticeable. Because we rely more on the low-variance sensor, while also taking the high-variance sensor's data into account.

In [11]:
different_data_sum = pd.DataFrame(
    {
        "mean": [np.mean(s) for s in samples_from_different_dists],
        "var": [np.var(s) for s in samples_from_different_dists],
        "weights": [1 / np.var(s) for s in samples_from_different_dists],
    }
)
different_data_sum
Out[11]:
mean var weights
0 27.420751 80.220913 0.012466
1 40.138350 693.570644 0.001442
2 37.333577 19038.164770 0.000053
3 45.710113 22407.137072 0.000045
In [12]:
weighted_mean_diff = np.sum(
    different_data_sum["mean"] * different_data_sum["weights"]
) / np.sum(different_data_sum["weights"])
weighted_mean_diff
Out[12]:
np.float64(28.82553227321503)

Here, the weighted mean calculated with IVW method is much closer to the actual mean than the grand mean because we are using different distributions (sensors) with different variances.

In [13]:
print("Actual mean:", actual_mean)
print("Weighted mean:", weighted_mean_diff)
print("Grand mean:", np.mean(different_data_sum["mean"]))
Actual mean: 30
Weighted mean: 28.82553227321503
Grand mean: 37.65069780282512

You may also ask why should we bother with high-variance data if we already have a decent low-variance data. The answer to this question is: Data is data, no matter what. In most scenarios, using high-variance data and giving it less weight is much more appropriate methodology than not using it at all.

>