Covariance, Correlation, and OLS

/personal-practices

Covariance

In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import linregress

First, we define two random variables with sample size ($n$) 500.

$$ \begin{gathered} X \sim Exp(\lambda=0.5) \\\\ \epsilon \sim \mathcal{N}(\mu=9, \sigma^2=4) \\\\ Y \sim 3X + \epsilon \end{gathered} $$

In [2]:
X = np.random.exponential(scale=2.0, size=500)
Y = 3.0 * X + np.random.normal(loc=9.0, scale=2.0, size=500)
In [3]:
mean_X = np.mean(X)
mean_Y = np.mean(Y)
In [4]:
print(f"E[X] = {mean_X}")
print(f"E[Y] = {mean_Y}")
E[X] = 2.1336007409114246
E[Y] = 15.363376328434354

Since we are interested in calculating the covariance, let’s calculate the deviations for both X and Y first.

$$ \begin{gathered} \text{dev_X} = X_i - \bar{X} \\\\ \text{dev_Y} = Y_i - \bar{Y} \end{gathered} $$

In [5]:
dev_X = X - mean_X
dev_X
Out[5]:
array([ 2.11478442e+00, -1.53484987e+00, -2.02658778e+00, -2.05792009e+00,
       -1.94684352e+00, -1.14041160e+00, -1.47674460e+00, -1.05657391e+00,
        2.41473913e+00, -1.81411083e+00, -3.37117460e-01,  2.29944576e-01,
        3.50483965e+00, -1.68885862e+00, -2.03515813e+00, -1.74901381e+00,
        4.71745646e+00,  5.96438710e-01,  5.62687083e+00, -1.52722965e+00,
        4.93381586e-01,  1.12311022e+00, -1.04925067e+00,  3.43084706e+00,
       -1.46298267e+00, -5.30741295e-01, -6.96607109e-01, -1.71850488e+00,
       -1.17331214e+00, -2.04231794e+00,  3.38878185e+00,  3.44906564e+00,
       -1.18537726e+00,  8.44203515e-02, -2.38919766e-02,  1.63878382e+00,
        1.19312901e+00, -1.42684800e+00,  9.42641987e-01,  2.81181948e+00,
        5.13223159e+00,  3.16212844e-01, -1.31559026e+00, -2.00921544e+00,
        2.45544099e+00,  2.35225703e+00,  2.01256012e-01,  9.34978871e-01,
       -2.10086552e+00,  1.13590459e+00, -1.88073195e+00,  8.36872853e-01,
       -5.92260594e-01, -1.66681963e+00, -1.52913109e+00, -1.98465426e+00,
       -7.62100850e-01, -1.00812033e+00,  7.43383456e-02,  1.07083611e+00,
       -8.49099219e-02, -9.64927908e-01, -3.52349839e-01,  4.46579218e-01,
        2.54533090e+00, -1.70098721e+00,  1.16999343e-01, -2.09873273e+00,
        5.45607834e+00, -9.74813107e-01, -9.22016116e-01, -1.59480638e+00,
       -1.19440583e+00, -1.41640841e+00, -1.67400524e+00,  1.86657135e+00,
       -1.34379314e+00, -1.32564002e+00, -1.75566799e+00,  6.87114062e-01,
       -6.90593634e-01, -2.09567883e+00,  1.24805501e+00, -1.64022164e+00,
        1.19350402e-01, -1.27438650e+00, -1.91190875e+00,  1.04254739e+00,
       -5.75782722e-01, -1.19604378e+00,  1.89595167e+00,  7.11111490e-01,
       -1.19955416e+00, -1.78728462e+00,  5.18140268e+00,  1.59627874e+00,
       -1.00337850e+00,  1.17986485e+00, -1.59447030e+00,  1.23165403e+00,
        1.24380476e+00, -1.92099839e+00,  6.29697156e-01,  8.92696068e+00,
       -2.10165856e+00, -1.15416959e+00, -2.03876527e+00,  1.61330298e+00,
        6.76097628e+00, -1.42513193e+00,  3.27451181e+00, -1.93591915e+00,
       -9.12205135e-01, -1.40713794e+00, -1.71575590e+00, -7.74419554e-02,
       -1.61842286e+00, -7.47000447e-01, -1.76041043e+00, -2.10019559e+00,
        1.30913491e+00,  2.10931943e+00,  1.64105284e+00, -1.30914971e+00,
        4.26949519e+00, -1.66240106e+00, -2.04655654e+00, -1.32349957e+00,
       -1.58913003e+00, -2.25441595e-01, -1.48799567e+00, -1.47203639e+00,
        3.52509756e+00, -4.25096794e-01, -8.07525935e-01,  3.56210741e+00,
       -1.13668420e+00, -9.91947869e-01,  7.89450907e+00,  1.30587436e+00,
        3.84158654e-01,  1.78284993e+00,  2.37327614e+00, -2.04474589e+00,
        2.41666259e+00, -1.26899149e+00, -1.21334106e+00,  3.54981568e+00,
       -1.59674569e+00, -9.10658277e-02, -1.79010825e-01,  8.05690656e-02,
       -1.78701673e+00, -1.94636212e-01, -1.23942636e+00, -3.06973600e-02,
        6.73575633e-02, -5.79767474e-01,  1.31374829e+00, -2.09949673e+00,
       -1.74304759e+00, -1.36122784e+00,  6.40727341e+00, -3.00342124e-01,
       -6.49617852e-01, -8.67844609e-01, -1.52879537e+00, -2.09013667e+00,
       -1.94620349e+00, -1.17496872e+00, -3.89764693e-01,  1.44419648e+00,
       -1.17128507e+00,  6.89293788e-01,  1.55955555e+00, -1.58220199e+00,
       -1.74147441e+00, -1.68302285e+00, -1.48508921e+00, -4.04915909e-01,
        2.27334006e+00,  5.45300736e-01, -4.71731767e-01, -1.94757285e+00,
        1.83003635e+00, -1.70968179e+00, -1.82673850e+00, -1.31902810e+00,
       -2.05296354e+00, -1.15738467e+00,  6.72792787e+00,  6.39478165e+00,
       -1.39698997e+00,  5.84512118e-01, -6.75076558e-01,  1.46175034e-01,
       -1.74782745e-01, -7.57141198e-01,  9.31370162e+00, -2.55109968e-01,
        1.65773978e+00, -1.65176831e+00, -1.72127499e+00,  1.07637373e+00,
        4.52005275e-01,  2.01393555e+00, -1.06858920e+00, -2.03946734e+00,
       -1.94370495e+00,  3.52408908e+00, -1.29859457e+00, -1.01717998e+00,
       -1.96290565e+00,  1.21201359e-01,  2.26037165e+00,  2.24126649e+00,
       -1.50676606e+00,  3.10300747e+00,  1.64710390e-01, -1.24637843e+00,
        6.39904433e+00, -2.02101797e+00, -1.89950435e+00, -1.77808096e+00,
       -2.08016561e+00,  8.47801386e-01,  1.37126411e+00,  7.68148389e+00,
       -5.88202782e-01, -1.20964596e+00,  3.53241017e-01,  3.56320337e-01,
       -1.57113835e+00, -1.08141659e+00, -2.36228733e-01, -1.67573017e+00,
        5.82360850e-01,  3.89444708e+00, -6.32146377e-01,  2.12637312e+00,
       -1.39494014e+00, -3.74631017e-01,  1.11748947e+00, -1.87076131e+00,
        6.55167839e+00, -8.43155281e-01, -1.25878096e+00, -2.06692524e+00,
       -1.97769504e+00,  9.03878175e-01, -1.78620758e+00,  1.84296993e+00,
       -2.06025031e+00, -1.79445380e-01,  4.06150607e-01, -1.87889806e+00,
       -1.07563494e+00, -1.32024736e+00, -3.20498862e-01, -3.94222598e-01,
       -6.40615913e-01,  8.63654781e-01,  9.12682951e-02, -1.17398704e+00,
        3.55158435e-02, -9.93763363e-01, -8.27974564e-01,  2.20963268e-01,
        4.29106769e+00, -1.36580535e+00,  3.54949015e+00,  1.55700702e-01,
        3.41840051e-01, -1.13128146e+00,  4.31960346e+00,  2.57212085e+00,
        3.39825687e-01,  2.49895712e+00,  1.13725847e-03, -5.43321596e-01,
       -1.14962761e+00,  6.08805717e-01,  3.18496059e+00, -1.00038578e-01,
       -1.94445967e+00,  4.44137817e+00, -1.31112975e+00, -1.97828907e-01,
        8.72826377e-01, -1.66044276e+00,  2.12335624e-01,  2.34828319e-01,
       -4.37614448e-01, -1.41976552e+00,  6.63987334e-01, -1.24598654e+00,
       -1.96652033e+00, -9.25176421e-01,  1.31133959e+00, -1.43560903e-01,
       -5.41096488e-01, -1.41550052e+00, -2.05707187e+00, -2.68286096e-03,
        3.66191202e+00, -6.45142952e-01, -2.01281908e+00, -1.97014724e+00,
       -1.47556988e+00,  2.31364908e+00, -1.66754887e+00, -1.70855721e+00,
        2.03663417e+00, -3.90542187e-02,  9.09148783e-02,  3.79002201e+00,
       -1.90642207e+00, -9.96532927e-01, -1.05310095e+00, -1.92524392e+00,
       -5.14939078e-02, -1.15367254e+00,  1.26698488e+00, -3.07291359e-01,
        1.50913263e+00, -2.06784119e+00, -1.25537553e+00, -1.80670242e+00,
        4.78049514e+00, -1.45503748e+00, -2.00314086e+00, -1.91749327e+00,
       -1.19308801e-01, -9.02513874e-01, -1.89625967e+00, -1.07806703e+00,
        2.62633574e+00, -1.40606422e+00, -1.01116733e+00, -1.99252437e+00,
        4.37968587e+00, -4.86077311e-01,  1.90090324e+00, -5.25424424e-01,
       -1.48887978e+00, -2.01747193e+00, -1.59161808e+00, -1.95683629e+00,
        2.90222492e+00,  6.46884749e-01, -1.22273111e+00, -1.41834995e+00,
       -1.59387356e+00, -2.01227917e+00, -1.09982197e+00,  1.04481428e+00,
       -3.68351190e-01,  4.94021988e+00, -8.16293346e-01,  2.93844597e-01,
       -1.25731355e-01,  3.06071241e+00, -9.49166194e-01,  7.19403037e+00,
       -1.19732578e+00, -9.66488765e-01, -1.35293972e+00,  5.72923719e-01,
       -2.09383014e+00,  5.96006691e-02,  1.55873592e+00, -1.67564369e+00,
       -1.47093628e+00,  1.22148246e-02, -1.07149046e+00,  7.50752055e-02,
        2.65614109e-01, -1.63052945e+00,  1.21792265e+00, -8.75308834e-02,
       -1.08801373e+00, -4.84780141e-01, -9.12020545e-01,  3.26332830e+00,
       -5.07257928e-01, -9.55044724e-01,  4.33878194e+00, -2.13272318e+00,
       -1.22300724e+00, -1.02621828e+00, -5.07035132e-01, -3.73096001e-01,
       -1.01870548e+00, -2.12075569e+00,  6.21866969e+00, -2.08967190e+00,
       -5.56722769e-01, -9.85787170e-01, -1.33228806e+00,  3.81313217e+00,
       -5.14471348e-01, -1.68038547e+00, -1.04173685e+00, -7.15589078e-01,
       -7.53063074e-01, -1.62481842e-01, -8.23463249e-01,  1.72746607e+00,
        1.69407686e-01,  1.58911103e+00,  1.87575983e-01, -6.47340032e-01,
       -1.10079384e-01,  3.06566897e+00,  1.50799515e+00,  7.55358883e-01,
        2.26548982e+00, -2.00626897e+00,  4.16981650e-01, -8.06986580e-01,
        1.22860855e-02, -1.78487505e+00, -5.24661587e-01, -1.89801619e+00,
       -1.50797229e+00, -1.57349965e-01, -1.91747452e+00,  3.17818869e+00,
       -1.93573154e+00, -2.85531459e-01, -6.24311234e-01, -9.20072717e-01,
       -4.04164979e-01, -1.74600388e+00, -2.00119197e+00, -2.41892269e-01,
        6.41716970e-01, -8.56851659e-01,  3.58491967e+00,  3.07831656e-01,
       -4.37588221e-01,  3.13962681e+00,  8.73091123e-02, -8.41376907e-01,
        2.76857274e+00, -9.97889702e-01, -2.02159425e+00, -1.22816442e+00,
        1.54870122e+00, -1.52078178e+00, -1.44676909e+00, -1.90508676e+00,
       -3.86243253e-01, -7.79757466e-01,  2.13348646e+00,  3.23707549e+00,
       -1.99482812e+00, -1.54861474e-01, -1.54481614e+00,  7.31532745e-01,
       -1.40108957e+00, -2.06056241e+00, -6.49705282e-01,  4.53544376e-01,
       -1.32185633e+00, -1.85617698e+00, -1.83284798e+00, -1.96912765e+00,
       -1.92942830e+00,  7.22930408e-01, -8.67273852e-01, -1.85840691e+00,
       -1.80349558e+00, -1.61128278e+00,  1.04996092e+01, -1.29365864e+00,
       -6.91128816e-01,  7.62808546e+00,  2.79954278e-01,  4.88783645e+00,
       -9.96432716e-01, -4.78050943e-01,  2.87464617e+00, -1.50415663e+00,
        3.97414788e+00, -2.12962306e+00, -1.15244575e+00, -2.02699967e+00,
       -1.42224286e+00,  8.87520640e+00, -1.57966534e+00,  7.01157359e-02,
       -1.23382621e+00, -1.97483525e+00, -5.73571603e-01, -1.83880385e+00,
       -1.65693174e+00, -1.67841238e+00, -1.93868938e+00, -8.17219729e-01])
In [6]:
dev_Y = Y - mean_Y
dev_Y
Out[6]:
array([ 3.88585677e+00, -3.04528790e+00, -9.64858525e+00, -5.80516384e+00,
       -8.77312578e+00,  6.68169742e-01, -3.80295835e+00, -1.11764635e+00,
        5.70851731e+00, -5.95639222e+00,  1.47866021e+00,  1.99343803e+00,
        1.16411119e+01, -7.80403576e+00, -4.93234180e+00, -5.94882290e+00,
        1.42406456e+01,  1.50904621e+00,  1.32482723e+01, -2.08783317e+00,
        4.97165379e-01,  3.08760511e-01, -2.77678354e+00,  8.83882346e+00,
       -6.89164443e+00, -4.98407225e-01,  1.38211273e+00, -5.24205794e+00,
       -3.49093894e+00, -6.93515428e+00,  1.07885983e+01,  1.17496130e+01,
       -2.23078144e+00,  1.22137438e+00, -3.10007470e+00,  6.39975384e+00,
        4.72069878e+00, -2.29531262e+00, -5.18419476e-01,  8.53307108e+00,
        1.67163365e+01,  1.31120760e+00, -7.47054745e+00, -6.60992778e+00,
        7.41052430e+00,  7.91011590e+00, -2.18461463e+00,  6.30888055e-01,
       -7.26738780e+00,  3.38536208e+00, -5.41686621e+00,  2.64684938e+00,
       -2.16841235e+00, -3.23080642e+00, -5.09863370e+00, -6.24336690e+00,
       -1.81197116e-01, -5.15089875e+00, -6.95531967e+00,  2.92505358e+00,
       -2.60127045e-02, -7.09395005e-01, -1.34111938e+00,  2.43458696e+00,
        8.92997046e+00, -5.63574303e+00,  4.17431190e-01, -6.38511361e+00,
        1.59367176e+01, -3.33314205e+00, -3.39846651e+00, -5.64199557e+00,
       -3.14647935e+00, -3.72230268e+00, -4.31720961e+00,  4.95078679e+00,
       -5.16942027e+00, -3.55857087e+00, -7.32081342e+00, -1.24914315e+00,
       -4.18429591e+00, -1.99168907e+00,  6.89687930e+00, -3.60567889e+00,
       -3.10553603e+00, -4.90120698e+00, -6.21181748e+00,  4.95765341e+00,
       -2.19724885e+00, -1.94424215e+00,  5.16754364e+00,  2.23502044e+00,
       -1.14038188e+00, -3.91558156e+00,  1.62659136e+01,  5.53326408e+00,
       -3.31290503e+00,  5.24719374e+00, -5.42786492e+00,  3.25465634e-01,
        1.94878211e+00, -1.24010310e+00,  2.51564795e+00,  2.76483224e+01,
       -9.02979246e+00, -1.95551286e+00, -6.90086354e+00,  4.30933855e+00,
        2.31775155e+01, -4.48091935e+00,  5.55773840e+00, -2.92750893e+00,
       -1.22997426e+00, -6.71811030e+00, -4.72934609e+00, -1.55701063e+00,
       -4.47713902e+00, -1.29793693e+00, -5.78427317e+00, -5.37844048e+00,
        4.48606984e+00,  5.20390501e+00,  4.77812464e+00, -4.04764062e+00,
        1.41366869e+01, -6.77374823e+00, -6.65902053e+00, -2.59051465e+00,
       -4.70507487e+00,  3.54534779e+00, -5.23423785e+00, -4.30354273e+00,
        1.09505576e+01, -1.68873142e+00, -1.53085306e+00,  8.99036876e+00,
        4.62774318e-02, -4.91566922e+00,  2.58197043e+01,  4.34986034e+00,
        1.19802519e+00,  5.91736126e+00,  5.61342730e+00, -1.01675291e+01,
        7.39632166e+00,  2.45790569e-01, -4.49727658e+00,  1.15318655e+01,
       -2.73160552e+00, -1.99845763e+00, -3.67709792e+00,  6.53567050e-01,
       -3.36502576e+00, -3.34428056e+00, -5.52208689e+00, -1.48043118e+00,
       -1.82653130e+00, -5.50027465e+00,  3.03196561e+00, -9.56148426e+00,
       -5.38410048e+00, -3.44558265e+00,  2.02459885e+01, -8.62258910e-01,
       -2.21318910e+00,  3.30311163e-01, -3.43876245e+00, -7.83463604e+00,
       -5.80925983e+00, -5.47889875e+00, -1.16233519e+00,  3.38523200e+00,
       -4.02777747e+00,  4.91154132e+00,  4.40822847e+00, -2.31268103e+00,
       -4.26692624e+00, -9.52433211e+00, -2.91205664e+00, -1.78639865e+00,
        4.89116164e+00, -4.15291225e-01, -1.11491953e+00, -7.41009875e+00,
        5.75874475e+00, -6.97946534e+00, -6.92830226e+00, -5.04835536e+00,
       -5.59711884e+00, -1.62332590e+00,  1.86027072e+01,  2.12401642e+01,
       -2.85871090e+00,  2.14466651e+00, -2.46521343e+00, -2.14253278e+00,
       -2.40491599e-01, -4.09754559e+00,  2.48173965e+01, -1.54344302e+00,
        5.99704027e+00, -2.45634188e+00, -6.04613150e+00,  4.45604005e+00,
        2.91355296e+00,  4.02712727e+00, -1.75568459e+00, -7.60162297e+00,
       -9.44258512e+00,  9.50561782e+00, -5.52032802e+00, -1.73293953e+00,
       -7.22614179e+00,  7.33722130e-01,  4.00514886e+00,  4.42460479e+00,
       -6.16112226e+00,  8.16511128e+00, -1.42252548e+00, -4.01631820e+00,
        2.16134094e+01, -8.29747981e+00, -6.53980032e+00, -3.35118296e+00,
       -7.55344606e+00,  3.06042582e+00,  7.65218150e+00,  1.94760971e+01,
       -5.70805741e+00, -3.61309348e+00,  6.09165436e-01,  1.63907779e+00,
       -3.57582346e+00, -4.98987124e+00, -2.82874272e+00, -4.16447028e+00,
       -3.49408823e-01,  9.88715976e+00, -5.90524754e-01,  6.34702177e+00,
       -3.59791008e+00,  1.42870736e+00,  2.50804083e+00, -5.79063564e+00,
        2.17479247e+01, -4.60503284e+00, -3.74146384e+00, -5.90766863e+00,
       -2.90916068e+00,  1.78203366e+00, -3.97638637e+00,  3.81696607e+00,
       -6.49655340e+00,  2.44107982e-01,  2.17548754e-01, -4.53580799e+00,
       -3.82708131e+00, -5.73120271e+00,  7.78872325e-01, -1.94621166e+00,
       -4.44620793e+00,  1.09977286e+00, -1.89793767e+00, -4.14552267e+00,
        3.90447245e+00, -2.66220225e+00, -4.33511177e+00,  2.64396338e+00,
        1.44899704e+01, -1.78137498e+00,  1.27385002e+01, -5.22076830e-01,
        1.22123420e+00, -2.60758064e+00,  1.40799371e+01,  8.99220070e+00,
        3.51457168e+00,  9.94235963e+00, -2.37954252e+00,  6.30649394e-01,
       -5.94808550e+00,  3.64921492e-01,  7.78819998e+00,  1.91975767e+00,
       -5.79663032e+00,  1.34589375e+01, -6.06096261e+00, -7.52943381e-01,
       -1.42492376e+00, -3.98243152e+00, -7.42891690e-01,  3.42681850e+00,
       -2.17393202e+00, -4.00049698e+00,  3.83330070e+00, -5.92278244e+00,
       -4.88082405e+00, -1.63334689e+00,  4.31774032e+00,  2.94913332e+00,
        9.95222934e-01, -2.79808286e+00, -4.58928422e+00,  1.33695388e+00,
        1.32835436e+01, -4.11274826e+00, -5.81422415e+00, -3.66321161e+00,
       -8.10197711e+00,  7.54558635e+00, -4.73278306e+00, -3.91286546e+00,
        7.40710147e+00,  1.03544523e+00,  2.04954578e+00,  1.07667535e+01,
       -4.53914415e+00, -1.61488364e+00, -1.14501194e+00, -3.74075140e+00,
       -9.06184271e-01, -4.94544673e+00,  5.94271358e+00, -1.54330373e+00,
        6.89550676e+00, -6.04804088e+00, -6.57056591e+00, -5.21262147e+00,
        1.31992455e+01, -2.11814153e+00, -5.69137533e+00, -5.53342547e+00,
       -1.86952312e+00, -3.67908407e+00, -1.00185484e+01, -2.31100311e+00,
        8.68677499e+00, -5.22735752e+00, -4.28158054e+00, -4.65739308e+00,
        1.35099126e+01, -8.05583420e-01,  7.38424104e+00,  6.44300571e-01,
       -2.89867190e+00, -7.86202600e+00, -3.02223899e+00, -7.39194131e+00,
        9.36441074e+00,  3.16585296e+00, -4.06415075e+00, -3.70748078e+00,
       -6.36706710e+00, -6.36070585e+00, -5.28638628e+00,  3.28731642e+00,
        1.04098986e+00,  1.59833912e+01,  2.15550365e+00, -3.40957767e+00,
       -4.10831103e+00,  1.36286591e+01,  9.49692840e-01,  2.52934073e+01,
       -3.18392540e+00, -2.62118212e+00, -4.28914741e+00,  7.70140760e-01,
       -7.21329476e+00, -1.27478565e+00,  4.35432632e+00, -3.76856379e+00,
       -7.12855878e+00,  3.29421967e+00, -6.94862646e-01, -5.06346536e-01,
        7.56580655e-01, -2.09754188e+00,  2.17192202e+00,  9.02427562e-01,
       -2.26699669e+00, -8.08407238e-01, -1.73183844e+00,  1.23071391e+01,
        1.74938189e+00, -4.58257480e+00,  1.41158628e+01, -3.92150993e+00,
       -3.86251441e+00, -6.43592587e+00, -1.98620008e+00,  7.62656624e-01,
       -4.59082858e+00, -4.48972135e+00,  2.03216662e+01, -8.14703568e+00,
       -1.03371353e+00, -3.14660572e+00, -5.18645419e+00,  1.12176114e+01,
       -6.78614312e-01, -5.77943275e+00, -4.04606660e+00, -1.33684548e+00,
       -9.09165749e-01, -1.43594778e+00, -4.21714942e-01,  4.34813805e+00,
        2.82820835e+00,  2.58896926e+00,  3.18962597e+00, -1.55563935e+00,
        6.33276297e-01,  9.53333664e+00,  4.61695070e+00,  4.22300031e+00,
        8.71009301e+00, -7.48985818e+00,  5.08325726e-01, -2.31951868e+00,
        3.99580765e-01, -4.73054037e+00, -7.19696068e+00, -8.52915575e+00,
       -5.62123661e+00, -1.68620447e+00, -8.40923713e+00,  9.64231050e+00,
       -5.60438401e+00, -3.69209720e+00,  2.23931994e+00, -4.56261676e+00,
       -1.61666295e+00, -6.62986419e+00, -2.83029814e+00, -2.99585716e+00,
        1.50694566e+00, -1.83742220e+00,  1.00454489e+01, -9.58432404e-01,
       -1.36473589e+00,  1.05038854e+01,  1.83076074e+00, -6.60014768e+00,
        9.29204702e+00, -4.28338663e+00, -4.58088163e+00, -5.95142442e+00,
        3.93380447e+00, -5.91045848e+00, -6.02749242e+00, -9.21040437e+00,
        1.91577060e+00, -1.91756845e+00,  5.66247853e+00,  1.01003804e+01,
       -3.99950569e+00, -1.50039007e+00, -3.43004901e+00,  1.07790579e+00,
       -2.42601927e+00, -8.02706787e+00,  7.96261226e-01, -7.06987516e-01,
       -8.41540997e-01, -7.20731754e+00, -1.10299396e+00, -4.00619615e+00,
       -5.48656753e+00,  4.98836795e+00, -1.28842100e-01, -3.49594966e+00,
       -5.89469083e+00, -6.95998333e+00,  3.67145338e+01, -3.45522142e+00,
       -2.43427031e+00,  2.39804522e+01,  1.95558874e+00,  8.89867148e+00,
       -1.05182707e+00,  3.10232938e-01,  1.13942049e+01, -6.46881990e+00,
        1.32491146e+01, -5.57454171e+00,  9.20997522e-01, -8.81218661e+00,
       -2.34866432e+00,  2.31619403e+01, -3.58094533e+00, -2.67131770e+00,
       -1.26481531e+00, -8.42925712e+00, -2.84433034e+00, -6.20243881e+00,
       -3.81319204e+00, -8.43865857e+00, -6.83610854e+00, -2.74329582e+00])

To find the covariance of X and Y, we calculate the mean of the product of the deviations of the two random variables that is the expected value.

$$ Cov(X,Y) = E[\text{dev_X}\cdot\text{dev_Y}] $$

In [7]:
cov_XY = np.mean(dev_X * dev_Y)
cov_XY
Out[7]:
np.float64(14.261783390722139)
In [8]:
cov_matrix = np.cov(X, Y, bias=True)
cov_matrix
Out[8]:
array([[ 4.68883875, 14.26178339],
       [14.26178339, 47.02138956]])

As we can see below, NumPy’s built-in method for calculating covariance also yields the same result as the one we derived.

In [9]:
np.isclose(cov_matrix[0][0], np.mean(dev_X * dev_X))
Out[9]:
np.True_

Correlation

The only difference between correlation and covariance is that correlation is normalized.

$$ \begin{gathered} z_X = \frac{X_i - E[X]}{\sigma_X} \\\\ z_Y = \frac{Y_i - E[Y]}{\sigma_Y} \end{gathered} $$

In [10]:
z_X = dev_X / np.std(X, ddof=1)
z_Y = dev_Y / np.std(Y, ddof=1)

While in covariance we directly calculate the mean of the deviations, here we calculate the mean of the Z-scores, which is actually the mean of the standardized deviations.

$$ \rho_{X,Y} = E[z_X z_Y] $$

In [11]:
corr_XY = np.mean(z_X * z_Y)
corr_XY
Out[11]:
np.float64(0.9585703669576935)

As we can see, NumPy’s built-in function also returns very similar values.

In [12]:
print(np.corrcoef(X, Y)[0][1], corr_XY)
0.9604913496570061 0.9585703669576935

Ordinary Least Squares

When creating an OLS model, we actually build it entirely using covariances and variances. Let's remember the model first:

$$ Y = \beta_0 + \beta_1 X $$

First, let’s define $\beta_1$:

$$ \beta_1 = \frac{Cov(X,Y)}{Var(X)} $$

The result of this calculation will tell us how much Y changes for every unit change in X. This makes perfect sense when we consider that covariance measures the “rate of joint change.”

In [13]:
slope = cov_XY / np.var(X)
slope
Out[13]:
np.float64(3.0416450981995125)

However, to determine the y-intercept, it is not enough to know the slope; we also need to know the value of Y when X is 0.

To do this, we first swap the variables in our model function:

$$ \beta_0 = Y - \beta_1 X $$

Then we substitute $\bar{X}$ and $\bar{Y}$ for X and Y.

In [14]:
intercept = mean_Y - (slope * mean_X)
intercept
Out[14]:
np.float64(8.873720093326272)
In [15]:
model = linregress(X, Y)
model.__getattribute__("slope")
Out[15]:
np.float64(3.041645098199512)

We can also see that the values we actually found are correct when we check them using the linregress function from scipy.stats.

In [16]:
print(
    "Is manuall found intercept and scipy's interecept close?",
    np.isclose(intercept, model.__getattribute__("intercept")),
)
print(
    "Is manuall found slope and scipy's slope close?",
    np.isclose(slope, model.__getattribute__("slope")),
)
Is manuall found intercept and scipy's interecept close? True
Is manuall found slope and scipy's slope close? True

Our goal in linear regression was to minimize the sum of squared errors; below, we can see the sum of squared errors we obtained.

In [17]:
Y_pred = intercept + (slope * X)
residuals = Y - Y_pred
print("Sum of squared errors:", np.sum(residuals**2))
Sum of squared errors: 1821.0530087903076

Here, too, we can see how our data points are distributed in the scatter plot. The line passing through them is the one that minimizes the sum of the squared errors, in other words, our linear regression line.

In [18]:
fig, ax = plt.subplots()
ax.scatter(X, Y, alpha=0.1, marker=".", color="red")
ax.plot(X, Y_pred, label="model")
ax.axvline(
    mean_X, ymax=mean_Y, linestyle="--", alpha=0.5, label=f"$\\bar{{X}}$: {mean_X:.2f}"
)
ax.axhline(
    mean_Y, xmax=mean_X, linestyle="--", alpha=0.5, label=f"$\\bar{{Y}}$: {mean_Y:.2f}"
)
ax.legend()
plt.show()
No description has been provided for this image

From our graph, we can also see that we have aligned the point where the line intersects the Y-axis ($\beta_0$) with the point where the line passes through the means of both X and Y.

In [19]:
np.isclose((intercept + (slope * mean_X)), mean_Y)
Out[19]:
np.True_
>