june 24, 2024
sick and tired of gentle introductions to *insert an ml concept*?
worry no more, for today, we will catapult you to the wolves of linear regression.
the derivation of the derivatives is relatively simple calculus, which actually took me way too long to do:
\[ MSE = \frac{1}{n} \sum^n_{i = 1} (y_i - \hat{y}_i)^2, \space e = (y_i - (mx_i + b))\]
-\[ \frac{\partial{e^2}}{\partial{b}} = \frac{\partial{(y_i - wx_i - b)}^2}{\partial{b}} = -1 \times 2(y_i - wx_i - b) = -2(y_i - wx_i - b) \]
+\[ \frac{\partial{e^2}}{\partial{b}} = \frac{\partial{(y_i - wx_i - b)}^2}{\partial{b}} \]
+\[ = -1 \times 2(y_i - wx_i - b) = -2(y_i - wx_i - b) \]
\[ \frac{\partial{MSE}}{\partial{b}} = \frac{1}{N} \sum^n_{i = 1} -2(y_i - wx_i - b)\]
-\[ \frac{\partial{e^2}}{\partial{w}} = \frac{\partial{(y_i - wx_i - b)}^2}{\partial{w}} = -x_i \times 2(y_i - wx_i - b) = -2x_i(y_i - wx_i - b) \]
+\[ \frac{\partial{e^2}}{\partial{w}} = \frac{\partial{(y_i - wx_i - b)}^2}{\partial{w}} \]
+\[ = -x_i \times 2(y_i - wx_i - b) = -2x_i(y_i - wx_i - b)\]
\[ \frac{\partial{MSE}}{\partial{w}} = \frac{1}{N} \sum^n_{i = 1} -2x_i(y_i - wx_i - b)\]
wielding one parameter in each hand, we can now improve our parameters by having them go a little bit towards the direction negative of their gradient:
@@ -107,23 +110,21 @@and... this is the actual algorithm for linear regression with stochastic gradient descent, this should be straightforward after all that buildup:
-- randomly initialize weights W and b in the correct shape - split the data into the train and test splits - for every epoch: - for every data point in the training set: - update the weights W - update the bias b - evaluate the model's predictions on the test set -+
the actual implementation of the above is left as an exercise for the reader...
-just kidding, you can find it here , written with just python and numpy.
+just kidding, you can find it here, written with just python and numpy.
-souces: d2l book, chapter 3 -