Backpropagation from Scratch

This is a toy MLP with one hidden layer for backprop.

The target vector y is fixed to an arbitrary value of [10,-2] for simplicity. It can be modified to depend on the input vector x for useful predictions.

Files

File	Desc
	Notebook → Download and modify the code! :)

Computation-Graph

with the Sigmoid activation 𝞹(z) = 1/(1+exp(-x)) and Loss = (1/2) * (y - o)²

Weight derivation

W1-Matrix gradient:

W2-Matrix gradient:

Gradient Calculations for Weights in Code

Gradient of Weight Matrix W⁽¹⁾

z = np.dot(W1, x)
a = (-y + out).T
b = np.dot(a, W2)
c = sigmoid(z) * (1 - sigmoid(z))
d = b.T * c
dL_dW1 = d * x.T

Gradient of Weight Matrix W⁽²⁾

h = sigmoid(np.dot(W1, x))
dL_dW2 = np.dot((-y + out), h.T)

Result:

We can see that the backpropagation is working and the correct gradients are calculated. The network is learning and decreasing its loss: