Unit 1 exercise 3

[avipul751]

Hi, I was working on the 3rd exercise in unit 1, and I observed that the learning rate affects training only when the weights and biases are initialised randomly and not when they are initialised to 0. Is this correct, or am I missing something?

[rasbt]

Nice! That's actually the expected result.

The rationale behind it is a bit complicated, but in a nutshell, that's because if all the weights are initialized to zero, the learning rate parameter, affects only the scale of the weight vector, not the direction. So this means that it does not change the decision boundary.

[harishb00]

Hi @rasbt . Could you please shed some light on what does the direction of a weight vector mean?

[rasbt]

Good but tricky question! In a nutshell, instead of looking at vectors as a list of numbers, we can also consider vectors as geometric objects. E.g., a 2D vector with the values [2, 1] is an arrow pointing from the origin to the coordinates x=2 and y=1 in a 2D cartesian coordinate system. Sounds complicated, but just think of drawing an arrow on a map or so.

Regarding the direction of vectors and the perceptron, I have a lecture on that from a few years back:
https://sebastianraschka.com/blog/2021/dl-course.html#l03-single-layer-neural-networks-the-perceptron-algorithm

• If you look at the PDF, it starts at slide 55
• "L3.5 The Geometric Intuition Behind the Perceptron" (18:43) has a link to a video recording of this

[harishb00]

Sure. I will go through the materials shared. Thanks.