Avatto>>DATA SCIENTIST>>SHORT QUESTIONS>>Deep Learning>>Gradient Descent Method
Gradient descent is the most popular and widely used optimization algorithms used for training neural networks. Yes, Gradient descent is the first-order optimization method because with gradient descent we calculate only the first-order derivative.
Gradient descent is an optimization method used for training the network. First, we compute the derivatives of the loss function with respect to the weights of the network and then update the weights of the network using the below update rule:
  • Weight = weight - learning rate x derivatives
The matrix is often called the Jacobian matrix if it contains the first-order partial derivatives.
When the learning rate is small then we take a very small step and it slows down attaining the convergence and when the learning rate is large then we take a very large step and it may cause us to miss out on the global minimum.
Gradient checking is used for debugging the gradient descent algorithm and to make sure that we have a correct implementation.

  That is, when we implement the gradient descent method for the complex neural network, even with buggy implementations, the network will learn something.

  But the buggy implementation will not be as optimal as a bug-free implementation. So to ensure that we have the bug free implementation of gradient descent we perform gradient checking.