Optimization Basics (Gradient Descent)

Summary

The iterative algorithm that adjusts model parameters in the direction that reduces loss the fastest — the core training loop of virtually every ML and DL model.

• 1Implement gradient descent from scratch on a simple 1D or 2D loss function.
• 2Visualize how learning rate affects convergence (too small, too large, just right).
• 3Learn the difference between batch, stochastic and mini-batch gradient descent.