Gradient Descent Variants (SGD, Adam)

Summary

Optimization algorithms that update model weights using gradients — SGD is the simplest, Adam adds momentum and adaptive learning rates and is the default for most deep learning.

• 1Train the same model with SGD and Adam and compare convergence speed.
• 2Learn what momentum and adaptive learning rates conceptually solve.
• 3Understand learning rate scheduling and why a fixed rate is often suboptimal.