Syllabus Map

• Study map: Syllabus Study Map

Core Idea

• Iterative method that updates parameters in the negative gradient direction.

Update Rule

Practical Notes

Learning rate controls stability-speed tradeoff.

• Too large diverges; too small is slow.

Mini-batch noise can aid exploration.

• Moderate stochasticity can escape poor regions.

Why This Matters for ML

• Gradient descent and its variants are the core training engines for most ML models.
• Update-rule behavior determines speed, stability, and final model performance.
• Batch vs mini-batch tradeoffs affect noise, generalization, and compute efficiency.
• Practical tuning of optimizers is impossible without gradient-descent fundamentals.