NN - 20 - Learning Rate Decay (with PyTorch code)

In this video we are going to look into Learning Rate decay, what is it, and what is the intuition behind it.
NN Playlist: bit.ly/3PvvYSF
Become a member and get full access to this online course:
meerkatstatist...
** 🎉 Special RUclips 60% Discount on Yearly Plan - valid for the 1st 100 subscribers; Voucher code: First100 🎉 **
"NN with Python" Course Outline:
Intro
Administration
Intro - Long
Notebook - Intro to Python
Notebook - Intro to PyTorch
Comparison to other methods
Linear Regression vs. Neural Network
Logistic Regression vs. Neural Network
GLM vs. Neural Network
Expressivity / Capacity
Hidden Layers: 0 vs. 1 vs. 2+
Training
Backpropagation - Part 1
Backpropagation - Part 2
Implement a NN in NumPy
Notebook - Implementation redo: Classes instead of Functions (NumPy)
Classification - Softmax and Cross Entropy - Theory
Classification - Softmax and Cross Entropy - Derivatives
Notebook - Implementing Classification (NumPy)
Autodiff
Automatic Differentiation
Forward vs. Reverse mode
Symmetries in Weight Space
Tanh & Permutation Symmetries
Notebook - Tanh, Permutation, ReLU symmetries
Generalization
Generalization and the Bias-Variance Trade-Off
Generalization Code
L2 Regularization / Weight Decay
DropOut regularization
Notebook - DropOut (PyTorch)
Notebook - DropOut (NumPy)
Notebook - Early Stopping
Improved Training
Weight Initialization - Part 1: What NOT to do
Notebook - Weight Initialization 1
Weight Initialization - Part 2: What to do
Notebook - Weight Initialization 2
Notebook - TensorBoard
Learning Rate Decay
Notebook - Input Normalization
Batch Normalization - Part 1: Theory
Batch Normalization - Part 2: Derivatives
Notebook - BatchNorm (PyTorch)
Notebook - BatchNorm (NumPy)
Activation Functions
Classical Activations
ReLU Variants
Optimizers
SGD Variants: Momentum, NAG, AdaGrad, RMSprop, AdaDelta, Adam, AdaMax, Nadam - Part 1: Theory
SGD Variants: Momentum, NAG, AdaGrad, RMSprop, AdaDelta, Adam, AdaMax, Nadam - Part 2: Code
Auto Encoders
Variational Auto Encoders
If you’re looking for statistical consultation, work on interesting projects, or training workshop, visit my website meerkatstatist... or contact me directly at [e-mail]
~~~~~ SUPPORT ~~~~~
Paypal me: paypal.me/Meer...
~~~~~~~~~~~~~~~~~
Intro/Outro Music: Dreamer - by Johny Grimes
• Johny Grimes - Dreamer