IT Repository

Vanilla SGD Momentum 개념 Momentum NAG Adaptive learning rate 개념 AdaGrad AdaDelta, RMSProp 위의 두 방법을 병합: ADAM (RMSProp + NAG) Adagrad (Adaptive Gradient)¶ Vanilla SGD : 일괄적인 Learning rate Adagrad : 각 파라미터마다 다른 Learning rate를 적용 (Adaptive Learning rate) $$\theta_{t+1} = \theta - \dfrac{\eta}{\sqrt{G_t + \epsilon}} \cdot \nabla_\theta J(\theta_t) \\ G_t = G_{t-1} + \left( \nabla_\theta J(\theta_t) \r..