5.6. clumsygrad.optimizer

This module contains common optimizers for optimizing parameters in a computational graph.

class clumsygrad.optimizer.Optimizer(parameters: list[Tensor])

Bases: ABC

Abstract base class for all optimizers.

__init__(parameters: list[Tensor])

step() → None

Update parameters. Must be implemented by subclasses.

zero_grad() → None

Zero gradients for all parameters.

class clumsygrad.optimizer.SGD(parameters: list[Tensor], lr: float = 0.01)

Bases: Optimizer

Stochastic Gradient Descent (SGD) optimizer.

This optimizer updates parameters using the formula: param -= lr * grad, where param is a parameter tensor, lr is the learning rate, and grad is the gradient of the parameter.

Reference: Robbins, H., & Monro, S. (1951). A stochastic approximation method.

__init__(parameters: list[Tensor], lr: float = 0.01)

Initialize the SGD optimizer.

Parameters:

• parameters (list[Tensor]) – List of parameter tensors to optimize.

• lr (float) – Learning rate for the optimizer.

step() → None

Update parameters.

class clumsygrad.optimizer.Adam(parameters: list[Tensor], lr: float = 0.001, beta1: float = 0.9, beta2: float = 0.999, eps: float = 1e-08)

Bases: Optimizer

Adam (Adaptive Moment Estimation) optimizer.

This optimizer combines the advantages of AdaGrad and RMSprop by computing adaptive learning rates for each parameter using estimates of first and second moments of the gradients.

Reference: Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization.

__init__(parameters: list[Tensor], lr: float = 0.001, beta1: float = 0.9, beta2: float = 0.999, eps: float = 1e-08)

Initialize the Adam optimizer.

Parameters:

• parameters (list[Tensor]) – List of parameter tensors to optimize.

• lr (float) – Learning rate.

• beta1 (float) – Exponential decay rate for first moment estimates.

• beta2 (float) – Exponential decay rate for second moment estimates.

• eps (float) – Small constant for numerical stability.

step() → None

Update parameters.