Gradient-Based Algorithms

First-order and second-order optimization algorithms that use gradient information.

Overview

Gradient-based optimizers use derivative information to guide the search toward optima. These methods are particularly effective for smooth, differentiable functions and are the foundation of modern deep learning.

Available Algorithms

First-Order Methods

• SGD with Momentum - Stochastic Gradient Descent with momentum
• Adam - Adaptive Moment Estimation
• AdamW - Adam with weight decay
• RMSprop - Root Mean Square Propagation
• Adagrad - Adaptive Gradient Algorithm
• Adadelta - Extension of Adagrad
• Nadam - Nesterov-accelerated Adam
• AMSGrad - Adam variant with long-term memory

Usage Example

from opt.gradient_based import Adam, SGDMomentum
from opt.benchmark.functions import rosenbrock

# Adam optimizer
adam = Adam(
    func=rosenbrock,
    lower_bound=-5,
    upper_bound=10,
    dim=10,
    max_iter=1000,
    learning_rate=0.01
)
best_solution, best_fitness = adam.search()

See Also

• API Reference - Complete API documentation
• Classical Methods - For quasi-Newton methods like BFGS