Stochastic Optimization

Stochastic optimization is a strong approach for determining the best parameters of a model by iteratively updating them using randomly selected subsets of training data. In contrast to standard optimization methods that consider the complete dataset in each iteration, stochastic optimization algorithms only employ a tiny fraction of the data, making them more appropriate for huge datasets and non-convex optimization problems.

Stochastic optimization techniques have a wide range of applications, including machine learning, deep learning, computer vision, natural language processing, and huge dataset optimization challenges. They are often used to train neural networks, optimize hyperparameters, choose features, and effectively handle complicated optimization problems.

Code:

Now we will explore some of the common stochastic optimization algorithms.

Importing Libraries

Reading the Dataset

Plotting the Dataset

Now, we will be plotting the dataset.

Output:

Model Implementation

We will build the model and then implement it.

Optimizers

Here is the part where we will look at the stochastic optimization algorithms.

1. Stochastic gradient

Stochastic gradient descent (SGD) is an iterative method for optimizing an objective function with appropriate smoothness qualities (for example, differentiable or subdifferentiable). It is a stochastic approximation of gradient descent optimization because it substitutes the real gradient (derived from the whole data set) with an estimate (calculated from a randomly chosen portion of the data). This minimizes the computing cost in high-dimensional optimization problems, resulting in quicker iterations but a lower convergence rate. descent (SGD).

2. Momentum

Stochastic gradient descent with momentum retains each iteration's update Δw and derives the next update as a linear combination of the gradient and the previous update.

3. Nesterov Momentum

Nesterov momentum, or Nesterov Accelerated Gradient (NAG), is a slightly modified form of Stochastic Gradient Descent Momentum that provides better theoretical convergence guarantees for convex functions. In practice, it has yielded somewhat better outcomes than traditional momentum.

4. AdaGrad

AdaGrad (adaptive gradient algorithm) is a modified stochastic gradient descent technique that uses a per-parameter learning rate. It was initially described in 2011. Informally, this boosts the learning rate for sparser parameters while decreasing it for those that are less sparse. This technique frequently outperforms traditional stochastic gradient descent in situations when data is sparse and sparse parameters are more informative.

5. RMSProp

RMSProp (for Root Mean Square Propagation) is another approach in which the learning rate is adjusted for each parameter. The concept is to divide a weight's learning rate by a running average of the magnitudes of its most recent gradients. First, the running average is determined in terms of means squared.

6. Adam

Adam (Adaptive Moment Estimation) is an upgrade to the RMSProp optimizer. This optimization approach employs running averages of both gradients and their second moments.

Training

Output:

From the above result we can say that:

• For the blobs dataset, all optimization techniques perform exceedingly well, obtaining complete accuracy.
• SGD, AdaGrad, RMSprop, and Adam perform well on the moons dataset, however, Momentum and Nesterov have poorer accuracy and larger variability.
• SGD performs best on the circles dataset, followed by AdaGrad, RMSprop, and Adam. Momentum and Nesterov score badly, suggesting that they may struggle with the dataset's nonlinearity.
• Overall, SGD, AdaGrad, RMSprop, and Adam demonstrate promising performance across datasets, with SGD being the most consistent. However, the algorithm used may be determined by the dataset's particular properties and optimization aims.