How Can Gradient Clipping Help Avoiding the Exploding Gradient Problem?

Deep neural networks are weak to the drawback of vanishing and exploding gradients. This is particularly true for Recurrent Neural Networks, that are generally used (RNNs). Because RNNs are usually utilized in conditions requiring short-term reminiscence, the weights could be simply exploited throughout coaching, leading to surprising outcomes equivalent to Nan or the mannequin failing to protection at the desired level. So, as a way to cut back this impact, varied strategies, equivalent to regularizers, are used. From all of these strategies, we are going to concentrate on the Gradient Clipping technique on this article and try to grasp it each theoretically and virtually. Below are the main factors listed which can be to be mentioned on this article.

Table Of Contents

The Exploding Gradient ProblemWhat is Gradient Clipping?How to Use Gradient Clipping?Implementing Gradient Clipping

Let’s begin the dialogue by understanding the drawback and its causes.

The Exploding Gradient Problem

The exploding gradient drawback is an issue that arises when utilizing gradient-based studying strategies and backpropagation to coach synthetic neural networks. An synthetic neural community, often known as a neural community or a neural internet, is a studying algorithm that employs a community of features to understand and translate knowledge enter into a selected output.  This sort of studying algorithm goals to duplicate the method neurons in the human mind work. 

When giant error gradients accumulate, exploding gradients happen, leading to very giant updates to neural community mannequin weights throughout coaching. Gradients are used to replace the community weights throughout coaching, however this course of usually works finest when the updates are small and managed.  When the magnitudes of the gradients add up, an unstable community is more likely to kind, leading to poor prediction outcomes or perhaps a mannequin that experiences nothing helpful in any respect. 

In the coaching of synthetic neural networks, exploding gradients could cause points. When gradients explode, the community turns into unstable, and the studying can’t be accomplished. The weights’ values can even develop to the level the place they overflow, leading to NaN values. 

The time period “not a quantity” refers to values that signify undefined or unrepresentable values. In order to appropriate the coaching, it’s useful to know tips on how to spot exploding gradients. Because recurrent networks and gradient-based studying strategies take care of giant sequences, it is a widespread prevalence. There are strategies for repairing exploding gradients, equivalent to gradient clipping and weight regularization, amongst others. In this put up, we are going to check out the Gradient Clipping technique.

What is Gradient Clipping?

Gradient clipping is a way for stopping exploding gradients in recurrent neural networks. Gradient clipping could be calculated in quite a lot of methods, however one in every of the commonest is to rescale gradients in order that their norm is at most a sure worth. Gradient clipping entails introducing a pre-determined gradient threshold after which cutting down gradient norms that exceed it to match the norm. 

This ensures that no gradient has a norm larger than the threshold, leading to the gradients being clipped. Although the gradient introduces a bias in the ensuing values, gradient clipping can maintain issues steady.

It could be troublesome to coach recurrent neural networks. Vanishing gradients and exploding gradients are two widespread issues when coaching recurrent neural networks. When the gradient turns into too giant, error gradients accumulate, leading to an unstable community. 

Vanishing gradients can happen when optimization turns into caught at a sure level as a consequence of a gradient that’s too small to progress. Gradient clipping can forestall these gradient points from messing up the parameters throughout coaching.

In normal, exploding gradients could be prevented by fastidiously configuring the community mannequin, equivalent to utilizing a small studying charge, scaling the goal variables, and utilizing a normal loss operate. However, in recurrent networks with numerous enter time steps, exploding gradients should be a problem.

How to Use Gradient Clipping?

Changing the error spinoff earlier than propagating it again by the community and utilizing it to replace the weights is a standard resolution to exploding gradients. By rescaling the error spinoff, the updates to the weights are additionally rescaled, decreasing the chance of an overflow or underflow dramatically.

Gradient scaling is the strategy of normalizing the error gradient vector in order that the vector norm (magnitude) equals a predefined worth, equivalent to 1.0. Gradient clipping is the strategy of forcing gradient values (element-by-element) to a selected minimal or most worth in the event that they exceed an anticipated vary. These strategies are continuously referred to collectively as “gradient clipping.”

It is widespread follow to make use of the similar gradient clipping configuration for all community layers. Nonetheless, there are some circumstances the place a wider vary of error gradients is permitted in the output layer than in the hidden layer.

Implementing Gradient Clipping

We now perceive why Exploding Gradients happen and the way Gradient Clipping can assist to resolve them. We additionally noticed two totally different strategies for making use of Clipping to your deep neural community. Let’s have a look at how each Gradient Clipping algorithms are carried out in main Machine Learning frameworks like Tensorflow and Pytorch.

We will use the Fashion MNIST dataset, which is an open-source digit classification knowledge set designed for picture classification.

Gradient clipping is easy to implement in TensorFlow fashions. All it’s important to do is go the parameter to the optimizer operate. To clip the gradients, all optimizers have ‘clipnorm’ and ‘clipvalue’ parameters.

Before continuing additional we rapidly focus on how we will clipnorm and clipvalue parameters.


Gradient norm scaling entails modifying the derivatives of the loss operate to have a specified vector norm when the gradient vector’s L2 vector norm (sum of squared values) exceeds a threshold worth. For instance, we might present a norm of 1.0, which implies that if the vector norm for a gradient exceeds 1.0, the vector values can be rescaled in order that the vector norm equals 1.0.


Gradient worth clipping entails clipping the derivatives of the loss operate to a selected worth if a gradient worth is lower than or larger than a destructive or constructive threshold. For occasion, we might outline a norm of 0.5, which implies that if a gradient worth is lower than -0.5, it’s set to -0.5, and whether it is larger than 0.5, it’s set to 0.5.

Now that we’ve understood what’s the precise position of those parameters. Start the implementation by importing the needed bundle and submodule.

import tensorflow as tf
from tensorflow.keras.datasets import mnist

Next load the Fashion MNIST dataset and pre-process it in order that the TF mannequin can deal with it.

# load the knowledge
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()

# make suitable for tensorflow
x_train, x_test = x_train / 255., x_test / 255. # scalling
train_data = tf.knowledge.Dataset.from_tensor_slices((x_train, y_train))
train_data = train_data.repeat().shuffle(5000).batch(32).prefetch(1)

Now we are going to outline and compile the mannequin with out gradient clipping, right here I’m deliberately limiting the numbers of layers and neurons for every layer in order to duplicate the conduct. 

# construct a mannequin
mannequin = tf.keras.fashions.Sequential([
tf.keras.layers.LSTM(10,input_shape=(28, 28)),


#compile a mannequin
# inside the optimizer we’re doing clipping

Next, we’ll match the mannequin and observe the loss and accuracy motion. 


Here is the outcome,

As we will see we’ve skilled for a number of epochs and during which mannequin is struggling to cut back loss and accuracy too. Now let’s whether or not Grading clipping will make any distinction right here. 

As we focus on earlier to implement gradient clipping we have to provoke the desired technique inside the optimizer. Here I’m transferring with the clipvalue technique.  

# inside the optimizer we’re doing clipping

Next, we’ll prepare the mannequin with gradient clipping and may observe loss and accuracies as,

Now it’s clear that clipping gradients worth can enhance the coaching efficiency of the mannequin. 

Final phrases 

Clipping the gradients hurries up coaching by permitting the mannequin to converge extra rapidly. This implies that the coaching reaches a minimal error charge sooner. Because the error diverges as the gradients explode, no world or native minima could be discovered. When the exploding gradients are clipped, the errors start to converge to a minimal level.

This put up has mentioned what exploding gradients are and why they occur. In order to come across this impact, we mentioned a way often known as Gradient clipping and noticed how this method can clear up the drawback each theoretically and virtually.


Recommended For You