Gradient Clipping
Created: 27 Apr 2023, 10:55 AM | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a")
Tags: knowledge, GeneralDL
How to fix exploding gradients: gradient clipping
There are a couple of techniques that focus on Exploding Gradient problems. One common approach isL2 Regularizationwhich applies “weight decay” in the cost function of the network. The regularization parameter gets bigger, the weights get smaller, effectively making them less useful, as a result making the model more linear.However, we’ll focus on a technique that’s far superior in terms of gaining results and easiness in implementation —Gradient Clipping.
Gradient Clippingis a method where theerror derivative is changed or clipped to a thresholdduring backward propagation through the network, and using the clipped gradients to update the weights.
By rescaling the error derivative, the updates to the weights will also be rescaled, dramatically decreasing the likelihood of an overflow or underflow.
Gradient clipping is a technique to prevent exploding gradients in very deep networks, usually inrecurrent neural networks.
There are many ways to compute gradient clipping, but a common one is to rescale gradients so that their norm is at most a particular value. With gradient clipping, pre-determined gradient threshold be introduced, and then gradients norms that exceed this threshold are scaled down to match the norm. This prevents any gradient to have norm greater than the threshold and thus the gradients are clipped. There is an introduced bias in the resulting values from the gradient, but gradient clipping can keep things stable.
From <https://deepai.org/machine-learning-glossary-and-terms/gradient-clipping>