Deep neural networks are vulnerable to the problem of vanishing and exploding gradients. This is especially true for Recurrent Neural Networks, which are commonly used (RNNs).