Analysis of Dropout

Overfitting is a problem in Deep Neural Networks (DNN): the model learns to classify only the training set, adapting itself to the training examples instead of learning decision boundaries capable of classifying generic instances. Many solutions to the overfitting problem have been presented during these years; one of them have overwhelmed the others due to its simplicity and its empirical good results: Dropout.


Dropout image

Visual representation of Dropout, right from the paper. On the left there's the network before applying Dropout, on the right the same network with Dropout applied.
The network on the left it's the same network used at test time, once the parameters have been learned.

The idea behind Dropout is to train an ensemble of DNNs and average the results of the whole ensemble instead of train a single DNN.

The DNNs are built dropping out neurons with probability, therefore keeping the others on with probability . When a neuron is dropped out, its output is set to zero, no matter what the input or the associated learned parameter is.

The dropped neurons do not contribute to the training phase in both the forward and backward phases of back-propagation: for this reason every time a single neuron is dropped out it’s like the training phase is done on a new network.

Quoting the authors:

In a standard neural network, the derivative received by each parameter tells it how it should change so the final loss function is reduced, given what all other units are doing. Therefore, units may change in a way that they fix up the mistakes of the other units. This may lead to complex co-adaptations. This in turn leads to overfitting because these co-adaptations do not generalize to unseen data. We hypothesize that for each hidden unit, Dropout prevents co-adaptation by making the presence of other hidden units unreliable. Therefore, a hidden unit cannot rely on other specific units to correct its mistakes.

In short: Dropout works well in practice because it prevents the co-adaption of neurons during the training phase.

Now that we got an intuitive idea behind Dropout, let’s analyze it in depth.

How Dropout works

As said before, Dropout turns off neurons with probability and therefore let the others turned on with probability .

Every single neuron has the same probability of being turned off. This means that:


  • a linear projection of a -dimensional input in a -dimensional output space.
  • an activation function

it’s possible to model the application of Dropout, in the training phase only, to the given projection as a modified activation function:

Where is a -dimensional vector of Bernoulli variables .

A Bernoulli random variable has the following probability mass distribution:

Where are the possible outcomes.

It’s evident that this random variable perfectly models the Dropout application on a single neuron. In fact, the neuron is turned off with probability and kept on otherwise.

It can be useful to see the application of Dropout on a generic i-th neuron:

where .

Since during train phase a neuron is kept on with probability , during the testing phase we have to emulate the behavior of the ensemble of networks used in the training phase.

To do this, the authors suggest scaling the activation function by a factor of during the test phase in order to use the expected output produced in the training phase as the single output required in the test phase. Thus:

Train phase:

Test phase:

Inverted Dropout

A slightly different approach is to use Inverted Dropout. This approach consists in the scaling of the activations during the training phase, leaving the test phase untouched.

The scale factor is the inverse of the keep probability: , thus:

Train phase:

Test phase:

Inverted Dropout is how Dropout is implemented in practice in the various deep learning frameworks because it helps to define the model once and just change a parameter (the keep/drop probability) to run train and test on the same model.

Direct Dropout, instead, force you to modify the network during the test phase because if you don’t multiply by the output the neuron will produce values that are higher respect to the one expected by the successive neurons (thus the following neurons can saturate or explode): that’s why Inverted Dropout is the more common implementation.

Dropout of a set of neurons

It can be easily noticed that a layer with neurons, in a single train step, can be seen as an ensemble of Bernoulli experiments, each one with a probability of success equals to .

Thus, the output of the layer have a number of dropped neurons equals to:

Since every neuron is now modeled as a Bernoulli random variable and all these random variables are independent and identically distributed, the total number of dropped neuron is a random variable too, called Binomial:

Where the probability of getting exactly success in trials is given by the probability mass distribution:

This formula can be easily explained:

  • is the probability of getting a single sequence of successes on trials and therefore failures.
  • is the binomial coefficient used to calculate the number of possible sequence of success.

We can now use this distribution to analyze the probability of dropping a specified number of neurons.

When using Dropout, we define a fixed Dropout probability for a chosen layer and we expect that a proportional number of neurons are dropped from it.

For example, if the layer we apply Dropout to has neurons and , we expect that 512 get dropped. Let’s verify this statement:

Thus, the probability of dropping out exactly neurons is of only !

A python 3 script can help us to visualize how neurons are dropped for different values of and a fixed value of . The code is commented.

Binomial distribution

The binomial distribution is very peaked around $$np$$

As we can see from the image above no matter what the value is, the number of neurons dropped on average is proportional to , in fact:

Moreover, we can notice that the distribution of values is almost symmetric around and the probability of dropping neurons increase as the distance from increase.

The scaling factor has been added by the authors to compensate the activation values, because they expect that during the training phase only a percentage of neurons have been kept. During the testing phase, instead, the of neurons are kept on, thus the value should be scaled down accordingly.

Dropout & other regularizers

Dropout is often used with L2 normalization and other parameter constraint techniques (such as Max Norm 1), this is not a case. Normalizations help to keep model parameters value low, in this way a parameter can’t grow too much.

In brief, the L2 normalization (for example) is an additional term to the loss, where is an hyper-parameter called regularization strength, is the model and is the error function between the real and the predicted value.

It’s easy to understand that this additional term, when doing back-propagation via gradient descent, reduces the update amount. If is the learning rate, the update amount of the parameter is

Dropout alone, instead, does not have any way to prevent parameter values from becoming too large during this update phase. Moreover, the inverted implementation leads the update steps to become bigger, as showed below.

Inverted Dropout and other regularizers

Since Dropout does not prevent parameters from growing and overwhelming each other, applying L2 regularization (or any other regularization technique that constraints the parameter values) can help.

Making explicit the scaling factor, the previous equation becomes:

It can be easily seen that when using Inverted Dropout, the learning rate is scaled by a factor of . Since has values in the ratio between and can vary between:

For this reason, from now on we’ll call boosting factor because it boosts the learning rate. Moreover, we’ll call the effective learning rate.

The effective learning rate, thus, is higher respect to the learning rate chosen: for this reason normalizations that constrain the parameter values can help to simplify the learning rate selection process.


  1. Dropout exists in two versions: direct (not commonly implemented) and inverted
  2. Dropout on a single neuron can be modeled using a Bernoulli random variable
  3. Dropout on a set of neurons can be modeled using a Binomial random variable
  4. Even if the probability of dropping exactly neurons is low, neurons are dropped on average on a layer of neurons.
  5. Inverted Dropout boost the learning rate
  6. Inverted Dropout should be using together with other normalization techniques that constrain the parameter values in order to simplify the learning rate selection procedure
  7. Dropout helps to prevent overfitting in deep neural networks.
  1. Max Norm impose a constraint to the parameters size. Chosen a value for the hyper-parameter it impose the constraint

Don't you want to miss the next article? Do you want to be kept updated?
Subscribe to the newsletter!

Related Posts

FaceCTRL: control your media player with your face

After being interrupted dozens of times a day while coding with my headphones on, I decided to find a solution that eliminates the stress of pausing and re-playing the song I was listening to. The solution is machine learning / computer vision application developed with TensorFlow 2, OpenCV, and Playerctl. This article will guide you trough the step required to develop such an application.

Hands-On Neural Networks with TensorFlow 2.0

The first book on TensorFlow 2.0 and neural networks is out now!

Analyzing tf.function to discover AutoGraph strengths and subtleties - part 3

In this third and last part, we analyze what happens when tf.function is used to convert a function that contains complex Python constructs in its body. Should we design functions thinking about how they are going to be converted?

Analyzing tf.function to discover AutoGraph strengths and subtleties - part 2

In part 1 we learned how to convert a 1.x code to its eager version, the eager version to its graph representation and faced the problems that arise when working with functions that create a state. In this second part, we’ll analyze what happens when instead of a tf.Variable we pass a tf.Tensor or a Python native type as input to a tf.function decorated function. Are we sure everything is going to be converted to the Graph representation we expect?

Analyzing tf.function to discover AutoGraph strengths and subtleties - part 1

AutoGraph is one of the most exciting new features of Tensorflow 2.0: it allows transforming a subset of Python syntax into its portable, high-performance and language agnostic graph representation bridging the gap between Tensorflow 1.x and the 2.0 release based on eager execution. As often happens all that glitters is not gold: although powerful, AutoGraph hides some subtlety that is worth knowing; this article will guide you through them using an error-driven approach.

Tensorflow 2.0: Keras is not (yet) a simplified interface to Tensorflow

In Tensorflow 2.0 Keras will be the default high-level API for building and training machine learning models, hence complete compatibility between a model defined using the old tf.layers and the new tf.keras.layers is expected. In version 2 of the popular machine learning framework the eager execution will be enabled by default although the static graph definition + session execution will be still supported. In this post, you'll see that the compatibility between a model defined using tf.layers and tf.keras.layers is not always guaranteed.

Fixed camera setup for object localization and measurement

A common task in Computer Vision is to use a camera for localize and measure certain objects in the scene. In the industry is common to use images of objects on a high contrast background and use Computer Vision algorithms to extract useful information. There's a lot of literature about the computer vision algorithm that we can use to extract the information, but something that's usually neglected is how to correctly setup the camera in order to correctly address the problem. This post aim is to shed light on this subject.

Tensorflow 2.0: models migration and new design

Tensorflow 2.0 will be a major milestone for the most popular machine learning framework: lots of changes are coming, and all with the aim of making ML accessible to everyone. These changes, however, require for the old users to completely re-learn how to use the framework: this article describes all the (known) differences between the 1.x and 2.x version, focusing on the change of mindset required and highlighting the pros and cons of the new implementation.

Understanding Tensorflow's tensors shape: static and dynamic

Describing computational graphs is just a matter connecting nodes correctly. Connecting nodes seems a trivial operation, but it hides some difficulties related to the shape of tensors. This article will guide you through the concept of tensor's shape in both its variants: static and dynamic.

Camera calibration guidelines

The process of geometric camera calibration (camera resectioning) is a fundamental step for machine vision and robotics applications. Unfortunately, the result of the calibration process can vary a lot depending on various factors. There are a lot of empirical guidelines that have to be followed in order to achieve good results: this post will drive you through them.