In the previous post, I wrote about expectation, a great tool that helps us capture notion of “center” for a random variable. In this post, I will write about variance, another great tool that represents the “spread” of a random variable.
Consider a random variable with two probability density functions (PDFs) and , shown in the figure below. under both PDFs has the same expected value (i.e., ). But, has a larger “spread” compared to . In other words, samples drawn from deviate more from their mean than samples drawn from . Variance helps us represent this notion of spread from mean.
The variance of a random variable is represented by and is defined by the expected value of the squared deviation from the mean :
is also often represented by as the variance is the square of the standard deviation denoted by .
The variance can also be expanded as:
Variance is closely related to covariance that represents the linear dependency between two variables. In fact, it is easy to see that:
Sign: Variance is always non-negative:
Shift invariance: Variance is invariant to shift:
where is a constant.
Sum: The variance of sum/minus is:
where denotes the covariance between two random variables. This can be easily generalized to the sum of many random variables:
Assume is a Normally-distributed random variable with mean and variance , i.e., . We use a Monte Carlo estimate of the mean by drawing samples from the distribution denoted by :
what are the mean and variance of ?
First, we should note that itself is a random variable. In other words, it has its own distribution with its own mean and variance. Second, is a stochastic estimation of . i.e., is fixed in our problem, but depends on the set of samples . If we calculate using a different set of samples, its value will be different.
Mean: Let’s use what we learned in the previous post to derive the mean of :
So, the expected value of is .
As you can see, the variance of depends on the sample size. As the sample size grows goes to zero. In fact the sum of Normally-distributed random variables has a Normal distribution itself. Thus, we have . As increases, the distribution of which is centered at , becomes narrower, and as a result, approaches .
Why variance is important in training:
Recall that in the previous post I mentioned that many training objective functions take the form of an expectation. The gradient of those objective functions is also in the form of an expectation. Since in general, we don’t have an analytic expression for the expected value of the gradient, we typically use a stochastic estimation of the gradient for optimizing the objective function. In this case, the variance of the gradient estimator plays a crucial role in training. High gradient variance can extremely slow down the training progress.
In practice, many factors impact the variance of a gradient estimator. Here, we are going to analyze the effect of the training batch size. Let’s consider a model with a single parameter. We represent this model by where is an input instance and is the parameter. For example, our model can be as simple as a linear function . Let’s denote the target variable by and the loss function measuring the mismatch between the model output and target by . The training objective is to find that has the lowest loss in average. This is formulated as minimizing the following expectation:
where is the joint distribution of input and target variables. We can use a gradient-based optimization method to minimize . For this, we require computing the gradient of the loss function:
As you can see both the objective function and the gradient are in the form of an expectation.
We don’t have the joint explicitly. Thus, we cannot compute the expectations analytically. However, the training dataset are samples drawn from the joint . We can define a Monte Carlo estimate of the gradient using a mini-batch of randomly-selected training datapoints:
where is the stochastic estimator of . As we saw in the example, where . We generally don’t have , so we cannot quantify . However, it is easy to see that reduces as the mini-batch size increases. In other words, increasing the training mini-batch size, in average, brings closer to the true gradient.
In this post, we learned about variance and its basic properties, and we saw how we can use them to derive the mean and variance of an estimator. In the next post, we will dig deeper into an ML problem to better understand how these fundamental concepts play together during training. If you like to stay updated with the future posts, you can use the form below to subscribe.