The purpose of this page is to provide a very brief intorduction to variation inference. This is very closely modeled after a series of 5 YouTube videos on this subject.


What we will need from information theory

This section provides some commonly used equations that will be encontered when studying variational inference.

First, the information associated with an event $x$ is

Now, if we have several events, we can calculate the weighted average of those events as,

which is commonly refered to as the entropy of the system.

Finally, when we move from the discrete domain to a continous one, we talk about differenital entropy, which is simpy the modification of the above equation to the continous domain,

An introduction to KL-Divergence

KL-Divergence is simply a measure of the distance between two probability distributions. If we have two distributions, then the KL-Diverence between them is written as,

where it should be noted that the KL-Divergence is not symmetric

One useful way to think about KL-Divergence is as a measure of relative entropy between two distributions,

which can be manipulated into the form generally presented as,

Why use KL-Divergence

Let’s say that we have an unkown distribution, $p(z|x)$, that we would like to estimate. We can use a new distribution, q(z), to estimate the desired distribution with KL-Divergence being the measure of closness.

Now, we can note that

so

Now, manipulating the equation, we get

Now, let’s call the second term on the left hand side of the equation above $\mathcal{L}$ for lower bound. Then, we can write the equation in it’s final form as,

The equation above gives us the key insight provided by variational inference, (i.e., we can maximize this lower bound and gain the same results as trying to minimize the KL-Divergence when trying to approximate a conditional probability). And this is beneficial because the KL-divergnce contains a conditional probability which can be intractable while the lower bound contains the joint distribution which we can easily calculate.

Goal ::

We want to find $q(z)$ such that $$ \mathcal{L} = \sum_N q(z) \frac{p(x,z)}{q(z)} $$ is maximized.