Where is the gradient vector of and determines how for to move along the gradient. The simplest numerical optimization algorithm to maximize a function is gradient ascent (descent if one is minimizing a function) which updates the current value of the parameters,, to a new value,, such that by following the gradient. In all but the simplest cases an analytic solution will not exist and we will need to resort to numerical methods. Along the way we will introduce some concepts from differential geometry and derive more efficient gradient directions to follow.įor the rest of this post we will assume the setting of variational inference so that the function to optimize is and the components of are the variational parameters.īasic calculus tells us that to optimize a continuous function (the objective function) we set the derivative of the function to zero and solve. This can cause slow convergence or convergence to inferior local modes in non-convex problems. In this post we will consider optimizing the parameters of a probability distribution and will see that using the gradient as in basic calculus can follow suboptimal directions. It can be shown that minimizing is equivalent to maximizing the evidence lower bound, denoted. The dominant assumption in machine learning for the form of is a product distribution, that is (where we assume there are variational parameters). Variational inference then proceeds to minimize the KL divergence from to. We will approximate with the variational distribution that is parameterized by the variational parameters. Suppose we are interested in a posterior distribution,, that we cannot compute analytically. Another example, and the motivating example for this post, is using variational inference to approximate a posterior distribution. For instance, finding maximum likelihood and maximum a posteriori estimates require maximizing the likilihood function and posterior distribution respectively. A common activity in statistics and machine learning is optimization.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |