@@ -147,6 +157,7 @@ $$
## Generative Modelling
+{:.no_toc}
Let's assume we have data samples $x_1, x_2, \ldots, x_n$ from a distribution of interest $q_1(x)$, which density is unknown. We're interested in using these samples to learn a probabilistic model approximating $q_1$. In particular, we want efficient generation of new samples (approximately ) distributed from $q_1$. This task is referred to as **generative modelling**.
@@ -186,10 +197,11 @@ The advancement in generative modelling methods over the past decade has been no
## Outline
+{:.no_toc}
Flow Matching (FM) models are in nature most closely related to (Continuous) Normalising Flows (CNFs). Therefore, we start this blogpost by briefly recapping the core concepts behind CNFs. We then continue by discussing the difficulties of CNFs and how FM models address them.
-## Basics: Normalising Flows
+# Normalising Flows
Let $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^d$ be a continuously differentiable function which transforms elements of $\mathbb{R}^d$, with a continously differentiable inverse $\phi^{-1}: \mathbb{R}^d \to \mathbb{R}^d$.
Let $q_0(x)$ be a density on $\mathbb{R}^d$ and let $p_1(\cdot)$ be the density induced by the following sampling procedure
@@ -208,12 +220,13 @@ $$
\begin{align}
\label{eq:changevar}
p_1(y) &= q_0(\phi^{-1}(y)) \abs{\det\left[\frac{\partial \phi^{-1}}{\partial y}(y)\right]} \\
+\label{eq:changevar-alt}
&= \frac{q_0(x)}{\abs{\det\left[\frac{\partial \phi}{\partial x}(x)\right]}} \quad \text{with } x = \phi^{-1}(y)
\end{align}
$$
where the last equality can be seen from the fact that $\phi \circ \phi^{-1} = \Id$ and a simple application of the chain rule[^chainrule].
The quantity $\frac{\partial \phi^{-1}}{\partial y}$ is the Jacobian of the inverse map. It is a matrix of size $d\times d$ containing $J_{ij} = \frac{d\phi^{-1}_i}{dx_j}$.
-Depending on the task at hand, evaluation of likelihood or sampling, one of the two formulation of $\eqref{eq:changevar}$ is preferred.
+Depending on the task at hand, evaluation of likelihood or sampling, the formulation in $\eqref{eq:changevar}$ or $\eqref{eq:changevar-alt}$ is preferred (Friedman, 1987; Chen & Gopinath, 2000).
-Figure : *Different paths with the same endpoints marginals[^interpolation].* +Figure 7: *Different paths with the same endpoints marginals[^interpolation].*
@@ -708,7 +727,7 @@ Figure : *Different paths with the same endpoints marginals[^interpolation].* ### Conditional Flows -First, let's remind that the *continuity equation* relates a vector field $u_t$ to (the time evolution of) a probability path $p_t$ +First, let's remind ourselves that the transport equation relates a vector field $u_t$ to (the time evolution of) a probability path $p_t$ $$ \begin{equation} \pdv{p_t(x)}{t} = - \nabla \cdot \big( u_t(x) p_t(x) \big), @@ -742,16 +761,16 @@ with $\sigmamin > 0$ small, and for whatever reference $p_0$ we choose, typicall-Figure 7: *Marginal vector field $u_t(x)$ vs. conditional vector field $u_t(x \mid x_1)$ for samples $x_1 \sim p_1$. Here $p_0 = p_1 = \mathcal{N}(0, 1)$ and the two trajectories are according to the marginal vector field $u_t(x)$. Samples $x_1$ transparency is given by the IS weight $p_t(x \mid x_1) / p_t(x)$.* +Figure 13: Marginal vector field $u_t(x)$ vs. conditional vector field $u_t(x \mid x_1)$ for samples $x_1 \sim p_1$. Here $p_0 = p_1 = \mathcal{N}(0, 1)$ and the two trajectories are according to the marginal vector field $u_t(x)$. Samples $x_1$ transparency is given by the IS weight $p_t(x \mid x_1) / p_t(x)$.
@@ -935,11 +1007,9 @@ $$ \nabla_\theta \mathcal{L}_{\mathrm{FM}}(\theta) = \nabla_\theta \mathcal{L}_{\mathrm{CFM}}(\theta), \end{equation} $$ -which implies that we can use ${\mathcal{L}}_{\text{CFM}}$ - -instead to train the parametric vector field $u_{\theta}$. +which implies that we can use $${\mathcal{L}}_{\text{CFM}}$$ instead to train the parametric vector field $u_{\theta}$. The defer the full proof to the footnote[^CFM], but show the key idea below. -By developing the squared norm in both losses, we can easily show that the squared terms are equal or independant of $\theta$. +By developing the squared norm in both losses, we can easily show that the squared terms are equal or independent of $\theta$. Let's develop inner product term for ${\mathcal{L}}_{\text{FM}}$ and show that it is equal to the inner product of ${\mathcal{L}}_{\text{CFM}}$: @@ -985,7 +1055,7 @@ As a result both endpoints constraint are satisfied since ones recovers+Figure 27: \eqref{eq:g2g} with uniformly sampled pairings (left) and with OT pairings (right). +
+ ++Figure 28: \eqref{eq:mog2mog} with uniformly sampled pairings (left) and with OT pairings (right). +
+ +