How to determine cond_fn in condition_mean

[cyuting940612]

Hello everyone,

I'm trying to implement the manipulate method. And I find the key to manipulate is to modify the cond in render_condition() function. And the condition_mean() function will calculate a new mean for diffusion for sampling. However, I can not see where the cond_fn comes from and how to determine it. Much appreciate if you have any clue!

def condition_mean(self, cond_fn, p_mean_var, x, t, model_kwargs=None):
    """
    Compute the mean for the previous step, given a function cond_fn that
    computes the gradient of a conditional log probability with respect to
    x. In particular, cond_fn computes grad(log(p(y|x))), and we want to
    condition on y.

    This uses the conditioning strategy from Sohl-Dickstein et al. (2015).
    """
    gradient = cond_fn(x, self._scale_timesteps(t), **model_kwargs)
    new_mean = (p_mean_var["mean"].float() +
                p_mean_var["variance"] * gradient.float())
    return new_mean

        pred_img = render_condition(self.conf,
                                    self.ema_model,
                                    noise,
                                    sampler=sampler,
                                    cond=cond)

[phizaz]

Can you please point to the line of the code in question?

[cyuting940612]

Thanks for your reply.

Yes of course. The condition_mean method is Line 403-416 in base.py

Also in the manipulate case:

cond2 = cond2 + 0.3 * math.sqrt(512) * F.normalize(cls_model.classifier.weight[cls_id][None, :], dim=1)

could you please share how you determine the coefficient (0.3*sqrt(512))?

Thanks for your help!

[phizaz]

I believe the line 403-416 was not used in DiffAE at all, and hence not related to DiffAE. It was a legacy code from the base repo that we built upon.

Regarding your second question, 0.3*sqrt(512). 0.3 is definitely tuned by hand based on qualitative results. sqrt(512) is related to the fact that F.normalize(cls_model.classifier.weight[cls_id][None, :], dim=1) is a unit vector (which is rather small, and depends on the number of dims). It's nice to have something that scales with dims (make the coefficient more robust). I think a Gaussian random vector with 512 dimensions has the norm of sqrt(512), multiplying a unit vector with sqrt(512) scales it as though it has the same size as a random Gaussian vector.

[cyuting940612]

Really appreciate your reply.

But I think line 403-416 actually used in img = model.render(xT, cond2, T=100) from block[33] of manipulate.ipynb.

In which the render method can be conditioned based on cond_fn. But I don't find anywhere cond_fn be defined.

So could you please provide me some clue on how to implement this conditional render?

Thank you!