Predictive Coding vs. HGF ?

[cgoemaere]

I found this repo while looking for a Predictive Coding library, inspired by this paper. However, the focus here seems to be more on Generalized Hierarchical Gaussian Filters (HGF).

Could you clarify how HGFs relate to Predictive Coding? Are HGFs a specific instance of Predictive Coding, or do they represent a broader framework?

[PTWaade]

This probably warrants a deeper both technical and conceptual investigation, and I hesitate to claim much expertise in predictive coding in general. That said, I would say that the generalized HGF can be seen as one instance of a predictive coding network, with some differences between the traditional types of predictive coding networks.
One advantage of the HGF over at least some types of predictive coding networks (primarily from talking to Rafal Bogacz; I may have understood plenty of things) is that the HGF uses one-step update equations where other approaches use gradient descent on free energy. This leads to the HGF having a similar but slightly different microcircuitry. More details: https://arxiv.org/abs/2305.10937
Another is that HGF'es can include temporal dynamics i.e. volatilities and drifts - and that it can include volatility learning for dynamically adjusting the precision and learning rates in the Bayesian update. But it doesn't have to. So in this regard, classical Gaussian predictive coding networks might be (or at least be similar to) a special type of the generalized HGF.

Finally, pyhgf as a package is made to be able to in principle implement other kinds of predictive coding networks than the HGF. So it should in principle be possible to implement other types of predictive coding networks too.

In general, it would be exciting to compare these different variations of predictive coding properly.
Let us know if I'm mistaken in any of the above; if there is any interest in comparing the HGF to other predictive coding networks; if there is anything else we can help with; and if there is anything interesting worth sharing with us :)

Thanks for writing!

[cgoemaere]

Thanks for the clarification!

From your explanation and the linked paper, the main distinction seems to be that HGFs are inherently dynamic, modeling time sequences as guided random walks with predicted mean and variance. The hierarchical aspect then allows layers to model higher-order dynamics of the data.

By contrast, Predictive Coding typically works on static data, indeed by minimizing an energy function, as you said. I can see a connection when static data is treated as an unchanging time sequence (then, each energy minimization step would correspond to an HGF update step). But I guess it's a superficial link, given the fundamentally different focus: Predictive Coding doesn't emphasize time dynamics, and HGFs are not meant for static data.

Regardless, it would be cool to see a unifying framework for these two models! I wonder whether such a thing could exist and what it could tell us about how the brain might operate.

[PTWaade]

(Sorry for the late response here!)

We haven't shown this formally, especially because I'm not familiar enough with the classic predictive coding networks - but I think that they might be special cases of the new version of the HGF. If the lambda parameter (in the article I sent before) is set to 0 in the generalized HGF, it corresponds to changing the Gaussian random walk to a stationary Gaussian distribution where the mean is determined by the node above (the value parent).

It would be nice to compare this numerically or formally - I don't think it should be too hard - but we haven't had the time yet. Do let us know if this is something you'd like to do, and we'd be happy to collaborate on it :)
Otherwise, I can return here once we get to doing it ourselves.

In any case - the interest is well-received :)