feat: add syncing models utility to ivy

[YushaArif99]

Wanted to have your thoughts on how to expose these utility functions? I think it makes sense to have them inside the ivy.functional.backends.tensorflow.module.py module. But this creates an issue as to how we import this helper. So there wont be any ivy.sync_models_torch_and_tf but instead we'll have to do:

from ivy.functional.backends.tensorflow.module import sync_models_torch_and_tf
sync_models_torch_and_tf(...)

What do you guys suggest we do here @Sam-Armstrong @hmahmood24

[hmahmood24]

Wanted to have your thoughts on how to expose these utility functions? I think it makes sense to have them inside the ivy.functional.backends.tensorflow.module.py module. But this creates an issue as to how we import this helper. So there wont be any ivy.sync_models_torch_and_tf but instead we'll have to do:

from ivy.functional.backends.tensorflow.module import sync_models_torch_and_tf
sync_models_torch_and_tf(...)

What do you guys suggest we do here @Sam-Armstrong @hmahmood24

My thinking was just exposing these as util functions so something like ivy.utils.sync_models_torch_and_tf or ivy.syncing_utils.sync_models_torch_and_tf seems cleaner to me rather than having to import the backends explicitly. What do you think @YushaArif99 ?

[Sam-Armstrong]

@YushaArif99 Couldn't we just store all helper functions like this in a new file like ivy/utils/syncing.py or something like that? Is there any need to have this in the backend?

Also, I'd suggest we could expose a general ivy.sync_models function, which will detect the relevant models types and route to the correct function - sync_models_torch_and_tf or whatever. If anything other than torch and tf models are passed, for now we can just throw a ivy.exceptions.IvyNotImplementedException. I think this would be the best UX.

[YushaArif99]

Sure! This is indeed a better UX. I was only inclined to not go this route as the logic contained within these utility helpers is framework-specific. And so it seems to deviate from the general convention of ivy API being an intermediary.
Then there's also a case of having dependency imports so will probably need to have the framework-specific imports as local imports.

Otherwise, this is definitely cleaner.

Based on this, should we still go forward with having these inside ivy.utils @hmahmood24 @Sam-Armstrong ?

[YushaArif99]

@YushaArif99 Couldn't we just store all helper functions like this in a new file like ivy/utils/syncing.py or something like that? Is there any need to have this in the backend?

Also, I'd suggest we could expose a general ivy.sync_models function, which will detect the relevant models types and route to the correct function - sync_models_torch_and_tf or whatever. If anything other than torch and tf models are passed, for now we can just throw a ivy.exceptions.IvyNotImplementedException. I think this would be the best UX.

Yeah that's a good idea 👍🏼

[Sam-Armstrong]

@YushaArif99 I don't see an issue with it, because you'd have to import torch into the tensorflow backend anyway, which is also against the general convention of ivy. But I guess if we do expose ivy.sync_models it wouldn't be super important where functions like sync_models_torch_and_tf are located anyway, as they wouldn't be used directly by the user. So I guess it's up to you really 👍

[YushaArif99]

@YushaArif99 I don't see an issue with it, because you'd have to import torch into the tensorflow backend anyway, which is also against the general convention of ivy. But I guess if we do expose ivy.sync_models it wouldn't be super important where functions like sync_models_torch_and_tf are located anyway, as they wouldn't be used directly by the user. So I guess it's up to you really 👍

Yeah I think exposing ivy.sync_models as the main interface is much cleaner. This way, we can have sync_models_torch_and_tf inside the TF module.py and similarily sync_models_torch_and_jax inside the JAX module.py which I feel is more intuitive.

I'll go ahead with this approach then. Thanks for the suggestions both!

[YushaArif99]

Hey @hmahmood24 @Sam-Armstrong, could you guys give this a final look and confirm whether we're all okay with this approach. I have added all the syncing logic inside ivy.stateful.utilities.py and have exposed ivy.sync_models_torch as the main interface for users.

[Sam-Armstrong]

lgtm! quick question though, could we generalise this to just be called ivy.sync_models, and in the longer term use this same function for syncing models from any framework? I feel like that would be a cleaner api. for now we can just throw a meaningful exception if a torch.nn.Module is not the first argument. what do you think @YushaArif99?

[YushaArif99]

question though, could we generalise this to just be called ivy.sync_models, and in the longer term use this same function for syncing models from any framework?

yeah we can definately do that. We can add source and target kwargs and then reroute to the appropriate implementation based on their values.

[Sam-Armstrong]

@YushaArif99 do we need source/target kwargs, can't we just infer based on the input types?

[hmahmood24]

@YushaArif99 LGTM. Agreed with @Sam-Armstrong's point though that keeping a single ivy.sync_models API should be cleaner if we can implement that 👍🏼

[YushaArif99]

@YushaArif99 do we need source/target kwargs, can't we just infer based on the input types?

@Sam-Armstrong that would require us importing all frameworks inside ivy.sync_models which is not ideal imo.

[Sam-Armstrong]

@YushaArif99 maybe we could check if each framework is in sys.modules when doing the check? so something like:

if "torch" in sys.modules:
    import torch

    if isinstance(x, torch.nn.Module):

[hmahmood24]

@YushaArif99 do we need source/target kwargs, can't we just infer based on the input types?

@Sam-Armstrong that would require us importing all frameworks inside ivy.sync_models which is not ideal imo.

@YushaArif99 Can't we just use this which shouldn't require us importing any fws