Synthcity

[laraPPr]

• link to support ticket: #2024062760001076
• website: https://github.com/vanderschaarlab/synthcity
• installation docs: https://github.com/vanderschaarlab/synthcity
• toolchain: foss/2023a
• easyblock to use: PythonBundle
• required dependencies:
  • ...
• optional dependencies:
  • ...
• notes:
  • There is already a version of synthcity on the HPC (synthcity/0.2.4-foss-2022a) which is from March 2023, but there have been some updates with functions that I need for my current work and are not available in this version
• effort: (TBD)
• other install methods
  • conda: yes (link?) / no
  • container image: yes (link?) / no
  • pre-built binaries (RHEL8 Linux x86_64): yes (link?) / no
  • easyconfig outside EasyBuild: yes / no

[laraPPr]

https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/s/synthcity/synthcity-0.2.4-foss-2022a.eb

[pavelToman]

There is a problem with version of PyTorch:

• synthcity need PyTorch v1.x but for foss/2023a there is only v2.x which is not compatible
• I can try to create PyTorch v1.x for foss/2023a but I have to either create MONAI with this version of PyTorch
• sythcity is not compatible with XGBoost v2.x neither PyTorch-Lightning v2.x

Synthcity v0.2.10 needs be-great >= 0.0.5 and it needs numpy>=1.24.2 and pandas>=1.4.4 so it can not be created for foss/2022a.

[pavelToman]

Part of log from pytest with errors for Synthcity-0.2.10-foss-2023a:

# ============================================== short test summary info ===============================================
        # FAILED tests/plugins/core/test_schema.py::test_schema_fail - AttributeError: module 'pydantic' has no attribute 'py...
            # -> move fix prom preinstallopts to runtest cmd - should fix this
            
            # def test_schema_fail() -> None:
            #     if pydantic.__version__ < "2":
            #         with pytest.raises(pydantic.error_wrappers.ValidationError):
            #             Schema(data="sdfsfs")
            #     else:
            #         with pytest.raises(pydantic.pydantic_core._pydantic_core.ValidationError):
            # tests/plugins/core/test_schema.py:17: 
            # _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/__init__.
            # py:363: in __getattr__
            #     return _getattr_migration(attr_name)
            # _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
            # > raise AttributeError(f'module {module!r} has no attribute {name!r}')
            # E AttributeError: module 'pydantic' has no attribute 'pydantic_core'
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/_migration.py:306: AttributeError

        # FAILED tests/plugins/privacy/test_decaf.py::test_plugin_generate[test_plugin0] - RuntimeError: Training with multip...
            # -> https://stackoverflow.com/questions/75907443/pytorch-lightning-works-on-cuda-but-throws-training-with-multiple-optimizers-is
            # -> probably skip this test or create PyTorch-Lightning-1.8.4.foss-2023a -> pytorch-lighning v1.8.4 created
            
            # tests/plugins/privacy/test_decaf.py:110: 
            # _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:55: in wrapper_function
            #     return vd.call(*args, **kwargs)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:150: in call
            #     return self.execute(m)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:222: in execute
            #     return self.raw_function(**d, **var_kwargs)
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/plugin.py:256: in fit
            #     output = self._fit(X, *args, **kwargs)
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/privacy/plugin_decaf.py:437: in _
            # fit
            #     trainer.fit(self.model, dm)
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/trainer/trainer.py:544: in fit
            #     call._call_and_handle_interrupt(
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/trainer/call.py:44: in _call_and_handle_interrupt
            #     return trainer_fn(*args, **kwargs)
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/trainer/trainer.py:580: in _fit_impl
            #     self._run(model, ckpt_path=ckpt_path)
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/trainer/trainer.py:963: in _run
            #     self.strategy.setup(self)
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/strategies/strategy.py:159: in setup
            #     self.setup_optimizers(trainer)
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/strategies/strategy.py:139: in setup_optimizers
            #     self.optimizers, self.lr_scheduler_configs = _init_optimizers_and_lr_schedulers(self.lightning_module)
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/py
            # thon3.11/site-packages/pytorch_lightning/core/optimizer.py:192: in _init_optimizers_and_lr_schedulers
            #     _validate_multiple_optimizers_support(optimizers, model)
            # _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
            # def _validate_multiple_optimizers_support(optimizers: List[Optimizer], model: "pl.LightningModule") -> None:
            #    if is_param_in_hook_signature(model.training_step, "optimizer_idx", explicit=True):
            #        raise RuntimeError(
            #             "Training with multiple optimizers is only supported with manual optimization. Remove the `optimizer_idx`"
            #             " argument from `training_step`, set `self.automatic_optimization = False` and access your optimizers"
            #             " in `training_step` with `opt1, opt2, ... = self.optimizers()`."
            #         )
            # E           RuntimeError: Training with multiple optimizers is only supported with manual optimization. 
            # Remove the `optimizer_idx` argument from `training_step`, set `self.automatic_optimization = False` and access your optimizers in `training_step` with `opt1, opt2, ... = self.optimizers()`.
            # /scratch/gent/vo/001/gvo00117/easybuild/RHEL8/cascadelake-ampere-ib/software/PyTorch-Lightning/2.2.1-foss-2023a/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py:351: RuntimeError
        # FAILED tests/plugins/privacy/test_decaf.py::test_plugin_generate[test_plugin1] - RuntimeError: Training with multip...
        # FAILED tests/plugins/privacy/test_decaf.py::test_plugin_generate[test_plugin2] - RuntimeError: Training with multip...
        
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate[True-test_plugin1] - AttributeError: 'DPOptimizer'...
            # -> seems DPOptimizer in Opacus do not have _optimizer_step_pre_hooks but Optimizer in Torch has -> problem -> can create PyTorch v1.13 but it is either in MONAI
            # -> DPOptimizer is in Opacus - but it gets the Torch Optimizer then and it has _optimizer_step_pre_hooks
            # ->https://github.com/pytorch/opacus/blob/v1.4.1/opacus/optimizers/optimizer.py
            # ->https://github.com/pytorch/pytorch/blob/v2.1.2/torch/optim/optimizer.py
            # maybe connected with this bug: https://github.com/pytorch/opacus/issues/649 or https://github.com/pytorch/opacus/issues/523
            
            # tests/plugins/privacy/test_dpgan.py:81: 
            # _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:55: in wrapper_function
            #     return vd.call(*args, **kwargs)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:150: in call
            #     return self.execute(m)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:222: in execute
            #     return self.raw_function(**d, **var_kwargs)
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/plugin.py:256: in fit
            #     output = self._fit(X, *args, **kwargs)
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/privacy/plugin_dpgan.py:287: in _
            # fit
            #     self.model.fit(X.dataframe(), cond=cond)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:55: in wrapper_function
            #     return vd.call(*args, **kwargs)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:150: in call
            #     return self.execute(m)
            # /apps/gent/RHEL8/cascadelake-ib/software/pydantic/2.5.3-GCCcore-12.3.0/lib/python3.11/site-packages/pydantic/deprecate
            # d/decorator.py:222: in execute
            #     return self.raw_function(**d, **var_kwargs)
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/models/tabular_gan.py:358: i
            # n fit
            #     self.model.fit(
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/models/gan.py:278: in fit
            #     self._train(
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/models/gan.py:616: in _train
            #     g_loss, d_loss = self._train_epoch(
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/models/gan.py:493: in _train
            # _epoch
            #     self._train_epoch_discriminator(
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/models/gan.py:462: in _train
            # _epoch_discriminator
            #     self.discriminator.optimizer.step()
            # ...
            # @functools.wraps(func)
            #     def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> R:
                #     self, *_ = args
                #     self = cast(Optimizer, self)
                #     profile_name = f"Optimizer.step#{self.__class__.__name__}.step"
                #     with torch.autograd.profiler.record_function(profile_name):
                #         # call optimizer step pre hooks
            # >           for pre_hook in chain(_global_optimizer_pre_hooks.values(), self._optimizer_step_pre_hooks.values()):
            # E           AttributeError: 'DPOptimizer' object has no attribute '_optimizer_step_pre_hooks'
            # /apps/gent/RHEL8/cascadelake-ib/software/PyTorch/2.1.2-foss-2023a/lib/python3.11/site-packages/torch/optim/optimizer.py:363: AttributeError
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate[True-test_plugin2] - AttributeError: 'DPOptimizer'...
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate[False-test_plugin0] - AttributeError: 'DPOptimizer...
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate[False-test_plugin1] - AttributeError: 'DPOptimizer...
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate[False-test_plugin2] - AttributeError: 'DPOptimizer...
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate_constraints[test_plugin0] - AttributeError: 'DPOpt...
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate_constraints[test_plugin1] - AttributeError: 'DPOpt...
        # FAILED tests/plugins/privacy/test_dpgan.py::test_plugin_generate_constraints[test_plugin2] - AttributeError: 'DPOpt...
        
        # FAILED tests/plugins/time_series/test_timegan.py::test_plugin_generate_static_cond[source0] - ValueError: setting a...
            # -> should be solved: it was moved to prerunopts so should be ok
                # FROM: https://stackoverflow.com/questions/67183501/setting-an-array-element-with-a-sequence-requested-array-has-an-inhomogeneous-sh
                # With dtype="object" it works for me. -> variable2 = np.asarray(variable1, dtype="object")
            # tests/plugins/time_series/test_timegan.py:112:
            # def unpack(self, as_numpy: bool = False, pad: bool = False) -> Any:
            #     if pad:
            #         (
            #             static_data,
            #             temporal_data,
            #             observation_times,
            #             outcome,
            #         ) = TimeSeriesDataLoader.pad_and_mask(
            #             self.data["static_data"],
            #             self.data["temporal_data"],
            #             self.data["observation_times"],
            #             self.data["outcome"],
            #         )
            #     else:
            #         static_data, temporal_data, observation_times, outcome = (
            #             self.data["static_data"],
            #             self.data["temporal_data"],
            #             self.data["observation_times"],
            #             self.data["outcome"],
            #         )
            
            #     if as_numpy:
            #         longest_observation_seq = max([len(seq) for seq in temporal_data])
            #         return (
            #             np.asarray(static_data),
            #             np.asarray(
            #                 temporal_data
            #             ),  # TODO: check this works with time series benchmarks
            #             # masked array to handle variable length sequences
            #             ma.vstack(
            #                 [
            #                     ma.array(
            #                         np.resize(ot, longest_observation_seq),
            #                         mask=[True for i in range(len(ot))]
            #                         + [False for j in range(longest_observation_seq - len(ot))],
            #                     )
            #                     for ot in observation_times
            #                 ]
            #             ),
            #             np.asarray(outcome),
            #         )
            # E           ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after
            # 1 dimensions. The detected shape was (50,) + inhomogeneous part.
            # /tmp/vsc47063/eb-2r7nzoxm/tmps38qxu1i/lib/python3.11/site-packages/synthcity/plugins/core/dataloader.py:936: ValueError
        # FAILED tests/plugins/time_series/test_timegan.py::test_plugin_generate_static_cond[source1] - ValueError: setting a...
        # ======== 14 failed, 2145 passed, 13 skipped, 103 deselected, 2 xfailed, 2350 warnings in 15515.30s (4:18:35) ========

[pavelToman]

For now Synthcity-0.2.10 can be installed with a toolchain foss/2022b, but not with foss/2023a, because of PyTorch and NumPy version.
As soon as Synthcity will support PyTorch v2 and newer Numpy it will be possible to install it with newer toolchain.

[pavelToman]

PR v0.2.11-foss-2023b:

• {lib}[foss/2023b] synthcity v0.2.11, pycodestyle v2.11.1, python-igraph v0.11.8, ... w/ PyTorch 2.1.2, llvmlite, ... easybuilders/easybuild-easyconfigs#21845

[boegel]

Problem to build ITK 5.4.0 for Synthcity-0.2.11-foss-2023b reported here:

* [Compilation error of v5.4.0 with GCC-13.2.0 - CastXML problem? InsightSoftwareConsortium/ITK#4911](https://github.com/InsightSoftwareConsortium/ITK/issues/4911)

@pavelToman How did you deal with this issue in the end?
You should definitely also follow up there, so they at least know how you worked around it...

[pavelToman]

Problem to build ITK 5.4.0 for Synthcity-0.2.11-foss-2023b reported here:

* [Compilation error of v5.4.0 with GCC-13.2.0 - CastXML problem? InsightSoftwareConsortium/ITK#4911](https://github.com/InsightSoftwareConsortium/ITK/issues/4911)

@pavelToman How did you deal with this issue in the end? You should definitely also follow up there, so they at least know how you worked around it...

I used MONAI-minimum to avoid use ITK as its dependency. Seems synthcity does not need it as a part of MONAI.

[pavelToman]

Problem to build ITK 5.4.0 for Synthcity-0.2.11-foss-2023b reported here:

* [Compilation error of v5.4.0 with GCC-13.2.0 - CastXML problem? InsightSoftwareConsortium/ITK#4911](https://github.com/InsightSoftwareConsortium/ITK/issues/4911)

@pavelToman How did you deal with this issue in the end? You should definitely also follow up there, so they at least know how you worked around it...

They fix it! Now the ITK-5.4.0 works with CastXML-0.6.10.
See easybuilders/easybuild-easyconfigs#21937