Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

module av_bench #40

Open
bliinmaker opened this issue Feb 7, 2025 · 7 comments
Open

module av_bench #40

bliinmaker opened this issue Feb 7, 2025 · 7 comments

Comments

@bliinmaker
Copy link

bliinmaker commented Feb 7, 2025

Hello, can you tell me where the avbenchmark project should be located so that I can access it in MMAudio? It is stored separately or inside MMAudio. I will be very grateful if you answer.

I am currently storing avbenchmark in the MM Audio folder, here is my mistake for your understanding:

(venv) tariell@Cinemamix-AI:/media/tariell/AI/AI/Audio_AI/MMAudio$ OMP_NUM_THREADS=4 torchrun --standalone --nproc_per_node=1 train.py exp_id=debug compile=False  debug=True example_train=True  batch_size=1
Traceback (most recent call last):
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/train.py", line 18, in <module>
    from mmaudio.runner import Runner
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/mmaudio/runner.py", line 12, in <module>
    from avbenchmark.av_bench.evaluate import evaluate
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/avbenchmark/av_bench/evaluate.py", line 9, in <module>
    from av_bench.metrics import compute_fd, compute_isc, compute_kl
ModuleNotFoundError: No module named 'av_bench'
E0208 00:11:29.127000 53067 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 53074) of binary: /media/tariell/AI/AI/Audio_AI/MMAudio/venv/bin/python3
Traceback (most recent call last):
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 138, in call
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-02-08_00:11:29
  host      : Cinemamix-AI
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 53074)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

(venv) tariell@Cinemamix-AI:/media/tariell/AI/AI/Audio_AI/MMAudio$
@hkchengrex
Copy link
Owner

It doesn't matter where you are putting it. Did you perform the pip install -e . step listed here https://github.com/hkchengrex/av-benchmark?

@bliinmaker
Copy link
Author

yes, I did, but I have a question how can I access the av_bench module in the mmaudio/runner file? That's why I'm asking where to put this module better.

@bliinmaker
Copy link
Author

bliinmaker commented Feb 7, 2025

initially, I installed av_bench due to an error

(venv) tariell@Cinemamix-AI:/media/tariell/AI/AI/Audio_AI/MMAudio$ OMP_NUM_THREADS=4 torchrun --standalone --nproc_per_node=1 train.py exp_id=debug compile=False  debug=True example_train=True  batch_size=1
Traceback (most recent call last):
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/train.py", line 18, in <module>
    from mmaudio.runner import Runner
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/mmaudio/runner.py", line 12, in <module>
    from av_bench.evaluate import evaluate
ModuleNotFoundError: No module named 'av_bench'
E0207 21:06:10.187000 49314 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 49321) of binary: /media/tariell/AI/AI/Audio_AI/MMAudio/venv/bin/python3
Traceback (most recent call last):
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 138, in call
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-02-07_21:06:10
  host      : Cinemamix-AI
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 49321)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
(venv) tariell@Cinemamix-AI:/media/tariell/AI/AI/Audio_AI/MMAudio$

@hkchengrex
Copy link
Owner

If you have installed it, python should find it just like any other pip-installed package. How about uninstalling and reinstalling it?

@bliinmaker
Copy link
Author

in other words, I can download av-bench anywhere. And to do this, it is important to execute the pip install -e command. in MMAudio, so that python finds the av-bench. Am I right? Thanks

@bliinmaker
Copy link
Author

bliinmaker commented Feb 8, 2025

the av_bench project has started, but an error occurs during the test run of training

(venv) tariell@Cinemamix-AI:/media/tariell/AI/AI/Audio_AI/MMAudio$ OMP_NUM_THREADS=4 torchrun --standalone --nproc_per_node=1 train.py exp_id=debug compile=False  debug=True example_train=True  batch_size=1
[2025-02-08 16:14:17][r0][INFO] - Initialized: local_rank=0, world_size=1
[2025-02-08 16:14:17][r0][INFO] - git - main 58a74a4bcdf75753a123adc6b7c37bf1ac9703f9
[2025-02-08 16:14:17][r0][INFO] - All configuration: {'data': {'VGGSound': {'root': '../data/video', 'subset_name': 'sets/vgg3-train.tsv', 'fps': 8, 'height': 384, 'width': 384, 'sample_duration_sec': 8.0}, 'VGGSound_test': {'root': '../data/video', 'subset_name': 'sets/vgg3-test.tsv', 'fps': 8, 'height': 384, 'width': 384, 'sample_duration_sec': 8.0}, 'VGGSound_val': {'root': '../data/video', 'subset_name': 'sets/vgg3-val.tsv', 'fps': 8, 'height': 384, 'width': 384, 'sample_duration_sec': 8.0}, 'ExtractedVGG': {'tsv': '../data/v1-16-memmap/vgg-train.tsv', 'memmap_dir': '../data/v1-16-memmap/vgg-train'}, 'ExtractedVGG_test': {'tag': 'test', 'gt_cache': '../data/eval-cache/vggsound-test', 'output_subdir': None, 'tsv': '../data/v1-16-memmap/vgg-test.tsv', 'memmap_dir': '../data/v1-16-memmap/vgg-test'}, 'ExtractedVGG_val': {'tag': 'val', 'gt_cache': '../data/eval-cache/vggsound-val', 'output_subdir': 'val', 'tsv': '../data/v1-16-memmap/vgg-val.tsv', 'memmap_dir': '../data/v1-16-memmap/vgg-val'}, 'AudioCaps': {'tsv': '../data/v1-16-memmap/audiocaps.tsv', 'memmap_dir': '../data/v1-16-memmap/audiocaps'}, 'AudioSetSL': {'tsv': '../data/v1-16-memmap/audioset_sl.tsv', 'memmap_dir': '../data/v1-16-memmap/audioset_sl'}, 'BBCSound': {'tsv': '../data/v1-16-memmap/bbcsound.tsv', 'memmap_dir': '../data/v1-16-memmap/bbcsound'}, 'FreeSound': {'tsv': '../data/v1-16-memmap/freesound.tsv', 'memmap_dir': '../data/v1-16-memmap/freesound'}, 'Clotho': {'tsv': '../data/v1-16-memmap/clotho.tsv', 'memmap_dir': '../data/v1-16-memmap/clotho'}, 'Example_video': {'tsv': './training/example_output/memmap/vgg-example.tsv', 'memmap_dir': './training/example_output/memmap/vgg-example'}, 'Example_audio': {'tsv': './training/example_output/memmap/audio-example.tsv', 'memmap_dir': './training/example_output/memmap/audio-example'}}, 'eval_data': {'AudioCaps': {'audio_path': '../data/AudioCaps-test-audioldm-ver', 'csv_path': '../data/AudioCaps-test-audioldm-ver/data.csv'}, 'AudioCaps_full': {'audio_path': '../data/AudioCaps-test-full-ver', 'csv_path': '../data/AudioCaps-test-full-ver/data.csv'}, 'MovieGen': {'video_path': '../data/MovieGen/MovieGenAudioBenchSfx/video_with_audio', 'jsonl_path': '../data/MovieGen/MovieGenAudioBenchSfx/metadata'}, 'VGGSound': {'video_path': '../data/test-videos', 'csv_path': '../data/vggsound.csv'}}, 'enable_email': False, 'model': 'small_16k', 'exp_id': 'debug', 'debug': True, 'cudnn_benchmark': True, 'compile': False, 'amp': True, 'weights': None, 'checkpoint': None, 'seed': 14159265, 'num_workers': 10, 'pin_memory': False, 'data_dim': {'text_seq_len': 77, 'clip_dim': 1024, 'sync_dim': 768, 'text_dim': 1024, 'latent_seq_len': 250, 'clip_seq_len': 64, 'sync_seq_len': 192}, 'ema': {'enable': True, 'sigma_rels': [0.05, 0.1], 'update_every': 1, 'checkpoint_every': 5000, 'checkpoint_folder': './output/${exp_id}/ema_ckpts', 'default_output_sigma': 0.05, 'start': 0}, 'sampling': {'mean': 0.0, 'scale': 1.0, 'min_sigma': 0.0, 'method': 'euler', 'num_steps': 25}, 'null_condition_probability': 0.1, 'cfg_strength': 4.5, 'vae_16k_ckpt': './ext_weights/v1-16.pth', 'vae_44k_ckpt': './ext_weights/v1-44.pth', 'bigvgan_vocoder_ckpt': './ext_weights/best_netG.pt', 'synchformer_ckpt': './ext_weights/synchformer_state_dict.pth', 'mini_train': False, 'example_train': True, 'vgg_oversample_rate': 3, 'log_text_interval': 200, 'log_extra_interval': 20000, 'val_interval': 5000, 'eval_interval': 20000, 'save_eval_interval': 40000, 'save_weights_interval': 10000, 'save_checkpoint_interval': 10000, 'save_copy_iterations': [], 'batch_size': 1, 'eval_batch_size': 256, 'num_iterations': 300000, 'learning_rate': 0.0001, 'linear_warmup_steps': 1000, 'lr_schedule': 'step', 'lr_schedule_steps': [240000,
270000], 'lr_schedule_gamma': 0.1, 'clip_grad_norm': 1.0, 'weight_decay': 1e-06}
[2025-02-08 16:14:17][r0][INFO] - Number of GPUs detected: 1
[2025-02-08 16:14:17][r0][INFO] - Number of dataloader workers (per GPU): 10
[2025-02-08 16:14:17][r0][INFO] - Training configuration: {'data': {'VGGSound': {'root': '../data/video', 'subset_name': 'sets/vgg3-train.tsv', 'fps': 8, 'height': 384, 'width': 384, 'sample_duration_sec': 8.0}, 'VGGSound_test': {'root': '../data/video', 'subset_name': 'sets/vgg3-test.tsv', 'fps': 8, 'height': 384, 'width': 384, 'sample_duration_sec': 8.0}, 'VGGSound_val': {'root': '../data/video', 'subset_name': 'sets/vgg3-val.tsv', 'fps': 8, 'height': 384, 'width': 384, 'sample_duration_sec': 8.0}, 'ExtractedVGG': {'tsv': '../data/v1-16-memmap/vgg-train.tsv', 'memmap_dir': '../data/v1-16-memmap/vgg-train'}, 'ExtractedVGG_test': {'tag': 'test', 'gt_cache': '../data/eval-cache/vggsound-test', 'output_subdir': None, 'tsv': '../data/v1-16-memmap/vgg-test.tsv', 'memmap_dir': '../data/v1-16-memmap/vgg-test'}, 'ExtractedVGG_val': {'tag': 'val', 'gt_cache': '../data/eval-cache/vggsound-val', 'output_subdir': 'val', 'tsv': '../data/v1-16-memmap/vgg-val.tsv', 'memmap_dir': '../data/v1-16-memmap/vgg-val'}, 'AudioCaps': {'tsv': '../data/v1-16-memmap/audiocaps.tsv', 'memmap_dir': '../data/v1-16-memmap/audiocaps'}, 'AudioSetSL': {'tsv': '../data/v1-16-memmap/audioset_sl.tsv', 'memmap_dir': '../data/v1-16-memmap/audioset_sl'}, 'BBCSound': {'tsv': '../data/v1-16-memmap/bbcsound.tsv', 'memmap_dir': '../data/v1-16-memmap/bbcsound'}, 'FreeSound': {'tsv': '../data/v1-16-memmap/freesound.tsv', 'memmap_dir': '../data/v1-16-memmap/freesound'}, 'Clotho': {'tsv': '../data/v1-16-memmap/clotho.tsv', 'memmap_dir': '../data/v1-16-memmap/clotho'}, 'Example_video': {'tsv': './training/example_output/memmap/vgg-example.tsv', 'memmap_dir': './training/example_output/memmap/vgg-example'}, 'Example_audio': {'tsv': './training/example_output/memmap/audio-example.tsv', 'memmap_dir': './training/example_output/memmap/audio-example'}}, 'eval_data': {'AudioCaps': {'audio_path': '../data/AudioCaps-test-audioldm-ver', 'csv_path': '../data/AudioCaps-test-audioldm-ver/data.csv'}, 'AudioCaps_full': {'audio_path': '../data/AudioCaps-test-full-ver', 'csv_path': '../data/AudioCaps-test-full-ver/data.csv'}, 'MovieGen': {'video_path': '../data/MovieGen/MovieGenAudioBenchSfx/video_with_audio', 'jsonl_path': '../data/MovieGen/MovieGenAudioBenchSfx/metadata'}, 'VGGSound': {'video_path': '../data/test-videos', 'csv_path': '../data/vggsound.csv'}}, 'enable_email': False, 'model': 'small_16k', 'exp_id': 'debug', 'debug': True, 'cudnn_benchmark': True, 'compile': False, 'amp': True, 'weights': None, 'checkpoint': None, 'seed': 14159265, 'num_workers': 10, 'pin_memory': False, 'data_dim': {'text_seq_len': 77, 'clip_dim': 1024, 'sync_dim': 768, 'text_dim': 1024, 'latent_seq_len': 250, 'clip_seq_len': 64, 'sync_seq_len': 192}, 'ema': {'enable': True, 'sigma_rels': [0.05, 0.1], 'update_every': 1, 'checkpoint_every': 5000, 'checkpoint_folder': './output/${exp_id}/ema_ckpts', 'default_output_sigma': 0.05, 'start': 0}, 'sampling': {'mean': 0.0, 'scale': 1.0, 'min_sigma': 0.0, 'method': 'euler', 'num_steps': 25}, 'null_condition_probability': 0.1, 'cfg_strength': 4.5, 'vae_16k_ckpt': './ext_weights/v1-16.pth', 'vae_44k_ckpt': './ext_weights/v1-44.pth', 'bigvgan_vocoder_ckpt': './ext_weights/best_netG.pt', 'synchformer_ckpt': './ext_weights/synchformer_state_dict.pth', 'mini_train': False, 'example_train': True, 'vgg_oversample_rate': 3, 'log_text_interval': 200, 'log_extra_interval': 20000, 'val_interval': 5000, 'eval_interval': 20000, 'save_eval_interval': 40000, 'save_weights_interval': 10000, 'save_checkpoint_interval': 10000, 'save_copy_iterations': [], 'batch_size': 1, 'eval_batch_size': 256, 'num_iterations': 300000, 'learning_rate': 0.0001, 'linear_warmup_steps': 1000, 'lr_schedule': 'step', 'lr_schedule_steps': [240000, 270000], 'lr_schedule_gamma': 0.1, 'clip_grad_norm': 1.0, 'weight_decay': 1e-06}
[2025-02-08 16:14:17][r0][INFO] - Batch size (per GPU): 1
Error executing job with overrides: ['exp_id=debug', 'compile=False', 'debug=True', 'example_train=True', 'batch_size=1']
Traceback (most recent call last):
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/train.py", line 88, in train
    dataset, sampler, loader = setup_training_datasets(cfg)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/mmaudio/data/data_setup.py", line 51, in setup_training_datasets
    video = load_vgg_data(cfg, cfg.data.Example_video)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/mmaudio/data/data_setup.py", line 30, in load_vgg_data
    dataset = ExtractedVGG(tsv_path=data_cfg.tsv,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/mmaudio/data/extracted_vgg.py", line 27, in init
    self.df_list = pd.read_csv(tsv_path, sep='\t').to_dict('records')
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1620, in init
    self._engine = self._make_engine(f, self.engine)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
                   ^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
             ^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './training/example_output/memmap/vgg-example.tsv'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[rank0]:[W208 16:14:17.619578307 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
E0208 16:14:18.248000 9711 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 9718) of binary: /media/tariell/AI/AI/Audio_AI/MMAudio/venv/bin/python3
Traceback (most recent call last):
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 138, in call
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/tariell/AI/AI/Audio_AI/MMAudio/venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-02-08_16:14:18
  host      : Cinemamix-AI
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 9718)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
(venv) tariell@Cinemamix-AI:/media/tariell/AI/AI/Audio_AI/MMAudio$

@hkchengrex
Copy link
Owner

FileNotFoundError: [Errno 2] No such file or directory: './training/example_output/memmap/vgg-example.tsv'

Do you have this file generated? It should be a product of extract_video_training_latents.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants