Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Availability of the Colab notebook #1168

Open
2 tasks done
34j opened this issue May 10, 2024 · 2 comments
Open
2 tasks done

Availability of the Colab notebook #1168

34j opened this issue May 10, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@34j
Copy link
Collaborator

34j commented May 10, 2024

Many issues have been opened, but I think the notebook is just barely working, although it is rubbish.

Describe the bug

  1. f-strings in commands no longer working
  2. chore(notebook): copy configs files from Google Drive #1064 might be reducing the the simplicity unfortunately
  3. Logs are not displayed
  4. Annoying popup: WARNING: Upgrading ipython, ipykernel, tornado, prompt-toolkit, pyzmq can cause your runtime to repeatedly crash. https://colab.research.google.com/ #1163

To Reproduce

Run notebook

Additional context

No response

Version

2024/05/10

Platform

Google Colab

Code of Conduct

  • I agree to follow this project's Code of Conduct.

No Duplicate

  • I have checked existing issues to avoid duplicates.
@34j 34j added the bug Something isn't working label May 10, 2024
@34j 34j changed the title Availability of Colab notebooks Availability of the Colab notebook May 10, 2024
@xD0135
Copy link

xD0135 commented Sep 29, 2024

I had the same experience trying to run the provided Colab notebook and decided on a simpler approach, as shown below. Each code block here represents a code block that you add in Colab, and I'm focusing only on training and not including inference, as the original Colab notebook has. Also, make sure that you select a GPU instance, otherwise this won't work.

#@title Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')
#@title Install Dependencies
!python -m pip install so-vits-svc-fork
#@title Verify Dependencies
!svc --help

For this next step, I created a folder MyDrive/TTS/sovits in my Google Drive, hence it listed in the path. Inside the folder I then added my training data in the expected location dataset_raw/{speaker_id}/**/{wav_file}.{any_format} before running the following:

#@title Generate Config
!cd /content/drive/MyDrive/TTS/sovits && svc pre-resample && svc pre-config

This will create a few folders, and because I'm running on the free T4 Colab instance, I optimized my config file located in my Google Drive under MyDrive/TTS/sovits/configs/44k/config.json with the following (only showing the modified lines)

{
  "train": {
    "epochs": 201,
    "batch_size": 16
  }
}

If you made changes to the config file, save/upload them before proceeding. It's worth noting that using 16 as the batch_size keeps the VRAM stable at around 11.9 / 15.0 GB. Anything over will cause it to run OOM.

#@title Generate Hubert
!cd /content/drive/MyDrive/TTS/sovits && svc pre-hubert
#@title Start Training
!cd /content/drive/MyDrive/TTS/sovits && svc train

And that's it! For the training step, if you want to load the TensorBoard so that you have a visual representation of the training, you can use the following instead of the previous:

#@title Start Training
%load_ext tensorboard
%tensorboard --logdir /content/drive/MyDrive/TTS/sovits/logs/44k
!cd /content/drive/MyDrive/TTS/sovits && svc train

For my dataset with about 13 minutes of training audio, it took just over an hour to complete. And now that I understand how long it takes to run, I can increase the epochs parameter to match my desired tradeoff of training data quality. Hope it helps!

@Gambozza
Copy link

I'm experiencing an issue while trying to train a model using so-vits-svc-fork on Google Colab. I followed the provided notebook instructions and structured my files in Google Drive as follows:

📂 MyDrive/TTS/sovits/
├── 📂 dataset_raw/{speaker_id}/**/*.wav
├── 📂 logs/44k/
├── 📂 configs/44k/config.json

After mounting Google Drive and installing dependencies, I executed the preprocessing steps successfully:

Mount Google Drive

from google.colab import drive
drive.mount('/content/drive')

Install dependencies

!python -m pip install so-vits-svc-fork

Generate config and preprocess data

!cd /content/drive/MyDrive/TTS/sovits && svc pre-resample && svc pre-config

Generate Hubert features

!cd /content/drive/MyDrive/TTS/sovits && svc pre-hubert

Start training

!cd /content/drive/MyDrive/TTS/sovits && svc train

The training command executes but doesn’t seem to progress. The output only shows:
[INFO] NumExpr defaulting to 2 threads.
and nothing else happens.

I checked TensorBoard logs at /content/drive/MyDrive/TTS/sovits/logs/44k/lightning_logs/version_0/, but no relevant data appears.
The dataset is correctly structured, and running ls dataset_raw/{speaker_id}/ confirms that all .wav files are present.

Additionally, GPU availability is confirmed (nvidia-smi shows Tesla T4 with CUDA 12.4), but it's not being utilized (GPU-Util: 0%).

I have tried:

Restarting the training process multiple times.
Checking logs in logs/44k/, but they don’t seem to update.
Running on CPU (CUDA_VISIBLE_DEVICES=-1), but the issue persists.
Ensuring TensorFlow and dependencies are up to date (pip show tensorflow shows version 2.15.0).
Attempting to run svc train --debug, but the flag isn’t recognized.

At this point, I’m unsure what is preventing the training from progressing. Has anyone encountered a similar issue on Colab? Is there a way to enable detailed logs or debug mode to identify the problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants