Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can we create a demo file to test trained models and visualize results? #8

Open
Chu-Orion opened this issue Jan 10, 2024 · 3 comments

Comments

@Chu-Orion
Copy link

Chu-Orion commented Jan 10, 2024

Greetings!!! I'm sorry to bother you again. Now that the model has been trained, I would like to ask for advice on how to test and generate a 3D voxel model using your own images. I will be grateful if model visualization script is collected in a test file.

@Chu-Orion Chu-Orion changed the title How to test and generate a 3D voxel model? How can we create a demo file to test trained models and visualize results? Jan 10, 2024
@LiyingCV
Copy link
Owner

You can add config --test in train.sh for testing. For visualization, please refer Pix2Vox

@Chu-Orion
Copy link
Author

Chu-Orion commented Jan 13, 2024

Thank you very much for your reply, following your instructions, I referred to Pix2Vox and got the following visualization script showed as below:
`

encoder = Encoder(cfg)
decoder = Decoder(cfg)
merger = Merger(cfg)

# cfg.CONST.WEIGHTS = '/home/caig/Desktop/pix2vox/Pix2Vox/pretrain/Pix2Vox-A-ShapeNet.pth'
cfg.CONST.WEIGHTS = './pths/UMIFormer.pth'
checkpoint = torch.load(cfg.CONST.WEIGHTS, map_location=torch.device('cpu'))

fix_checkpoint = {}
fix_checkpoint['encoder_state_dict'] = OrderedDict(
    (k.split('module.')[1:][0], v) for k, v in checkpoint['encoder_state_dict'].items())
fix_checkpoint['decoder_state_dict'] = OrderedDict(
    (k.split('module.')[1:][0], v) for k, v in checkpoint['decoder_state_dict'].items())
fix_checkpoint['merger_state_dict'] = OrderedDict(
    (k.split('module.')[1:][0], v) for k, v in checkpoint['merger_state_dict'].items())
# fix_checkpoint['refiner_state_dict'] = OrderedDict(
#     (k.split('module.')[1:][0], v) for k, v in checkpoint['refiner_state_dict'].items())

epoch_idx = checkpoint['epoch_idx']
encoder.load_state_dict(fix_checkpoint['encoder_state_dict'])
decoder.load_state_dict(fix_checkpoint['decoder_state_dict'])
merger.load_state_dict(fix_checkpoint['merger_state_dict'])
# if cfg.NETWORK.USE_REFINER:
#     print('Use refiner')
#     refiner.load_state_dict(fix_checkpoint['refiner_state_dict'])
# if cfg.NETWORK.USE_MERGER:
#     print('Use merger')
#     merger.load_state_dict(fix_checkpoint['merger_state_dict'])

encoder.eval()
decoder.eval()
merger.eval()
# refiner.eval()


# img1_path = './datasets/ShapeNet/ShapeNetRendering/03211117/1a92363c2a155ed3c397356311cbeea4/rendering/17.png'
img1_path = './datasets/Pix3D/img/chair/0016.png'
img1_np = cv2.imread(img1_path, cv2.IMREAD_UNCHANGED).astype(np.float32) / 255.

sample = np.array([img1_np])

IMG_SIZE = cfg.CONST.IMG_H, cfg.CONST.IMG_W
CROP_SIZE = cfg.CONST.CROP_IMG_H, cfg.CONST.CROP_IMG_W

test_transforms = utils.data_transforms.Compose([
    utils.data_transforms.CenterCrop(IMG_SIZE, CROP_SIZE),
    utils.data_transforms.RandomBackground(cfg.TEST.RANDOM_BG_COLOR_RANGE),
    utils.data_transforms.Normalize(mean=cfg.DATASET.MEAN, std=cfg.DATASET.STD),
    utils.data_transforms.ToTensor(),
])

rendering_images = test_transforms(rendering_images=sample)
rendering_images = rendering_images.unsqueeze(0)

with torch.no_grad():
    image_features = encoder(rendering_images)
    raw_features, generated_volume = decoder(image_features)
    # if cfg.NETWORK.USE_MERGER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_MERGER:
    #     generated_volume = merger(raw_features, generated_volume)
    # else:
    #     generated_volume = torch.mean(generated_volume, dim=1)
    # if cfg.NETWORK.USE_REFINER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_REFINER:
    #     generated_volume = refiner(generated_volume)
    generated_volume = merger(raw_features, generated_volume)


generated_volume = generated_volume.squeeze(0)

img_dir = './output/TestResult'
gv = generated_volume.cpu().numpy()
gv_new = np.swapaxes(gv, 2, 1)
rendering_views = utils.binvox_visualization.get_volume_views(gv_new, os.path.join(img_dir),
                                                              epoch_idx)`

I am using this script in the runner.py
`

# Start train/test process
if not args.test and not args.batch_test:
    train_net(cfg)
else:
    if 'WEIGHTS' in cfg.CONST and os.path.exists(cfg.CONST.WEIGHTS):
        if args.test:
            test_net(test_single)
        elif args.batch_test:
            batch_test(cfg)
    else:
        logging.error('Please specify the file path of checkpoint.')
        sys.exit(2)

`
however, there is still something wrong when I try to generate the voxel model, and the Traceback info

_warnings.warn(
Traceback (most recent call last):
  File "runner.py", line 20, in <module>
    from core.test_single_image import test_single
  File "/root/UMIFormer/core/test_single_image.py", line 16, in <module>
    class test_single():
  File "/root/UMIFormer/core/test_single_image.py", line 18, in test_single
    encoder = Encoder(cfg)
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 21, in __init__
    self.encoder = self.create_model(
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 57, in create_model
    return self._create_vision_transformer(
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 43, in _create_vision_transformer
    model = self.decoupling_type(
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 200, in __init__
    if torch.distributed.get_rank() == 0:
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 822, in get_rank
    default_pg = _get_default_group()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 410, in _get_default_group
    raise RuntimeError(
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 13409) of binary: /root/miniconda3/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
runner.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-13_12:59:56
  host      : autodl-container-d82911a03c-d427ecf0
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 13409)
  error_file: <N/A>_

so, what should I do to fix this? Thank you very much!!! (I'm really sorry for my weak coding skills (༎ຶ ෴ ༎ຶ))

@LiyingCV
Copy link
Owner

Thank you very much for your reply, following your instructions, I referred to Pix2Vox and got the following visualization script showed as below: `

encoder = Encoder(cfg)
decoder = Decoder(cfg)
merger = Merger(cfg)

# cfg.CONST.WEIGHTS = '/home/caig/Desktop/pix2vox/Pix2Vox/pretrain/Pix2Vox-A-ShapeNet.pth'
cfg.CONST.WEIGHTS = './pths/UMIFormer.pth'
checkpoint = torch.load(cfg.CONST.WEIGHTS, map_location=torch.device('cpu'))

fix_checkpoint = {}
fix_checkpoint['encoder_state_dict'] = OrderedDict(
    (k.split('module.')[1:][0], v) for k, v in checkpoint['encoder_state_dict'].items())
fix_checkpoint['decoder_state_dict'] = OrderedDict(
    (k.split('module.')[1:][0], v) for k, v in checkpoint['decoder_state_dict'].items())
fix_checkpoint['merger_state_dict'] = OrderedDict(
    (k.split('module.')[1:][0], v) for k, v in checkpoint['merger_state_dict'].items())
# fix_checkpoint['refiner_state_dict'] = OrderedDict(
#     (k.split('module.')[1:][0], v) for k, v in checkpoint['refiner_state_dict'].items())

epoch_idx = checkpoint['epoch_idx']
encoder.load_state_dict(fix_checkpoint['encoder_state_dict'])
decoder.load_state_dict(fix_checkpoint['decoder_state_dict'])
merger.load_state_dict(fix_checkpoint['merger_state_dict'])
# if cfg.NETWORK.USE_REFINER:
#     print('Use refiner')
#     refiner.load_state_dict(fix_checkpoint['refiner_state_dict'])
# if cfg.NETWORK.USE_MERGER:
#     print('Use merger')
#     merger.load_state_dict(fix_checkpoint['merger_state_dict'])

encoder.eval()
decoder.eval()
merger.eval()
# refiner.eval()


# img1_path = './datasets/ShapeNet/ShapeNetRendering/03211117/1a92363c2a155ed3c397356311cbeea4/rendering/17.png'
img1_path = './datasets/Pix3D/img/chair/0016.png'
img1_np = cv2.imread(img1_path, cv2.IMREAD_UNCHANGED).astype(np.float32) / 255.

sample = np.array([img1_np])

IMG_SIZE = cfg.CONST.IMG_H, cfg.CONST.IMG_W
CROP_SIZE = cfg.CONST.CROP_IMG_H, cfg.CONST.CROP_IMG_W

test_transforms = utils.data_transforms.Compose([
    utils.data_transforms.CenterCrop(IMG_SIZE, CROP_SIZE),
    utils.data_transforms.RandomBackground(cfg.TEST.RANDOM_BG_COLOR_RANGE),
    utils.data_transforms.Normalize(mean=cfg.DATASET.MEAN, std=cfg.DATASET.STD),
    utils.data_transforms.ToTensor(),
])

rendering_images = test_transforms(rendering_images=sample)
rendering_images = rendering_images.unsqueeze(0)

with torch.no_grad():
    image_features = encoder(rendering_images)
    raw_features, generated_volume = decoder(image_features)
    # if cfg.NETWORK.USE_MERGER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_MERGER:
    #     generated_volume = merger(raw_features, generated_volume)
    # else:
    #     generated_volume = torch.mean(generated_volume, dim=1)
    # if cfg.NETWORK.USE_REFINER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_REFINER:
    #     generated_volume = refiner(generated_volume)
    generated_volume = merger(raw_features, generated_volume)


generated_volume = generated_volume.squeeze(0)

img_dir = './output/TestResult'
gv = generated_volume.cpu().numpy()
gv_new = np.swapaxes(gv, 2, 1)
rendering_views = utils.binvox_visualization.get_volume_views(gv_new, os.path.join(img_dir),
                                                              epoch_idx)`

I am using this script in the runner.py `

# Start train/test process
if not args.test and not args.batch_test:
    train_net(cfg)
else:
    if 'WEIGHTS' in cfg.CONST and os.path.exists(cfg.CONST.WEIGHTS):
        if args.test:
            test_net(test_single)
        elif args.batch_test:
            batch_test(cfg)
    else:
        logging.error('Please specify the file path of checkpoint.')
        sys.exit(2)

` however, there is still something wrong when I try to generate the voxel model, and the Traceback info

_warnings.warn(
Traceback (most recent call last):
  File "runner.py", line 20, in <module>
    from core.test_single_image import test_single
  File "/root/UMIFormer/core/test_single_image.py", line 16, in <module>
    class test_single():
  File "/root/UMIFormer/core/test_single_image.py", line 18, in test_single
    encoder = Encoder(cfg)
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 21, in __init__
    self.encoder = self.create_model(
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 57, in create_model
    return self._create_vision_transformer(
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 43, in _create_vision_transformer
    model = self.decoupling_type(
  File "/root/UMIFormer/models/encoder/encoder_vit_ivdb.py", line 200, in __init__
    if torch.distributed.get_rank() == 0:
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 822, in get_rank
    default_pg = _get_default_group()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 410, in _get_default_group
    raise RuntimeError(
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 13409) of binary: /root/miniconda3/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
runner.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-13_12:59:56
  host      : autodl-container-d82911a03c-d427ecf0
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 13409)
  error_file: <N/A>_

so, what should I do to fix this? Thank you very much!!! (I'm really sorry for my weak coding skills (༎ຶ ෴ ༎ຶ))

It seems like you made some mistakes about the initialization of Encoder. Please check your code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants