How to use MMSelfSup to train a model on my own dataset?

[tonysy]

Hi, how to use MMSelfSup to train a model on my own dataset?

[tonysy]

There is an example for COCO dataset. #511

[tonysy]

We provide an example of MAE on my own dataset, which have no labels.

Please use the 1.x branch of MMSelfSup.

The documentation website is: https://mmselfsup.readthedocs.io/en/1.x/
Please follow this documentaion to install the latest version of MMSelfsup

_base_ = [
    '../_base_/models/mae_vit-base-p16.py',
    # '../_base_/datasets/imagenet_mae.py',  # removed
    '../_base_/schedules/adamw_coslr-200e_in1k.py',
    '../_base_/default_runtime.py',
]

#<<<<<<<  modified <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# custom dataset
dataset_type = 'mmcls.CustomDataset'

data_root = 'data/sbu/'
file_client_args = dict(backend='disk')
train_pipeline = [
    dict(type='LoadImageFromFile', file_client_args=file_client_args),
    dict(
        type='RandomResizedCrop',
        size=224,
        scale=(0.2, 1.0),
        backend='pillow',
        interpolation='bicubic'),
    dict(type='RandomFlip', prob=0.5),
    dict(type='PackSelfSupInputs', meta_keys=['img_path'])
]

# dataset 8 x 512
train_dataloader = dict(
    batch_size=512,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    collate_fn=dict(type='default_collate'),
    dataset=dict(
        type=dataset_type,
        data_root=data_root,
        # ann_file='meta/train.txt', # removed if you don't have the annotation file
        data_prefix=dict(img_path='./'),
        pipeline=train_pipeline))

#>>>>>>>>>> modified >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

# optimizer wrapper
optimizer = dict(
    type='AdamW', lr=1.5e-4 * 4096 / 256, betas=(0.9, 0.95), weight_decay=0.05)
optim_wrapper = dict(
    type='OptimWrapper',
    optimizer=optimizer,
    paramwise_cfg=dict(
        custom_keys={
            'ln': dict(decay_mult=0.0),
            'bias': dict(decay_mult=0.0),
            'pos_embed': dict(decay_mult=0.),
            'mask_token': dict(decay_mult=0.),
            'cls_token': dict(decay_mult=0.)
        }))

# learning rate scheduler
param_scheduler = [
    dict(
        type='LinearLR',
        start_factor=1e-4,
        by_epoch=True,
        begin=0,
        end=40,
        convert_to_iter_based=True),
    dict(
        type='CosineAnnealingLR',
        T_max=360,
        by_epoch=True,
        begin=40,
        end=400,
        convert_to_iter_based=True)
]

# runtime settings
# pre-train for 400 epochs
train_cfg = dict(max_epochs=400)
default_hooks = dict(
    logger=dict(type='LoggerHook', interval=100),
    # only keeps the latest 3 checkpoints
    checkpoint=dict(type='CheckpointHook', interval=1, max_keep_ckpts=3))

# randomness
randomness = dict(seed=0, diff_rank_seed=True)
resume = True

Feel free to ask question for using custom dataset.

[kshitijd20]

I am using the above config, but I would like to initialize from weights in model zoo. So I added following line in config
load_from = CHECKPOINT_PATH

However, I am getting following warnings and it seems that state dicts are different due to additional prefix backbone . Where can I add or remove this prefix to load the weights correctly?

The model and loaded state dict do not match exactly

unexpected key in source state_dict: cls_token, pos_embed, patch_embed.projection.weight, patch_embed.projection.bias, layers.0.ln1.weight, layers.0.ln1.bias, layers.0.attn.qkv.weight, layers.0.attn.qkv.bias, layers.0.attn.proj.weight, layers.0.attn.proj.bias, layers.0.ln2.weight, layers.0.ln2.bias, layers.0.ffn.layers.0.0.weight, layers.0.ffn.layers.0.0.bias, layers.0.ffn.layers.1.weight, layers.0.ffn.layers.1.bias, layers.1.ln1.weight, layers.1.ln1.bias, layers.1.attn.qkv.weight, layers.1.attn.qkv.bias, layers.1.attn.proj.weight, layers.1.attn.proj.bias, layers.1.ln2.weight, layers.1.ln2.bias, layers.1.ffn.layers.0.0.weight, layers.1.ffn.layers.0.0.bias, layers.1.ffn.layers.1.weight, layers.1.ffn.layers.1.bias, layers.2.ln1.weight, layers.2.ln1.bias, layers.2.attn.qkv.weight, layers.2.attn.qkv.bias, layers.2.attn.proj.weight, layers.2.attn.proj.bias, layers.2.ln2.weight, layers.2.ln2.bias, layers.2.ffn.layers.0.0.weight, layers.2.ffn.layers.0.0.bias, layers.2.ffn.layers.1.weight, layers.2.ffn.layers.1.bias, layers.3.ln1.weight, layers.3.ln1.bias, layers.3.attn.qkv.weight, layers.3.attn.qkv.bias, layers.3.attn.proj.weight, layers.3.attn.proj.bias, layers.3.ln2.weight, layers.3.ln2.bias, layers.3.ffn.layers.0.0.weight, layers.3.ffn.layers.0.0.bias, layers.3.ffn.layers.1.weight, layers.3.ffn.layers.1.bias, layers.4.ln1.weight, layers.4.ln1.bias, layers.4.attn.qkv.weight, layers.4.attn.qkv.bias, layers.4.attn.proj.weight, layers.4.attn.proj.bias, layers.4.ln2.weight, layers.4.ln2.bias, layers.4.ffn.layers.0.0.weight, layers.4.ffn.layers.0.0.bias, layers.4.ffn.layers.1.weight, layers.4.ffn.layers.1.bias, layers.5.ln1.weight, layers.5.ln1.bias, layers.5.attn.qkv.weight, layers.5.attn.qkv.bias, layers.5.attn.proj.weight, layers.5.attn.proj.bias, layers.5.ln2.weight, layers.5.ln2.bias, layers.5.ffn.layers.0.0.weight, layers.5.ffn.layers.0.0.bias, layers.5.ffn.layers.1.weight, layers.5.ffn.layers.1.bias, layers.6.ln1.weight, layers.6.ln1.bias, layers.6.attn.qkv.weight, layers.6.attn.qkv.bias, layers.6.attn.proj.weight, layers.6.attn.proj.bias, layers.6.ln2.weight, layers.6.ln2.bias, layers.6.ffn.layers.0.0.weight, layers.6.ffn.layers.0.0.bias, layers.6.ffn.layers.1.weight, layers.6.ffn.layers.1.bias, layers.7.ln1.weight, layers.7.ln1.bias, layers.7.attn.qkv.weight, layers.7.attn.qkv.bias, layers.7.attn.proj.weight, layers.7.attn.proj.bias, layers.7.ln2.weight, layers.7.ln2.bias, layers.7.ffn.layers.0.0.weight, layers.7.ffn.layers.0.0.bias, layers.7.ffn.layers.1.weight, layers.7.ffn.layers.1.bias, layers.8.ln1.weight, layers.8.ln1.bias, layers.8.attn.qkv.weight, layers.8.attn.qkv.bias, layers.8.attn.proj.weight, layers.8.attn.proj.bias, layers.8.ln2.weight, layers.8.ln2.bias, layers.8.ffn.layers.0.0.weight, layers.8.ffn.layers.0.0.bias, layers.8.ffn.layers.1.weight, layers.8.ffn.layers.1.bias, layers.9.ln1.weight, layers.9.ln1.bias, layers.9.attn.qkv.weight, layers.9.attn.qkv.bias, layers.9.attn.proj.weight, layers.9.attn.proj.bias, layers.9.ln2.weight, layers.9.ln2.bias, layers.9.ffn.layers.0.0.weight, layers.9.ffn.layers.0.0.bias, layers.9.ffn.layers.1.weight, layers.9.ffn.layers.1.bias, layers.10.ln1.weight, layers.10.ln1.bias, layers.10.attn.qkv.weight, layers.10.attn.qkv.bias, layers.10.attn.proj.weight, layers.10.attn.proj.bias, layers.10.ln2.weight, layers.10.ln2.bias, layers.10.ffn.layers.0.0.weight, layers.10.ffn.layers.0.0.bias, layers.10.ffn.layers.1.weight, layers.10.ffn.layers.1.bias, layers.11.ln1.weight, layers.11.ln1.bias, layers.11.attn.qkv.weight, layers.11.attn.qkv.bias, layers.11.attn.proj.weight, layers.11.attn.proj.bias, layers.11.ln2.weight, layers.11.ln2.bias, layers.11.ffn.layers.0.0.weight, layers.11.ffn.layers.0.0.bias, layers.11.ffn.layers.1.weight, layers.11.ffn.layers.1.bias, ln1.weight, ln1.bias

missing keys in source state_dict: backbone.cls_token, backbone.pos_embed, backbone.patch_embed.projection.weight, backbone.patch_embed.projection.bias, backbone.layers.0.ln1.weight, backbone.layers.0.ln1.bias, backbone.layers.0.attn.qkv.weight, backbone.layers.0.attn.qkv.bias, backbone.layers.0.attn.proj.weight, backbone.layers.0.attn.proj.bias, backbone.layers.0.ln2.weight, backbone.layers.0.ln2.bias, backbone.layers.0.ffn.layers.0.0.weight, backbone.layers.0.ffn.layers.0.0.bias, backbone.layers.0.ffn.layers.1.weight, backbone.layers.0.ffn.layers.1.bias, backbone.layers.1.ln1.weight, backbone.layers.1.ln1.bias, backbone.layers.1.attn.qkv.weight, backbone.layers.1.attn.qkv.bias, backbone.layers.1.attn.proj.weight, backbone.layers.1.attn.proj.bias, backbone.layers.1.ln2.weight, backbone.layers.1.ln2.bias, backbone.layers.1.ffn.layers.0.0.weight, backbone.layers.1.ffn.layers.0.0.bias, backbone.layers.1.ffn.layers.1.weight, backbone.layers.1.ffn.layers.1.bias, backbone.layers.2.ln1.weight, backbone.layers.2.ln1.bias, backbone.layers.2.attn.qkv.weight, backbone.layers.2.attn.qkv.bias, backbone.layers.2.attn.proj.weight, backbone.layers.2.attn.proj.bias, backbone.layers.2.ln2.weight, backbone.layers.2.ln2.bias, backbone.layers.2.ffn.layers.0.0.weight, backbone.layers.2.ffn.layers.0.0.bias, backbone.layers.2.ffn.layers.1.weight, backbone.layers.2.ffn.layers.1.bias, backbone.layers.3.ln1.weight, backbone.layers.3.ln1.bias, backbone.layers.3.attn.qkv.weight, backbone.layers.3.attn.qkv.bias, backbone.layers.3.attn.proj.weight, backbone.layers.3.attn.proj.bias, backbone.layers.3.ln2.weight, backbone.layers.3.ln2.bias, backbone.layers.3.ffn.layers.0.0.weight, backbone.layers.3.ffn.layers.0.0.bias, backbone.layers.3.ffn.layers.1.weight, backbone.layers.3.ffn.layers.1.bias, backbone.layers.4.ln1.weight, backbone.layers.4.ln1.bias, backbone.layers.4.attn.qkv.weight, backbone.layers.4.attn.qkv.bias, backbone.layers.4.attn.proj.weight, backbone.layers.4.attn.proj.bias, backbone.layers.4.ln2.weight, backbone.layers.4.ln2.bias, backbone.layers.4.ffn.layers.0.0.weight, backbone.layers.4.ffn.layers.0.0.bias, backbone.layers.4.ffn.layers.1.weight, backbone.layers.4.ffn.layers.1.bias, backbone.layers.5.ln1.weight, backbone.layers.5.ln1.bias, backbone.layers.5.attn.qkv.weight, backbone.layers.5.attn.qkv.bias, backbone.layers.5.attn.proj.weight, backbone.layers.5.attn.proj.bias, backbone.layers.5.ln2.weight, backbone.layers.5.ln2.bias, backbone.layers.5.ffn.layers.0.0.weight, backbone.layers.5.ffn.layers.0.0.bias, backbone.layers.5.ffn.layers.1.weight, backbone.layers.5.ffn.layers.1.bias, backbone.layers.6.ln1.weight, backbone.layers.6.ln1.bias, backbone.layers.6.attn.qkv.weight, backbone.layers.6.attn.qkv.bias, backbone.layers.6.attn.proj.weight, backbone.layers.6.attn.proj.bias, backbone.layers.6.ln2.weight, backbone.layers.6.ln2.bias, backbone.layers.6.ffn.layers.0.0.weight, backbone.layers.6.ffn.layers.0.0.bias, backbone.layers.6.ffn.layers.1.weight, backbone.layers.6.ffn.layers.1.bias, backbone.layers.7.ln1.weight, backbone.layers.7.ln1.bias, backbone.layers.7.attn.qkv.weight, backbone.layers.7.attn.qkv.bias, backbone.layers.7.attn.proj.weight, backbone.layers.7.attn.proj.bias, backbone.layers.7.ln2.weight, backbone.layers.7.ln2.bias, backbone.layers.7.ffn.layers.0.0.weight, backbone.layers.7.ffn.layers.0.0.bias, backbone.layers.7.ffn.layers.1.weight, backbone.layers.7.ffn.layers.1.bias, backbone.layers.8.ln1.weight, backbone.layers.8.ln1.bias, backbone.layers.8.attn.qkv.weight, backbone.layers.8.attn.qkv.bias, backbone.layers.8.attn.proj.weight, backbone.layers.8.attn.proj.bias, backbone.layers.8.ln2.weight, backbone.layers.8.ln2.bias, backbone.layers.8.ffn.layers.0.0.weight, backbone.layers.8.ffn.layers.0.0.bias, backbone.layers.8.ffn.layers.1.weight, backbone.layers.8.ffn.layers.1.bias, backbone.layers.9.ln1.weight, backbone.layers.9.ln1.bias, backbone.layers.9.attn.qkv.weight, backbone.layers.9.attn.qkv.bias, backbone.layers.9.attn.proj.weight, backbone.layers.9.attn.proj.bias, backbone.layers.9.ln2.weight, backbone.layers.9.ln2.bias, backbone.layers.9.ffn.layers.0.0.weight, backbone.layers.9.ffn.layers.0.0.bias, backbone.layers.9.ffn.layers.1.weight, backbone.layers.9.ffn.layers.1.bias, backbone.layers.10.ln1.weight, backbone.layers.10.ln1.bias, backbone.layers.10.attn.qkv.weight, backbone.layers.10.attn.qkv.bias, backbone.layers.10.attn.proj.weight, backbone.layers.10.attn.proj.bias, backbone.layers.10.ln2.weight, backbone.layers.10.ln2.bias, backbone.layers.10.ffn.layers.0.0.weight, backbone.layers.10.ffn.layers.0.0.bias, backbone.layers.10.ffn.layers.1.weight, backbone.layers.10.ffn.layers.1.bias, backbone.layers.11.ln1.weight, backbone.layers.11.ln1.bias, backbone.layers.11.attn.qkv.weight, backbone.layers.11.attn.qkv.bias, backbone.layers.11.attn.proj.weight, backbone.layers.11.attn.proj.bias, backbone.layers.11.ln2.weight, backbone.layers.11.ln2.bias, backbone.layers.11.ffn.layers.0.0.weight, backbone.layers.11.ffn.layers.0.0.bias, backbone.layers.11.ffn.layers.1.weight, backbone.layers.11.ffn.layers.1.bias, backbone.ln1.weight, backbone.ln1.bias, neck.mask_token, neck.decoder_pos_embed, neck.decoder_embed.weight, neck.decoder_embed.bias, neck.decoder_blocks.0.ln1.weight, neck.decoder_blocks.0.ln1.bias, neck.decoder_blocks.0.attn.qkv.weight, neck.decoder_blocks.0.attn.qkv.bias, neck.decoder_blocks.0.attn.proj.weight, neck.decoder_blocks.0.attn.proj.bias, neck.decoder_blocks.0.ln2.weight, neck.decoder_blocks.0.ln2.bias, neck.decoder_blocks.0.ffn.layers.0.0.weight, neck.decoder_blocks.0.ffn.layers.0.0.bias, neck.decoder_blocks.0.ffn.layers.1.weight, neck.decoder_blocks.0.ffn.layers.1.bias, neck.decoder_blocks.1.ln1.weight, neck.decoder_blocks.1.ln1.bias, neck.decoder_blocks.1.attn.qkv.weight, neck.decoder_blocks.1.attn.qkv.bias, neck.decoder_blocks.1.attn.proj.weight, neck.decoder_blocks.1.attn.proj.bias, neck.decoder_blocks.1.ln2.weight, neck.decoder_blocks.1.ln2.bias, neck.decoder_blocks.1.ffn.layers.0.0.weight, neck.decoder_blocks.1.ffn.layers.0.0.bias, neck.decoder_blocks.1.ffn.layers.1.weight, neck.decoder_blocks.1.ffn.layers.1.bias, neck.decoder_blocks.2.ln1.weight, neck.decoder_blocks.2.ln1.bias, neck.decoder_blocks.2.attn.qkv.weight, neck.decoder_blocks.2.attn.qkv.bias, neck.decoder_blocks.2.attn.proj.weight, neck.decoder_blocks.2.attn.proj.bias, neck.decoder_blocks.2.ln2.weight, neck.decoder_blocks.2.ln2.bias, neck.decoder_blocks.2.ffn.layers.0.0.weight, neck.decoder_blocks.2.ffn.layers.0.0.bias, neck.decoder_blocks.2.ffn.layers.1.weight, neck.decoder_blocks.2.ffn.layers.1.bias, neck.decoder_blocks.3.ln1.weight, neck.decoder_blocks.3.ln1.bias, neck.decoder_blocks.3.attn.qkv.weight, neck.decoder_blocks.3.attn.qkv.bias, neck.decoder_blocks.3.attn.proj.weight, neck.decoder_blocks.3.attn.proj.bias, neck.decoder_blocks.3.ln2.weight, neck.decoder_blocks.3.ln2.bias, neck.decoder_blocks.3.ffn.layers.0.0.weight, neck.decoder_blocks.3.ffn.layers.0.0.bias, neck.decoder_blocks.3.ffn.layers.1.weight, neck.decoder_blocks.3.ffn.layers.1.bias, neck.decoder_blocks.4.ln1.weight, neck.decoder_blocks.4.ln1.bias, neck.decoder_blocks.4.attn.qkv.weight, neck.decoder_blocks.4.attn.qkv.bias, neck.decoder_blocks.4.attn.proj.weight, neck.decoder_blocks.4.attn.proj.bias, neck.decoder_blocks.4.ln2.weight, neck.decoder_blocks.4.ln2.bias, neck.decoder_blocks.4.ffn.layers.0.0.weight, neck.decoder_blocks.4.ffn.layers.0.0.bias, neck.decoder_blocks.4.ffn.layers.1.weight, neck.decoder_blocks.4.ffn.layers.1.bias, neck.decoder_blocks.5.ln1.weight, neck.decoder_blocks.5.ln1.bias, neck.decoder_blocks.5.attn.qkv.weight, neck.decoder_blocks.5.attn.qkv.bias, neck.decoder_blocks.5.attn.proj.weight, neck.decoder_blocks.5.attn.proj.bias, neck.decoder_blocks.5.ln2.weight, neck.decoder_blocks.5.ln2.bias, neck.decoder_blocks.5.ffn.layers.0.0.weight, neck.decoder_blocks.5.ffn.layers.0.0.bias, neck.decoder_blocks.5.ffn.layers.1.weight, neck.decoder_blocks.5.ffn.layers.1.bias, neck.decoder_blocks.6.ln1.weight, neck.decoder_blocks.6.ln1.bias, neck.decoder_blocks.6.attn.qkv.weight, neck.decoder_blocks.6.attn.qkv.bias, neck.decoder_blocks.6.attn.proj.weight, neck.decoder_blocks.6.attn.proj.bias, neck.decoder_blocks.6.ln2.weight, neck.decoder_blocks.6.ln2.bias, neck.decoder_blocks.6.ffn.layers.0.0.weight, neck.decoder_blocks.6.ffn.layers.0.0.bias, neck.decoder_blocks.6.ffn.layers.1.weight, neck.decoder_blocks.6.ffn.layers.1.bias, neck.decoder_blocks.7.ln1.weight, neck.decoder_blocks.7.ln1.bias, neck.decoder_blocks.7.attn.qkv.weight, neck.decoder_blocks.7.attn.qkv.bias, neck.decoder_blocks.7.attn.proj.weight, neck.decoder_blocks.7.attn.proj.bias, neck.decoder_blocks.7.ln2.weight, neck.decoder_blocks.7.ln2.bias, neck.decoder_blocks.7.ffn.layers.0.0.weight, neck.decoder_blocks.7.ffn.layers.0.0.bias, neck.decoder_blocks.7.ffn.layers.1.weight, neck.decoder_blocks.7.ffn.layers.1.bias, neck.ln1.weight, neck.ln1.bias, neck.decoder_pred.weight, neck.decoder_pred.bias

[fangyixiao18]

I used python mmselfsup/tools/train.py config_folder/my_config.py

It seems that your mmcv installation might have some problem. How did you install your mmcv?

[thangld201]

When installing mmcv, there is this line that popped up /home/ubuntu/.conda/envs/user/lib/python3.8/site-packages/torch/utils/cpp_extension.py:782: UserWarning: The detected CUDA version (11.0) has a minor version mismatch with the version that was used to compile PyTorch (11.3). Most likely this shouldn't be a problem. warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
Other than that, I just followed the instruction:

git clone -b 2.x https://github.com/open-mmlab/mmcv.git
cd mmcv
pip install -v -e .

Did I do something wrong with this ? @fangyixiao18

[fangyixiao18]

It might have some compiling problem during installation.

Please try to install mmcv again with the command below:

It is much more convenient.

[thangld201]

Thanks a lot, I was able to resolve the issue. By the way, if I want to use other transformations (mmcls, torchvision, etc) in mmselfsup, do I have to re-implement them myself in transformations.py, or are they supported out of the box ? (I know that we can use mmcv's without further implementation) @fangyixiao18

[fangyixiao18]

Thanks a lot, I was able to resolve the issue. By the way, if I want to use other transformations (mmcls, torchvision, etc) in mmselfsup, do I have to re-implement them myself in transformations.py, or are they supported out of the box ? (I know that we can use mmcv's without further implementation) @fangyixiao18

If you want to use transform from mmcls, you can write mmcls.xxxx in config, like below

pipeline = [
    dict(type='LoadImageFromFile', file_client_args=file_client_args),
    dict(type='mmcls.ResizeEdge', scale=256, edge='short', backend='pillow'),
    dict(type='mmcls.CenterCrop', crop_size=224),
    dict(type='PackSelfSupInputs'),
]