Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with Training polygon based model in Alvis #443

Open
Bergylta opened this issue Nov 6, 2024 · 2 comments
Open

Issue with Training polygon based model in Alvis #443

Bergylta opened this issue Nov 6, 2024 · 2 comments
Assignees
@Diewertje11
Labels
bug Something isn't working Support

Comments

@Bergylta
Copy link
Collaborator

Bergylta commented Nov 6, 2024

🐛 Bug

The workaround might have worked, but a new issue showed up in the train_models.ipynb notebook, unsure if this has to do with the baseline model, Alvis, SNIC, mlflow or something else entirely.

To Reproduce (REQUIRED)

Batch size: 8
Epochs: 50
mlp.registry= mlflow
Model: YoloV8 Segmentation
Input:

mlp.train_yolo(
    exp_name=exp_name.value,
    weights=weights.artifact_path,
    project=mlp.project_name,
    epochs=epochs.value,
    batch_size=batch_size.value,
    img_size=img_h.value,  # this requires an int
)

a = torch.tensor([5])
c = a / 0

Output:

Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8m.pt to 'yolov8m.pt'...
100%|██████████| 49.7M/49.7M [00:00<00:00, 71.7MB/s]
New https://pypi.org/project/ultralytics/8.3.27 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.0.200 🚀 Python-3.8.18 torch-2.4.1+cu121 CUDA:0 (NVIDIA A40, 45739MiB)
engine/trainer: task=detect, mode=train, model=yolov8m.pt, data=/mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/Koster_Seafloor_Obs_11_42_08.yaml, epochs=50, patience=50, batch=8, imgsz=640, save=True, save_period=-1, cache=False, device=None, workers=8, project=koster_seafloor_obs, name=KSO_Paulina_oldmodel_2, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, vid_stride=1, stream_buffer=False, line_width=None, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, boxes=True, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, tracker=botsort.yaml, save_dir=koster_seafloor_obs/KSO_Paulina_oldmodel_2
Overriding model.yaml nc=80 with nc=8

                   from  n    params  module                                       arguments                     
  0                  -1  1      1392  ultralytics.nn.modules.conv.Conv             [3, 48, 3, 2]                 
  1                  -1  1     41664  ultralytics.nn.modules.conv.Conv             [48, 96, 3, 2]                
  2                  -1  2    111360  ultralytics.nn.modules.block.C2f             [96, 96, 2, True]             
  3                  -1  1    166272  ultralytics.nn.modules.conv.Conv             [96, 192, 3, 2]               
  4                  -1  4    813312  ultralytics.nn.modules.block.C2f             [192, 192, 4, True]           
  5                  -1  1    664320  ultralytics.nn.modules.conv.Conv             [192, 384, 3, 2]              
  6                  -1  4   3248640  ultralytics.nn.modules.block.C2f             [384, 384, 4, True]           
  7                  -1  1   1991808  ultralytics.nn.modules.conv.Conv             [384, 576, 3, 2]              
  8                  -1  2   3985920  ultralytics.nn.modules.block.C2f             [576, 576, 2, True]           
  9                  -1  1    831168  ultralytics.nn.modules.block.SPPF            [576, 576, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 12                  -1  2   1993728  ultralytics.nn.modules.block.C2f             [960, 384, 2]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 15                  -1  2    517632  ultralytics.nn.modules.block.C2f             [576, 192, 2]                 
 16                  -1  1    332160  ultralytics.nn.modules.conv.Conv             [192, 192, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 18                  -1  2   1846272  ultralytics.nn.modules.block.C2f             [576, 384, 2]                 
 19                  -1  1   1327872  ultralytics.nn.modules.conv.Conv             [384, 384, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 21                  -1  2   4207104  ultralytics.nn.modules.block.C2f             [960, 576, 2]                 
 22        [15, 18, 21]  1   3780328  ultralytics.nn.modules.head.Detect           [8, [192, 384, 576]]          
Model summary: 295 layers, 25860952 parameters, 25860936 gradients, 79.1 GFLOPs

Transferred 469/475 items from pretrained weights
TensorBoard: Start with 'tensorboard --logdir koster_seafloor_obs/KSO_Paulina_oldmodel_2', view at http://localhost:6006/
Freezing layer 'model.22.dfl.conv.weight'
AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n...
/opt/conda/envs/myenv/lib/python3.8/site-packages/ultralytics/nn/tasks.py:567: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(file, map_location='cpu'), file  # load
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs
/opt/conda/envs/myenv/lib/python3.8/site-packages/ultralytics/utils/checks.py:558: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(True):
AMP: checks passed ✅
/opt/conda/envs/myenv/lib/python3.8/site-packages/ultralytics/engine/trainer.py:239: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  self.scaler = amp.GradScaler(enabled=self.amp)
train: Scanning /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/labels.cache... 129 images, 0 backgrounds, 129 corrupt: 100%|██████████| 129/129 [00:00<?, ?it/s]
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/01211c55-4dd6-4453-981a-2bc85bbbbb18.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0400de70-02e4-4f44-9dd1-f0a1c654a328.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/04016f4b-c2e2-465e-accd-8fc2db8f624a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/048bc083-6d15-4105-b83b-e95bf810d3c7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/059fbf23-5a21-4bc5-9456-9f7efc400932.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0637fb53-8ad4-4188-8124-875b3741939a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/06873e3c-1cdf-4468-a420-cd4bc958af6c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0727ebd2-72d9-4727-9a7e-c9b0434ac3a1.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0b6c1fb5-31f7-4fcb-87fb-db7b7631607a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0bdc28b1-aa4d-48ff-9ed2-2f4518c746f6.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0c2ba32b-9bda-43d4-9dce-22183a7060f4.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0e1b964b-a1de-4ebc-8d64-be220aee85d3.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/13a9539a-f53f-4809-a623-5db747b76485.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/15181051-477d-42ee-8901-c27954e799cd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/19516876-d94b-43e2-b12f-23129cdcba5f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/1d0acf63-30ab-4bb1-a603-23dcc48e3003.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/1ea603fe-aab4-4c0b-aea1-1b27623dd653.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/23be80d8-307a-4906-9ca2-ecb1b22f92d8.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/27d2cd3d-daf4-4b74-90cb-a73b84e13404.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/29832828-e270-4f98-9b98-d239147c903d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2adaaadf-11c9-4b49-a741-5612cac31889.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2b0abc51-33f9-4ec2-8970-75576deada25.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2e0b0a6f-7554-4aef-bbdd-0f0b20463fdc.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2fe79d9e-dd94-4f03-9f64-df136a318303.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/30836d9e-4f12-41d7-85d4-458d38323fc5.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/31fc17b2-347e-4681-bc63-5385102905a7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/353143aa-c89d-4ec7-ab43-ae7779ed2785.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/36872e87-858d-4859-9324-277021c44cc0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/39d69402-471c-4953-9531-c9169bd7f4ea.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/3c777de5-617e-4627-bd35-212233828b17.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/3c9b97a4-0125-4c12-bb96-45dd3b6e523b.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/3d303da6-bc75-4805-9265-168862aa6b62.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/41431376-8845-444d-a0bd-b0d0d9c65199.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4a75bf9d-2e5e-4051-a670-35644e0e7d9d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4c4a531d-4a53-4b6b-926f-7d7f20d4d768.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4f250bd2-95a0-4639-a00f-5f1365c0303e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4fdb58f6-765f-43c8-9adc-72526985b9ee.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/520f17c1-8a22-460e-88d5-e583ec19ee55.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/535e679b-c3fa-4802-926c-58c109affef2.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/54f55aed-2954-4a00-aa0e-22d1f30fb336.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/593d8971-5158-4509-8f1f-cb6647ae923c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/5d574ec6-e08c-47e7-b516-f197bf658bee.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6036f363-2d21-42e0-b4d7-3c807bbaf491.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6215867d-654b-491d-8798-01d0d1740077.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6269399d-e6fe-465e-bac7-fbd39868ef9f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/64a5680a-b0cc-4ffb-8017-569e33da441e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6c4a4912-da9f-4c32-9728-b6c515f242d7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/70a2b982-b455-4e55-9d05-d6e0b08ee686.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/71051f70-2e82-407d-95f1-1c00804a797a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/72842bf5-92d2-4c79-8c67-1b5d5c968fe4.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/73a5f098-a742-45cf-98a7-66a62167d2d0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/74eff295-7ae5-4056-b139-ffc9738bef90.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/75edb4dc-d804-4eea-a079-20e1f5718d61.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/7c85f00c-9c10-4aad-82d5-5be915463cb0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/7e1697cd-a490-401f-b7dc-97c807ce7aee.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/800f2b27-8be5-4d0a-aa37-32bf981fa475.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/82efb338-f70a-4df7-a0bb-70050f421c31.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/85ab9efc-51c1-4da5-9520-79eafd5a98f2.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/86fbbf12-0f59-4733-97dd-64c927108319.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/87de2613-bcc5-4a3a-b954-661895eec80d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8829c67d-ebae-4c1b-91f9-449e0862e319.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/885805c3-e559-4003-a3a8-6cb433d21496.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8955a583-4ee6-41a5-a957-fc34f843d6cf.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/89b89a28-f645-451f-93cb-7cd25f372fea.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8ad5146a-ff5b-44d5-96fd-d7903ec10cbf.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8b4d9e1f-1f00-4a95-8025-1f00596b1392.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8baf41df-67f9-439d-b4d3-cb9485eb320a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8c35cec3-9d63-49d6-b869-5f5e3895caa7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/928449a4-5b9a-418f-aac4-89b1cc3d5e40.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/92a5bd3e-30b4-4b41-998a-e56d87c4f94c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/92ca90f3-1c20-4cfd-af82-c616f5971fea.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/9425fac5-f192-4fe4-b4df-3121b413d8a0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/98e208d5-1bf3-4858-8f6b-a8c9169a72a0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/98f7395c-2e78-4471-95e4-52fc803fe003.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/9d1c17db-4a36-43ab-ba55-e4038d8945fd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/9d657191-dc44-40b6-be76-7efad95b0ddd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a1f2df08-61d7-46c0-afc6-f48bb99d5126.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a65c0e61-bfbd-4289-8354-1dc6a140a572.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a6aaef68-83fb-4609-bd73-2ec908eb9dcd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a7788144-0d71-4f86-8c8e-7048f8be018b.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/aacab416-b803-44d0-8767-f7f78effaaec.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ab21420b-b2ae-437e-b61a-85d3b0025639.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/abd006b4-6cfc-4dd3-af76-10e5256d431e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/acbec617-7359-43fc-af3b-53c95d817929.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ad9229f8-0ae8-4400-94b5-249249f47cdb.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b273b6f0-85dc-4587-ae6b-cee1f58b8fcd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b51b4e6d-b38e-4892-a23d-93c3ea93861d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b5255532-3976-4484-8b3a-1e1657dcc80c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b60a7359-e1ad-40fe-b609-3a2ea31a1351.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/bc3ac1dc-124d-4cfa-9e62-1cbe52bddab5.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c422b70a-51c4-4f63-8b29-374bf4434871.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c52d8c64-e7a1-43ae-966b-889d3b41f243.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c6e7d8e4-a018-41f7-9eae-dbb12613099e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c8b82793-5a0c-4fd9-9a9c-23952279f9d7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ca2399b8-946c-4a5c-b706-8f8ae11915d1.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/cb9ef33f-2e14-4105-9982-a9db9e7d6d75.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/cbf3257f-b5b5-40de-bb60-6d13f24ababd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ccc06186-57f3-4380-bf3b-db259ded1402.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ce2926dc-a549-4fdb-ab40-9bca29a1fd8b.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ce3ad9d6-bada-404b-b2a4-df4b2005ea7f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/cea07903-89b3-4f2b-a7b7-047588580c21.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d2d4632e-a0c2-4d5a-90ff-54d936d32e3c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d31e2ebf-122e-4675-9fe0-d56da54a12bc.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d34eaad8-eb92-4371-a922-07455329bfaf.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d5792a2a-00ba-4068-94a4-952c41e12129.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d7f37c06-02cd-4e58-9ccd-896c58806da0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dbe19372-8b54-41f5-b187-ab6ef2bb727e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dc184311-de22-46fb-bdc1-0fc779a1cbbe.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dcf3d6d0-a71c-4211-aa33-edc51aa7ddf6.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dd3cb739-7e74-4191-b199-cc0d6bbe7769.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e0e11c98-f320-44df-881a-58fc64c7a01a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e23ea91e-ea34-454a-86d0-03ca78f99cad.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e431c828-10ce-418e-a548-ae219f5716b5.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e4f92f2b-78f6-443a-bf95-c56c2f926150.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e5435fc8-e49b-47fc-a118-ebac9367b745.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e65150fc-1778-4472-878b-73a8b95ea9f7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e6f33643-0f80-4463-9061-f772d1ef76d7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e8547cdf-9a4f-4f5b-a288-93a633d81f36.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ea0204f2-2683-4bd4-96d8-6629ca55b663.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ea8d038b-642a-4768-b5f2-11e3a14e7524.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/eb0fb5e3-eb39-4bfd-9115-9dd1c780c8ad.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ee1ac174-bf1a-4c73-8461-9d2eed31b040.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/f471dccd-04b3-4174-9f88-3f6be8d16d23.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/f894a137-cc72-486f-b998-8d85065a3433.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fa0e4b5a-1f07-4c4c-93ef-c75724dfbe10.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fac84af3-4e73-449b-a27e-f8b4c2893b94.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fba49ae1-bf11-4836-b28d-2e33c0408c02.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fc1939bd-8361-4bba-8aea-89fd83b03d58.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fdfad66b-81fc-4b3e-9dad-9909082e523f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
WARNING ⚠️ No images found in /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/labels.cache, training may not work correctly. See https://docs.ultralytics.com/datasets/detect for dataset formatting guidance.
INFO:root:Training failed due to: not enough values to unpack (expected 3, got 0)
View run KSO_Paulina_oldmodel_1 at: https://wandb.ai/koster/koster_seafloor_obs/runs/qaowczru
View project at: https://wandb.ai/koster/koster_seafloor_obs
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: /mimer/NOBACKUP/groups/snic2021-6-9/wandb/run-20241101_111940-qaowczru/logs

Additional context

Add any other context about the problem here.
@Bergylta Bergylta added bug Something isn't working Support labels Nov 6, 2024
@Diewertje11
Copy link
Collaborator

Is my following understanding correct?
I think that you are trying to retrain a model with Paulina's data, by taking an already existing baseline model and now retraining it. Is that correct?

From the following message: train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/01211c55-4dd6-4453-981a-2bc85bbbbb18.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
I understand it as that the data from Paulina does not fit the model. It says that the data from Paulina is 'non-normalized or out of bounds'. So I guess (but not sure) that that means that the values for every pixel in the image does not match the model. For example, that the model was trained on data that have pixel values from 1-255 and that Paulina's data goes from 0-1 for example. Or something like that. But I am not sure about this as I am not that familiar with the training process and the specific model. Does this ring any bells and that you know how to trouble shoot?
Or do you want me to further investigate?

@Bergylta
Copy link
Collaborator Author

Bergylta commented Nov 6, 2024 •
edited
Loading

@Diewertje11 In this case, we are taking a baseline model (yolov8 baseline segmentation) and using it to train on paulinas data that she has annotated. I think you are correct in that the pixels might not match, there was something that happened when zooniverse did a update a while back, and that our bounding boxes were not centred above the object, but instead was a bit off, which might be related to this (also unsure). But this is a new issue that i haven't seen before i think, nor do i know how to troubleshoot it sadly.
We can put this at prio 2-3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Diewertje11 Diewertje11

Labels
bug Something isn't working Support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Diewertje11 @Bergylta