Help with precompute_knn

[Kampmarsvin]

precompute_knns.py looks for a specific folder which does not exist. It looks for STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val which is not among the downloaded datasets. I understand that the path is somehow generated on the basis of the config file, but how do I point it to one of the existing datasets instead?

[tanveer6715]

Change your data path train_config.yml file. Then it will generate nns folder by itself.

[Kampmarsvin]

Maybe my question was a bit unclear. I tried changing the path to potsdam, but still get this weird autogenerated path. Here is the error message
output_root: ../
pytorch_data_dir: /home/STEGO/src/datadrive/pytorch-data
experiment_name: exp1
log_dir: potsdam
azureml_logging: true
submitting_to_aml: false
num_workers: 12
max_steps: 5000
batch_size: 16
num_neighbors: 7
dataset_name: potsdam
dir_dataset_name: null
dir_dataset_n_classes: 5
has_labels: false
crop_type: None
crop_ratio: 0.5
res: 224
loader_crop_type: center
extra_clusters: 0
use_true_labels: false
use_recalibrator: false
model_type: vit_small
arch: dino
use_fit_model: false
dino_feat_type: feat
projection_type: nonlinear
dino_patch_size: 8
granularity: 1
continuous: true
dim: 70
dropout: true
zero_clamp: true
lr: 0.0005
pretrained_weights: null
use_salience: false
stabalize: false
stop_at_zero: true
pointwise: true
feature_samples: 11
neg_samples: 5
aug_alignment_weight: 0.0
correspondence_weight: 1.0
neg_inter_weight: 0.63
pos_inter_weight: 0.25
pos_intra_weight: 0.67
neg_inter_shift: 0.46
pos_inter_shift: 0.12
pos_intra_shift: 0.18
rec_weight: 0.0
repulsion_weight: 0.0
crf_weight: 0.0
alpha: 0.5
beta: 0.15
gamma: 0.05
w1: 10.0
w2: 3.0
shift: 0.0
crf_samples: 1000
color_space: rgb
reset_probe_steps: null
n_images: 5
scalar_log_freq: 10
checkpoint_freq: 50
val_freq: 100
hist_freq: 100

Global seed set to 0
../data
../
Since no pretrained weights have been provided, we load the reference pretrained DINO weights.
/home/STEGO/src/datadrive/pytorch-data/nns/nns_vit_small_cocostuff27_val_five_224.npz not found, computing
Error executing job with overrides: []
Traceback (most recent call last):
File "precompute_knns.py", line 78, in my_app
cfg=cfg,
File "/home/STEGO/src/data.py", line 495, in init
target_transform=target_transform, **extra_args)
File "/home/STEGO/src/data.py", line 380, in init
self.num_images = len(os.listdir(self.img_dir))
FileNotFoundError: [Errno 2] No such file or directory: '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'

[tanveer6715]

pytorch_data_dir

You should change your data directory here to the potsdam data where it exists in your system.

[Kampmarsvin]

The path '/home/STEGO/src/datadrive/pytorch-data' which I have provided in PyTorch_data_dir is the path to the potsdam directory. Yet precompute_knns.py apparently looks for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. I cannot figure out, why that path is being generated. Where in the config file is anything indicating that I want to look for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'.
Can you @tanveer6715 show me an example of a config file, which points to potsdam?

[tanveer6715]

The path '/home/STEGO/src/datadrive/pytorch-data' which I have provided in PyTorch_data_dir is the path to the potsdam directory. Yet precompute_knns.py apparently looks for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. I cannot figure out, why that path is being generated. Where in the config file is anything indicating that I want to look for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. Can you @tanveer6715 show me an example of a config file, which points to potsdam?

Actually I just trained it with custom dataset not with Potsdam. I simply change data directory to my dataset path. Then it generate precompute_knns successfully. I am not sure why it looks for that path in potsdam dataset.

[Kampmarsvin]

Ok. Thank you. Maybe I'll try testing with a custom set

[Guzaiwang]

The path '/home/STEGO/src/datadrive/pytorch-data' which I have provided in PyTorch_data_dir is the path to the potsdam directory. Yet precompute_knns.py apparently looks for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. I cannot figure out, why that path is being generated. Where in the config file is anything indicating that I want to look for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. Can you @tanveer6715 show me an example of a config file, which points to potsdam?

You may first run the following code:
python crop_datasets.py

[Kampmarsvin]

@Guzaiwang but crop_dataset.py is for training on custom dataset right? I am simply trying to follow the explicit steps in the readme file, and it says to call precompute_knns.py when using pretrained model. So I have the models downloaded, and I have the datasets. But precompute_knns doesn't work because it looks for a different folder than the one specified in the config file. How do I change the path, so that it finds the path to the potsdam dataset?

[KennyChen880127]

@Kampmarsvin Hellos sir, Do you fix this issue? I have same problem now,it drives me crazy.

[Kampmarsvin]

@Kampmarsvin Hellos sir, Do you fix this issue? I have same problem now,it drives me crazy.

Unfortunately not. I haven't had the time to do it. But I found out that there are som folders that need to be unzipped, so I got a little closer

[innat-asj]

precompute_knns.py looks for a specific folder which does not exist. It looks for STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val which is not among the downloaded datasets. I understand that the path is somehow generated on the basis of the config file, but how do I point it to one of the existing datasets instead?

In the precompute_knn.py file, this supported data are hard coded. If you want to test one particular data among this (let's say potsdam), you need to update as follows:

# dataset_names = ["cocostuff27", "cityscapes", "potsdam"]
dataset_names = ["potsdam"]

Then run (instruction)

python precompute_knns.py