Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#IndexError: list index out of range when running "python -m scripts.preprocess_libero --suite libero_spatial" #7

Open
Khlann opened this issue Jul 26, 2024 · 8 comments · May be fixed by #9
Open

#IndexError: list index out of range when running "python -m scripts.preprocess_libero --suite libero_spatial" #7

Khlann opened this issue Jul 26, 2024 · 8 comments · May be fixed by #9

Comments

@Khlann
Copy link

Khlann commented Jul 26, 2024

Dear author:
I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range
@Robot-K
Copy link

Robot-K commented Jul 26, 2024

same issue

@ahadjawaid
Copy link

You have to run the preprocessing step where they split the dataset into validation and training set. You can do this by doing python3 scripts/split_libero_dataset.py --folder data/atm_libero --train_ratio 0.9.

@hanbit-o
Copy link

#7 (comment)
this didn't work to me.

@AlvinWen428
Copy link
Collaborator

Hi, I guess this problem is because the downloaded libero dataset is not correctly placed into the folder ATM/data/. Would you please check the folder structure of ATM/data/? It should be like this:

data

libero_spatial

pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate_demo.hdf5
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate_demo.hdf5
...

libero_object
libero_goal
libero_10
libero_90

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

@lib36157
Copy link

Hi, I guess this problem is because the downloaded libero dataset is not correctly placed into the folder ATM/data/. Would you please check the folder structure of ATM/data/? It should be like this:

data

libero_spatial

pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate_demo.hdf5
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate_demo.hdf5
...

libero_object
libero_goal
libero_10
libero_90

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

same error. I don't think it's the problem of data structure. Should be some other reasons.

@lib36157
Copy link

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

just put all these data files into a folder named libero. The directory should be ATM/data/libero/data files. That works for me.

@Robot-K
Copy link

Robot-K commented Aug 17, 2024

Solved it, there's a .DS_Store file under the location, use
for source_h5 in os.listdir(suite_dir):
source_h5_path = os.path.join(suite_dir, source_h5)
if 'hdf5' not in source_h5: # to pass files like DS_store
continue

@Jendker Jendker linked a pull request Aug 20, 2024 that will close this issue
Open
@RenMing-Huang
Copy link

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

just put all these data files into a folder named libero. The directory should be ATM/data/libero/data files. That works for me.

thanks, it works for me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix to accomodate for preprocess_libero.py path Jendker/ATM
7 participants
@AlvinWen428 @RenMing-Huang @ahadjawaid @Robot-K @Khlann @lib36157 @hanbit-o