How to extract RGB frames from Physionv1.5?

[duxiaodan]

Hello authors of Physion dataset,

There isn't a documentation for Physionv1.5(https://physion-benchmark.github.io/physion_v15) data structure yet. Could you please share a code snippet on how to extract mp4 videos or RGB frames from the hdf5 file? Thank you very much

Best,

Xiaodan Du

[rahulvenkk]

Hi Xiaodan, sorry for the delay in response.

Here's some code to extract frames from the hdf5 files:

import h5py as h5
import io
from PIL import Image
import numpy as np

def get_image(raw_img, pil=False):
    '''
    raw_img: binary image to be read by PIL
    returns: HxWx3 image
    '''
    img = Image.open(io.BytesIO(raw_img))

    if pil:
        return img
    
    return np.array(img)

def index_img(h5_file, index, suffix='', pil=False):

    # print(index)
    if index > len(h5_file['frames']) - 1:
        #set index to last frame
        # print("inside if")
        index = len(h5_file['frames']) - 1

    img0 = h5_file['frames'][str(index).zfill(4)]['images']
    
    rgb_img = get_image(img0['_img' + suffix][:], pil=pil)

    return rgb_img


h5file = '<path to hdf5 file>'

with h5.File(h5file) as hf:
    rgb_image = index_img(hf, 10, suffix='_cam0')

We will also be releasing the pipeline for model evaluations on PhysionV1.5 soon. We will notify you when that is ready.

[duxiaodan]

Thank you very much. Looking forward to checking out how to get the binary GT labels from the hdf5 etc.

[rahulvenkk]

Hi Xiaodan, we released our evaluation pipeline for Physion1.5 recently: https://github.com/neuroailab/physion_evaluator/tree/main

We also have a colab notebook that simplifies the evaluation process: https://github.com/neuroailab/cwm_dynamics?tab=readme-ov-file#colab-notebook

[duxiaodan]

Thank you for letting me know about the update! Will check it out.