add efficient AV decoding

[jon-barker]

There is still tidying and documentation to be done. I am creating the draft PR to get initial feedback on design and big changes required.

[voegtlel]

Thanks for adding this code! A great addition! A few comments mainly on the integration

[voegtlel]

Should these be constant and not rather be set when fetching the frames in the user code? I think there should be a dynamic option as well.
E.g. Open-Sora group-batches videos into different sized outputs/frames.

[jon-barker]

yep, agreed and i like your suggested design below

[voegtlel]

I'd suggest to move this to a more top-level package. Maybe under megatron.energon.video?

[jon-barker]

ack - will do

[voegtlel]

Maybe we should have an alternative decoder as well, which returns the decoder itself, so the user can decide in user code (=encode_sample) which frames to read?

Like this:

# This function is to be registered as decoder
def read_av_data(key: str, data: bytes):
    if key in ("mp3", ...):
        return AVData(data)


# This class is now passed to the user's `encode_sample` function (i.e. the raw video
# bytes are essentially passed through). This allows the user to decide on the 
# parameters on the fly (e.g. for open-sora).
class AVData:
    def __init__(self, raw: bytes):
        ...

    def get_frames(
            self,
            audio_convert_to_melspec: bool = False,
            audio_clip_duration: int = 1,
            audio_num_clips: int = -1,
            audio_target_rate: int = 16000,
            video_decode_audio: bool = False,
            video_num_frames: int = 64,
            video_out_frame_size: tuple = (224, 224),
        ) -> AudioVideoData:
            ...

WDYT?

[jon-barker]

i like it - happy to make this change

[voegtlel]

I feel, the functions below may have their own file, and also reside in the fastseek package?

[jon-barker]

yes. i'd like to see if i can get rid of the torchaudio dependency here too. i'll revise and relocate this code

[voegtlel]

Generally, we have all parameters statically typed. Also all class variables are typically typed.

[jon-barker]

ack - will fix