Full timeline for playlists

[frankrousseau]

Would reviewers be interested in having a timeline for the entire playlist?

This timeline would allow to:

• Jump directly to a shot
• Show a preview of when the mouse is over
• Show the progress at the full movie level
• Show the status of the playlisted task in the timeline

I attached a screenshot of the timeline below:

Please let me know what you think about it.

[tetsuoanimation]

This reminds me a little of blenders watchtower and I absolutely loved it.

Having to check the 'hookups' between shots faily often, this and being able to scrub seamlessly between shots would 100% be super helpful.

Currently shot transitions are not always smooth - will this pre-encode or cache the whole timeline to smooth it?

[frankrousseau]

By default, it won't manage the transitions. It is simply a different view of the playlist. But if you run a playlist build, you will be able to see the transitions. Fewer options would be available (like comparisons), but you will be able to browse the whole film.

[frankrousseau]

The PR is open but the transition checkingss are tedious to manage while keeping the progress bar properly synced. It seems that FFMPEG adds frames sometimes at transitions.

[tetsuoanimation]

Can you post your testvideos? I am running ffmpeg to stitch videos in another context and I could try looking into that issue.

[tetsuoanimation]

I'm not sure what you tested etc, so please forgive me if I am posting exactly what you already tried here :D

Here's the part of my code generating the ffmpeg commands:

Step 1: Conform videos

I do this for every clip. You'll probably want to set some kind of output codec etc here. Maybe even convert it into an image sequence so you're sure there are no extra frames.

Get fps:

out = subprocess.check_output([ffprobe_bin,str(self.clip_path),"-v","0","-select_streams","v","-print_format","flat","-show_entries","stream=r_frame_rate"], text=True)
rate = out.split('=')[1].strip()[1:-1].split('/')

Get duration:

out = subprocess.check_output([ffprobe_bin,str(self.clip_path),"-v", "error","-select_streams","v","-print_format","flat","-show_entries","stream=duration"],text=True)
duration = float(out.split('=')[1].strip()[1:-1])

You won't need the filter and 'duration' and 'out_fps' are gathered with ffprobe in a previous step

"{ffmpeg_bin} -y -hide_banner -stats -loglevel error "
"-filter_complex {filter_string} "
"-t {duration} "
"-r {out_fps} "
"{output_name}"

Step 2:

Build the 'final' edit by splicing the conformed videos ( or using the image sequence ).
As you know there are two ways - splicing or re-encoding

Re-encoding:

ffmpeg_cmd = (
            "{ffmpeg_bin} -y -hide_banner -stats -loglevel error "
            "{input} "
            "-filter_complex \"{concatfilter}[0];[0]{sequencemaskfilter}\" "
            #"-movflags faststart "
            "{output_name}"
        ).format(
            ffmpeg_bin=ffmpeg_bin, 
            input=inputlist, 
            concatfilter=concatfilter,
            sequencemaskfilter=sequencemask.generateFilterString(),
            output_name=outputpath
            )

Edit:

The concatfilterlist was missing. It is a little weird as it is autogegnerated from a whole range of -i commands in ffmpeg. Here's how I do it:

This generates two strings - one that is basically all input files in a list like: "-i <file_one.mp4> -i <file_two.mp4> -i etc."
The other one uses the input indices to grab the video streams for the in the concat filter: "[0:0][1:0][2:0]...."

inputlist=''
concatfilterlist=''

        for i, clip in enumerate(self.edit):
                inputlist+= '-i "{}" '.format(clip.converted_clip_path)
                concatfilterlist+='[{index}:0] '.format(index=i)

        concatfilter='{concatfilterlist}concat=n={clipnum}:v=1:a=0'.format(concatfilterlist=concatfilterlist, clipnum=len(self.edit))

Splicing:

ffmpeg_cmd = (
            "{ffmpeg_bin} -y -hide_banner -stats -loglevel error "
            "-f concat -safe 0 "
            "-i {input} "
            "-c copy "
            "{output_name}"
        ).format(
            ffmpeg_bin=ffmpeg_bin, 
            input=editfile, 
            output_name=outputpath
            )

For me, this results in no extra frames ( as long as the input-fps for all clips is the same and matching the output-fps)

[frankrousseau]

Thank you a lot. It's the first time, someone agrees to share ffmpeg commands.

For step 1, Kitsu is missing the duration parameter, we'll have a look at it.

The step2 looks like what we already do. But the code on our side is quite complex. I'm not sure we have the same parameters.

Can you explain us how do you generate a mask with ffmpeg?

[tetsuoanimation]

Haha if I'd had known ffmpeg is such a hidden black-magic art I would have thought twice about sharing it ( nah I am kidding ;) )

You might not need the duration parameter - this basically 'cuts' the video after a certain run time. I use it to trim handles or make sure a clip is not longer than it reports.

For the mask, I am using ffmpeg's -filter_complex. I am generating two rectangles for the top and bottom and then place the text on them. I split this into multiple 'options' in my code that get concatenated. It's a bit messy but here's an example from the ffmpeg docs to burn in timecode ( http://trac.ffmpeg.org/wiki/FilteringGuide )

ffmpeg -i in.mp4 -vf "drawtext=fontfile=/usr/share/fonts/truetype/DroidSans.ttf: timecode='09\:57\:00\:00': r=25: \
x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000000@1" -an -y out.mp4 

Here's how I use it in my code:

# generates the filter string for ffmpeg - has two modes, 'clip' for the clip data and 'sequence' for the sequence data
    def generateFilterString(self, mode: str='') -> str:
        if not mode:
            mode=self.mode
        # set up the placement of the logo. we use the second -i input of ffmpeg for this and use an 'overlay' filter to put it on top
        logofilter="[1:v]scale=h={mask_size}-{logopadding}:force_original_aspect_ratio=1".format(mask_size=self.mask_size, logopadding=self.mask_size/6) if self.logo_path else ""
        logooverlay="overlay=x={mask_padding}:y={logopadding}/2".format(mask_padding=self.mask_padding, logopadding=self.mask_size/6) if self.logo_path else ""
        
        # setup the video so it matches what we expect in aspect ratio, size and framelength. 
        # We get this data from our edit description or production management system.
        # conform size and length of the clip to what we defined for it:
        # scale: scale to width and height we expect, keep original aspect ratio
        # pad: artists might playblast with the wrong aspect ratio. Pad it to make sure the video fits the expected size
        # color: extend the video to match the expected frame length with an opaque color
        sizefilter=(
            "scale=w={width}:h={height}:force_original_aspect_ratio=1[0];"
            "[0]pad=width={width}:height={height}:x=-1:y=-1:color=black[0];"
            "color={missing_frame_color}:{width}x{height}:r={fps}[c];"
            "[c][0]overlay=eof_action=pass"
        ).format(width=self.scale[0], height=self.scale[1], logofilter=logofilter, missing_frame_color=self.missing_frame_color, fps=self.fps)
        
        # draw the black rectangle masks we use as the 'background' for our text. we use two 'drawbox' filters here and calculate the size dynamically using the mask-size ( mask size is in pixels here I think )
        drawmaskfilter=(
            "drawbox=x=0:y=0:w=-1:h={mask_size}:color=black@{mask_opacity}:t=fill[0];"
            "[0]drawbox=x=0:y=ih-h:w=-1:h={mask_size}:color=black@{mask_opacity}:t=fill"
        ).format(mask_size=self.mask_size, mask_opacity=self.mask_opacity, logooverlay=logooverlay)

        if mode == "clip":
            # draw all the text for a 'clip'.
            # clips get an overlay with the clip information
            # we use the consola.ttf font available on windows. You can drop the fontfile definiition and just use the systemfonts here. 
            # the text is placed using its size, the mask size and basic percentages
            drawtextfilter=(
                "drawtext=fontsize={fontsize_small}:fontcolor=white:fontfile='C\\:/Windows/fonts/consola.ttf':text='{pass_name}':x=(w-text_w)/2:y=({mask_size}/2)-(text_h/2)[0];"
                "[0]drawtext=fontsize={fontsize_large}:fontcolor=white:fontfile='C\\:/Windows/fonts/consola.ttf':text='{shot_name}':x=w-text_w-{mask_padding}:y=({mask_size}/2)-(text_h/2)[0];"
                "[0]drawtext=fontsize={fontsize_small}:fontcolor=white:fontfile='C\\:/Windows/fonts/consola.ttf':text='{shot_date}':x={mask_padding}:y=h-(text_h/2)-({mask_size}/3)[0];"
                "[0]drawtext=fontsize={fontsize_large}:fontcolor=white:fontfile='C\\:/Windows/fonts/consola.ttf':text='%{{frame_num}}':start_number={start_frame}:x=w-text_w-{mask_padding}:y=h-(text_h/2)-({mask_size}/2)[0];"
                "[0]drawtext=fontsize={fontsize_small}:fontcolor=white:fontfile='C\\:/Windows/fonts/consola.ttf':text={shot_file_name}':x={mask_padding}:y=h-(text_h/2)-(({mask_size}/3)*2)"
            ).format(fontsize_small=self.fontsize_small, fontsize_large=self.fontsize_large, mask_size=self.mask_size, mask_padding=self.mask_padding, start_frame=self.in_frame, pass_name=self.pass_name, shot_name=self.shot_name, shot_date=self.date, shot_file_name=os.path.basename(self.file_name))
        # in 'seqeunce' mode we add only the sequence timecode to the image
        elif mode == "sequence":
            drawtextfilter=(
                "drawtext=fontsize={fontsize_small}:fontcolor=white:fontfile='C\\:/Windows/fonts/consola.ttf':timecode='00\:00\:00\:00':rate={fps}:x=(w-text_w)/2:y=h-(text_h/2)-({mask_size}/2)"
            ).format(fontsize_small=self.fontsize_small, fontsize_large=self.fontsize_large, fps=self.fps, mask_size=self.mask_size, mask_padding=self.mask_padding)
        # if we don't specify a mode, no text is added
        else:
             drawtextfilter=""
        
        # finally, we concatenate all of the filters to generate the final filterstring that is passed to ffmpegs -filter_complex
        if mode == "clip":
            return '"{logofilter}[1];{sizefilter}[0];[0][1]{logooverlay}[0];[0]{drawmaskfilter}[0];[0]{drawtextfilter}"'.format(
                logofilter=logofilter,
                logooverlay=logooverlay,
                sizefilter=sizefilter, 
                drawmaskfilter=drawmaskfilter,
                drawtextfilter=drawtextfilter
                )
        elif mode == "sequence":
            return '"{drawtextfilter}"'.format(drawtextfilter=drawtextfilter)
        elif mode == "resizeonly":
            return '"{sizefilter}"'.format(sizefilter=sizefilter)
        else:
            return False

[tetsuoanimation]

I just noticed that the above depends on some class variables. Here's how the class is defined:

@dataclass
class ShotMask:
    mode: str
    scale: tuple=(1920,1080)
    mask_size: int=60
    mask_padding: int=25
    mask_opacity: float=0.5
    logo_path: str=''
    pass_name: str='defaultpass'
    shot_name: str='defaultshot'
    file_name: str='defaultfilename.mp4' 
    in_frame: int=0
    fps: int=60
    timecode: str='00\:00\:00\:00'
    date: str='yyyy-mm-dd'
    fontsize_large: int=25
    fontsize_small: int=16
    missing_frame_color: str='orange'
    
    # FUNCTIONS

[frankrousseau]

Thank you, I don't know if we will implement it anytime soon but at least we have good reference now!

[frankrousseau]

I forgot to mention here that the development is available but works only for new videos because it needs extra information that was not stored in the previous versions of Kitsu. For the previous videos a command is necessary:

/opt/zou/zouenv/bin/zou generate-preview-extra --with-tiles --with-metadata

The processing can be very long depending on the number of previews for which movie metadata and tiles are computed.

[frankrousseau]

Some feedback from the Normaal studio:

• Make the timeline bigger, it's hard to grab
• Add shortcuts:
  • Home => jump to the beginning of the shot
  • End => jump to the end of the shot
  • Alt + <- => Jump to last frame of previous shot
  • Alt + -> => Jump to the first frame of the next shot

[tetsuoanimation]

Talking about shortcuts here's another idea/request I'd be using super often:

• , => jump to previous drawing
• . => jump to next drawing

( or up and down arrow maybe - but some browsers override that with scrolling the page )