Temporal Coherence Workflow

[Trentonom0r3]

While running some tests, I came across a workflow to achieve temporal coherence almost entirely from the SD output. A bit of post processing, and temporal coherence has been achieved on a level almost reaching EBSynth.

Discoveries;
Img2img alt runs faster if you're using images generated by SD.

Img2img alt accurately turns stylized and/or animated frames into realistic images.

Workflow:

1. Run a single Img2Img generation on frame 1 of your video, using controlnet. (I used depth leres++, and canny)

After you have a seed you like, set the seed, and proceed to step 2.

2. Using the same controlnets as in step 1, run a batch generation on the whole video. Enable reference only on CN 3, and add the generated frame 1 you created. Enable loopBack on reference only.

3. Run the batch generation.
   (Optional) using the same process, run the generated images through img2img alt to get a photorealistic version.
   (After this step, the style flicker is minimal, and easily corrected using deflickering and/or temporal smoothing.

4. Using optical flow data and occlusion masks, interpolate extra frames in the generated video to account for choppiness.

5. Perform a final deflicker and temporal smoothing pass, with optional color correction step.

You now have a much more temporally coherent video!

[AugmentedRealityCat]

I must finish my current contract asap so I can test this ! It sounds like you found something really impressive, and I can't wait to try that technique.

[Trentonom0r3]

I'm still figuring out the full workflow, but I found a method that may produce even better results. This is from an early test with a single keyframe from SD;

[AugmentedRealityCat]

Today is the day I'll finally have time to test this ! I'll be back with feedback this evening.

Is this magic ? It looks like magic !

[Trentonom0r3]

Today is the day I'll finally have time to test this ! I'll be back with feedback this evening.

Is this magic ? It looks like magic !

It sure feels like magic!
Clever use of the Content-Aware Fill tool gets you results like above!

I need to figure out how to account for the rest of the inconsistencies, because after that, it'll be perfect!!

[AugmentedRealityCat]

Do you have a very simple example scene with a working controlNet link in it that you can share ?

For some reason I cannot get controlNet to work at all, and without controlNet, there is no magic :(

I wrote an "issue" post about it already.

[Trentonom0r3]

Do you have a very simple example scene with a working controlNet link in it that you can share ?

For some reason I cannot get controlNet to work at all, and without controlNet, there is no magic :(

I wrote an "issue" post about it already.

For the coherence tests, generate a single keyframe,

Create rotoscope around your subject, invert it.

Go to Tracking>Surface>create reference layer

Pre-comp reference layer, and place SD image under the reference frame.

Generate the fill layer.

I'm sure you'll see the inconsistencies when you try it, but the results are promising, it's just a matter of circumventing the limitations!

[AugmentedRealityCat]

Should I apply the AESD effect to a new adjustment layers or to the video layer I want to use as a reference ?
And then should I apply the rotoscope to that video layer, or to the adjustment layer (if one was required) or to the new generated frame ?

[AugmentedRealityCat]

OK, I think I understand better now - I'll give it another try.

[AugmentedRealityCat]

No luck.
The rotoscopy is not a problem. That's covered.

But I cannot get AESD to generate a batch of frames - it only ever generate a single frame, and it's the last frame from the clip I am refering to. This triggers a Javas Script Alert message that says "error occured" telling there should be more details in the console, but there are not any details there at all (I have activated the console with ctrl-f12).

Should I transform my video into a sequence of still pictures first ?

[AugmentedRealityCat]

Good news: I did manage to use AE to "animate" and fill-in the empty roscoped areas using the technique you describe !

The only problem is that I had to do it from a single frame (the last from my video) since I cannot render batches.

Basically, this is very close to having EBSynth, but directly in After Effects, isn't it ?

[Trentonom0r3]

Yes, exactly! If anything, I think the algorithm is a bit more advanced. I think with a combination of tracking/puppet pins, and good reference frames, you can get an amazing output.

[Trentonom0r3]

Details will actually be in the console log file in your temp files--

Check your project folder-- See if all of your frame .png files exist there. If they dont, change the viewer quality or one of the sliders and it should output them.

Make sure you've enabled batch.

Also, I've noticed there is an issue with getting the current frame, where it will somewhat offset the frame it is supposed to grab. You could try moving your timeline indicator to the very beginning, then running. Also, make sure the layer you have selected is the layer that contains the AESD effect-- that could cause the alert as well.

[Trentonom0r3]

can make a short tutorial!

That would be really appreciated. Nothing fancy, but ideally with each and every step. I see I am almost there - I understand what the trick should do but not how to do it. Somehow I have never used the Content Aware fill function in After Effects before, and it's been years since I had to use the rotoscoping tools - it's much easier now, but there are a few things that I am missing, clearly.

Add me on discord!
It is:
.spigon

[AugmentedRealityCat]

Here is my first result with the technique you found and shared ! Nothing groundbreaking content-wise, but technically impressive. This is extremely smooth.

And done from a single keyframe, which is this video's very last frame.

Very similar workflow and result to what you get from EBSynth, but all done directly in AE.

redcarloop_aesd_roto_fill.mp4

Thank you so much Master @Trentonom0r3 !

[Trentonom0r3]

Ran a quick test, wasn't trying to match SD Keyframes too much, and since inpainting is broken, I couldn't use that. However!
With two keyframes (one at the start, and one in the middle) we get this;

2keyframe.test.mp4

[Trentonom0r3]

Here is my first result with the technique you found and shared ! Nothing groundbreaking content-wise, but technically impressive. This is extremely smooth.

And done from a single keyframe, which is this video's very last frame.

Very similar workflow and result to what you get from EBSynth, but all done directly in AE.

redcarloop_aesd_roto_fill.mp4
Thank you so much Master @Trentonom0r3 !

So, I started doing some tests on your footage-- The issue seems to be that the comp start was not set to 0, for some reason. After changing that batch works properly!
I'll look into how I could account for this in the extension

[Trentonom0r3]

And another test-- this time done with 3 keyframes and some simple matching for extra consistency.

3keyframetest.mp4

[Trentonom0r3]

3.frames.inpaint.CN.mp4