Endless-gif-Fixer-NERF

Authors:

• Yaad
• Almog Hadad

Given a GIF that endlessly repeats, but with detectable end and start points, the Endless-GIF-Fixer-NERF corrects the GIF by adding or removing new camera poses and smoothing the route through the construction of a 3D image using NERF networks and retrieving the necessary poses.

using instant-ngp (NERF) repo (submodule in our project): https://github.com/NVlabs/instant-ngp

projrct steps:

1. split the git into images

2. perform COLMAP and get camera poses for each of the images

3. Process the data output of the COLMAP to the nerf network

4. train and run NERF

5. retrive the missing posses from the last frame to the first one

   • cut overlap poses (if exists)
   • perform interpolation using a fully connected neural network

6. extract the missing poses from the NERF

7. creating fixed gif

result

original:

after our algorithm:

solution ways - gap between the end and start point

try 1 - perform linear interpolation - between last and first point

try 2 - perform Polynomial interpolation

try 3 - interpolation with nueral network:

solution ways - overlap video, needed to cut poses

sulotion 1 - try 1 - widest angle of 2 last points

choosing the point that creates the most obtuse angle (less than 180 degrees) with the last two points, and deleting all the points from the beginning of the gif to it. then, rebuilding by using polynimial interpolation

sulotion 1 - try 2 - better fitted to nerf with widest angle

instead of taling the 2 last point, we find the first point that over lap and we start to cut images. the reason is that in this section, there are more images so the images that created by the network, will be in better shape. It is important to mention that the cut is both the positions at the beginning and at the end.

sulotion 2 - the smallest sum of distances

Start with the first three points and create a circle that will result in the smallest sum of distances between the points and the arc of the circle. At each step, the next point is added and the sum of the distances is updated. At the point where the sum of the distances increases, cut off the end of the video and rebuild it by polynomial interpolation.

Installation

To get started with the Endless-GIF-Fixer-NERF, follow these steps:

Clone the project repository:

git clone https://github.com/AlmogHadad/Endless-gif-Fixer-NERF.git

Initialize and update the submodule:

cd Endless-gif-Fixer-NERF git submodule init git submodule update

Navigate to the instant-ngp directory:

cd instant-ngp

Setup instant-ngp following the instructions provided in the repository at: https://github.com/NVlabs/instant-ngp#building-instant-ngp-windows--linux

Install the required Python dependencies:

pip install -r requirements.txt

With the installation completed, you should now have all the necessary components and dependencies ready to run the Endless-GIF-Fixer-NERF project.

image from nerf (from train set)

guess the fake image:

Additional Notes

The Endless-GIF-Fixer-NERF project utilizes the instant-ngp (NERF) repository as a submodule. Make sure to check the linked repository for more details on NERF implementation.

Please follow the project steps carefully and ensure that all dependencies are correctly installed before running the solution.

Acknowledgments

This project is inspired by the amazing work in the field of neural rendering, and it builds upon the advancements made by the instant-ngp (NERF) repository.

License

This project is licensed under the MIT License. Feel free to use, modify, and distribute it according to the terms of the license.