transforms.json to base_cam.json

[kajc10]

Hi,
I'm quite confused about the different coordinate systems.
Is there a way to convert transforms.json's positions to base_cam.json's format?
(I'd like to render exactly the same frames)

[felixgeen]

Ah I have this question too! This would be a really nice to have feature.

[kajc10]

A little update for anyone who is interested in this question.
When I checked run.py I realized there are some extra arguments.
Now I'm running the script with the following (the last 3 are the important ones).

python scripts/run.py \
--mode nerf \
--scene data/nerf/Dataset01B \
--n_steps 20000 \
--save_mesh data/nerf/Dataset01B/Dataset01B.obj \
--save_snapshot data/nerf/Dataset01B/Dataset01B.msgpack \
--marching_cubes_res 256 \
--screenshot_transforms data/nerf/Dataset01B/eval.json  \
--screenshot_dir data/nerf/Dataset01B/output \
--test_transforms data/nerf/Dataset01B/eval.json

The eval.json contains 1 section from transforms.json.
Current status:

• ref.png and out.png are generated, but not to the given folder. Also if a .json is passed with multiple frames, still only 1 out and 1 ref is generated.
• generating diff.png failed (but I can calculate PSNR myself so it's ok)
  Output:

Training: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 19984/20000 [03:45<00:00, 88.75step/s, loss=0.000235]
Saving snapshot  data/nerf/Dataset01B/Dataset01B.msgpack
Evaluating test transforms from  data/nerf/Dataset01B/eval.json
Rendering test frame:   0%|                                                                                                                                       | 0/1 [00:22<?, ?images/s]
Traceback (most recent call last):
  File "scripts/run.py", line 268, in <module>
    diffimg = np.absolute(image - ref_image)
ValueError: operands could not be broadcast together with shapes (744,1395,4) (744,1395,3)

[Tom94]

See #354

run.py's screenshot functionality does not work with RGB images (as opposed to RGBA). You could either manually add an alpha channel of 1 to your images, or modify the code to automatically add one to the loaded RGB images.

Cheers!

[kajc10]

Yep, this solves it:

ref_image = read_image(ref_fname)
if ref_image.shape[2] == 3:
  alpha = np.full((ref_image.shape[0], ref_image.shape[1]), 1, dtype=np.uint8)
  ref_image = np.dstack((ref_image, alpha))

[0xiNach]

Hope this helps!

import json
import numpy as np
from scipy.spatial.transform import Rotation as R

def nerf_to_ngp(xf):
    mat = np.copy(xf)
    mat = mat[:-1,:]
    mat[:,1] *= -1 # flip axis
    mat[:,2] *= -1
    mat[:,3] *= 0.33 #scale
    mat[:,3] += [0.5, 0.5, 0.5] #offset
    
    mat = mat[[1,2,0], :] # swap axis
    
    rm = R.from_matrix(mat[:,:3]) 
    
    # quaternion (x, y, z, w) and translation
    return rm.as_quat(), mat[:,3] + 0.025

def smooth_camera_path(path_to_transforms, ):
    out = {"path":[], "time":1.0}
    with open(path_to_transforms + '/transforms.json') as f:
        data = json.load(f)
    
    n_frames = len(data['frames'])
    
    xforms = {}
    for i in range(n_frames):
        file = int(data['frames'][i]['file_path'].split('/')[-1][:-4])
        xform = data['frames'][i]['transform_matrix']
        xforms[file] = xform
        
    xforms = dict(sorted(xforms.items()))
    
    # linearly take 12 transformation from transfroms.json
    for ind in np.linspace(1, n_frames, 12, endpoint=True, dtype=int):
        q, t = nerf_to_ngp(np.array(xforms[ind]))
        
        out['path'].append({
            "R": list(q),
            "T": list(t),
            "dof":0.0,
            "fov":43,
            "scale":0,
            "slice":0.0
        })
        
    with open(path_to_transforms+'/base_cam.json', "w") as outfile:
        json.dump(out, outfile, indent=2)
        
smooth_camera_path('/home/ubuntu/instant-ngp/data/nerf/fox')

[PascalHbr]

For some reason this only creates a black video to me. Anyone else experiences this issue or know what might be the reason for this?

[0xiNach]

@PascalHbr Try to adjust scale and fov parameter.

[SummerpanKing]

Dear 0xiNach,
I have a question about the function "nerf_to_ngp", can you please explain the purpose of the #scale and #offset?
My task is generating a "transforms.json" from "transforms_base_cam.json"(from GUI), so i write an inverse function "ngp_to_nerf", but the results show that there exsists a little difference between generated images and images shown in GUI.
So I want to know the algorithm of your function, thanks for your time.

`def ngp_to_nerf(q, t):
# -- From cam to transforms --
rm = R.from_quat(q)
mat_q = np.array(rm.as_matrix())
mat_t = np.array(t - 0.025).reshape((3, 1))

mat = np.hstack((mat_q, mat_t))

mat = mat[[2, 0, 1], :]
mat[:, 3] -= [0.5, 0.5, 0.5]
mat[:, 3] /= 0.33
mat[:, 2] *= -1
mat[:, 1] *= -1

buttom = np.array([0, 0, 0, 1])
xf = np.vstack((mat, buttom))

return xf`

[0xiNach]

@SummerpanKing instant-ngp's NeRF implementation by default only marches rays through a unit bounding box that is from [0, 0, 0] to [1, 1, 1]. The data loader by default takes the camera transforms in the input JSON file, and scales the positions by 0.33 and offsets by [0.5, 0.5, 0.5] in order to map the origin of the input data to the center of this cube. The scale factor is chosen to suit the synthetic datasets in the original NeRF paper, as well as the output of our scripts/colmap2nerf.py script.

[SummerpanKing]

Got it, thank you!!!