How can obtain .r3d data format by recording from USB live streaming (iPad)?

[jeonhuhuhu]

We extracted .r3d data format using the existing Record3d app.

However, we have now succeeded in streaming in real-time by connecting the iPad (USB) to Linux,

but we do not know how to record and save it in .r3d data format like the existing Record3d app.

I would appreciate it if you could tell me how to save r3d data like the app provided by existing Apple apps.

[marek-simonik]

I think it would be best if you would "dissect" a .r3d file produced by Record3D, see what is inside, and then create your own .r3d files according to that.

The .r3d files are just ZIP files with .r3d extension instead of .zip extension. So I recommend that you export a 3D video into .r3d in Record3D, copy the .r3d file to your computer, unzip the .r3d file and see what is inside.

The contents of a typical (unzipped) .r3d file:

• rgbd folder: contains files in the format %d.jpg, %d.depth and %d.conf per each frame of the video, where %d is frame number, starting from 0
• icon file: a JPG thumbnail of the 3D video
• metadata file: a JSON text file containing metadata
• sound.m4a file: contains audio file; this file is optional, you can remove it if you do not want the 3D video to contain audio track

A single .depth file contains the raw float32 bytes (stored in row-major convention), compressed with lzfse. See this comment: #7 (comment)

The .conf files contain raw uint8 bytes (stored in row-major convention), compressed with lzfse.

You can compress the raw float32 depthmap bytes using lzfse, and then write the compressed bytes into a .depth file. Analogously for the .conf files.

The .jpg files are self-explanatory.

To construct the metadata JSON file, I recommend that you inspect the metadata JSON file inside any .r3d file to see which pieces of data are required — the USB streaming API should now provide all data necessary to construct a metadata JSON file yourself.

See #33 (comment) and #27 to learn how to encode the camera poses into Record3D's storage format (stored under the "poses" key in the metadata JSON file).

Since Record3D 1.10.3, Record3D includes even the relative timestamp of the camera in the USB stream. See #91 for more details. The frame timestamps are stored under the frameTimestamps key in the metadata file.

Caveat: the .r3d files do not contain a root folder. This means that when creating the ZIP/r3d file, you should compress the rgbd folder and the icon, metadata & sound.m4a files directly, and not enclose them into a folder!

I think the information above is enough for you to develop what you need.

[jeonhuhuhu]

I think it would be best if you would "dissect" a .r3d file produced by Record3D, see what is inside, and then create your own .r3d files according to that.

The .r3d files are just ZIP files with .r3d extension instead of .zip extension. So I recommend that you export a 3D video into .r3d in Record3D, copy the .r3d file to your computer, unzip the .r3d file and see what is inside.

The contents of a typical (unzipped) .r3d file:

• rgbd folder: contains files in the format %d.jpg, %d.depth and %d.conf per each frame of the video, where %d is frame number, starting from 0
• icon file: a JPG thumbnail of the 3D video
• metadata file: a JSON text file containing metadata
• sound.m4a file: contains audio file; this file is optional, you can remove it if you do not want the 3D video to contain audio track

A single .depth file contains the raw float32 bytes (stored in row-major convention), compressed with lzfse. See this comment: #7 (comment)

The .conf files contain raw uint8 bytes (stored in row-major convention), compressed with lzfse.

You can compress the raw float32 depthmap bytes using lzfse, and then write the compressed bytes into a .depth file. Analogously for the .conf files.

The .jpg files are self-explanatory.

To construct the metadata JSON file, I recommend that you inspect the metadata JSON file inside any .r3d file to see which pieces of data are required — the USB streaming API should now provide all data necessary to construct a metadata JSON file yourself.

See #33 (comment) and #27 to learn how to encode the camera poses into Record3D's storage format (stored under the "poses" key in the metadata JSON file).

Since Record3D 1.10.3, Record3D includes even the relative timestamp of the camera in the USB stream. See #91 for more details. The frame timestamps are stored under the frameTimestamps key in the metadata file.

Caveat: the .r3d files do not contain a root folder. This means that when creating the ZIP/r3d file, you should compress the rgbd folder and the icon, metadata & sound.m4a files directly, and not enclose them into a folder!

I think the information above is enough for you to develop what you need.

The Record3D app you developed is very useful and I appreciate it.

That's fully answered, thank you!