This repository includes the download instuction and the code for the paper FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation.
Usage is limited to non-commercial research/development use. Users need to sign the license form to access the videos, metadata and annotation. Please sign to the license agreement and submit via this form.
Link to the dataset and required credentials will be sent by e-mail after approval.
24/09/12 Update: The object detection annotation (finebio_coco_annotations.zip) has been updated because there were missing annotation. Please re-download it if you are using the older version.
Video and annotation files are named by the following rules:
# First-person view
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>.mp4
# Third-person view
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>_T<camera_id (1-5)>.mp4
# Atomic operation annotation
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>.txt
# Object detection images
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>_<frame_num>.jpg
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>__T<camera_id (1-5)>_frame_num>.jpg
Please refer to the README for each benchmark for details.
| Backbone | Model | Acc | Edit | F1@10 | F1@25 | F1@50 | F1@75 | Weights |
|---|---|---|---|---|---|---|---|---|
| I3D | MS-TCN++ | 90.2 | 96.7 | 97.4 | 96.7 | 93.5 | 73.4 | Link |
| I3D | ASFormer | 87.2 | 94.8 | 94.2 | 92.7 | 86.5 | 67.0 | Link |
| Backbone | Model | [email protected] | 0.4 | 0.5 | 0.6 | 0.7 | Avg. | Weights |
|---|---|---|---|---|---|---|---|---|
| I3D | ActionFormer (multi-head + set classification) | 45.2 | 41.7 | 36.5 | 28.4 | 18.7 | 34.1 | Link |
| Model | AP | AP50 | AP_manipulated | AP_affected | Weights |
|---|---|---|---|---|---|
| DINO | 53.3 | 77.4 | 55.9 | 51.6 | Link |
| Deformable DETR | 56.1 | 78.5 | 64.0 | 58.8 | Link |
| Hand | H + Manipulated | H + M + Affected | Weights | |
|---|---|---|---|---|
| Left Hand | 96.8 | 6.5 | 5.9 | Link |
| Right Hand | 94.5 | 22.2 | 10.7 |
Please cite the necessary methods if you used our baseline models.
If you find any problem, please report to Takuma Yagi (takuma.yagi[at]aist.go.jp) or by this repository's issue.
Please cite our work if you have used our data or code:
@article{yagi2024finebio,
title={Finebio: a fine-grained video dataset of biological experiments with hierarchical annotation},
author={Yagi, Takuma and Ohashi, Misaki and Huang, Yifei and Furuta, Ryosuke and Adachi, Shungo and Mitsuyama, Toutai and Sato, Yoichi},
year={2024},
journal={arXiv preprint arXiv:2402.00293},
}
The code is licensed under the MIT License.