README.html

<h1 id="epic-kitchens-55-dataset"><a href="https://epic-kitchens.github.io/2018">EPIC KITCHENS-55 Dataset</a></h1>
<!-- start badges -->
<!-- end badges -->
<blockquote>
<p><a href="https://epic-kitchens.github.io/">EPIC-KITCHENS-55</a> is the largest dataset in first-person (egocentric) vision; 55 hours of multi-faceted, non-scripted recordings in native environments - i.e. the wearers’ homes, capturing all daily activities in the kitchen over multiple days. Annotations are collected using a novel `live’ audio commentary approach.</p>
</blockquote>
<h2 id="authors">Authors</h2>
<p>Dima Damen (1) Hazel Doughty (1) Giovanni Maria Farinella (3) Sanja Fidler (2) Antonino Furnari (3) Evangelos Kazakos (1) Davide Moltisanti (1) Jonathan Munro (1) Toby Perrett (1) Will Price (1) Michael Wray (1)</p>
<ul>
<li>(1 University of Bristol)</li>
<li>(2 University of Toronto)</li>
<li>(3 University of Catania)</li>
</ul>
<p><strong>Contact:</strong> <a href="mailto:uob-epic-kitchens@bristol.ac.uk">uob-epic-kitchens@bristol.ac.uk</a></p>
<h2 id="citing">Citing</h2>
<p>When using the dataset, kindly reference:</p>
<pre><code>@INPROCEEDINGS{Damen2018EPICKITCHENS,
   title={Scaling Egocentric Vision: The EPIC-KITCHENS Dataset},
   author={Damen, Dima and Doughty, Hazel and Farinella, Giovanni Maria  and Fidler, Sanja and 
           Furnari, Antonino and Kazakos, Evangelos and Moltisanti, Davide and Munro, Jonathan 
           and Perrett, Toby and Price, Will and Wray, Michael},
   booktitle={European Conference on Computer Vision (ECCV)},
   year={2018}
} </code></pre>
<p>(Check publication <a href="https://epic-kitchens.github.io">here</a>)</p>
<h2 id="dataset-details">Dataset Details</h2>
<h3 id="ground-truth">Ground Truth</h3>
<p>We provide ground truth for action segments and object bounding boxes.</p>
<ul>
<li><strong>Objects:</strong> Full bounding boxes of narrated objects for every annotated frame.</li>
<li><strong>Actions:</strong> Split into narrations and action labels:
<ul>
<li>Narrations containing the narrated sentence with the timestamp.</li>
<li>Action labels containing the verb and noun labels along with the start and end times of the segment.</li>
</ul></li>
</ul>
<h3 id="dataset-splits">Dataset Splits</h3>
<p>The dataset is comprised of three splits with the corresponding ground truth:</p>
<ul>
<li>Training set - Full ground truth.</li>
<li>Seen Kitchens (S1) Test set - Start/end times only.</li>
<li>Unseen Kitchens (S2) Test set - Start/end times only.</li>
</ul>
<p>Initially we are only releasing the full ground truth for the training set in order to run action and object challenges.</p>
<h3 id="important-files">Important Files</h3>
<ul>
<li><code>README.md (this file)</code></li>
<li><code>README.html</code></li>
<li><code>README.pdf</code></li>
<li><a href="#license"><code>license.txt</code></a></li>
<li><a href="EPIC_train_action_labels.csv"><code>EPIC_train_action_labels.csv</code></a> (<a href="#epic_train_action_labelscsv">Info</a>) (<a href="EPIC_train_action_labels.pkl">Pickle</a>)</li>
<li><a href="EPIC_train_object_labels.csv"><code>EPIC_train_object_labels.csv</code></a> (<a href="#epic_train_object_labelscsv">Info</a>)</li>
<li><a href="EPIC_test_s1_timestamps.csv"><code>EPIC_test_s1_timestamps.csv</code></a> (<a href="#epic_test_s1_timestampscsv">Info</a>) (<a href="EPIC_test_s1_timestamps.pkl">Pickle</a>)</li>
<li><a href="EPIC_test_s2_timestamps.csv"><code>EPIC_test_s2_timestamps.csv</code></a> (<a href="#epic_test_s2_timestampscsv">Info</a>) (<a href="EPIC_test_s2_timestamps.pkl">Pickle</a>)</li>
<li><a href="EPIC_train_object_action_correspondence.csv"><code>EPIC_train_object_action_correspondence.csv</code></a> (<a href="#epic_train_object_action_correspondencecsv">Info</a>) (<a href="EPIC_train_object_action_correspondence.pkl">Pickle</a>)</li>
<li><a href="EPIC_test_s1_object_action_correspondence.csv"><code>EPIC_test_s1_object_action_correspondence.csv</code></a> (<a href="#epic_test_s1_object_action_correspondencecsv">Info</a>) (<a href="EPIC_test_s1_object_action_correspondence.pkl">Pickle</a>)</li>
<li><a href="EPIC_test_s2_object_action_correspondence.csv"><code>EPIC_test_s2_object_action_correspondence.csv</code></a> (<a href="#epic_test_s2_object_action_correspondencecsv">Info</a>) (<a href="EPIC_test_s2_object_action_correspondence.pkl">Pickle</a>)</li>
<li><a href="EPIC_test_s1_object_video_list.csv"><code>EPIC_test_s1_object_video_list.csv</code></a> (<a href="#epic_test_s1_object_video_listcsv">Info</a>)</li>
<li><a href="EPIC_test_s2_object_video_list.csv"><code>EPIC_test_s2_object_video_list.csv</code></a> (<a href="#epic_test_s2_object_video_listcsv">Info</a>)</li>
<li><a href="EPIC_noun_classes.csv"><code>EPIC_noun_classes.csv</code></a> (<a href="#epic_noun_classescsv">Info</a>)</li>
<li><a href="EPIC_verb_classes.csv"><code>EPIC_verb_classes.csv</code></a> (<a href="#epic_verb_classescsv">Info</a>)</li>
</ul>
<h3 id="additional-files">Additional Files</h3>
<ul>
<li><a href="EPIC_train_invalid_labels.csv"><code>EPIC_train_invalid_labels.csv</code></a> (<a href="#epic_train_invalid_labelscsv">Info</a>) (<a href="EPIC_train_invalid_labels.pkl">Pickle</a>)</li>
<li><a href="EPIC_train_action_narrations.csv"><code>EPIC_train_action_narrations.csv</code></a> (<a href="#epic_train_action_narrationscsv">Info</a>)</li>
<li><a href="EPIC_descriptions.csv"><code>EPIC_descriptions.csv</code></a> (<a href="#epic_descriptionscsv">Info</a>)</li>
<li><a href="EPIC_many_shot_verbs.csv"><code>EPIC_many_shot_verbs.csv</code></a> (<a href="#epic_many_shot_verbscsv">Info</a>)</li>
<li><a href="EPIC_many_shot_nouns.csv"><code>EPIC_many_shot_nouns.csv</code></a> (<a href="#epic_many_shot_nounscsv">Info</a>)</li>
<li><a href="EPIC_many_shot_actions.csv"><code>EPIC_many_shot_actions.csv</code></a> (<a href="#epic_many_shot_actionscsv">Info</a>)</li>
<li><a href="EPIC_video_info.csv"><code>EPIC_video_info.csv</code></a> (<a href="#epic_video_infocsv">info</a>)</li>
</ul>
<p>We direct the reader to <a href="https://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d">RDSF</a> for the videos and rgb/flow frames.</p>
<p>We provide html and pdf alternatives to this README which are auto-generated.</p>
<h2 id="files-structure">Files Structure</h2>
<h3 id="epic_train_action_labels.csv">EPIC_train_action_labels.csv</h3>
<p>CSV file containing 14 columns:</p>
<table>
<colgroup>
<col style="width: 10%" />
<col style="width: 15%" />
<col style="width: 8%" />
<col style="width: 65%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>uid</code></td>
<td>int</td>
<td><code>6374</code></td>
<td>Unique ID of the segment.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P03_01</code></td>
<td>Video the segment is in.</td>
</tr>
<tr class="odd">
<td><code>narration</code></td>
<td>string</td>
<td><code>close fridge</code></td>
<td>English description of the action provided by the participant.</td>
</tr>
<tr class="even">
<td><code>start_timestamp</code></td>
<td>string</td>
<td><code>00:23:43.847</code></td>
<td>Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td><code>stop_timestamp</code></td>
<td>string</td>
<td><code>00:23:47.212</code></td>
<td>End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td><code>start_frame</code></td>
<td>int</td>
<td><code>85430</code></td>
<td>Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td><code>stop_frame</code></td>
<td>int</td>
<td><code>85643</code></td>
<td>End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="even">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P03</code></td>
<td>ID of the participant.</td>
</tr>
<tr class="odd">
<td><code>verb</code></td>
<td>string</td>
<td><code>close</code></td>
<td>Parsed verb from the narration.</td>
</tr>
<tr class="even">
<td><code>noun</code></td>
<td>string</td>
<td><code>fridge</code></td>
<td>First parsed noun from the narration.</td>
</tr>
<tr class="odd">
<td><code>verb_class</code></td>
<td>int</td>
<td><code>3</code></td>
<td>Numeric ID of the parsed verb’s class.</td>
</tr>
<tr class="even">
<td><code>noun_class</code></td>
<td>int</td>
<td><code>10</code></td>
<td>Numeric ID of the parsed noun’s class.</td>
</tr>
<tr class="odd">
<td><code>all_nouns</code></td>
<td>list of string (1 or more)</td>
<td><code>['fridge']</code></td>
<td>List of all parsed nouns from the narration.</td>
</tr>
<tr class="even">
<td><code>all_noun_classes</code></td>
<td>list of int (1 or more)</td>
<td><code>[10]</code></td>
<td>List of numeric IDs corresponding to all of the parsed nouns’ classes from the narration.</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_train_invalid_labels.csv">EPIC_train_invalid_labels.csv</h3>
<p>CSV file containing 14 columns:</p>
<table>
<colgroup>
<col style="width: 10%" />
<col style="width: 15%" />
<col style="width: 8%" />
<col style="width: 65%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>uid</code></td>
<td>int</td>
<td><code>6374</code></td>
<td>Unique ID of the segment.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P03_01</code></td>
<td>Video the segment is in.</td>
</tr>
<tr class="odd">
<td><code>narration</code></td>
<td>string</td>
<td><code>close fridge</code></td>
<td>English description of the action provided by the participant.</td>
</tr>
<tr class="even">
<td><code>start_timestamp</code></td>
<td>string</td>
<td><code>00:23:43.847</code></td>
<td>Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td><code>stop_timestamp</code></td>
<td>string</td>
<td><code>00:23:47.212</code></td>
<td>End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td><code>start_frame</code></td>
<td>int</td>
<td><code>85430</code></td>
<td>Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td><code>stop_frame</code></td>
<td>int</td>
<td><code>85643</code></td>
<td>End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="even">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P03</code></td>
<td>ID of the participant.</td>
</tr>
<tr class="odd">
<td><code>verb</code></td>
<td>string</td>
<td><code>close</code></td>
<td>Parsed verb from the narration.</td>
</tr>
<tr class="even">
<td><code>noun</code></td>
<td>string</td>
<td><code>fridge</code></td>
<td>First parsed noun from the narration.</td>
</tr>
<tr class="odd">
<td><code>verb_class</code></td>
<td>int</td>
<td><code>3</code></td>
<td>Numeric ID of the parsed verb’s class.</td>
</tr>
<tr class="even">
<td><code>noun_class</code></td>
<td>int</td>
<td><code>10</code></td>
<td>Numeric ID of the parsed noun’s class.</td>
</tr>
<tr class="odd">
<td><code>all_nouns</code></td>
<td>list of string (1 or more)</td>
<td><code>['fridge']</code></td>
<td>List of all parsed nouns from the narration.</td>
</tr>
<tr class="even">
<td><code>all_noun_classes</code></td>
<td>list of int (1 or more)</td>
<td><code>[10]</code></td>
<td>List of numeric IDs corresponding to all of the parsed nouns’ classes from the narration.</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_train_action_narrations.csv">EPIC_train_action_narrations.csv</h3>
<p>CSV file containing 5 columns:</p>
<p><em>Note: The start/end timestamp refers to the start/end time of the narration, not the action itself.</em></p>
<table>
<colgroup>
<col style="width: 17%" />
<col style="width: 6%" />
<col style="width: 15%" />
<col style="width: 62%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P03</code></td>
<td>ID of the participant.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P03_01</code></td>
<td>Video the segment is in.</td>
</tr>
<tr class="odd">
<td><code>start_timestamp</code></td>
<td>string</td>
<td><code>00:23:43.847</code></td>
<td>Start time in <code>HH:mm:ss.SSS</code> of the narration.</td>
</tr>
<tr class="even">
<td><code>stop_timestamp</code></td>
<td>string</td>
<td><code>00:23:47.212</code></td>
<td>End time in <code>HH:mm:ss.SSS</code> of the narration.</td>
</tr>
<tr class="odd">
<td><code>narration</code></td>
<td>string</td>
<td><code>close fridge</code></td>
<td>English description of the action provided by the participant.</td>
</tr>
</tbody>
</table>
<h3 id="epic_train_object_labels.csv">EPIC_train_object_labels.csv</h3>
<p>CSV file containing 6 columns:</p>
<table>
<colgroup>
<col style="width: 11%" />
<col style="width: 19%" />
<col style="width: 17%" />
<col style="width: 51%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>noun_class</code></td>
<td>int</td>
<td><code>20</code></td>
<td>Integer value representing the class in noun-classes.csv.</td>
</tr>
<tr class="even">
<td><code>noun</code></td>
<td>string</td>
<td><code>bag</code></td>
<td>Original string name for the object.</td>
</tr>
<tr class="odd">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P01</code></td>
<td>ID of participant.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_01</code></td>
<td>Video the object was annotated in.</td>
</tr>
<tr class="odd">
<td><code>frame</code></td>
<td>int</td>
<td><code>056581</code></td>
<td>Frame number of the annotated object.</td>
</tr>
<tr class="even">
<td><code>bounding_boxes</code></td>
<td>list of 4-tuple (0 or more)</td>
<td><code>&quot;[(76, 1260, 462, 186)]&quot;</code></td>
<td>Annotated boxes with format <code>(&lt;top:int&gt;,&lt;left:int&gt;,&lt;height:int&gt;,&lt;width:int&gt;)</code>.</td>
</tr>
</tbody>
</table>
<h3 id="epic_train_object_action_correspondence.csv">EPIC_train_object_action_correspondence.csv</h3>
<p>CSV file containing 5 columns:</p>
<table>
<colgroup>
<col style="width: 12%" />
<col style="width: 5%" />
<col style="width: 10%" />
<col style="width: 72%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P01</code></td>
<td>ID of participant.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_01</code></td>
<td>Video the frames are part of.</td>
</tr>
<tr class="odd">
<td><code>object_frame</code></td>
<td>int</td>
<td><code>56581</code></td>
<td>Frame number of the object detection image from <code>object_detection_images</code>.</td>
</tr>
<tr class="even">
<td><code>action_frame</code></td>
<td>int</td>
<td><code>56638</code></td>
<td>Frame number of the corresponding image in the released frames for action recognition in <code>frames_rgb_flow</code>.</td>
</tr>
<tr class="odd">
<td><code>timestamp</code></td>
<td>string</td>
<td><code>00:00:00.00</code></td>
<td>Timestamp in <code>HH:mm:ss.SS</code> corresponding to the frame.</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_test_s1_object_action_correspondence.csv">EPIC_test_s1_object_action_correspondence.csv</h3>
<p>CSV file containing 5 columns:</p>
<table>
<colgroup>
<col style="width: 12%" />
<col style="width: 5%" />
<col style="width: 10%" />
<col style="width: 72%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P01</code></td>
<td>ID of participant.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_11</code></td>
<td>Video containing the object s1 test frames.</td>
</tr>
<tr class="odd">
<td><code>object_frame</code></td>
<td>int</td>
<td><code>33601</code></td>
<td>Frame number of the object detection image from <code>object_detection_images</code>.</td>
</tr>
<tr class="even">
<td><code>action_frame</code></td>
<td>int</td>
<td><code>33635</code></td>
<td>Frame number of the corresponding image in the released frames for action recognition in <code>frames_rgb_flow</code>.</td>
</tr>
<tr class="odd">
<td><code>timestamp</code></td>
<td>string</td>
<td><code>00:09:20.58</code></td>
<td>Timestamp in <code>HH:mm:ss.SS</code> corresponding to the frames.</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_test_s2_object_action_correspondence.csv">EPIC_test_s2_object_action_correspondence.csv</h3>
<p>CSV file containing 5 columns:</p>
<table>
<colgroup>
<col style="width: 12%" />
<col style="width: 5%" />
<col style="width: 10%" />
<col style="width: 72%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P09</code></td>
<td>ID of participant.</td>
</tr>
<tr class="even">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P09_05</code></td>
<td>Video containing the object s2 test frames.</td>
</tr>
<tr class="odd">
<td><code>object_frame</code></td>
<td>int</td>
<td><code>15991</code></td>
<td>Frame number of the object detection image from <code>object_detection_images</code>.</td>
</tr>
<tr class="even">
<td><code>action_frame</code></td>
<td>int</td>
<td><code>16007</code></td>
<td>Frame number of the corresponding image in the released frames for action recognition in <code>frames_rgb_flow</code>.</td>
</tr>
<tr class="odd">
<td><code>timestamp</code></td>
<td>string</td>
<td><code>00:04:26.78</code></td>
<td>Timestamp in <code>HH:mm:ss.SS</code> corresponding to the frames.</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_test_s1_object_video_list.csv">EPIC_test_s1_object_video_list.csv</h3>
<p>CSV file listing the videos used to obtain the object s1 test frames. The frames can be obtained from <a href="https://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d">RDSF</a> under <code>object_detection_images/test</code>. Please test all frames from this folder for the videos listed in this csv.</p>
<table>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_11</code></td>
<td>Video containing the object s1 test frames.</td>
</tr>
<tr class="even">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P01</code></td>
<td>ID of the participant.</td>
</tr>
</tbody>
</table>
<h3 id="epic_test_s2_object_video_list.csv">EPIC_test_s2_object_video_list.csv</h3>
<p>CSV file listing the videos used to obtain the object s2 test frames. The frames can be obtained from <a href="https://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d">RDSF</a> under <code>object_detection_images/test</code>. Please test all frames from this folder for the videos listed in this csv.</p>
<table>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_11</code></td>
<td>Video containing the object s2 test frames.</td>
</tr>
<tr class="even">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P01</code></td>
<td>ID of the participant.</td>
</tr>
</tbody>
</table>
<h3 id="epic_test_s1_timestamps.csv">EPIC_test_s1_timestamps.csv</h3>
<p>CSV file containing 7 columns:</p>
<table>
<colgroup>
<col style="width: 10%" />
<col style="width: 3%" />
<col style="width: 9%" />
<col style="width: 75%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>uid</code></td>
<td>int</td>
<td><code>1924</code></td>
<td>Unique ID of the segment.</td>
</tr>
<tr class="even">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P01</code></td>
<td>ID of the participant.</td>
</tr>
<tr class="odd">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_11</code></td>
<td>Video the segment is in.</td>
</tr>
<tr class="even">
<td><code>start_timestamp</code></td>
<td>string</td>
<td><code>00:00:00.000</code></td>
<td>Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td><code>stop_timestamp</code></td>
<td>string</td>
<td><code>00:00:01.890</code></td>
<td>End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td><code>start_frame</code></td>
<td>int</td>
<td><code>1</code></td>
<td>Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td><code>stop_frame</code></td>
<td>int</td>
<td><code>93</code></td>
<td>End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_test_s2_timestamps.csv">EPIC_test_s2_timestamps.csv</h3>
<p>CSV file containing 7 columns:</p>
<table>
<colgroup>
<col style="width: 10%" />
<col style="width: 3%" />
<col style="width: 9%" />
<col style="width: 75%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>uid</code></td>
<td>int</td>
<td><code>15582</code></td>
<td>Unique ID of the segment.</td>
</tr>
<tr class="even">
<td><code>participant_id</code></td>
<td>string</td>
<td><code>P09</code></td>
<td>ID of the participant.</td>
</tr>
<tr class="odd">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P09_01</code></td>
<td>Video the segment is in.</td>
</tr>
<tr class="even">
<td><code>start_timestamp</code></td>
<td>string</td>
<td><code>00:00:01.970</code></td>
<td>Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td><code>stop_timestamp</code></td>
<td>string</td>
<td><code>00:00:03.090</code></td>
<td>End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td><code>start_frame</code></td>
<td>int</td>
<td><code>118</code></td>
<td>Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td><code>stop_frame</code></td>
<td>int</td>
<td><code>185</code></td>
<td>End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.</p>
<h3 id="epic_noun_classes.csv">EPIC_noun_classes.csv</h3>
<p>CSV file containing 3 columns:</p>
<p><em>Note: a colon represents a compound noun with the more generic noun first. So pan:dust should be read as dust pan.</em></p>
<table>
<colgroup>
<col style="width: 9%" />
<col style="width: 23%" />
<col style="width: 25%" />
<col style="width: 41%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>noun_id</code></td>
<td>int</td>
<td><code>2</code></td>
<td>ID of the noun class.</td>
</tr>
<tr class="even">
<td><code>class_key</code></td>
<td>string</td>
<td><code>pan:dust</code></td>
<td>Key of the noun class.</td>
</tr>
<tr class="odd">
<td><code>nouns</code></td>
<td>list of string (1 or more)</td>
<td><code>&quot;['pan:dust', 'dustpan']&quot;</code></td>
<td>All nouns within the class (includes the key).</td>
</tr>
</tbody>
</table>
<h3 id="epic_verb_classes.csv">EPIC_verb_classes.csv</h3>
<p>CSV file containing 3 columns:</p>
<table>
<colgroup>
<col style="width: 9%" />
<col style="width: 22%" />
<col style="width: 29%" />
<col style="width: 38%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>verb_id</code></td>
<td>int</td>
<td><code>3</code></td>
<td>ID of the verb class.</td>
</tr>
<tr class="even">
<td><code>class_key</code></td>
<td>string</td>
<td><code>close</code></td>
<td>Key of the verb class.</td>
</tr>
<tr class="odd">
<td><code>verbs</code></td>
<td>list of string (1 or more)</td>
<td><code>&quot;['close', 'close-off', 'shut']&quot;</code></td>
<td>All verbs within the class (includes the key).</td>
</tr>
</tbody>
</table>
<h3 id="epic_descriptions.csv">EPIC_descriptions.csv</h3>
<p>CSV file containing 4 columns:</p>
<table>
<colgroup>
<col style="width: 10%" />
<col style="width: 5%" />
<col style="width: 39%" />
<col style="width: 44%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>video_id</code></td>
<td>string</td>
<td><code>P01_01</code></td>
<td>ID of the video.</td>
</tr>
<tr class="even">
<td><code>date</code></td>
<td>string</td>
<td><code>30/04/2017</code></td>
<td>Date on which the video was shot.</td>
</tr>
<tr class="odd">
<td><code>time</code></td>
<td>string</td>
<td><code>13:49:00</code></td>
<td>Local recording time of the video.</td>
</tr>
<tr class="even">
<td><code>description</code></td>
<td>string</td>
<td><code>prepared breakfast with soy milk and cereals</code></td>
<td>Description of the activities contained in the video.</td>
</tr>
</tbody>
</table>
<h3 id="epic_many_shot_verbs.csv">EPIC_many_shot_verbs.csv</h3>
<p>CSV file containing the many shot verbs. A verb class is considered many shot if it appears more than 100 times in training. (NOTE: this file is derived from <code>EPIC_train_action_labels.csv</code>, checkout the <a href="https://github.com/epic-kitchens/epic-many-shot-classes">accompanying notebook</a> demonstrating how we compute these classes)</p>
<table>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>verb_class</code></td>
<td>int</td>
<td><code>1</code></td>
<td>Numeric ID of the verb class</td>
</tr>
<tr class="even">
<td><code>verb</code></td>
<td>string</td>
<td><code>put</code></td>
<td>Verb corresponding to the verb class</td>
</tr>
</tbody>
</table>
<h3 id="epic_many_shot_nouns.csv">EPIC_many_shot_nouns.csv</h3>
<p>CSV file containing the many shot nouns. A noun class is considered many shot if it appears more than 100 times in training. (NOTE: this file is derived from <code>EPIC_train_action_labels.csv</code>, checkout the <a href="https://github.com/epic-kitchens/epic-many-shot-classes">accompanying notebook</a> demonstrating how we compute these classes)</p>
<table>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>noun_class</code></td>
<td>int</td>
<td><code>3</code></td>
<td>Numeric ID of the noun class</td>
</tr>
<tr class="even">
<td><code>noun</code></td>
<td>string</td>
<td><code>tap</code></td>
<td>Noun corresponding to the noun class</td>
</tr>
</tbody>
</table>
<h3 id="epic_many_shot_actions.csv">EPIC_many_shot_actions.csv</h3>
<p>CSV file containing the many shot actions. An action class (composed of a verb class and noun class) is considered many shot if BOTH the verb class and noun class are many shot AND the action class appears in training at least once. (NOTE: this file is derived from <code>EPIC_train_action_labels.csv</code>, checkout the <a href="https://github.com/epic-kitchens/epic-many-shot-classes">accompanying notebook</a> demonstrating how we compute these classes)</p>
<table>
<colgroup>
<col style="width: 11%" />
<col style="width: 7%" />
<col style="width: 38%" />
<col style="width: 42%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>action_class</code></td>
<td>(int, int)</td>
<td><code>(9, 84)</code></td>
<td>Numeric Pair of IDs, first the verb, then the noun</td>
</tr>
<tr class="even">
<td><code>verb_class</code></td>
<td>int</td>
<td><code>9</code></td>
<td>Numeric ID of the verb class</td>
</tr>
<tr class="odd">
<td><code>verb</code></td>
<td>string</td>
<td><code>move</code></td>
<td>Verb corresponding to the verb class</td>
</tr>
<tr class="even">
<td><code>noun_class</code></td>
<td>int</td>
<td><code>84</code></td>
<td>Numeric ID of the noun class</td>
</tr>
<tr class="odd">
<td><code>noun</code></td>
<td>string</td>
<td><code>sausage</code></td>
<td>Noun corresponding to the noun class</td>
</tr>
</tbody>
</table>
<h3 id="epic_video_info.csv">EPIC_video_info.csv</h3>
<p>CSV file containing information for each video</p>
<table>
<colgroup>
<col style="width: 11%" />
<col style="width: 7%" />
<col style="width: 38%" />
<col style="width: 42%" />
</colgroup>
<thead>
<tr class="header">
<th>Column Name</th>
<th>Type</th>
<th>Example</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>video</code></td>
<td>(string)</td>
<td><code>P01_01</code></td>
<td>Video ID</td>
</tr>
<tr class="even">
<td><code>resolution</code></td>
<td>(string)</td>
<td><code>1920x1080</code></td>
<td>Resolution of the video, format is <code>WIDTHxHEIGHT</code></td>
</tr>
<tr class="odd">
<td><code>duration</code></td>
<td>(float)</td>
<td><code>1652.152817</code></td>
<td>Duration of the video, in seconds</td>
</tr>
<tr class="even">
<td><code>fps</code></td>
<td>(float)</td>
<td><code>59.9400599400599</code></td>
<td>Frame rate of the video</td>
</tr>
</tbody>
</table>
<h2 id="file-downloads">File Downloads</h2>
<p>Due to the size of the dataset we provide scripts for downloading parts of the dataset:</p>
<ul>
<li><a href="https://github.com/epic-kitchens/download-scripts/blob/master/download_videos.sh">videos</a> (750GB)</li>
<li><a href="https://github.com/epic-kitchens/download-scripts/blob/master/download_frames_rgb_flow.sh">frames</a> (320GB)
<ul>
<li><a href="https://raw.githubusercontent.com/epic-kitchens/download-scripts/master/frames_rgb_flow/download_rgb.sh">rgb-frames</a> (220GB)</li>
<li><a href="https://raw.githubusercontent.com/epic-kitchens/download-scripts/master/frames_rgb_flow/download_flow.sh">flow-frames</a> (100GB)</li>
</ul></li>
<li><a href="https://github.com/epic-kitchens/download-scripts/blob/master/download_object_detection_images.sh">object annotation images</a> (80GB)</li>
</ul>
<p><em>Note: These scripts will work for Linux and Mac. For Windows users a bash installation should work.</em></p>
<p>These scripts replicate the folder structure of the dataset release, found <a href="https://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d">here</a>.</p>
<p>If you wish to download part of the dataset instructions can be found <a href="https://github.com/epic-kitchens/download-scripts">here</a>.</p>
<h2 id="video-information">Video Information</h2>
<p>Videos are recorded in 1080p at 59.94 FPS on a GoPro Hero 5 with linear field of view. There are a minority of videos which were shot at different resolutions, field of views, or FPS due to participant error or camera. These videos identified using <code>ffprobe</code> are:</p>
<ul>
<li>1280x720: <code>P12_01</code>, <code>P12_02</code>, <code>P12_03</code>, <code>P12_04</code>.</li>
<li>2560x1440: <code>P12_05</code>, <code>P12_06</code></li>
<li>29.97 FPS: <code>P09_07</code>, <code>P09_08</code>, <code>P10_01</code>, <code>P10_04</code>, <code>P11_01</code>, <code>P18_02</code>, <code>P18_03</code></li>
<li>48 FPS: <code>P17_01</code>, <code>P17_02</code>, <code>P17_03</code>, <code>P17_04</code></li>
<li>90 FPS: <code>P18_09</code></li>
</ul>
<p>The GoPro Hero 5 was also set to drop the framerate in low light conditions to preserve exposure leading to variable FPS in some videos. If you wish to extract frames we suggest you resample at 60 FPS to mitigate issues with variable FPS, this can be achieved in a single step with FFmpeg:</p>
<pre><code>ffmpeg -i &quot;P##_**.MP4&quot; -vf &quot;scale=-2:256&quot; -q:v 4 -r 60 &quot;P##_**/frame_%010d.jpg&quot;</code></pre>
<p>where <code>##</code> is the Participant ID and <code>**</code> is the video ID.</p>
<p>Optical flow was extracted using a fork of <a href="https://github.com/feichtenhofer/gpu_flow"><code>gpu_flow</code></a> made <a href="https://github.com/dl-container-registry/furnari-flow">available on github</a>. We set the parameters: stride = 2, dilation = 3, bound = 25 and size = 256.</p>
<h2 id="license">License</h2>
<p>All files in this dataset are copyright by us and published under the Creative Commons Attribution-NonCommerial 4.0 International License, found <a href="https://creativecommons.org/licenses/by-nc/4.0/">here</a>. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.</p>
<h2 id="disclaimer">Disclaimer</h2>
<p>EPIC-KITCHENS-55 and EPIC-KITCHENS-100 were collected as a tool for research in computer vision, however, it is worth noting that the dataset may have unintended biases (including those of a societal, gender or racial nature).</p>
<h2 id="changelog">Changelog</h2>
<p>See release history for changelog.</p>