+
+
+ The CLAMS Vocabulary defines an ontology of terms for a core of objects and features exchanged amongst tools that process multi-media data. It is based on the LAPPS Web Service Exchange Vocabulary at
+
+
+ http://vocab.lappsgrid.org.
+
+
+ The vocabulary is being developed bottom-up on an as-needed basis for use in the development of the CLAMS platform.
+
+
+ In the hierarchy below annotation types are printed with the properties defined for them, metadata properties are printed between square brackets.
+
+
+
+
+
+ |
+
+ Thing (v1)
+
+
+ : id
+
+ |
+
+
+ |
+ |
+
+ |
+
+ |
+
+
+
+
+
+ |
+
+ Alignment (v1)
+
+
+ : [sourceType, targetType]
+
+
+ : source, target
+
+ |
+
+
+ |
+ |
+
+ |
+
+ |
+
+ |
+
+
+ |
+
+
+
+
+
+The result of this processing is a MMIF document with an image document and a view that contains a *BoundingBox* annotation where the bounding box has the `boxType` property set to "text":
+
+```json
+{
+ "documents": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/ImageDocument/v1",
+ "properties": {
+ "id": "m1",
+ "mime": "image/jpeg",
+ "location": "file:///var/archive/image-0012.jpg" }
+ }
+ ],
+ "views": [
+ {
+ "id": "v1",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/east/0.2.2",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/BoundingBox/v1": {
+ "document": "m1" } }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb1",
+ "coordinates": [[10,20], [40,20], [10,30], [40,30]],
+ "boxType": "text" }
+ }
+ ]
+ }
+ ]
+}
+```
+
+Tesseract will then add a view to this MMIF document that contains a text document as well as an *Alignment* type that specifies that the text document is aligned with the bounding box from view "v1".
+
+```json
+{
+ "id": "v2",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/tesseract/0.2.2",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TextDocument/v1" : {},
+ "http://mmif.clams.ai/vocabulary/Alignment/v1": {} }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td1",
+ "text": {
+ "@value": "yelp" } }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "source": "v1:bb1",
+ "target": "td1" }
+ }
+ ]
+}
+```
+
+The text document annotation is the same kind of objects as the text document objects in the toplevel `documents` property, it has the same type and uses the same properties. Notice also that the history of the text document, namely that it was derived from a particular bounding box in a particular image, can be traced via the alignment of the text document with the bounding box.
+
+An alternative for using an alignment would be to use a `textSource` property on the document or perhaps to reuse the `location` property. That would require less space, but would introduce another ways to align annotations.
+{: .box-note}
+
+Now this text document can be input to language processing. An NER component will not do anything interesting with this text so let's say we have a semantic typing component that has *"dog-sound"* as one of its categories. That hypothetical semantic typing component would add a new view to the list. That semantic typing component would add a new view to the list:
+
+```json
+{
+ "id": "v3",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/semantic-typer/0.2.4",
+ "contains": {
+ "http://vocab.lappsgrid.org/SemanticTag": {
+ "document": "v2:td1" } }
+ },
+ "annotations": [
+ {
+ "@type": "http://vocab.lappsgrid.org/SemanticTag",
+ "properties": {
+ "id": "st1",
+ "category": "dog-sound",
+ "start": 0,
+ "end": 4 }
+ }
+ ]
+}
+```
+
+This view encodes that the span from character offset 0 to character offset 4 contains a semantic tag and that the category is "dog-sound". This type can be traced to *TextDocument* "td1" in view "v2" via the `document` metadata property, and from there to the bounding box in the image.
+
+See ["examples" section](#mmif-examples) with the MMIF examples for a more realistic and larger example.
+
+We are here abstracting away from how the actual processing would proceed since we are focusing on the representation. In short, the CLAMS platform knows what kind of input an application requires and it would now that an NLP application requires a *TextDocument* to run on and it knows how to find all instance of *TextDocument* in a MMIF file.
+{: .box-note}
+
+
+
+
+### Multiple text documents in a view
+
+The image with the dog in the previous section just had a bounding box for the part of the image with the word *yelp*, but there were three other image regions that could have been input to OCR as well. With more boxes we just add more text documents and more alignments, here shown for one additional box:
+
+```json
+{
+ "id": "v2",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/tesseract/1.0.1",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TextDocument/v1": {},
+ "http://mmif.clams.ai/vocabulary/Alignment/v1": {} }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td1",
+ "text": {
+ "@value": "yelp" } }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "source": "v1:bb1",
+ "target": "td1" }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td2",
+ "text": {
+ "@value": "woof" } }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "source": "v1:bb2",
+ "target": "td2" }
+ }
+ ]
+}
+```
+
+This of course assumes that view "v1" has a bounding box identified by "v1:bb2".
+
+Now if you run the semantic tagger you would get tags with the category set to "dog-sound":
+
+```json
+{
+ "id": "v3",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/semantic-typer/0.2.4",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/SemanticTag/v1": {} }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/SemanticTag/v1",
+ "properties": {
+ "id": "st1",
+ "category": "dog-sound",
+ "document": "V2:td1",
+ "start": 0,
+ "end": 4 }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/SemanticTag/v1",
+ "properties": {
+ "id": "st2",
+ "category": "dog-sound",
+ "document": "V2:td2",
+ "start": 0,
+ "end": 4 }
+ }
+ ]
+}
+```
+
+Notice how the document to which the *SemanticTag* annotations point is not expressed by the metadata `document` property but by individual `document` properties on each semantic tag. This is unavoidable when we have multiple text documents that can be input to language processing.
+
+The above glances over the problem that we need some way for Tesseract to know what bounding boxes to take. We can do that by either introducing some kind of type or use the `app` property in the metadata or maybe by introducing a subtype for BoundingBox like TextBox. In general, we may need to solve what we never really solved for LAPPS which is what view should be used as input for an application.
+{: .box-note}
+
+
+
+## MMIF and the Vocabularies
+
+The structure of MMIF files is defined in the [schema](schema/mmif.json) and described in this document. But the semantics of what is expressed in the views are determined by the [CLAMS Vocabulary](vocabulary). Each annotation in a view has two fields: `@type` and `properties`. The value of the first one is typically an annotation type from the vocabulary. Here is a *BoundingBox* annotation as an example:
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb1",
+ "coordinates": [[0,0], [10,0], [0,10], [10,10]]
+ }
+}
+```
+
+The value of `@type` refers to the URL [http://mmif.clams.ai/vocabulary/BoundingBox/v1](vocabulary/BoundingBox) which is a page in the published vocabulary. That page will spell out the definition of *BoundingBox* as well as list all properties defined for it, whether inherited or not. On the page we can see that `id` is a required property inherited from *Annotation* and that `coordinates` is a required property of *BoundingBox*. Both are expressed in the `properties` dictionary above. The page also says that there is an optional property `timePoint`, but it is not used above.
+
+You might also have noticed by now that these URL-formatted values to this key end with some version number (e.g. `/v1`), which is different from the version of this document. That is because each individual annotation type (and document type in `documents` list) has its own version independent of the MMIF version. The independent versioning of annotation types enables type checking mechanism in CLAMS pipelines. See [versioning notes](../versioning) for more details.
+
+As displayed in the vocabulary, annotation types are hierarchically structured with `is-a` inheritance relations. That is, all properties from a parent type are *inherited* to their children. The top-level type in the CLAMS vocabulary is [http://mmif.clams.ai/vocabulary/Annotation](vocabulary/Annotation), and it can be generally used for attaching a piece of information (annotation) to a source document, using `document` property to indicate the source document. If an annotation is specifically about (or derived from) a part of the document (for example, a certain sentence in the text or a certain area of the image, etc.), one should consider one of the *Annotation*'s children that can anchor to the part that suits semantics and purpose of the annotation. Again, the annotation object can (and probably should) use the `document` property with a source document identifier, as long as the type is a sub-type of the *Annotation*. We will see concrete examples in the below.
+
+The [http://mmif.clams.ai/vocabulary/Thing](vocabulary/Thing) type is designed only as a placeholder and is not intended to be used to represent actual annotations.
+{: .box-note}
+
+The vocabulary also defines `metadata` properties. For example, the optional property `timeUnit` can be used for a *TimeFrame* to specify what unit is used for the start and end time points in instances of *TimeFrame*. This property is not expressed in the annotation but in the metadata of the view with the annotation type in the `contains` dictionary:
+
+As aforementioned, the *Annotation* type and its children can put the source document identifier in the `contains` dictionary, using `document` metadata property. Namely, there are two ways to express the source document of annotations: at individual object level or at the view level. Unless there is a good reason to specify document information for each and every annotation objects, using the view-level representation is recommended to save space when the MMIF is serialized to a JSON file.
+{: .box-warning}
+
+```json
+{
+ "metadata": {
+ "app": "http://apps.clams.ai/some_time_segmentation_app/1.0.3",
+ "timestamp": "2020-05-27T12:23:45",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TimeFrame/v2": {
+ "document": "m12",
+ "timeUnit": "milliseconds" } }
+ }
+}
+```
+
+Annotations in a MMIF file often refer to the LAPPS Vocabulary at [http://vocab.lappsgrid.org](http://vocab.lappsgrid.org). In that case, the annotation type in `@type` will refer to a URL just as with CLAMS annotation types, the only difference is that the URL will be in the LAPPS Vocabulary. Properties and metadata properties of LAPPS annotation types are defined and used the same way as described above for CLAMS types.
+
+Using a LAPPS type is actually an instance of the more general notion that the value of `@type` can be any URL (actually, any IRI). You can use any annotation category defined elsewhere, for example, you can use categories defined by the creator of an application or categories from other vocabularies. Here is an example with a type from [https://schema.org](https://schema.org):
+
+```json
+{
+ "@type": "https://schema.org/Clip",
+ "properties": {
+ "id": "clip-29",
+ "actor": "Geena Davis"
+ }
+}
+```
+
+This assumes that [https://schema.org/Clip](https://schema.org/Clip) defines all the features used in the `properties` dictionary. One little disconnect here is that in MMIF we insist on each annotation having an identifier in the `id` property and as it happens [https://schema.org](https://schema.org) does not define an `id` attribute, although it does define `identifier`.
+
+The CLAMS Platform does not require that a URL like [https://schema.org/Clip](https://schema.org/Clip) actually exists, but if it doesn't users of an application that creates the *Clip* type will not know exactly what the application creates.
+
+
+
+## MMIF Examples
+
+To finish off this document we provide some examples of complete MMIF documents:
+
+
+| example | description |
+| --------------------------------------------------------- | ------------------------------------------------------------ |
+| [bars-tones-slates](samples/bars-tones-slates) | A couple of time frames and some minimal text processing on a transcript. |
+| [east-tesseract-typing](samples/east-tesseract-typing) | EAST text box recognition followed by Tesseract OCR and semantic typing. |
+| [segmenter-kaldi-ner](samples/segmenter-kaldi-ner) | Audio segmentation followed by Kaldi speech recognition and NER. |
+| [everything](samples/everything) | A big MMIF example with various multimodal AI apps for video/audio as well as text. |
+
+Each example has some comments and a link to a raw JSON file.
+
+As we move along integrating new applications, other examples will be added with other kinds of annotation types.
+
diff --git a/docs/1.0.1/pi78oGjdT-annotated.jpg b/docs/1.0.1/pi78oGjdT-annotated.jpg
new file mode 100644
index 00000000..88875a55
Binary files /dev/null and b/docs/1.0.1/pi78oGjdT-annotated.jpg differ
diff --git a/docs/1.0.1/pi78oGjdT.jpg b/docs/1.0.1/pi78oGjdT.jpg
new file mode 100644
index 00000000..69e03023
Binary files /dev/null and b/docs/1.0.1/pi78oGjdT.jpg differ
diff --git a/docs/1.0.1/samples/bars-tones-slates/index.md b/docs/1.0.1/samples/bars-tones-slates/index.md
new file mode 100644
index 00000000..bc52b80b
--- /dev/null
+++ b/docs/1.0.1/samples/bars-tones-slates/index.md
@@ -0,0 +1,33 @@
+---
+layout: page
+title: MMIF Specification
+subtitle: Version 1.0.1
+---
+
+# Example: Bars and Tones and Slates
+
+To see the full example scroll down to the end or open the [raw json file](raw.json).
+
+This is a minimal example that contains two media documents, one pointing at a video and the other at a transcript. For the first document there are two views, one with bars-and-tone annotations and one with slate annotations. For the second document there is one view with the results of a tokenizer. This example file, while minimal, has everything required by MMIF.
+
+Some notes:
+
+- The metadata just specify the MMIF version.
+- Both media documents in the *documents* list refer to a location on a local disk or a mounted disk. If this document is not on a local disk or mounted disk then URLs should be used.
+- Each view has some metadata spelling out several kinds of things:
+ - The application that created the view.
+ - A timestamp of when the view was created.
+ - What kind of annotations are in the view and what metadata are there on those annotations (for example, in the view with id=v2, the *contains* field has a property "http://mmif.clams.ai/vocabulary/TimeFrame/v2" with a dictionary as the value and that dictionary contains the metadata. Here the metadata specify what document the annotations are over what unit is used for annotation offsets.
+
+Only one annotation is shown for each view, this is to keep the file as small as possible. Of course, often the bars-and-tones and slate views often have only one annotation so it is likely only the tokens view where annotations were left out.
+
+
+
+## Full MMIF File
+
+```json
+{% include_relative raw.json %}
+```
+
+
+
diff --git a/docs/1.0.1/samples/bars-tones-slates/raw.json b/docs/1.0.1/samples/bars-tones-slates/raw.json
new file mode 100644
index 00000000..7dc69067
--- /dev/null
+++ b/docs/1.0.1/samples/bars-tones-slates/raw.json
@@ -0,0 +1,96 @@
+{
+ "metadata": {
+ "mmif": "http://mmif.clams.ai/1.0.1"
+ },
+ "documents": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/VideoDocument/v1",
+ "properties": {
+ "id": "m1",
+ "mime": "video/mp4",
+ "location": "file:///var/archive/video-0012.mp4"
+ }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "m2",
+ "mime": "text/plain",
+ "location": "file:///var/archive/video-0012-transcript.txt"
+ }
+ }
+ ],
+ "views": [
+ {
+ "id": "v1",
+ "metadata": {
+ "app": "http://apps.clams.ai/bars-and-tones/1.0.5",
+ "timestamp": "2020-05-27T12:23:45",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TimeFrame/v2": {
+ "document": "m1",
+ "timeUnit": "seconds"
+ }
+ }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v2",
+ "properties": {
+ "id": "s1",
+ "start": 0,
+ "end": 5,
+ "frameType": "bars-and-tones"
+ }
+ }
+ ]
+ },
+ {
+ "id": "v2",
+ "metadata": {
+ "app": "http://apps.clams.ai/slates/1.0.3",
+ "timestamp": "2020-05-27T12:23:45",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TimeFrame/v2": {
+ "document": "m1",
+ "timeUnit": "seconds"
+ }
+ }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v2",
+ "properties": {
+ "id": "s1",
+ "start": 25,
+ "end": 38,
+ "frameType": "slate"
+ }
+ }
+ ]
+ },
+ {
+ "id": "v3",
+ "metadata": {
+ "app": "http://apps.clams.ai/spacy/1.3.0",
+ "timestamp": "2020-05-27T12:25:15",
+ "contains": {
+ "http://vocab.lappsgrid.org/Token": {
+ "document": "m2"
+ }
+ }
+ },
+ "annotations": [
+ {
+ "@type": "http://vocab.lappsgrid.org/Token",
+ "properties": {
+ "id": "s1",
+ "start": 0,
+ "end": 3,
+ "word": "The"
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/docs/1.0.1/samples/east-tesseract-typing/index.md b/docs/1.0.1/samples/east-tesseract-typing/index.md
new file mode 100644
index 00000000..e1d08c04
--- /dev/null
+++ b/docs/1.0.1/samples/east-tesseract-typing/index.md
@@ -0,0 +1,177 @@
+---
+layout: page
+title: MMIF Specification
+subtitle: Version 1.0.1
+---
+
+
+
+# Example: EAST, Tesseract and Typing
+
+This example contains one image document which points to this image:
+
+
+
+In addition, there are three views, one created by EAST, one by Tesseract and one by a semantic typing component. We now give fragments of the four relevant parts of the MMIF file, each with some comments.
+
+To see the full example scroll down to the end or open the [raw json file](raw.json).
+
+### Fragment 1: the documents list
+
+```json
+{
+ "documents": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/ImageDocument/v1",
+ "properties": {
+ "id": "m1",
+ "mime": "image/jpg",
+ "location": "/var/archive/image-fido-barks.jpg" }
+ }
+ ]
+}
+```
+This is simply just a list with one *ImageDocument* which points at the file with the barking dog image.
+
+### Fragment 2: the EAST view
+
+Here are the metadata in this view:
+
+```json
+{
+ "app": "http://mmif.clams.ai/apps/east/0.2.1",
+ "contains": {
+ "http://mmif.clams.ai/1.0.1/BoundingBox": {
+ "timeUnit": "pixels",
+ "document": "m1" } }
+}
+```
+
+It simply says that EAST created the view and that all bounding box annotations are over document "m1" using pixels as the unit.
+
+And here is the annotations list:
+
+
+```json
+[
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb1",
+ "coordinates": [[10,20], [40,20], [10,30], [40,30]],
+ "boxType": "text" }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb2",
+ "coordinates": [[210,220], [240,220], [210,230], [240,230]],
+ "boxType": "text" }
+ }
+]
+```
+
+EAST has found two text boxes: one for "Arf" and one for "yelp" (although after EAST runs we do not know yet what the actual text is). Text boxes are encoded simply by specifying what the type of the bounding box is. For the sake of a somewhat smaller example file we are assuming here that EAST does not find text boxes when the text slants down. Note also that the coordinates are made up and bear little relation to what the real coordinates are.
+
+### Fragment 3: the Tesseract view
+
+Metadata:
+
+```json
+{
+ "app": "http://mmif.clams.ai/apps/tesseract/0.2.1",
+ "contains": {
+ "http://mmif.clams.ai/0.1.0/vocabulary/TextDocument": {},
+ "http://mmif.clams.ai/0.1.0/vocabulary/Alignment": {} }
+}
+```
+
+Tesseract creates text documents from bounding boxes with type equal to "text" and creates alignment relations between the documents and the boxes. The interesting thing here is compared to the metadata for the view created by EAST is that here no *document* metadata property is defined. This is because neither *TextDocument* nor *Alignment* need to be directly anchored into a document.
+
+Annotations list:
+
+```json
+[
+ {
+ "@type": "http://mmif.clams.ai/0.1.0/vocabulary/TextDocument",
+ "properties": {
+ "id": "td1",
+ "text": {
+ "@value": "Arf" } }
+ },
+ {
+ "@type": "http://mmif.clams.ai/0.1.0/vocabulary/Alignment",
+ "properties": {
+ "id": "a1",
+ "source": "v1:bb1",
+ "target": "td1" }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td2",
+ "text": {
+ "@value": "yelp" } }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "id": "a2",
+ "source": "v1:bb2",
+ "target": "td2" }
+ }
+]
+```
+
+The text documents just have identifiers and store the text, they themselves are not aware of where they came from. The alignments link the text documents to bounding boxes in the view created by EAST.
+
+### Fragment 4: the Semantic Typer view
+
+Metadata:
+
+```json
+{
+ "app": "http://mmif.clams.ai/apps/semantic-typer/0.2.4",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/SemanticTag/v1": {} },
+}
+
+```
+
+Nothing spectacular here. Like the previous view no *document* property is used, but in this case it is because the semantic tags in the annotation list each refer to a different document.
+
+Annotations list:
+
+```json
+[
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/SemanticTag/v1",
+ "properties": {
+ "id": "st1",
+ "category": "dog-sound",
+ "document": "V2:td1",
+ "start": 0,
+ "end": 4 }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/SemanticTag/v1",
+ "properties": {
+ "id": "st2",
+ "category": "dog-sound",
+ "document": "V2:td2",
+ "start": 0,
+ "end": 4 }
+ }
+]
+```
+
+Now each annotation needs to have its own *document* property so we know what documents each semantic tag is anchored to.
+
+
+
+## Full MMIF File
+
+```json
+{% include_relative raw.json %}
+```
+
diff --git a/docs/1.0.1/samples/east-tesseract-typing/raw.json b/docs/1.0.1/samples/east-tesseract-typing/raw.json
new file mode 100644
index 00000000..fc74fbac
--- /dev/null
+++ b/docs/1.0.1/samples/east-tesseract-typing/raw.json
@@ -0,0 +1,124 @@
+{
+ "metadata": {
+ "mmif": "http://mmif.clams.ai/1.0.1"
+ },
+ "documents": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/ImageDocument/v1",
+ "properties": {
+ "id": "m1",
+ "mime": "image/jpeg",
+ "location": "file:///var/archive/image-fido-barks.jpg"
+ }
+ }
+ ],
+ "views": [
+ {
+ "id": "v1",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/east/0.2.1",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/BoundingBox/v1": {
+ "timeUnit": "pixels",
+ "document": "m1"
+ }
+ }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb1",
+ "coordinates": [ [10, 20], [40, 20], [10, 30], [40, 30] ],
+ "boxType": "text"
+ }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb2",
+ "coordinates": [ [210, 220], [240, 220], [210, 230], [240, 230] ],
+ "boxType": "text"
+ }
+ }
+ ]
+ },
+ {
+ "id": "v2",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/tesseract/0.2.1",
+ "contains": {
+ "http://mmif.clams.ai/0.1.0/TextDocument": {},
+ "http://mmif.clams.ai/0.1.0/Alignment": {}
+ }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td1",
+ "text": {
+ "@value": "Arf"
+ }
+ }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "id": "a1",
+ "source": "v1:bb1",
+ "target": "td1"
+ }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td2",
+ "text": {
+ "@value": "yelp"
+ }
+ }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "id": "a2",
+ "source": "v1:bb2",
+ "target": "td2"
+ }
+ }
+ ]
+ },
+ {
+ "id": "v3",
+ "metadata": {
+ "app": "http://mmif.clams.ai/apps/semantic-typer/0.2.4",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/SemanticTag/v1": {}
+ }
+ },
+ "annotations": [
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/SemanticTag/v1",
+ "properties": {
+ "id": "st1",
+ "category": "dog-sound",
+ "document": "V2:td1",
+ "start": 0,
+ "end": 4
+ }
+ },
+ {
+ "@type": "http://mmif.clams.ai/vocabulary/SemanticTag/v1",
+ "properties": {
+ "id": "st2",
+ "category": "dog-sound",
+ "document": "V2:td2",
+ "start": 0,
+ "end": 4
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/docs/1.0.1/samples/everything/images/newshour-loud-dogs.jpg b/docs/1.0.1/samples/everything/images/newshour-loud-dogs.jpg
new file mode 100644
index 00000000..6c7c3cad
Binary files /dev/null and b/docs/1.0.1/samples/everything/images/newshour-loud-dogs.jpg differ
diff --git a/docs/1.0.1/samples/everything/index.md b/docs/1.0.1/samples/everything/index.md
new file mode 100644
index 00000000..c784c1e4
--- /dev/null
+++ b/docs/1.0.1/samples/everything/index.md
@@ -0,0 +1,236 @@
+---
+layout: page
+title: MMIF Specification
+subtitle: Version 1.0.1
+---
+
+# Example: Everything and the kitchen sink
+
+To see the full example scroll down to the end or open the [raw json file](raw.json).
+
+This is an example with a bunch of different annotations created by a variety of tools. For the input we have a short totally made up video which starts with some bars-and-tone and a simple slate. Those are followed by about a dozen seconds of a talking head followed by an image of a barking dog.
+
+
+
+The timeline includes markers for seconds. In the views below all anchors will be using milliseconds.
+
+We apply the following processing tools:
+
+1. Bars-and-tone extraction
+1. Slate extraction
+1. Audio segmentation
+1. Kaldi speech recognition and alignment
+1. EAST text box recognition
+1. Tesseract OCR
+1. Named entity recognition
+1. Slate parsing
+
+Following now are short explanations of some frgaments of the full MMIF file, some application output was explained in more detail in other examples, refer to those for more details.
+
+### Extracting time frames
+
+The first three steps are straightforward and all result in views with time frame annotations (views with id=v1, id=v2 and id=v3). The bars-and-tone and slate extraction applications each find one time frame and the audio segmenter finds two segments with the second one being a speech time frame that starts at about 5500ms from the start.
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v2",
+ "properties": {
+ "id": "tf2",
+ "frameType": "speech",
+ "start": 5500,
+ "end": 22000 }
+}
+```
+
+This time frame will provide the input to Kaldi.
+
+### Kaldi speech recognition
+
+Kaldi creates one view (with id=v4) which has
+
+- a text document
+- an alignment of that document with the speech time frame from the segmenter
+- a list of tokens for the document
+- a list of time frames corresponding to each token
+- a list of alignments between the tokens and the time frames
+
+In the metadata it spells out that the offsets of all tokens are taken to be offsets in "td1", which is a text document in the same view. We can do this instead of the alternative (using the *document* property on all tokens) because all tokens are for the same text document.
+
+```json
+{
+ "app": "http://mmif.clams.ai/apps/kaldi/0.2.1",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TextDocument/v1": {},
+ "http://vocab.lappsgrid.org/Token": {
+ "document": "td1" },
+ "http://mmif.clams.ai/vocabulary/TimeFrame/v2": {
+ "timeUnit": "milliseconds",
+ "document": "m1" },
+ "http://mmif.clams.ai/vocabulary/Alignment/v1": {}
+ }
+}
+```
+
+Note that a text document can refer to its text by either using the *text* property which contains the text verbatim or by referring to an external file using the *location* property, here we use the second approach:
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td1",
+ "mime": "text/plain",
+ "location": "/var/processed/transcript-002.txt" }
+}
+```
+
+For the sake of argument we assume perfect speech recognition, and the content of the external file is as follows.
+
+> Hello, this is Jim Lehrer with the NewsHour on PBS. In the nineteen eighties, barking dogs have increasingly become a problem in urban areas.
+
+This text is aligned with the second time frame from the segmenter.
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "id": "a1",
+ "source": "v3:tf2",
+ "target": "td1" }
+}
+```
+
+See the full example below for all the tokens, time frames for each token and the alignment between the token and the time frame.
+
+### EAST and Tesseract
+
+EAST adds bounding boxes anchored to the video document with id=m1:
+
+```json
+{
+ "app": "http://mmif.clams.ai/apps/east/0.2.1",
+ "contains": {
+ "http://mmif.clams.ai/1.0.1/BoundingBox": { "document": "m1" }
+}
+```
+
+Let's assume that EAST runs on frames sampled from the video at 1 second intervals. For our example that means that EAST finds boxes at time offsets 3, 4, 5 and 21 seconds. Let's assume decent performance where EAST finds all the boxes in the slate and just the caption in the image (but not the barking sounds). Here is one example box annotation:
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/BoundingBox/v1",
+ "properties": {
+ "id": "bb9",
+ "timePoint": 4000,
+ "coordinates": [[180, 110], [460, 110], [180, 170], [460, 170]],
+ "boxType": "text" }
+}
+```
+
+Due to the nature of the input many of the bounding boxes will have identical or near-identical coordinates. For example, there are two more bounding boxes with the coordinates above, one for the box with time offset 3000 and one for the box with time offset 5000.
+
+Tesseract now runs on all those boxes and creates a text document for each of them. In doing so, it will add these to a new view:
+* text documents from each text box
+* alignment of that documents to their originating boxes
+
+Thus, the metadata of the new view would be:
+
+
+```json
+{
+ "app": "http://mmif.clams.ai/apps/tesseract/0.4.4",
+ "contains": {
+ "http://mmif.clams.ai/vocabulary/TextDocument/v1": {},
+ "http://mmif.clams.ai/vocabulary/Alignment/v1": {
+ "sourceType": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "targetType": "http://mmif.clams.ai/vocabulary/BoundingBox/v1"
+ }
+ }
+}
+```
+
+Unlike the alignment annotations in the Kaldi view, Tesseract specifies types of both ends of the alignments in the `contains` metadata. This is only allowed because all alignment annotations in the view have the same source type and target types. This information can help, for example, machines search for certain alignments more quickly.
+{: .box-note}
+
+Now the recognition results are recorded as text documents, here's one:
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
+ "properties": {
+ "id": "td1",
+ "text": { "@value": "DATE" } }
+}
+```
+
+And here is the corresponding alignment from the bounding box to the text document:
+
+```json
+{
+ "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
+ "properties": {
+ "id": "a1",
+ "source": "v5:bb1",
+ "target": "td1" }
+}
+```
+
+The source is in another view, hence the prefix on the identifier.
+
+### Named entity recognition
+
+After Kaldi and Tesseract have added text documents we now have all text extracted from audiovisual elements and we can run NLP tools like named entity recognizers over them. Each entity annotation refers to a text document, either the one in the Kaldi view or one of the documents in the Tesseract view, this examples refers to one of the documents in the Tesseract view:
+
+```json
+{
+ "@type": "http://vocab.lappsgrid.org/NamedEntity",
+ "properties": {
+ "id": "ne1",
+ "document": "v6:td2",
+ "start": 0,
+ "end": 10,
+ "category": "Date",
+ "text": "1982-05-12" }
+}
+```
+
+Note that since there were three text boxes with the date and therefore three documents with the actual text, there are also three named entities for this date.
+
+### Slate parsing
+
+This section is somewhat speculative since we have not yet made any decisions on what the output of a slate paser will look like.
+
+Slate parsing applies to frames in the slate segment found in the slate view (id=v2) and uses several kinds of information obtained from two or three other views:
+
+- The EAST view has text bounding boxes with coordinates for all those boxes.
+- The Tesseract view has the text values for all those boxes.
+- The NER view has named entity classes for some of those text values, which may in some cases be useful for slate parsing.
+
+A minimal option for the slate parser is to create a particular semantic tag dat describes value fields in a slate. For that it may use the the category of the named entity that is anchored to the field or the text in adjacent field. For example, if we have the text "1982-05-12" and we know it was tagged as a *Date* then this may indicate that that value is the air time of the video. Similary, if that value occurs next to a text that has the text "DATE" in it we may also derive that the value was a *Date*.
+
+Here is the tag annotation on the same document as the named entity annotation above:
+
+```json
+{
+ "@type": "http://vocab.lappsgrid.org/SemanticTag",
+ "properties": {
+ "id": "st1",
+ "document": "v6:td2",
+ "start": 0,
+ "end": 10,
+ "tagName": "Date",
+ "text": "1982-05-12" }
+}
+```
+
+Note that the *tagName* property has the same value as the *category* property on the named entity. This is a coincidence in that there is a named entity category *Date* as well as a slate category *Date*.
+
+Similar to what we saw for the named entities, there will be multiple versions of this data tag due to multiple text boxes with the same text.
+
+
+## Full MMIF File
+
+```json
+{% include_relative raw.json %}
+```
+
+
diff --git a/docs/1.0.1/samples/everything/pbcore.md b/docs/1.0.1/samples/everything/pbcore.md
new file mode 100644
index 00000000..9fa246e4
--- /dev/null
+++ b/docs/1.0.1/samples/everything/pbcore.md
@@ -0,0 +1,144 @@
+# MMIF and PBCore
+
+Some notes on the mappings of elements from the MMIF file in the "everything and the kitched sink" example to PBCore (see [index.md](index) and [raw.json](raw.json)).
+
+The relevant information that we have in MMIF is in the following types:
+
+1. Instances of *TimeFrame* with frameType "bars-and-tone" or "slate". These directly refer back to time slices in the video.
+2. Instances of *SemanticTag* with tagName "Date", "Title", "Host" or "Producer". These can be traced back to the part of the video where the information was obtained (that is, the location of the slate), but this is not needed here because it is not required by PBCore (or even allowed in the PBCore elements that we would be using).
+3. Instances of *NamedEntity* with category "Person", "Location" or "Organization". These do need to be traced back because we want to index on the locations in the video where a subject occurs.
+
+Tracing back to the source location requires some processing because while the information is available in the MMIF file it is not explicitly stated in the *NamedEntity* annotation.
+
+A note on collaboration on this. The CLAMS team could do one of the following:
+
+1. Provide code and an API that makes it easy to get the information from a MMIF file that is needed.
+2. Create code that extracts the needed in formation from a MMIF file and outputs it in some kind of generic format.
+3. Create code that extracts the information and creates PBCore output.
+
+Most of the work is in item 1 and that would be the minimal thing to do for CLAMS, but this document assumes for now that the CLAMS team also creates PBCore output.
+
+### Mappings from MMIF to PBCore
+
+The PBCore to be created has a top-level *pbcoreDescriptionDocument* element:
+
+```xml
+