-
Notifications
You must be signed in to change notification settings - Fork 19
adjudication
- Introduction
- Getting Started
- Defining an Annotation Task
- Annotation Workflow
- Adjudication Workflow
- IAA calculator
As of 2.0, the adjudication tool, MAI, is integrated into MAE. To start an adjudication job, adjudicators first should open all the annotation XML files, using [File] > [Open Document] menu. When all annotations to adjudicate are open, they can start adjudication with [File] > [Start Adjudication] menu.
Note that adjudication only starts when all open annotations are sharing the common primary text.
When switching to adjudication, MAE will ask adjudicators whether they want to create an empty gold standard file or to load an existing gold standard. Also note that once in the adjudication mode, users cannot switch the document tab back to annotation file without exiting adjudication mode. So it is recommended to save all changes before starting adjudication.
In adjudication mode, the tables in the bottom work differently from the annotation mode. The table is populated with only tags that anchored on currently selected text. If no text is selected at all, non-consuming tags will fill in the table.
The text area works differently as well.
Unlike the annotation mode, where each tag type is assigned an identifying color, in adjudication each document, namely each annotator, gets an unique color.
These colors are shown in the text area to visualize the spans of tags of the current type (selected from the bottom table).
If a certain region of the text gets the color of a particular annotator, that means only that annotator marked up the region as a tag of currently selected type.
Gray color means at least two annotators mark the same region as a tag, and an underlined gray region indicates
For link tags, the color visualization is done based on arguments of the tags.
By dragging over some text span, adjudicators can see what tags annotators made.
To make a tag as a gold standard tag, they can use right-click on the text span or on the corresponding row in the table in the bottom to make a copy of a tag to gold standard file.
To make a clear distinction between existing tags and gold standard tags, non-gold tags are grayed out, right-click menu only has [Copy to GS].
Adjudicators can edit gold standard tags using the table, however, editing existing annotators' tags is not allowed in the adjudication mode.
When two or more tags are found under the selected text, MAE will show the agreement of attributes among the tags as well. This will help adjudicators find out the majority attribute value.
- Green glowing indicates the attribute value is either unanimity or holding the majority.
- Yellow glowing shows the value is holding the plurality (relative majority)
- Orange glowing means either the all relevant values are equally distributed or the values are not holding the majority nor the minority
- Red glowing means the value is in the smallest subset.
- Although creating new tags, including using dedicated modes to capture special instances, is supported while in adjudication as well, it is not recommended to create tags instead of taking tags from existing annotation.
- The text colors in adjudication are assigned based on tags of currently selected type. Thus adjudicators should pay more attention when using argument selection mode to create a link tag; extent tags are not visually identifiable.
- Adjudicating link tags that links between different sets of extent tags might add an extra complexity to the creation of gold standard. As a way to avoid such troubles, an annotation task can be separated out into two phases, one for annotating and adjudicating all extent tags, which is followed by annotation and adjudication of link tags.