DeleteMapAnnotationContext: handling large number of multi-linked annotations

[sbesson]

See https://trello.com/c/kRPgBR36/20-linked-annotation-deletion-is-very-slow-due-to-53-graph-deletion-rules

The map annotation deletion context is first doing a query a list of the <Object>AnnotationLink objects to delete followed by batches of graph operations to delete them. This works fine in the default bulkmap configuration use case where each type of map annotation is duplicated per image.

When the bulkmap configuration file contains primary group keys as in https://github.com/IDR/idr-metadata/blob/0.5.0/idr0038-held-kidneylightsheet/experimentA/idr0038-experimentA-bulkmap-config.yml#L83, canonical map annotations are re-used and linked to multiple objects. In the current case of IDR, this typically leads to millions of links to a single map annotation like Homo Sapiens.

With OMERO 5.4 graph rule, when deleting links with the default options, the server traverses the graph to determine whether the map annotation ends up orphaned and deletes it if appropriate. With the numbers above, this leads to extreme times and/or timeouts.

To work around this issue, the solution is to separate the deletion operations:

• delete the canonical map annotation links only using e.g. --localcfg '{"ns":"openmicroscopy.org/mapr/organism", "typesToIgnore":["Annotation"]}' (with b208d37, released as 0.3.0)
• delete the other annotations links and their annotations using the regular deletemap command
• optionally find and delete the orphaned map annotations from step 1

Since the knowledge of the canonical map annotations is currently stored in the bulkmap configuration, the logic above could be handled internally by the deletion context by re-using the logic of BulkToMapAnnotationContext to retrieve the primary keys namespace.