Design for Authorities' tables of contents

[abartov]

Generation

I have come to realize that basing authority tables of contents on iterating through a root_collection is a less robust approach: it requires a manual step of associating collections with an authority's root collection, and that quickly becomes a hassle when inputting heterogeneous works.

Instead, I am thinking of a hybrid solution:

• The first level of tables of contents is entirely procedurally generated — by:
  1. searching for any Collections with the authority as an involved_authority
  2. in addition to whole Collections associated with the authority, and completely uncollected texts associated with the authority, there will be texts that are by that authority but which are in some Collection that isn't wholly associated with that authority:
  • the two main cases there are:
    1. a preface, afterword, etc. by authority X for a volume primarily by authority Y.
    2. an item in a completely heterogenous volume (an anthology, a periodical issue, etc.).
  • So, in ToC generation, texts by our Authority that are inside Collections not by our authority will be added to the ToC, under the heading/metadata of the Collection they are from, but without the rest of the contents of that Collection.
  • something like this, for a work by authority X in a collection edited by authority Y:

  ## from "The Best Ukrainian Poetry of the 20th century", edited by Y
  ...
  <title> / X
  ...
  

  • (if X has more than one item in that collection, more than one can appear, of course)
  3. any texts (Manifestations) not included in the Collections above, to be grouped under a catch-all (and created, if it doesn't exist) "Uncollected works" Collection for that authority. (This way, future runs of the generation logic will find those uncollected works under the "Uncollected works" Collection, so only any newly-added texts that were not added into a Collection yet would be found and added.
• Within each Collection, human editors would be able to re-arrange CollectionItems, add paratexts and placeholders, etc.
• But the arranging of volume-type Collections in the Authority's ToC would be entirely automatic, by a given sort order (date of publication by default, probably).
• Where I would need your help is in figuring out the data queries (and any model/index changes that may be required) to support this fetching. I do expect it to be aggressively cached.
• Essentially, it can be pre-generated and even persisted in the DB, because the only time it needs to be re-generated is when a new text by this authority is created (ingestion), or when a text is destroyed, or has X removed from its involved authorities. Those well-known places can trigger re-generation, or at least invalidate the cache so it can be lazily re-calculated next time it is shown.

Initial migration from current (manual markdown ToCs)

• Current (pre-Collections) Authority ToCs are manually-edited markdown buffers linking to Manifestations, with no form of aggregation (the volumes or series texts belong to are expressed via markdown headings, but not in any database entity or relation).
• Before the automated generation described above can replace status quo, an assisted migration process would need to take place, wherein human editors would be asked to group texts under (new) Collection entities (primarily volumes, and optionally sub-volume divisions like sections or cycles as well). Once a particular authority's migration is marked complete, the generation logic should be used from then on (and the legacy manual ToC entities would eventually be removed).
  • the migration process should do a best-effort grouping of texts deduced from markdown headings, and human editors will confirm or amend those guesses before approving the creation of new (Collection and CollectionItem) entities.
• The goal is to allow a period of co-existence of legacy (manually-maintained) Authority TOC pages alongside Collection-based generated pages. That period should be as short as we can practically make it, but would definitely be long weeks, probably a few months.
• This would allow gradual and careful migration, at the cost of a couple of months of inconsistent UI (e.g. authorities that have completed the migration would have Collection-enabled features such as download a whole volume, whereas authorities not yet migrated would not).

[damisul]

Some questions:

1. So we still do want to have actual Collection (record in collections table with corresponding items) to be created as root_collection for every authority. And they will replace existing markdown tocs, Correct?

And probably we want to have so 'generated_root_collection' to be created (as described in #390):

This generated tree is also used to create permanent root collections, and to augment existing ones. (i.e. if an authority has a root collection, and new texts are uploaded that involve this authority, but did not already exist as placeholders in one of the root collection's collections, those new texts should be automatically added to the root collection under appropriate collections as described above.

If so we'll need to add new collection_type 'generated_root'? Or this generated root should never be actually persisted?

2. Regarding:

any texts (Manifestations) not included in the Collections above, to be grouped under a catch-all (and created, if it doesn't exist) "Uncollected works" Collection for that authority. (This way, future runs of the generation logic will find those uncollected works under the "Uncollected works" Collection, so only any newly-added texts that were not added into a Collection yet would be found and added.

I believe we'll need to add 'uncollected_works' collection type as well?

Within each Collection, human editors would be able to re-arrange CollectionItems, add paratexts and placeholders, etc.

But the arranging of volume-type Collections in the Authority's ToC would be entirely automatic, by a given sort order (date of publication by default, probably).

So... Am I understand correctly, that Authority's root collection should only contain other collections, where authority is either involved in whole collection or in individual items.

I wonder if there could be situation when same work is included in different collections?
E.g. some poem A is included in 'Poems of Author X' and something like 'Best poems of 19th century' (and probably there could be much more collections).

So in this case we in TOC we'll have something like:

• 'Poems of Author X' (collection link)
• Poem A from 'Best poems of 19th century' (manifestation link)

?

Where I would need your help is in figuring out the data queries (and any model/index changes that may be required) to support this fetching. I do expect it to be aggressively cached.

Ok, sure I'll help with this. But I'd propose not to invest much into a caching now. Let's concentrate on implementing functionality first, and optimize/introduce caches later.

Actually... It could turn out that such collection generation will not be so heavy so maybe we can avoid storing root collection in DB at all, and only cache generated HTML layout...

Essentially, it can be pre-generated and even persisted in the DB, because the only time it needs to be re-generated is when a new text by this authority is created (ingestion), or when a text is destroyed, or has X removed from its involved authorities. Those well-known places can trigger re-generation, or at least invalidate the cache so it can be lazily re-calculated next time it is shown.

For now I'd propose to add button somewhere to authority show page to pregenerate root collection.

the migration process should do a best-effort grouping of texts deduced from markdown headings, and human editors will confirm or amend those guesses before approving the creation of new (Collection and CollectionItem) entities.

This existing markdown parsing could be a complex stuff... Worst case is when there are works by different authors in same collection. So for each author it will create own copy of collection... Perhaps we can think about this later, when new TOC mechanics will be ready.

The goal is to allow a period of co-existence of legacy (manually-maintained) Authority TOC pages alongside Collection-based generated pages. That period should be as short as we can practically make it, but would definitely be long weeks, probably a few months.

Well... Implementing this co-existance should not be a big problem. E.g. if author has 'root collection' we render it, if not - render legacy TOC.
But I understand your desire to speed it up.

Another approach to speed it up I can imagine is to not create complex UI in a web app (which has to be removed), but make this grouping of uncollected works in some external tool, e.g. in Excel, and then feed CSV file to some script which will create collections and groups items from it.

[damisul]

And additional question regarding root collection (if we'll decide to use persisted root collections).

Let's suppose we have an author who writes original Hebrew works as well as translates other authors to Hebrew.
The question arises from the fact that we want to have InvolvedAuthorities records to be associated with a root collection.

I wonder if we should create TWO root collections for this author, one for original works and one for translated works? So in first collection we'll have person specified as author, and in second as translator?

And if we have an author who e.g. written an article in Russian and then translated it to Hebrew himself, then perhaps this article should get into both collections.

Another option, is to use single collection, and specify writer both as author and translator, but we should always remember, that for root collections roles does not apply to each included work/sub-collection. It simply means that among all works person player role of author and translator (and possibly editor, illustrator, etc).

Another option is to not specify and roles on root collection at all. We know that this collection is associated with an authority via authorities.root_collection_id.

[damisul]

And all those discussion regarding root_collection is applicable to future 'uncollected' collection we want to use for works not included in any collection.

[abartov]

Again, I am thinking of removing the idea of a persisted root collection, and generating the ToC tree from individual collections' involved_authorities (including the person's "uncollected works" collection).

That generated collection would display the items it consists of, in whatever sort order. Within items that are themselves collections (volumes etc.), the order of their items is of course persisted in their CollectionItems' sequence fields, and displayed accordingly.

If a person wrote in a foreign language and then translated themselves into Hebrew, they could be an involved_authority twice on that text, once in author role and once in translator role, no problem.

[abartov]

Answers:

1. I am thinking of zero customization of the ToC between volumes, i.e. the root collection should be 100% calculable (with selectable sort order), and then whether or not it's persisted becomes a mere caching question. In other words, "losing" the root collection will never result in loss of information, as it can always be regenerated.
2. Yes, thank you for submitting a PR for the 'uncollected works' collection type!
3. Yes, correct.
4. Yes, agreed. Premature optimization is the root of all evil. :)
5. the button is fine and convenient for testing, sure.
6. yes, the migration will be complex, which is why we'll do it only semi-automatically, with human editors verifying the conversion to collections. But also, not that right now we don't have any heterogeneous collections in the system -- we have been postponing uploading such works precisely because we wanted this implementation to be there first. So the migration is still 99.9% of works by the TOC's owner, or by someone else and translated by the TOC's owner. (Note that when we have, say, a translated poem by Shakespeare, we only care about what volume (collection) it belongs to in translation, not what volume(s) it was part of in the original English publication.)
7. I think grouping the works and planning the migration outside the system would be far too error prone. So I do expect we will be creating some basic migration workbench for the human editors. Even though it's a one-time process, it would still involve thousands and thousands of volumes, so...