receivers: implement autocomplete #3006

iulianav · 2017-11-29T09:49:07Z

Description

Implemented the back-end for autocompletion using ES completion type suggesters for several required fields.

Related Issue

inspirehep/record-editor#261
closes #2755

Motivation and Context

Needed for the front-end implementation of the autocompletion..

Checklist:

I have all the information that I need (if not, move to RFC and look for it).
I linked the related issue(s) in the corresponding commit logs.
I wrote good commit log messages.
My code follows the code style of this project.
I've added any new docs if API/utils methods were added.
I have updated the existing documentation accordingly.
I have added tests to cover my changes.
All new and existing tests passed.

chris-asl · 2017-11-29T12:51:57Z

inspirehep/modules/records/mappings/records/hep.json

@@ -281,6 +281,9 @@
                        },
                        "volume": {
                            "type": "string"
+                        },
+                        "book_series_suggest": {


Since, this field is inside book_series, adding the prefix book_series is duplicating information.
I would suggest having that solely as suggest.

chris-asl · 2017-11-29T12:52:20Z

inspirehep/modules/records/mappings/records/hep.json

@@ -310,6 +313,9 @@
                        },
                        "value": {
                            "type": "string"
+                        },
+                        "collaboration_suggest": {


Same goes for this field.

chris-asl · 2017-11-29T12:53:23Z

inspirehep/modules/records/receivers.py

@@ -292,7 +354,8 @@ def populate_abstract_source_suggest(sender, json, *args, **kwargs):


 def populate_title_suggest(sender, json, *args, **kwargs):
-    """Populate the ``title_suggest`` field of Journals records."""
+    """Populate the ``title_suggest`` field of


Nit: formatting

michamos · 2017-12-06T14:59:47Z

inspirehep/modules/records/mappings/records/conferences.json

@@ -79,6 +79,10 @@
                "conferenceautocomplete": {


should this still be used after this PR (I have no idea)? if not you should remove it.

michamos · 2017-12-06T14:59:57Z

inspirehep/modules/records/mappings/records/experiments.json

@@ -132,6 +132,10 @@
                "experimentautocomplete": {


same comment.

michamos · 2017-12-06T15:02:02Z

inspirehep/modules/records/receivers.py

@@ -148,7 +153,7 @@ def enhance_after_index(sender, json, *args, **kwargs):


 def populate_bookautocomplete(sender, json, *args, **kwargs):


so what's the difference between *autocomplete and *_suggest? if there is none, the names should probably made consistent one way or another.

@michamos we will stick to the *_suggest as this is the proper way of implementing the suggesters. We will also remove all *autocomplete suggesters.

michamos · 2017-12-06T15:11:16Z

inspirehep/modules/records/receivers.py

+        return
+
+    doc_types = json.get('document_type', [])
+    if all(doc_type not in doc_types for doc_type in ['book', 'thesis', 'proceedings']):


I find if not set(doc_types) & {'book', 'thesis', 'proceedings'} clearer.

michamos · 2017-12-06T15:15:34Z

inspirehep/modules/records/receivers.py

                'title': titles,
            },
        },
    })


+def populate_book_series_suggest(sender, json, *args, **kwargs):


there seems to be a book series autocomplete in the submission forms. It is broken however, as it autocompletes from all journals, instead of only those having book_series=True combined with the book series in Literature records. Can you check whether you can salvage that one instead of creating a new one?

@michamos I would say that we should probably try to use this one in oreder to have consistent suggesters and remove the old one. @jacquerie, @harunurhan ? Also, the old one seems overly-complicated and confusing I would say.

michamos · 2017-12-06T15:19:46Z

inspirehep/modules/records/receivers.py

                'title': titles,
            },
        },
    })


+def populate_book_series_suggest(sender, json, *args, **kwargs):
+    """Populate the ``book_series.book_series_suggest`` field of Literature records."""
+    if 'hep.json' not in json.get('$schema'):


is this really one should check for a given record type (cc @jacquerie)? I would expect there to be a helper somewhere that gets you the type from the record. If not, it should probably exist.

@michamos is this related: #1355 ?

michamos · 2017-12-06T15:22:22Z

inspirehep/modules/records/receivers.py

+
+    json.update({
+        'conference_suggest': {
+            'input': cnum,


autocompleting on cnum is not very useful: either you already know it and you can type it out, or you don't and you won't learn anything from the list of suggestions. You should add more information from the conference record:
acronyms, address, series.name, titles, opening_date.

michamos · 2017-12-06T15:25:42Z

inspirehep/modules/records/receivers.py

+
+    input_values = []
+    input_values.append(legacy_name)
+    input_values.append(long_name)


I would rather do input_values = [legacy_name, long_name]

michamos · 2017-12-06T15:26:45Z

inspirehep/modules/records/receivers.py

@@ -306,14 +417,17 @@ def populate_title_suggest(sender, json, *args, **kwargs):
    input_values.extend(title_variants)


michamos · 2017-12-07T08:18:55Z

This is probably relevant: https://github.com/inveniosoftware/invenio/wiki/Invenio-Sprint-Reviews#completion-suggesters

iulianav · 2017-12-07T16:16:44Z

@michamos should we change to the V5 way of doing things that uses _source?

michamos · 2017-12-07T16:53:07Z

not my decision :) @jacquerie ?

harunurhan · 2017-12-08T08:26:16Z

If you remove the payload, it will be a breaking change. For record-editor, it will be small and quick PR to fix it, no problem but I don't know much about other parts.

iulianav · 2017-12-08T15:30:05Z

inspirehep/modules/records/receivers.py

@@ -170,20 +175,106 @@ def populate_bookautocomplete(sender, json, *args, **kwargs):
    input_values.extend(titles)
    input_values = [el for el in input_values if el]

-    ref = get_value(json, 'self.$ref')
+    record = get_value(json, 'self.$ref', '')

    json.update({
        'bookautocomplete': {
            'input': input_values,
            'payload': {
                'authors': authors,


@michamos I need more indications for bookautocomplete. Currently it has no output and we need to decide on it and what should be in the payload, right @harunurhan ?

input seems fine, output for parent_isbn should be isbns.value[0].

iulianav · 2017-12-08T17:03:28Z

Includes also and closes: #2795

iulianav · 2017-12-11T12:27:57Z

Includes and closes #2837

jacquerie · 2017-12-11T12:30:48Z

Includes and closes #2837

This doesn't work: annotations like "(c|C)loses #XXX" only take effect if they are part of the first message of the PR (#3006 (comment)) or as part of any commit. See: https://help.github.com/articles/closing-issues-using-keywords/.

Signed-off-by: Iuliana Voinea <[email protected]>

Added unit tests for hep records suggesters: collaboration_suggest and bokk_series_suggest, and for experiments records experiment_suggest suggester. Signed-off-by: Iuliana Voinea <[email protected]>

Signed-off-by: Iuliana Voinea <[email protected]>

Added 'long_name' and 'name_variants' to the input of experiment_suggest. Signed-off-by: Iuliana Voinea <[email protected]>

Signed-off-by: Iuliana Voinea <[email protected]>

So far, even if a record has several report numbers associated with it, the output will contain only the first one. Signed-off-by: Iuliana Voinea <[email protected]>

Signed-off-by: Iuliana Voinea <[email protected]>

…mplete` Signed-off-by: Iuliana Voinea <[email protected]>

The autocompletion is now done using a completion suggester. `experiment_suggest`. Signed-off-by: Iuliana Voinea <[email protected]>

Signed-off-by: Iuliana Voinea <[email protected]>

I replaced the old `conferenceautocomplete` with `conference_suggest`. Also, there was a bug in the JavaScript which called the `conference_suggest` endpoint every time the submission form was laoded, which I also hopefully fixed. Lastly, I removed some commented/unused autocompletion functions from the forms module and added `TODO`. comments which mention the fact that we need a consistent way of performing autocompletion (with ES compeltion suggeters), instead of using the old way this was implemented in `INSPIREForm`. Signed-off-by: Iuliana Voinea <[email protected]>

Signed-off-by: Iuliana Voinea <[email protected]>

iulianav · 2018-01-31T10:49:21Z

Some acceptance tests for conference and advisors autocompletion were failing and I had to modify them. This is due to the fact that when trying to autocomplete for the conference IN2P3 School of Statistics, 2012-05-28, Autrans, FR for example, the test tries to do so by typing autrans, but the way the ES completion suggester works does not allow for this as we would need an arbitrary offset, but I do not think this is possible to configure: The completion suggester is a so-called PREFIX suggester. It would have to type IN2 in order to work. Also, the tests expected an output that is no longer accurate.

However, for an author Smith, John the tests might pass for both typing Smi or Joh ( note that the required minimum length for autocompletion is 3 characters ) because there is a field called preferred_name which when populated seems to contain the value John Smith and is also added to the input_values needed for the suggester. Please someone correct me if I am wrong!

Signed-off-by: Iuliana Voinea <[email protected]>

iulianav added Status: Ready for review labels Nov 29, 2017

iulianav self-assigned this Nov 29, 2017

iulianav requested review from jacquerie, michamos and harunurhan November 29, 2017 09:49

iulianav force-pushed the implement-autocomplete branch from 0eab17c to 6f087d0 Compare November 29, 2017 09:51

chris-asl reviewed Nov 29, 2017

View reviewed changes

iulianav removed the Status: Ready for review label Nov 29, 2017

iulianav force-pushed the implement-autocomplete branch 2 times, most recently from 5ab546a to 2f8863a Compare December 1, 2017 10:09

michamos requested changes Dec 6, 2017

View reviewed changes

iulianav force-pushed the implement-autocomplete branch 2 times, most recently from bda7ccf to fb97a92 Compare December 8, 2017 13:47

iulianav commented Dec 8, 2017

View reviewed changes

iulianav force-pushed the implement-autocomplete branch 3 times, most recently from e7d7b33 to 32460aa Compare December 11, 2017 12:13

iulianav force-pushed the implement-autocomplete branch from 6b5853e to 482fb69 Compare December 11, 2017 13:01

This was referenced Dec 12, 2017

Fix experiments autocomplete #2837

Closed

Fix crash in autocomplete API for Institutions #2795

Closed

Iuliana Voinea added 18 commits January 31, 2018 10:29

records: implement book_series title autocomplete

ead4b15

Signed-off-by: Iuliana Voinea <[email protected]>

records: implement experiment legacy_name autocomplete

15a70f6

Signed-off-by: Iuliana Voinea <[email protected]>

tests: added unit tests

ddbc28e

Added unit tests for hep records suggesters: collaboration_suggest and bokk_series_suggest, and for experiments records experiment_suggest suggester. Signed-off-by: Iuliana Voinea <[email protected]>

records: implement conference cnum autocomplete

0b8be2d

Signed-off-by: Iuliana Voinea <[email protected]>

Extend input of experiment_suggest

e3c9d0b

Added 'long_name' and 'name_variants' to the input of experiment_suggest. Signed-off-by: Iuliana Voinea <[email protected]>

Add $ref to payloads of journals suggester

4be51a8

Signed-off-by: Iuliana Voinea <[email protected]>

A first implementation of report_number_suggest

7d1a076

So far, even if a record has several report numbers associated with it, the output will contain only the first one. Signed-off-by: Iuliana Voinea <[email protected]>

Apply review suggestions and refactor

d7053d2

Signed-off-by: Iuliana Voinea <[email protected]>

Change bookautocomplete to book_suggest and remove `journalautoco…

7741362

…mplete` Signed-off-by: Iuliana Voinea <[email protected]>

Fix experiments autocomplete

5421f6f

The autocompletion is now done using a completion suggester. `experiment_suggest`. Signed-off-by: Iuliana Voinea <[email protected]>

experiment_suggest: add more relevant fields to input_values

cd848ab

Signed-off-by: Iuliana Voinea <[email protected]>

tests: refactor experiment_suggest unit tests

456752e

Signed-off-by: Iuliana Voinea <[email protected]>

Remove and replace affautocomplete field with affiliation_suggest

a73cae8

Signed-off-by: Iuliana Voinea <[email protected]>

tests: refactor unit tests for receivers

3dadf5e

Signed-off-by: Iuliana Voinea <[email protected]>

Add relevant input fields to conference_suggest

7da4d6e

Signed-off-by: Iuliana Voinea <[email protected]>

Solve rebase conflicts

2c9316a

Signed-off-by: Iuliana Voinea <[email protected]>

Make conference_suggest meet front-end requirements.

c14fded

Signed-off-by: Iuliana Voinea <[email protected]>

iulianav force-pushed the implement-autocomplete branch 2 times, most recently from 7bb48e2 to 92ed6f0 Compare January 31, 2018 12:17

Improve the author autocompletion and ammend conference acceptnace test

3afcd77

Signed-off-by: Iuliana Voinea <[email protected]>

iulianav force-pushed the implement-autocomplete branch from 92ed6f0 to 3afcd77 Compare January 31, 2018 13:06

drjova removed Status: Ready for review labels Feb 6, 2018

jacquerie unassigned jacquerie and iulianav Feb 23, 2018

david-caro force-pushed the master branch from f170a67 to 90c2063 Compare May 2, 2018 16:42

drjova closed this Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

receivers: implement autocomplete #3006

receivers: implement autocomplete #3006

iulianav commented Nov 29, 2017 •

edited by michamos

Loading

chris-asl Nov 29, 2017

chris-asl Nov 29, 2017

chris-asl Nov 29, 2017

michamos Dec 6, 2017

michamos Dec 6, 2017

michamos Dec 6, 2017

iulianav Dec 7, 2017 •

edited

Loading

michamos Dec 6, 2017

michamos Dec 6, 2017

iulianav Dec 8, 2017 •

edited

Loading

michamos Dec 6, 2017

iulianav Dec 8, 2017

michamos Dec 6, 2017 •

edited

Loading

michamos Dec 6, 2017

michamos Dec 6, 2017

michamos commented Dec 7, 2017

iulianav commented Dec 7, 2017

michamos commented Dec 7, 2017

harunurhan commented Dec 8, 2017 •

edited

Loading

iulianav Dec 8, 2017

michamos Dec 8, 2017 •

edited

Loading

iulianav commented Dec 8, 2017 •

edited

Loading

iulianav commented Dec 11, 2017

jacquerie commented Dec 11, 2017 •

edited

Loading

iulianav commented Jan 31, 2018 •

edited

Loading

		@@ -148,7 +153,7 @@ def enhance_after_index(sender, json, args, *kwargs):


		def populate_bookautocomplete(sender, json, args, *kwargs):

		@@ -306,14 +417,17 @@ def populate_title_suggest(sender, json, args, *kwargs):
		input_values.extend(title_variants)

receivers: implement autocomplete #3006

receivers: implement autocomplete #3006

Conversation

iulianav commented Nov 29, 2017 • edited by michamos Loading

Description

Related Issue

Motivation and Context

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iulianav Dec 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iulianav Dec 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michamos Dec 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michamos commented Dec 7, 2017

iulianav commented Dec 7, 2017

michamos commented Dec 7, 2017

harunurhan commented Dec 8, 2017 • edited Loading

Choose a reason for hiding this comment

michamos Dec 8, 2017 • edited Loading

Choose a reason for hiding this comment

iulianav commented Dec 8, 2017 • edited Loading

iulianav commented Dec 11, 2017

jacquerie commented Dec 11, 2017 • edited Loading

iulianav commented Jan 31, 2018 • edited Loading

iulianav commented Nov 29, 2017 •

edited by michamos

Loading

iulianav Dec 7, 2017 •

edited

Loading

iulianav Dec 8, 2017 •

edited

Loading

michamos Dec 6, 2017 •

edited

Loading

harunurhan commented Dec 8, 2017 •

edited

Loading

michamos Dec 8, 2017 •

edited

Loading

iulianav commented Dec 8, 2017 •

edited

Loading

jacquerie commented Dec 11, 2017 •

edited

Loading

iulianav commented Jan 31, 2018 •

edited

Loading