add is_incomplete_or_stale to the search index

[abeglova]

What are the relevant tickets?

https://github.com/mitodl/hq/issues/5658

Description (What does it do?)

This adds is_incomplete_or_stale as a field to the search. It will be used as a sort term in the default sort so that stale or incomplete resources are not shown at the top of the results when there is no search term. For now this only adds the field, the sort will be updated in a followup pr after we reindex rc and prod

How can this be tested?

run ./manage.py recreate_index --all
Comment out line 420 in learning_resources_search/constants.py so is_incomplete_or_stale is shown in the results

run ./manage.py backpopulate_ocw_data --course-name sp-248-neet-ways-of-thinking-fall-2023
go to http://api.open.odl.local:8063/api/v1/learning_resources_search/?q=%22NEET%20ways%20of%20thinking%22
The is_incomplete_or_stale should be true for the resource

run ./manage.py backpopulate_ocw_data --course-name 21m-302-harmony-and-counterpoint-ii-spring-2005
go to http://localhost:8063/api/v1/learning_resources_search/?q=%22Harmony%20and%20Counterpoint%20II%22
The is_incomplete_or_stale should be true for the resource

run ./manage.py backpopulate_ocw_data 6-100l-introduction-to-cs-and-programming-using-python-fall-2022
got to http://api.open.odl.local:8063/api/v1/learning_resources_search/?q=%22Introduction%20to%20CS%20and%20Programming%20using%20Python%22
The is_incomplete_or_stale should be false for the resource

[shanbady]

@abeglova I wasnt seeing is_incomplete_or_stale show up in the search endpoint via GET. for good measure i tried this in a codespace as well. does it need to get added to the SearchResponseSerializer as a field?

[shanbady]

Still seeing something very strange locally and I'm not sure where it stems from yet. I am now able to see "is_complete_or_stale" when going to the search api results - however - it seems that some resources have this defined and some dont. I added a print statement to see what opensearch was directly giving me and here is an opensearch response where one resource does have this field and the other resource doesnt

{'took': 55, 'timed_out': False, '_shards': {'total': 14, 'successful': 14, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 2, 'relation': 'eq'}, 'max_score': 50.92838, 'hits': [

{'_index': 'micromasters_program_f3ef0941645440ac9008e1779bcdaf30', '_id': '79', '_score': 50.92838, '_source': {'continuing_ed_credits': None, 'readable_id': 'micromasters-program-4', 'description': None, 'program': {'course_count': 0}, 'availability': 'dated', 'completeness': 1.0, 'title': 'Statistics and Data Science', 'platform': {'code': 'edx', 'name': 'edX'}, 'professional': False, 'full_description': None, 'id': 79, 'departments': [], 'prices': ['1500.00'], 'free': False, 'last_modified': None, 'course_feature': [], 'views': 0, 'image': {'alt': None, 'description': None, 'id': 77, 'url': 'https://d34pclujt4iir0.cloudfront.net/images/idss_thumb.690x530.2e16d0ba.fill-300x186.jpg'}, 'delivery': [{'code': 'online', 'name': 'Online'}], 'languages': None, 'learning_path_parents': [], 'topics': [], 'pace': [], 'format': [{'code': 'asynchronous', 'name': 'Asynchronous'}], 'resource_type': 'program', 'is_incomplete_or_stale': False, 'certification_type': {'code': 'micromasters', 'name': 'Micromasters Credential'}, 'offered_by': {'code': 'mitx', 'name': 'MITx', 'channel_url': 'http://open.odl.local:8062/c/unit/mitx/'}, 'published': True, 'ocw_topics': [], 'url': 'http://micromasters.mit.edu/ds/', 'certification': True, 'license_cc': False, 'resource_category': 'program', 'next_start_date': None, 'runs': [{'end_date': None, 'year': None, 'enrollment_end': None, 'description': None, 'availability': 'dated', 'title': 'Statistics and Data Science', 'instructors': [{'full_name': 'Devavrat Shah', 'last_name': None, 'id': 99, 'first_name': None}, {'full_name': 'Patrick Jaillet', 'last_name': None, 'id': 100, 'first_name': None}, {'full_name': 'John Tsitsiklis', 'last_name': None, 'id': 101, 'first_name': None}, {'full_name': 'Philippe Rigollet', 'last_name': None, 'id': 102, 'first_name': None}, {'full_name': 'Regina Barzilay', 'last_name': None, 'id': 103, 'first_name': None}, {'full_name': 'Tommi S. Jaakkola', 'last_name': None, 'id': 104, 'first_name': None}, {'full_name': 'Stefanie Jegelka', 'last_name': None, 'id': 105, 'first_name': None}, {'full_name': 'Caroline Uhler', 'last_name': None, 'id': 106, 'first_name': None}, {'full_name': 'Esther Duflo', 'last_name': None, 'id': 29, 'first_name': None}, {'full_name': 'Sara Fisher Ellison', 'last_name': None, 'id': 30, 'first_name': None}], 'full_description': None, 'checksum': None, 'id': 79, 'prices': ['1500.00'], 'last_modified': None, 'slug': None, 'start_date': None, 'image': None, 'delivery': [{'code': 'online', 'name': 'Online'}], 'run_id': 'micromasters-program-4', 'languages': None, 'level': [], 'pace': [], 'format': [{'code': 'asynchronous', 'name': 'Asynchronous'}], 'published': True, 'url': None, 'enrollment_start': None, 'semester': None}], 'user_list_parents': []}}, 

{'_index': 'micromasters_course_594f2d3543ad4e729c164229cd262de1', '_id': '507', '_score': 4.5644984, '_source': {'continuing_ed_credits': None, 'readable_id': '18.650+fall_2016', 'description': 'This course offers an in-depth the theoretical foundations for statistical methods that are useful in many applications. The goal is to understand the role of mathematics in the research and development of efficient statistical methods.', 'availability': 'anytime', 'completeness': 0.6500000000000001, 'title': 'Statistics for Applications ', 'platform': {'code': 'ocw', 'name': 'MIT OpenCourseWare'}, 'professional': False, 'full_description': None, 'course': {'course_numbers': [{'listing_type': 'primary', 'department': {'department_id': '18', 'school': None, 'name': 'Mathematics', 'channel_url': None}, 'value': '18.650'}, {'listing_type': 'cross-listed', 'department': {'department_id': '18', 'school': None, 'name': 'Mathematics', 'channel_url': None}, 'value': '18.6501'}]}, 'id': 507, 'departments': [{'department_id': '18', 'school': {'name': 'School of Science', 'id': 5, 'url': 'https://science.mit.edu/'}, 'name': 'Mathematics', 'channel_url': 'http://open.odl.local:8062/c/department/mathematics/'}], 'prices': [], 'free': True, 'last_modified': '2024-09-12T10:13:35Z', 'course_feature': ['Lecture Notes', 'Lecture Videos', 'Problem Sets'], 'views': 0, 'image': {'alt': '2D plot for data fitting.', 'description': '', 'id': 502, 'url': 'https://ocw.mit.edu/courses/18-650-statistics-for-applications-fall-2016/de1506da626801272f3f35bb8412e638_18-650f16.jpg'}, 'delivery': [{'code': 'online', 'name': 'Online'}], 'languages': None, 'learning_path_parents': [], 'topics': [], 'pace': [{'code': 'self_paced', 'name': 'Self-paced'}], 'format': [{'code': 'asynchronous', 'name': 'Asynchronous'}], 'resource_type': 'course', 'certification_type': {'code': 'none', 'name': 'No Certificate'}, 'offered_by': {'code': 'ocw', 'name': 'MIT OpenCourseWare', 'channel_url': 'http://open.odl.local:8062/c/unit/ocw/'}, 'published': True, 'ocw_topics': ['Mathematics', 'Probability and Statistics'], 'url': 'https://ocw.mit.edu/courses/18-650-statistics-for-applications-fall-2016', 'certification': False, 'license_cc': True, 'resource_category': 'course', 'next_start_date': None, 'runs': [{'end_date': None, 'year': 2016, 'enrollment_end': None, 'description': 'This course offers an in-depth the theoretical foundations for statistical methods that are useful in many applications. The goal is to understand the role of mathematics in the research and development of efficient statistical methods.', 'availability': 'anytime', 'title': 'Statistics for Applications ', 'instructors': [{'full_name': 'Prof. Philippe Rigollet', 'last_name': 'Rigollet', 'id': 673, 'first_name': 'Philippe'}], 'full_description': None, 'checksum': None, 'id': 507, 'prices': [], 'last_modified': '2024-09-12T10:13:35Z', 'slug': 'courses/18-650-statistics-for-applications-fall-2016', 'start_date': None, 'image': {'alt': '2D plot for data fitting.', 'description': '', 'id': 502, 'url': 'https://ocw.mit.edu/courses/18-650-statistics-for-applications-fall-2016/de1506da626801272f3f35bb8412e638_18-650f16.jpg'}, 'delivery': [{'code': 'online', 'name': 'Online'}], 'run_id': '56d18bbf4ea5ace047883cafba9043da', 'languages': None, 'level': [{'code': 'undergraduate', 'name': 'Undergraduate'}], 'pace': [{'code': 'self_paced', 'name': 'Self-paced'}], 'format': [{'code': 'asynchronous', 'name': 'Asynchronous'}], 'published': True, 'url': 'https://ocw.mit.edu/courses/18-650-statistics-for-applications-fall-2016', 'enrollment_start': None, 'semester': 'Fall'}], 'user_list_parents': []}}]}, 'suggest': {'description.trigram': [{'text': 'Statistics and Data Science', 'offset': 0, 'length': 27, 'options': [{'text': 'statistics and data science', 'score': 4.3255448e-05, 'collate_match': False}]}], 'title.trigram': [{'text': 'Statistics and Data Science', 'offset': 0, 'length': 27, 'options': [{'text': 'statistics and data science', 'score': 0.020119302, 'collate_match': True}]}

]}} 

is there any way for some resources to not have this attribute at all?

[shanbady]

resolving some failed indexing jobs got it to work 👍