Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data rearchitecture] Stop creating complete universe of article course timeslices #6069

Conversation

gabina
Copy link
Member

@gabina gabina commented Dec 18, 2024

What this PR does

This PR stops calling create_timeslices_for_new_article_course_records when new articles are ingested. Having the complete universe of article course timeslices for courses with a huge number of articles implies a lot of disk space that is not useful since most timeslices are empty. For this reason, this PR implements a new approach, in which article course timeslices are only created if they are non-empty (i.e. when there is a revision for that article and that date). This new approach should considerably decrease the disk space used in the data-rearchitecture instance.

Open questions and concerns

This approach could be replicated for course user wiki timeslices in the future, although the course_user_wiki_timeslices table does not appear to take up considerable space.

…meslices. Instead of that, we only create ac timeslices on demand, based on ingested revisions. This is to avoid using a huge amount of disk space for article course timeslices table, when most rows are actually empty.
@gabina
Copy link
Member Author

gabina commented Dec 19, 2024

Failing specs are not related to changes.

Failures:

  1) Assigned Articles view lets users submit feedback about articles
     Failure/Error: find('textarea.feedback-form').fill_in with: 'This is a great article!'

     Capybara::ElementNotFound:
       Unable to find css "textarea.feedback-form"
     # ./spec/features/assigned_articles_spec.rb:26:in `block (3 levels) in <top (required)>'
     # ./spec/features/assigned_articles_spec.rb:22:in `block (2 levels) in <top (required)>'

@gabina gabina marked this pull request as ready for review December 19, 2024 02:51
@gabina gabina changed the title [WIP] [Data rearchitecture] Stop creating complete universe of article course timeslices [Data rearchitecture] Stop creating complete universe of article course timeslices Dec 19, 2024
@gabina gabina merged commit 8c4862a into WikiEducationFoundation:data-rearchitecture-for-dashboard Dec 19, 2024
1 check passed
@gabina gabina deleted the data-rearchitecture-stop-creating-timeslices-for-all-articles branch December 19, 2024 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant