-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support batch_size per class #9
base: main
Are you sure you want to change the base?
Conversation
fd772e1
to
6c23a4a
Compare
fcbbf81
to
3144c2e
Compare
62f6643
to
c5f6ee3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah man turns out I forgot to hit the review button.... anyway - the batch changes look all good but I'm wondering about the cooldown. I'm a bit worried this will block a queue for potentially a long time and we could leverage queued jobs for this instead?
f624dd0
to
5fbe31a
Compare
5fbe31a
to
062eaaa
Compare
062eaaa
to
97cfe2c
Compare
97cfe2c
to
987de78
Compare
Converting back to draft because I've had a thought that I'd like to look further in to.
Edit, after some investigation:
I think the feature (as we have it right now in this PR) is a good middleground that doesn't cause any breaking changes. I like the way that this new feature configures batch sizes per class, and I think that would persist (even if we move where the batching occurs) Raised a follow up Issue: |
5693c1f
to
f35c662
Compare
fc64778
to
4a6b23c
Compare
5261e5d
to
7dc5d9f
Compare
7dc5d9f
to
177030b
Compare
Batch size by class and index
Different classes can require different levels of server resource to process, for example, Files are often more process intensive than (say) Pages.
Provide developers with a way to define a
batch_size
to be used when re-indexing batches of certain classes.Unit test changes
I'm sure they were fine before, but they used features I don't really understand - so, some have been updated to use test mechanisms that I understand.