feat: Nested optimization for lists and connections #540

bellini666 · 2024-06-02T00:41:28Z

This is going to enable optimization of nested relations even when using filters/ordering/pagination, which previously would mean throwing the results away and fetching new ones, causing more than n+1 issues.

Note: Although nested connections are being optimized as well, we are only doing that for those annotated with ListConnection and ListConnectionWithTotalCount, as custom connections may contain more complex code which are not safe to be optimized the way we are doing here.

We are also dropping support for Django versions older than 4.2 as we cannot filter over window functions in older versions. That is also a recommendation from Django itself when 5.0 was released: https://docs.djangoproject.com/en/5.0/releases/5.0

Fix #337
Fix #340
Fix #345
Fix #389
Fix #521

sourcery-ai

Hey @bellini666 - I've reviewed your changes and they look great!

Here's what I looked at during the review

🟡 General issues: 7 issues found
🟢 Security: all looks good
🟡 Testing: 3 issues found
🟢 Complexity: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

strawberry_django/optimizer.py

strawberry_django/relay.py

strawberry_django/pagination.py

strawberry_django/fields/types.py

tests/test_optimizer.py

tests/filters/test_filters.py

strawberry_django/optimizer.py

codecov-commenter · 2024-06-02T00:44:31Z

Codecov Report

Attention: Patch coverage is 88.98305% with 13 lines in your changes missing coverage. Please review.

Project coverage is 88.67%. Comparing base (eec919c) to head (305cacb).

Files	Patch %	Lines
strawberry_django/optimizer.py	83.67%	8 Missing ⚠️
strawberry_django/relay.py	89.65%	3 Missing ⚠️
strawberry_django/fields/field.py	80.00%	1 Missing ⚠️
strawberry_django/pagination.py	95.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #540      +/-   ##
==========================================
+ Coverage   88.60%   88.67%   +0.07%     
==========================================
  Files          41       41              
  Lines        3439     3524      +85     
==========================================
+ Hits         3047     3125      +78     
- Misses        392      399       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

strawberry_django/fields/field.py

strawberry_django/optimizer.py

strawberry_django/pagination.py

patrick91 · 2024-06-02T15:56:39Z

strawberry_django/pagination.py

+    if limit >= 0:
+        queryset = queryset.filter(_strawberry_row_number__lte=offset + limit)


should we enforce a limit?

This is just adding support to do the pagination from the optimizer.

I agree that we maybe should enforce a limit, but I'd do that in another PR focused on that

patrick91 · 2024-06-02T15:57:06Z

strawberry_django/pagination.py


    def get_queryset(
        self,
        queryset: _QS,
        info: Info,
        *,
        pagination: Optional[object] = None,
+        _strawberry_related_field_id: Optional[str] = None,


why is this private? 😊

This is passed in **kwargs. Wanted to make sure this would not conflict at all with some user-given argument in the resolver.

But maybe the leading strawberry_ is enough? What do you think?

probably, but it is fine to keep the underscore

strawberry_django/relay.py

strawberry_django/fields/field.py

strawberry_django/optimizer.py

patrick91 · 2024-06-02T21:30:51Z

strawberry_django/pagination.py


    def get_queryset(
        self,
        queryset: _QS,
        info: Info,
        *,
        pagination: Optional[object] = None,
+        _strawberry_related_field_id: Optional[str] = None,


probably, but it is fine to keep the underscore

Eraldo · 2024-06-04T13:34:56Z

Note: Although nested connections are being optimized as well, we are only doing that for those annotated with ListConnection and ListConnectionWithTotalCount, as custom connections may contain more complex code which are not safe to be optimized the way we are doing here.

Does this also apply to connections that inherit from ListConnection or ListConnectionWithTotalCount? 🤔

strawberry-graphql/strawberry-django#540

Eraldo · 2024-06-05T11:26:11Z

@bellini666 As you suggested, I tried using this PR.
I upgraded my minimal reproduction and it does not solve issue: #535

bellini666 · 2024-06-06T17:22:14Z

Note: Although nested connections are being optimized as well, we are only doing that for those annotated with ListConnection and ListConnectionWithTotalCount, as custom connections may contain more complex code which are not safe to be optimized the way we are doing here.

Does this also apply to connections that inherit from ListConnection or ListConnectionWithTotalCount? 🤔

Yes, because it is not safe if you are doing customizations, as the optimizer will actually change what nodes is

But if you are doing FruitConnection = ListConnectionWithTotalCount[Fruit] then you are safe.

But if you still need to optimize a connection subclass, we can think of a way of doing that... maybe by defining a ALLOW_UNSAFE_OPTIMIZATIONS = True in the class?

patrick91

🔥

bellini666 self-assigned this Jun 2, 2024

bellini666 requested a review from patrick91 June 2, 2024 00:41

sourcery-ai bot reviewed Jun 2, 2024

View reviewed changes

bellini666 force-pushed the nested_optimizer branch from 74e9f84 to 3faaa30 Compare June 2, 2024 00:55

patrick91 reviewed Jun 2, 2024

View reviewed changes

bellini666 force-pushed the nested_optimizer branch from 3faaa30 to 5e18fb5 Compare June 2, 2024 20:44

bellini666 requested a review from patrick91 June 2, 2024 20:47

patrick91 reviewed Jun 2, 2024

View reviewed changes

Eraldo added a commit to Eraldo/strawberry-django-lazy-connection-missing-filters-mre that referenced this pull request Jun 5, 2024

feat: Upgrade to pr and add pytest

5927a8d

strawberry-graphql/strawberry-django#540

bellini666 force-pushed the nested_optimizer branch from 5e18fb5 to 6747fa3 Compare June 6, 2024 17:30

bellini666 requested a review from patrick91 June 6, 2024 17:31

bellini666 force-pushed the nested_optimizer branch 2 times, most recently from 70d1032 to 26ac62f Compare June 6, 2024 17:35

feat: Nested optimization for lists and connections

305cacb

bellini666 force-pushed the nested_optimizer branch from 26ac62f to 305cacb Compare June 7, 2024 14:51

patrick91 approved these changes Jun 10, 2024

View reviewed changes

bellini666 merged commit eecec51 into main Jun 10, 2024
22 checks passed

bellini666 deleted the nested_optimizer branch June 10, 2024 16:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Nested optimization for lists and connections #540

feat: Nested optimization for lists and connections #540

bellini666 commented Jun 2, 2024 •

edited

Loading

sourcery-ai bot left a comment

codecov-commenter commented Jun 2, 2024 •

edited

Loading

patrick91 Jun 2, 2024

bellini666 Jun 2, 2024

patrick91 Jun 2, 2024

bellini666 Jun 2, 2024

patrick91 Jun 2, 2024

patrick91 Jun 2, 2024

Eraldo commented Jun 4, 2024

Eraldo commented Jun 5, 2024 •

edited

Loading

bellini666 commented Jun 6, 2024

patrick91 left a comment

		if limit >= 0:
		queryset = queryset.filter(_strawberry_row_number__lte=offset + limit)

feat: Nested optimization for lists and connections #540

feat: Nested optimization for lists and connections #540

Conversation

bellini666 commented Jun 2, 2024 • edited Loading

sourcery-ai bot left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 2, 2024 • edited Loading

Codecov Report

patrick91 Jun 2, 2024

Choose a reason for hiding this comment

bellini666 Jun 2, 2024

Choose a reason for hiding this comment

patrick91 Jun 2, 2024

Choose a reason for hiding this comment

bellini666 Jun 2, 2024

Choose a reason for hiding this comment

patrick91 Jun 2, 2024

Choose a reason for hiding this comment

patrick91 Jun 2, 2024

Choose a reason for hiding this comment

Eraldo commented Jun 4, 2024

Eraldo commented Jun 5, 2024 • edited Loading

bellini666 commented Jun 6, 2024

patrick91 left a comment

Choose a reason for hiding this comment

bellini666 commented Jun 2, 2024 •

edited

Loading

codecov-commenter commented Jun 2, 2024 •

edited

Loading

Eraldo commented Jun 5, 2024 •

edited

Loading