Skip to content

feat: Add deduplication to add_batch_of_requests #534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 56 commits into
base: master
Choose a base branch
from
Open

feat: Add deduplication to add_batch_of_requests #534

wants to merge 56 commits into from

Conversation

Pijukatel
Copy link
Contributor

@Pijukatel Pijukatel commented Aug 7, 2025 •
edited
Loading

Description

  • Ensure that already known requests are excluded from api_client.batch_add_requests calls to avoid expensive and pointless API calls.
  • Add all new requests to the cache when calling batch_add_requests.
  • Add test with real API usage measurement.

Issues

Testing

  • Added new integration tests to verify reduced API usage.
  • Comparing benchmark actor based on master vs this PR. Actor is a simple ParselCrawler that crawls the whole crawlee.dev, which contains many duplicate links, as the documentation is cross-linked thoroughly. Results:
    • Massive reduction of cost from request queue.
    • Significant overall speed up due to reduced API calls.
image

vdusek and others added 30 commits June 26, 2025 08:22
@vdusek
Rm old Apify storage clients
5c437c9
@vdusek
Add init version of new Apify storage clients
bf55338
@vdusek
Move specific models from Crawlee to SDK
6b2f82b
@vdusek
Adapt to Crawlee v1
38bef68
@vdusek
Adapt to Crawlee v1 (p2)
1f85430
@vdusek
Fix default storage IDs
a3d68a2
@vdusek
Fix integration test and Not implemented exception in purge
c77e8d5
@vdusek
Fix unit tests
8731aff
@vdusek
fix lint
8dfaffb
@vdusek
add KVS record_exists not implemented
53fad07
@vdusek
update to apify client 1.12 and implement record exists
5869f8e
@vdusek
Move default storage IDs to Configuration
82e65fc
@vdusek
opening storages get default id from config
8de950b
@vdusek
Addressing more feedback
98b76c5
@vdusek
Fixing integration test test_push_large_data_chunks_over_9mb
7b5ee07
@vdusek
Abstract open method is removed from storage clients
afcb8c7
@vdusek
fixing generate public url for KVS records
3bacab7
@vdusek
add async metadata getters
287a119
@vdusek
Merge branch 'master' into new-apify-storage-clients
e45d65b
@vdusek
better usage of apify config
51178ca
@vdusek
renaming
3cd7dfe
@vdusek
Merge branch 'master' into new-apify-storage-clients
6fe9eb3
@vdusek
fixes after merge commit
1547cbd
@vdusek
Merge branch 'master' into new-apify-storage-clients
bb47efc
@Pijukatel
Change from orphan commit to master in crawlee version
4e4fa93
@vdusek
Merge branch 'master' into new-apify-storage-clients
683cb31
@vdusek
fix encrypted secrets test
e5b2bc4
@vdusek
Add Apify's version of FS client that keeps the INPUT json
638756f
@vdusek
update metadata fixes
931b0ca
@vdusek
Merge branch 'master' into new-apify-storage-clients
ad7c0d8
@Pijukatel Pijukatel changed the title Add deduplication to add_batch_of_requests and test feat: Add deduplication to add_batch_of_requests and test Aug 7, 2025
@Pijukatel Pijukatel added enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team. labels Aug 7, 2025
@Pijukatel Pijukatel changed the title feat: Add deduplication to add_batch_of_requests and test feat: Add deduplication to add_batch_of_requests Aug 7, 2025
Base automatically changed from new-apify-storage-clients to master August 12, 2025 16:45
@github-actions github-actions bot added this to the 121st sprint - Tooling team milestone Aug 13, 2025
@github-actions github-actions bot added the tested Temporary label used only programatically for some analytics. label Aug 13, 2025
@Pijukatel Pijukatel requested review from vdusek and Mantisus August 13, 2025 12:48
@Pijukatel Pijukatel marked this pull request as ready for review August 13, 2025 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vdusek vdusek Awaiting requested review from vdusek

@Mantisus Mantisus Awaiting requested review from Mantisus

At least 1 approving review is required to merge this pull request.

Assignees

@Pijukatel Pijukatel

Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.
Projects
None yet
Milestone
121st sprint - Tooling team
Development

Successfully merging this pull request may close these issues.

Ensure that duplicate links are handled in a cost effective way when using Apify RequestQueue
2 participants
@Pijukatel @vdusek