Skip to content

fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler #1496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Oct 23, 2025
Merged

fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler #1496

merged 5 commits into from
Oct 23, 2025

Conversation

@Mantisus
Copy link
Collaborator

@Mantisus Mantisus commented Oct 17, 2025 •
edited
Loading

Description

  • Remove the use of deepcopy for kwargs in AdaptivePlaywrightCrawler for compatibility with SqlStorageClient

Issues

@Mantisus Mantisus self-assigned this Oct 17, 2025
@Mantisus Mantisus requested review from janbuchar and vdusek October 17, 2025 00:24
@vdusek vdusek changed the title fix: Add special deepcopy logic for SqlStorageClient for compatibility with AdaptivePlaywrightCrawler. fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler Oct 17, 2025
vdusek
vdusek reviewed Oct 18, 2025
View reviewed changes
@Mantisus Mantisus requested a review from vdusek October 19, 2025 01:46
vdusek
vdusek approved these changes Oct 19, 2025
View reviewed changes
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, lgtm

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 19, 2025
@janbuchar
Copy link
Collaborator

Huh. Huh. While this does fix the issue, I think the root cause is the unfortunate way deepcopy is used when constructing the subcrawlers in adaptive crawler. Could we fix that instead?

janbuchar
janbuchar requested changes Oct 22, 2025
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comment above

@Mantisus
Copy link
Collaborator Author

Huh. Huh. While this does fix the issue, I think the root cause is the unfortunate way deepcopy is used when constructing the subcrawlers in adaptive crawler. Could we fix that instead?

Updated.

@Pijukatel could you take a look at this as well? You are the most familiar with the AdaptivePlaywrightCrawler. I just want to make sure I'm not missing any important cases where deepcopy was used.

janbuchar
janbuchar reviewed Oct 22, 2025
View reviewed changes
@Mantisus Mantisus requested a review from janbuchar October 22, 2025 15:54
janbuchar
janbuchar approved these changes Oct 22, 2025
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@janbuchar
Copy link
Collaborator

Perhaps we could add a test that shows that adaptive crawler can be instantiated with sql storage client, that wouldn't be too hard 🙂

@Pijukatel Pijukatel merged commit ce172c4 into apify:master Oct 23, 2025
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdusek vdusek vdusek approved these changes

@janbuchar janbuchar janbuchar approved these changes

@Pijukatel Pijukatel Pijukatel approved these changes

Assignees

@Mantisus Mantisus

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Crawler's storage_client errors when using SqlStorageClient

4 participants

@Mantisus @janbuchar @vdusek @Pijukatel