-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix!: merge payload and data fields of Request #542
base: master
Are you sure you want to change the base?
Conversation
JS version have only |
Also, since this is breaking, maybe we should wait, and merge it once we have more breaking changes than a simple one like this (or some feature worth the minor bump). |
9d83dcb
to
49243db
Compare
I learned about |
It also says "Alternatively, you can send form data as URL parameters using the query_params argument." btw, not sure if that is something related or not. |
We will have the first version of fingerprints for Playwright soon 🤞.
Thanks. Of course, I forgot to update the docs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Just a couple of questions...
c04ecb2
to
414ac66
Compare
649eaa5
to
4c3d891
Compare
I've separated the Curl impersonation fix and other minor updates into a separate PR (#543), allowing us to release a patch version for 0.3. The |
### Description - Version 1.7.2 of `curl-cffi` introduces breaking changes. - This update includes adopting the new version, adding type aliases for `Request` fields, and incorporating other minor changes from PR #542. ### Issues - N/A (ad-hoc fix) ### Testing - The current set of unit tests should cover the changes. ### Checklist - [x] CI passed
4c3d891
to
f0ea79c
Compare
46e1e5e
to
4078303
Compare
5f7ac3d
to
62b332e
Compare
### Description - Unify the HTTP headers type across the project. - ~I used just the type alias plus custom validator in the `Request` model to reach the same result (lowercase & sorted).~ - ~Of course, now it is only in the `Request`, but I believe it is not a problem.~ - Take it out from #542. ### Issues - N/A ### Testing - N/A ### Checklist - [x] CI passed
62b332e
to
c371b29
Compare
c371b29
to
d1b9abb
Compare
A new model for |
d1b9abb
to
2752643
Compare
2752643
to
260acd1
Compare
async def request_handler(context: HttpCrawlingContext) -> None: | ||
response = json.loads(context.http_response.read()) | ||
# The httpbin.org/post endpoint returns the form data in the response. | ||
assert response['form'] == form_data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you also assert that the request_handler was called?
@@ -127,15 +127,17 @@ class BaseRequestData(BaseModel): | |||
method: HttpMethod = 'GET' | |||
"""HTTP request method.""" | |||
|
|||
headers: Annotated[HttpHeaders, Field(default_factory=HttpHeaders())] = HttpHeaders() | |||
headers: Annotated[HttpHeaders, Field(default_factory=HttpHeaders)] = HttpHeaders() | |||
"""HTTP request headers.""" | |||
|
|||
query_params: Annotated[HttpQueryParams, Field(alias='queryParams', default_factory=dict)] = {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably get rid of this one, or make it a property that changes the url
field. Unfortunately, Apify API does not accept queryParams
.
Nice work @vdusek, and sorry for taking multiple rounds of review to pinpoint all issues. |
Description
data
andpayload
fields on theRequest
model.payload
was not being provided to the HTTP clients, only thedata
field.In this PR, I'm merging them together, keeping only thedata
field (use the same naming asHTTPX
&CurlImpersonate
).payload
field (use the same naming as in JS Crawlee).I also defined type aliases for HTTP data and HTTP query parameters and used them across the project.Some struggle with Pydantic & serialization, but should be OK now...Issues
Testing
Checklist