Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

possible to perform a combined req_perform_sequential together with req_perform_iterative? #568

Closed
deschen1 opened this issue Oct 21, 2024 · 1 comment

Comments

@deschen1
Copy link

httr2 offers options for paginated requests (req_perform_iterative) as well as requests that need to be structured as a list (request_sequential). For the latter one, a common use case example would be an API that provides data from countries and a single API call accepts only one country in the request body.

I can build up such a request "loop" and fetch the data as follows:

countries <- c("DE", "US", "FR")

req_countries <- map(.x = countries,
                     .f = ~request("https://apis.test.com") |>
                       req_headers("Accept" = "application/json",
                                   'auth1' = "abc",
                                   'apikey' = "xyz"))

resp_countries <- req_perform_sequential(req = req_countries)

However, imagine the API uses pagination for each country, i.e. each response has several pages. I can use the above request and map it with req_perform_iterative:

is_complete <- function(resp)
{
  resp_body_json(resp)$pagination$hasNext == FALSE
}

resp_countries <- req_countries |> 
  map(.f = ~req_perform_iterative(req = .x,
                                  next_req = iterate_with_offset(param_name = "pageNumber",
                                                                 start = 0,
                                                                 offset = 1,
                                                                 resp_complete = is_complete),
                                  max_reqs = Inf,
                                  on_error = "return"))

So I need to map twice, once for building the request and then again when fetching the data.

I'm wondering if there is a way or if it would be a valuable new feature for httr2 to offer a combined function, e.g. by adding the req_perform_iterative functionality directly in req_perform_sequential?

So that the call could look like:

resp_countries <- req_countries |> 
  req_perform_sequential(req = req_countries,
                         iterative = TRUE,
                         next_req = iterate_with_offset(param_name = "pageNumber",
                                                        start = 0,
                                                        offset = 1,
                                                        resp_complete = is_complete),
                         max_reqs = Inf,
                         on_error = "return")
@hadley
Copy link
Member

hadley commented Oct 21, 2024

This would be a special case of #456

@hadley hadley closed this as completed Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants