refactor: remove unmaintained surf for reqwest

[KolbyML]

What was wrong?

fixes #1446 surf has been unmaintained for over 2 years now, it is also using a old version of packages which prevents us from compiling Trin to Risc-v #1444

How was it fixed?

Just convert surf -> reqwest equivalents, which was pretty straight forward

[morph-dev]

Mostly looks good, but there is one big issue in portal-bridge (see comment).

Also, light-client crate also depends on reqwest, so we should update that dependency (should use workspace version)

[morph-dev]

The surf client supported setting base_url, while reqwest doesn't. Based on the name of this function, I think that's the expected behavior. In fact, all uses of this function (consensus.rs and execution.rs) now only pass relative path, and they won't work.

Since this function is doing more than just setting base_url and correct headers (in case of pandaops), I would say we should do following:

• have fn create_client() -> Result<ClientWithMiddleware, String> that does everything that this function do, except setting pandaops headers.
• have different function that accepts &mut Request. that adds pandaops headers if appropriate

Something like:

fn add_pandaops_headers(request: &mut Request) {
    let host = request.url().host_str().unwrap_or("");
    if !host.contains("pandaops.io") {
        return;
    }
    let client_id = ...;
    request.headers_mut().insert("CF-Access-Client-Id", ...);
    let client_secret = ...;
    request.headers_mut().insert("CF-Access-Client-Secret", ...);
}

// and in other places:
let client = create_client()?;
let mut request = client.post("some-url").build()?;
add_pandaops_headers(&mut request);
let result = client.execute().await?;

Alternatively, we can have a function combines last 3 lines into something like this:

async fn add_pandaops_headers_and_execute(request_builder: RequestBuilder) -> Result<Response, Error> {
    let (client, request) = request_builder.build_split();

    //
    // setting pandaops headers
    //

     client.execute().await
}

[KolbyML]

I will just implement base_url for Reqwest

[njgheorghita]

Comment on line 276 should be removed, and probably some testing to make sure that reqwest can handle the async load, though it's a very popular library so I expect it will

[KolbyML]

@njgheorghita @morph-dev ready for another review. I believe I resolved all the PR concerns


Here are a few pictures of history latest nothing seems wrong

Running backfill using infura I see what you mean nick when you said shisui uTP failures. It looks good as well

[njgheorghita]

Cool. I'm guessing this comment predates era1 files, so back then when in backfill mode, we'd be making quite a bit of parallel http requests. But, now that we have era1 files (and plan to use some version of era files for everything from merge -> latest?) I actually don't think this is much of an issue anymore. Unless there's somewhere else in the bridge that I'm missing where we'll have large amounts of parallel http requests....

[KolbyML]

large amounts of parallel http requests

I just ran backfill using infura as I said before, so it was making a large amount of parallel http requests. Reqwest's get 100 times more downloads a month than surf and is also built on hyper so I don't see why it wouldn't be able to handle large amounts of parallel http requests

[morph-dev]

LGTM. But wait for Nick's approval as well.

[ogenev]

LGTM