How can I post with chunked data?

[ysak-y]

Overview

Hi, I love this crate, thank you for providing eco-system to implement firmware by Rust.
btw, now I'm worrying how I send POST request with chunked data that is achieved by using Transfer-Encoding: chunked header and without Content-Length header basically.

Goal

I want to write data multiple times for the request as follow.

    let mut request = client.post(url, &headers).unwrap();
    request.write(b"a").unwrap();
    request.write(b"b").unwrap();
    request.write(b"c").unwrap();
    request.flush().unwrap(); -> Send request with "abc" payload.

Problem

To send chunked data, I want to add Transfer-Encoding: chunked to the header and don't want to add Content-Length because I can't measure the total size of chunked data while sending requests.
But it seems this crate add Content-Length with 0 value if I don't set it to the header.

Example is as follows.

    let payload = b"{\"hello\": \"world\"}";

    // Prepare headers and URL
    let headers = [
        ("content-type", "application/json"),
        ("Transfer-Encoding", "chunked"),
    ];
    let url = "http://192.168.86.48";
    let mut client = Client::wrap(EspHttpConnection::new(&Default::default()).unwrap());

    // Send request
    let mut request = client.post(url, &headers).unwrap();
    request.write(payload).unwrap();

    request.flush().unwrap();
    request.submit().unwrap();

Above implementation sends follow request (I captured it by Wireshark). It has Content-Length: 0 in its header.

POST / HTTP/1.1
User-Agent: ESP32 HTTP Client/1.0
Host: 192.168.86.48
content-type: application/json
Transfer-Encoding: chunked
Content-Length: 0

HTTP/1.1 400 Bad Request
Connection: close

{"hello": "world"}

Solution

Don't add Content-Length: 0 to the header even if there is no Content-Length. Or maybe some other ways to send request with chunked data?

[ysak-y]

I guess this line sets Content-Length to 0.
https://github.com/esp-rs/esp-idf-svc/blob/master/src/http/client.rs#L249

This value is used for esp_http_client_open method, maybe I need to investigate about it.
https://github.com/esp-rs/esp-idf-svc/blob/master/src/http/client.rs#L251

[Vollbrecht]

If i understand correctly the open method just wants one parameter that sets an upper limit on a request size for all requests. So it probably internally allocate space for it. We still want the option to set a content length of 0 to indicate a read only connection, but you may can patch it to not unwrap to 0 but rather unwrap to some max_value like an MTU size or such.

In the case the user wants 0 length he can just pass ["content-length","0"] and in the othere case where it should not be set one could use ["content-length",""] and filter it out here

You can simply clone this repository and patch your project with your cloned variant with

[patch.crates-io]
esp-idf-svc = { path = "../path/to/esp-idf-svc" }

insides your project Cargo.toml and running cargo update
to test it out

[ysak-y]

Thanks for your reply @Vollbrecht !! I'm happy to implement patch if I can. But I want to understand actual specification of it before implement.

As you say, it may work by just setting larger value to content-length because it is max size I can write. But it isn't what I'm looking for because it isn't streaming post.
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_http_client.html#http-stream

I investigated implementation of esp_http_client_open, content-length value (= write_len in esp_http_client_open) is passed to esp_http_client_request_send
https://github.com/espressif/esp-idf/blob/b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e/components/esp_http_client/esp_http_client.c#L1553

And then, it is passed to http_client_prepare_first_line.
https://github.com/espressif/esp-idf/blob/b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e/components/esp_http_client/esp_http_client.c#L1457

Finally, it is used as follows. This implementation shows library sets Content-Length with write_len value is it is larger than 0, otherwise, sets Transfer-Encoding: chunked what I'm looking for.
https://github.com/espressif/esp-idf/blob/b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e/components/esp_http_client/esp_http_client.c#L1418-L1422

So setting negative value to request_content_len would be the solution of this issue...? I'm not an expert of esp-idf so I should investigate more...

[ivmarkov]

Closed with 0769782