How to decode POST requests with different charset

[staticssleever668]

I have an old application I'm making a back-end server for. This application sends data in not-URL-encoded Shift-JIS, and whenever there are CJK characters in that data, Rocket skips the request:

POST / application/x-www-form-urlencoded:
   >> Matched: (greet_route) POST /
   >> Data guard `rocket :: form :: Form < GreetingData >` failed: Errors([Error { name: None, value: None, kind: Io(Custom { kind: InvalidData, error: "stream did not contain valid UTF-8" }), entity: Form }]).
   >> Outcome: Failure
   >> No 400 catcher registered. Using Rocket default.
   >> Response succeeded.

I thought adding charset=Shift_JIS to the content type header would fix the issue, but it doesn't help.

An example:

#[derive(rocket::FromForm)]
struct GreetingData {
    name: String
}

#[rocket::post("/", data = "<data>")]
fn greet_route(data: rocket::form::Form<GreetingData>) -> String {
    format!("Hello, {}", data.name)
}

#[rocket::launch]
fn rocket() -> _ {
    rocket::build().mount("/", rocket::routes![greet_route,])
}

Make two files with different encodings, data-sjis.txt with Shift_JIS and data-utf8.txt with UTF-8, put テスト in them, then make requests using curl like so: curl -X POST "http://localhost:8000/" -d @data-sjis.txt.

Result:
Using curl -X POST "http://localhost:8000/" -d @data-utf8.txt works fine, but curl -X POST "http://localhost:8000/" -d @data-sjis.txt gives the aforementioned problem. Adding charset to the content type like so doesn't help: curl -X POST "http://localhost:8000/" -H "Content-Type: application/x-www-form-urlencoded; charset=Shift_JIS" -d @data-sjis.txt.

A bit different but related problem:
In some requests that application percent-encodes those CJK characters, but it still uses Shift_JIS for them. This way Rocket doesn't skip the route, but the characters get garbled.

[the10thWiz]

To make a long story short: I don't expect Rocket to ever support this, since it's not standards conforming. https://url.spec.whatwg.org/#application/x-www-form-urlencoded has details on the content type, but the tl;dr is x-www-form-urlencoded doesn't actually have a charset argument, and is always utf-8. It does note that servers supporting legacy applications may have to support other charsets, but I don't expect this to be part of Rocket's form parser.

One solution is to write a custom form parser that accepts Shift-JIS, but this is time consuming and error prone. Alternatively, you could write a wrapper that first converts the form data to UTF-8, and then uses Rocket's form parser to parse the UTF-8. I've gone ahead and written an implementation, which seems to fix the issue as outlined above:

use encoding::Encoding;
use rocket::{
    data::{Data, FromData, ToByteUnit},
    form::{self, Error, Form, FromForm},
    http::Status,
    request::{Request, local_cache},
    response::Responder,
    outcome::{try_outcome, IntoOutcome},
};

pub struct CharsetAwareForm<T>(T);

#[rocket::async_trait]
impl<'r, T> FromData<'r> for CharsetAwareForm<T>
where
    T: FromForm<'r>,
{
    type Error = <Form<T> as FromData<'r>>::Error;

    async fn from_data(
        req: &'r Request<'_>,
        data: Data<'r>,
    ) -> rocket::data::Outcome<'r, Self, Self::Error> {
        if req.format().map_or(false, |format| {
            format.is_form()
                && format
                    .params()
                    .any(|(k, v)| k == "charset" && v == "Shift-JIS")
        }) {
            let by = try_outcome!(data
                .open(
                    req.rocket()
                        .config()
                        .limits
                        .get("form")
                        .unwrap_or(2usize.mebibytes())
                )
                .into_bytes()
                .await
                .map_err(|e| {
                    let mut errors = rocket::form::Errors::new();
                    println!("E: {:?}", e);
                    errors.push(Error {
                        name: None,
                        value: None,
                        kind: form::prelude::ErrorKind::Io(e),
                        entity: form::prelude::Entity::Form,
                    });
                    errors
                })
                .into_outcome(Status::BadRequest));
            let s = local_cache!(
                req,
                encoding::all::WINDOWS_31J
                    .decode(&by, encoding::DecoderTrap::Replace)
                    .unwrap()
            );
            Form::parse(s).map(|t| Self(t)).into_outcome(Status::UnprocessableEntity)
        } else {
            <Form<T> as FromData<'r>>::from_data(req, data)
                .await
                .map(|f| Self(f.into_inner()))
        }
    }
}

impl<T> std::ops::Deref for CharsetAwareForm<T> {
    type Target = T;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

It uses the encoding crate (which doesn't have a specific Shift-JIS encoder, since it can actually refer to multiple different encodings. You will likely need to test and select the appropriate encoder for your text, but the WINDOWS_31J encoder worked on the sample you provided. You may also wish to adjust the DecoderTrap depending on how you want to handle invalid characters.

This doesn't handle percent encoded characters at all, but it could be added.

[staticssleever668]

This is a drop-in solution, thank you very much!

---

Not completely related to your code, I think I found something strange with adding headers in fairings. My application doesn't send a content type at all, so I was inserting it in a fairing like this:

req.add_header(ContentType::Form.with_params);

So obviously I tried adding the charset parameter like so:

req.add_header(ContentType::Form.with_params(("charset", "Shift-JIS")));

But this didn't work, I added req.format() to the start of from_data and it was empty. Looking at req.headers(), I saw that the MediaType had empty params field.

So I tried making a custom content type like so and it worked:

req.add_header(
    ContentType::new("application", "x-www-form-urlencoded")
        .with_params(("charset", "Shift-JIS")),
);

I'm not sure if this is a bug, but this is definitely not obvious.

[staticssleever668]

About handling percent encoding characters, could you guide me in the right direction? I was only able to implement it by modifying the percent_encoding crate to use encoding::all::WINDOWS_31J.decode instead of String::from_utf8_lossy(bytes), but not only does it feel dirty, it actually doesn't work correctly, because plus signs are not replaced by spaces (and I don't understand why this is the case as I don't see anything related in the section of code I modified). Is there a better way?

[the10thWiz]

ContentType::with_params should replace the params, so it not working for you when using the Form constant is a bug.

I don't know how you're modifying percent_encoding, but I agree that it feels like a hacky solution. On the other hand, I suspect the issue you're running into is that the percent encoding crate is encoding the plus signs with percent encoding, even if they were a literal plus in the original text. This means that Rocket will decode them as a literal plus (since they were percent encoded). This is a little more complicated, but the I suspect the ideal solution would be to modify the percent decode logic to skip decoding +, and remove + from the list of encoded chars. Overall, I suspect modifying percent_encoding is actually your best bet, at least for now.

[staticssleever668]

About plus signs: I think percent_encoding doesn't handle them at all, as it doesn't even decode "%2B" which is supposed to be a plus sign (maybe it implements a different specification?), so I just decided to handle them myself by replacing "+" -> "%20" (whitespace), "%2B" -> "+".

---

If anyone ever has the same issue, here is how I did it:
Modification to percent_encoding: https://git.sr.ht/~showy_fence/rust-url/commit/6daf7c45d19bfad287fb1475c7ffb47392e9f216
Edit to this CharsetAwareForm: https://git.sr.ht/~showy_fence/new-lr2ir-rust/commit/382091adbf18bdfc216d775039ef317967035aff.

EDIT: it actually doesn't work because this way bytes are decoded with Shift_JIS twice, and by the second time it's already in UTF-8, so it decoded UTF-8 as Shift_JIS and characters become garbled. The solution is to only use the decode method in percent_encoding, which will change the encoding of the rest of the bytes as well.
Here is this fix: https://git.sr.ht/~showy_fence/new-lr2ir-rust/commit/7eb04095b84ae75686ce99e0ab4e54a3097f09ae. I don't yet understand how to prevent allocations in Rust so this probably very inefficient, but it finally fully works.

[the10thWiz]

Honestly, I think you're doing the best you can here. I don't see any obvious allocations to remove, so it should actually be pretty good.