AssetReaderError::Io when loading more than one web asset

[jf908]

Bug

I am getting this error when trying to load more than one web asset.

ERROR bevy_asset::server: Encountered an I/O error while loading asset: unexpected status code 500 while loading https://example.png: Head byte length should be less than 8kb

This error is misleading and the head byte length is less than 8kb. There is an issue on the surf repo http-rs/surf#289 which points to it being a bug in http-rs/async-h1's DDOS protection but from a cursory glance, I can't see the cause. I could create an issue downstream but its looking unmaintained.

Reproduction

It was reproducing 100% of the time in my project and I think I boiled it down to it happening when:

1. First, a request has to be made to a web server that 404s but has a body to it.
2. Next, a request to an asset over 8kb will result in this error.

Since asset meta checks 404s on every asset, this bug happens every time I load more than one asset. I noticed that this bug does not happen on all web servers and I am assuming this is something to do with the size of the 404 response. For testing purposes, I noticed images from Apple's website also return 404s with html bodies when .meta is appended to the end of an image URL and can be used to reproduce this bug.

Workarounds

1. Use .insert_resource(AssetMetaCheck::Never) to avoid 404s. I cannot use this one because I am using processed mode for local assets.
2. Compile bevy_web_asset with surf = { features = [ "curl-client" ] }