WASM FDW url resolution issue

[akacase]

Bug report

I have a few custom FDW's written with the postgres_fdw template provided. I've been working on an interesting error:

Query 1 ERROR at Line 1: : ERROR: wasmtime error: WebAssembly translation error

There seems to be some incorrect logic in handling http(s) related remote connections.

wrappers/wrappers/src/fdw/wasm_fdw/wasm_fdw.rs

Lines 46 to 122 in 8a30794

	fn download_component(
	rt: &Runtime,
	engine: &Engine,
	url: &str,
	name: &str,
	version: &str,
	checksum: Option<&str>,
	) -> WasmFdwResult<Component> {
	if let Some(file_path) = url.strip_prefix("file://") {
	return load_component_from_file(engine, file_path);
	}
	if url.starts_with("warg://") || url.starts_with("wargs://") {
	let url = url
	.replacen("warg://", "http://", 1)
	.replacen("wargs://", "https://", 1);
	// download from warg registry
	let config = warg::Config {
	disable_interactive: true,
	..Default::default()
	};
	let client = rt.block_on(warg::FileSystemClient::new_with_config(
	Some(&url),
	&config,
	None,
	))?;
	let pkg_name = warg_protocol::registry::PackageName::new(name)?;
	let ver = semver::VersionReq::parse(version)?;
	let pkg = rt
	.block_on(client.download(&pkg_name, &ver))?
	.ok_or(format!("{}@{} not found on {}", name, version, url))?;
	return load_component_from_file(engine, pkg.path);
	}
	// otherwise, download from custom url if it is not in local cache
	// calculate file name hash and make up cache path
	let hash = Sha256::digest(format!(
	"{}:{}:{}@{}",
	unsafe { pg_sys::GetUserId().as_u32() },
	url,
	name,
	version
	));
	let file_name = hex::encode(hash);
	let mut path = dirs::cache_dir().expect("no cache dir found");
	path.push(file_name);
	path.set_extension("wasm");
	if !path.exists() {
	// package checksum must be specified
	let option_checksum = checksum.ok_or("package checksum option not specified".to_owned())?;
	// download component wasm from remote and check its checksum
	let resp = rt.block_on(reqwest::get(url))?;
	let bytes = rt.block_on(resp.bytes())?;
	let bytes_checksum = hex::encode(Sha256::digest(&bytes));
	if bytes_checksum != option_checksum {
	return Err("package checksum not match".to_string().into());
	}
	// save the component wasm to local cache
	if let Some(parent) = path.parent() {
	// create all parent directories if they do not exist
	fs::create_dir_all(parent)?;
	}
	fs::write(&path, bytes)?;
	}
	load_component_from_file(engine, &path).inspect_err(|_| {
	// remove the cache file if it cannot be loaded as component
	let _ = fs::remove_file(&path);
	})
	}

Describe the bug

Remote supabase postgres (15.6) database issues a: Query 1 ERROR at Line 1: : ERROR: wasmtime error: WebAssembly translation error after defining a remote server and foreign table pointing at a https file hosted outside of the "release process" on GitHub actions. Local postgres (16.6) crashes.

To Reproduce

first test:

1. compile fdw: cargo component build --release --target wasm32-unknown-unknown
2. create foreign wrapper with said output (I know checksum calculations aren't used for file:///)
3. test and works

second test:

1. compile fdw: cargo component build --release --target wasm32-unknown-unknown
2. serve the file via a simple Python server locally or in my case Deno (2.1.2).
3. test and works http://localhost:8000/fdw/test_fdw.wasm

third test:

1. stick compiled fdw anywhere; on a supabase S3 bucket, apache httpd server, even a public release on GitHub
   after compiling locally via gh release
2. create server, foreign table and see error prop up.

fourth test:

1. use local postgres
2. point to remotely hosted wasm
3. create server and table
4. postgres crashes

behavior is also present (no database crash) but WASM error on Postgres 15.6 (hosted on Supabase)

This reliably works on all supabase wasm-fdw's built locally with: cargo component build --release --target wasm32-unknown-unknown and served via any of the aforementioned endpoints: paddle, notion, etc.

case@corp paddle_fdw % wasm-tools validate -vv target/wasm32-unknown-unknown/release/paddle_fdw.wasm 
[2024-12-05T18:31:28Z INFO ] read module in 733.167µs
[2024-12-05T18:31:28Z INFO ] module structure validated in 3.883958ms
[2024-12-05T18:31:28Z INFO ] functions validated in 1.401333ms

case@corp paddle_fdw % curl -I https://wiw.org/~case/paddle_fdw.wasm   
HTTP/1.1 200 OK
Date: Thu, 05 Dec 2024 18:32:10 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
Last-Modified: Thu, 05 Dec 2024 18:16:20 GMT
ETag: "28629-62889e5289c1e"
Accept-Ranges: bytes
Content-Length: 165417

< HTTP/2 200 
< content-type: application/octet-stream
< content-disposition: attachment; filename=paddle_fdw.wasm
< content-length: 133622
< accept-ranges: bytes
< cache-control: public, max-age=3600
< last-modified: Thu, 05 Dec 2024 02:16:47 GMT
< access-control-allow-origin: *
< vary: Accept-Encoding
< date: Thu, 05 Dec 2024 18:39:14 GMT
< via: http/2 edgeproxy-h

headers off another remote with should be the correct headers to indicate to reqwest to allow a successful download. I believe the current document deployment process is one where the GitHub action provided (softprops/action-gh-release@v2):

https://github.com/softprops/action-gh-release/blob/92bc83c4214f780e84c7d2a85464a2ca05ebc2f0/src/github.ts#L143-L154

sets the correct mime type and serves it correctly as my test on manually issuing a release via gh release with the compiled output which is confirmed to be correct is also failing.

Here's the current headers off the actual released artifact:

curl -L -I https://github.com/supabase/wrappers/releases/download/wasm_paddle_fdw_v0.1.1/paddle_fdw.wasm

HTTP/2 200 
content-type: application/octet-stream
last-modified: Wed, 14 Aug 2024 06:57:09 GMT
etag: "0x8DCBC2E5148BA20"
server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: ddf6a6fa-f01e-0011-3d14-407be1000000
x-ms-version: 2024-08-04
x-ms-creation-time: Wed, 14 Aug 2024 06:57:09 GMT
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: BlockBlob
content-disposition: attachment; filename=paddle_fdw.wasm
x-ms-server-encrypted: true
accept-ranges: bytes
date: Thu, 05 Dec 2024 18:56:53 GMT
via: 1.1 varnish
age: 8
x-served-by: cache-iad-kjyo7100088-IAD
x-cache: HIT
x-cache-hits: 1
x-timer: S1733425014.504189,VS0,VE1
content-length: 163743

The testing of local HTTP caching seems to be off as I frequently have to remove the cache rm * ~/Library/Caches/*.wasm to get it to reload even over local HTTP.

Final output:

case@corp paddle_fdw % curl -L https://github.com/supabase/wrappers/releases/download/wasm_paddle_fdw_v0.1.1/paddle_fdw.wasm > wut.wasm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  159k  100  159k    0     0   490k      0 --:--:-- --:--:-- --:--:--  490k
case@corp paddle_fdw % wasm-tools validate -v wut.wasm                                                                                 
[2024-12-05T18:59:01Z INFO ] read module in 78.625µs
[2024-12-05T18:59:01Z INFO ] module structure validated in 5.570375ms
[2024-12-05T18:59:01Z INFO ] functions validated in 1.703584ms
case@corp paddle_fdw % curl -L https://wiw.org/~case/paddle_fdw.wasm > why.wasm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  161k  100  161k    0     0   121k      0  0:00:01  0:00:01 --:--:--  121k
case@corp paddle_fdw % wasm-tools validate -v why.wasm                         
[2024-12-05T18:59:41Z INFO ] read module in 148.5µs
[2024-12-05T18:59:41Z INFO ] module structure validated in 933.417µs
[2024-12-05T18:59:41Z INFO ] functions validated in 703.166µs

Feel free to use the server mentioned in this docs, here's a simple test:

create server paddle_server
  foreign data wrapper wasm_wrapper
  options (
    fdw_package_url 'http://wiw.org/~case/paddle_fdw.wasm',
    fdw_package_name 'supabase:paddle-fdw',
    fdw_package_version '0.1.1',
    fdw_package_checksum '36ac1cf5eebfa57a70a8d068a21d858a32771cac07721633e3c15ee8841510df',
    api_url 'https://sandbox-api.paddle.com', -- Use https://api.paddle.com for live account
    api_key_id 'test'
);

create foreign table test (
  id text,
  name text,
  email text,
  status text,
  custom_data jsonb,
  created_at timestamp,
  updated_at timestamp,
  attrs jsonb
)
  server paddle_server
  options (
    object 'customers',
    rowid_column 'id'
 );

select * from test;

drop server if exists paddle_server cascade;

Expected behavior

All files served via http(s):// if wasm-tools reports that the file is correct and is working locally should just work

System information

# macOS
case@corp paddle_fdw % uname -a
Darwin corp.local 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:02:26 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T8122 arm64
# local postgres
case@corp paddle_fdw % psql -c "SELECT version();" 
                                                           version                                                            
------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 16.6 (Homebrew) on aarch64-apple-darwin24.1.0, compiled by Apple clang version 16.0.0 (clang-1600.0.26.4), 64-bit
# supabase
case@corp paddle_fdw % psql $POSTGRES_URL -c "SELECT version();"
                                      version                                       
------------------------------------------------------------------------------------
 PostgreSQL 15.6 on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 13.2.0, 64-bit

[akacase]

I’m not at my computer at the moment, but I have a diff or a PR that fixes this issue I can add in a few hours.

[burmecia]

Thank you very much for reporting this issue! We've noticed this issue and identified it was caused by different version of Rust wasm compiler. The GitHub release workflow always used the latest version of Rust, but the Wasm runtime used in Supabase was compiled with older version of Rust, so they are not compatible with each other.

This issue has been fixed in #369, which fixed Rust version to 1.81.0 in the workflow. The same issue hasn't been fixed in the template repo, that's why you saw this issue when using it. I've raised another PR to fix it. Once it is in place, the compiled wasm file using the template should be working on Supabase platform.

[akacase]

Ah, so the issue on the Supabase platform was caused by binary incompatibility, that's interesting. The remote downloading issue is still an issue with 1.83 as the base from remote locations though, so even though the incompatibility issue will no longer persist, we may end up with incorrect semantics, as local postgres would crash without the new changes applied. Please feel free to test or close if you think it's not an issue.