Figure out a way around CORS

[sebamarynissen]

Now that the Simtropolis channel is nicely filling up, I started playing a bit with what a script could look like that would add an "Install with sc4pac" button to the STEX upload. You can find the proof of concept below. It is extremely basic and doesn't include things like caching the channel contents - which definitely should be done, neither does it support both the default and simtropolis channel - but it shows what the basic structure might look like.

Proof of concept

function h(tag, attrs = {}, children = []) {
  let node = document.createElement(tag);
  for (let name of Object.keys(attrs)) {
    node.setAttribute(name, attrs[name]);
  }
  for (let child of children) {
    if (typeof child === 'string') {
      node.appendChild(new Text(child));
    } else {
      node.appendChild(child);
    }
  }
  return node;
}

async function install(packages) {
  let payload = packages.map(pkg => {
    return {
      package: `${pkg.group}:${pkg.name}`,
      channelUrl: 'https://sc4pac.sebamarynissen.dev',
    };
  });
  let body = JSON.stringify(payload);
  try {
    await fetch(`http://localhost:51515/packages.open`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Content-Length': body.length,
      },
      body,
    });
  } catch (e) {
    if (e instanceof TypeError) {
      alert('The gui needs to be running in order to use this button!');
    }
  }
}

(async function() {
  let res = await fetch('https://sc4pac.sebamarynissen.dev/sc4pac-channel-contents.json');
  let json = await res.json();

  let packagesById = {};
  for (let pkg of json.packages) {
    let { stex = [] } = pkg.externalIds || {};
    for (let id of stex) {
      if (!packagesById[id]) packagesById[id] = [];
      packagesById[id].push(pkg);
    }
  }

  let id = new URL(window.location.href)
    .pathname
    .replace(/\/$/, '')
    .split('/')
    .at(-1)
    .split('-')
    .at(0);
  
  if (packagesById[id]) {

    let button = h('button', {
      class: 'ipsButton ipsButton_fullWidth ipsButton_large',
      style: 'font-weight: 600; background: black; color: white; display: flex; align-items: center; justify-content: center;',
    }, [
      h('i', { class: 'fa fa-download fa-lg' }),
      '\u00A0\u00A0Install with sc4pac',
    ]);
    button.addEventListener('click', () => install(packagesById[id]));

    let a = [...document.querySelectorAll('a')].find(a => a.textContent.includes('Download File'));
    let li = h('li', {
      style: 'filter:drop-shadow(0px 3px 3px #000000);'
    }, [button]);
    a.closest('ul').appendChild(li);
  }

})();

There is however one big problem with this approach: you get a CORS error if you use the default channel url:

Access to fetch at 'https://memo33.github.io/sc4pac/channel/sc4pac-channel-contents.json' from origin 'https://community.simtropolis.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

This can be solved by sending the Access-Control-Allow-Origin: * header, but unfortunately GitHub pages does not support setting custom headers. There are a few solutions to this problem:

1. Move the default channel to sc4pac.com

This is my preferred solution.

Pros

• Allows adding a header rewrite rule on CloudFlare to add the Access-Control-Allow-Origin: * header. This is what I did to get the proof of concept working with sc4pac.sebamarynissen.dev
• Allows using a fixed url for the channel - e.g. channel.sc4pac.com - which gives greater flexibility when deciding to move the channel to a different hosting provider in the future.

Cons

• Requires a domain name, which comes at a yearly cost. I am however willing to bear this cost. I can use my company whisthub.com for it to register it as an expense, and at roughly $10 a year at CloudFlare, it's definitely bearable. Heck, I'd pay $200 to register it for the next 20 years if possible. I can also give access on CloudFlare for the domain name to other people, to avoid being too dependent on me.

2. Move the deployment to CloudFlare pages

Pros

• CloudFlare pages is free, and allows you to specify a _headers file where you can customize the headers.

Cons

• Deployment becomes harder to setup
• Requires everyone to update the channel url from https://memo33.github.io/sc4pac/channel to something like https://sc4pac.pages.dev. Could obviously be combined with the domain name approach of above.

3. Provide the channel as .js file with JSONP

A way around CORS is that you could load the channel in a <script> tag. Hence, along with sc4pac-channel-contents.json, the channel could also provide something like sc4pac-channel-contents.js which looks like

loadChannel('https://memo33.github.io/sc4pac/channel', { ...json });

The calling script should provide a global loadChannel() function in that case.

Pros

• Allows loading channels easily with <script src="https://memo33.github.io/sc4pac/channel/sc4pac-channel-contents.js">.

Cons

• Feels like a hack
• Is a hack
• Vulnerable to XSS attacks if someone manages to update sc4pac-channel-contents.js

4. Limit integration to a browser plugin

I'm not entirely sure, but I suppose you can get around the CORS limitations when making the request as a browser plugin service worker, rather than a content script.

Pros

• Requires no changes to the channel configuration

Cons

• Requires maintenance of multiple browser plugins (Chrome, Firefox, ...)
• Friction for users as they have to install a plugin first

@Zasco you might be interested in this as well, as it's also relevant for your plugin I guess

[memo33]

I'm confused. I don't see the problem in the first place. The website already fetches json files from different channels (i.e. different origins) in the client browser.

In fact, GitHub does set access-control-allow-origin: *. For example:

curl --header "Origin: https://example.org" -I "https://memo33.github.io/sc4pac/channel/sc4pac-channel-contents.json"

Response:

HTTP/2 200
server: GitHub.com
content-type: application/json; charset=utf-8
permissions-policy: interest-cohort=()
last-modified: Tue, 31 Dec 2024 09:35:12 GMT
access-control-allow-origin: *
etag: "6773bad0-93251"
expires: Tue, 31 Dec 2024 13:41:57 GMT
cache-control: max-age=600
x-proxy-cache: MISS
x-github-request-id: D598:4D6EE:100DF6E6:102B48D7:6773F24C
accept-ranges: bytes
date: Tue, 31 Dec 2024 13:34:39 GMT
via: 1.1 varnish
age: 162
x-served-by: cache-fra-eddf8230064-FRA
x-cache: HIT
x-cache-hits: 1
x-timer: S1735652080.620346,VS0,VE2
vary: Accept-Encoding
x-fastly-request-id: 04832f31cf7e83bcfce74ce69ea7a2a47b36f0c3
content-length: 602705

[sebamarynissen]

Alright, I see what's happening. I never actually tried it with the default channel, only with https://sebamarynissen.github.io/simtropolis-channel/sc4pac-channel-contents.json. However, GitHub redirects this response to https://sc4pac.sebamarynissen.dev/sc4pac-channel-contents.json because I have mapped that domain to it, but it does not include the access-control-allow-origin: * header in it's 301 response, hence the CORS error.

curl --header "Origin: https://example.org" -I "https://sebamarynissen.github.io/simtropolis-channel/sc4pac-channel-contents.json"

HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 162
Server: GitHub.com
Content-Type: text/html
permissions-policy: interest-cohort=()
Location: https://sc4pac.sebamarynissen.dev/sc4pac-channel-contents.json
X-GitHub-Request-Id: 7271:4DD5A:104FF41E:106D83A1:677401E6
Accept-Ranges: bytes
Age: 0
Date: Tue, 31 Dec 2024 14:38:31 GMT
Via: 1.1 varnish
X-Served-By: cache-bru1480063-BRU
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1735655911.188738,VS0,VE107
Vary: Accept-Encoding
X-Fastly-Request-ID: 71b54d0b607d2ed40c2f7790e37d7c62f4211952

I think we're safe then. I will verify and close if confirmed.

[sebamarynissen]

Verified, and it does work indeed.

[Zasco]

I had the same problem while trying to fetch the exchange IDs mapping lists for my browser extension. I used JSDeliver feature to wrap a request in valid headers.

[sebamarynissen]

@Zasco I think we should avoid relying on my manually curated list of available packages in sebamarynissen/sc4pac-helpers and go with the approach of directly fetching the channels instead, as @memo33 also mentioned in sebamarynissen/sc4pac-helpers#4. I don't plan on keeping that list up to date forever, and it kind of defeats the purpose of having everything automated as much as possible.

Relying on the channels itself is by definition the only reliable way to tell whether a package can be installed or not, plus it has the benefit of not throwing any CORS errors.

[Zasco]

@Zasco I think we should avoid relying on my manually curated list of available packages in sebamarynissen/sc4pac-helpers and go with the approach of directly fetching the channels instead, as @memo33 also mentioned in sebamarynissen/sc4pac-helpers#4. I don't plan on keeping that list up to date forever, and it kind of defeats the purpose of having everything automated as much as possible.

Relying on the channels itself is by definition the only reliable way to tell whether a package can be installed or not, plus it has the benefit of not throwing any CORS errors.

Already planned (Zasco/sc4pac-browser-extension#1). I might even do it today... It was just so you know about this service if you ever face that problem again.