Support Edge Runtime (no http module)

[kristojorg]

Scope

Improves an existing behavior

Compatibility

• This is a breaking change

Feature description

NextJS has introduced an Edge Runtime (more docs here, which is a stricter node.js runtime based on web standards to run "on the edge". This is used, for example, but their middleware.

I am performing user authentication in this middleware, and would like to mock the fetch requests I make in my middleware. This means I need MSW to be compatible with the edge runtime and support intercepting fetch requests made there.

I have cloned the with-msw example and added a middleware.ts file which makes a simple fetch request. The failure can be seen in my repository: with-msw-app

The error I see when trying to run this is:

error - unhandledRejection: Error: The edge runtime does not support Node.js 'http' module.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
    at Object.get (webpack-internal:///(middleware)/./node_modules/next/dist/server/web/adapter.js:172:19)
    at __extends (webpack-internal:///(middleware)/./node_modules/@mswjs/interceptors/lib/interceptors/ClientRequest/NodeClientRequest.js:15:73)
    at eval (webpack-internal:///(middleware)/./node_modules/@mswjs/interceptors/lib/interceptors/ClientRequest/NodeClientRequest.js:102:5)
    at eval (webpack-internal:///(middleware)/./node_modules/@mswjs/interceptors/lib/interceptors/ClientRequest/NodeClientRequest.js:441:2)
    at Object.(middleware)/./node_modules/@mswjs/interceptors/lib/interceptors/ClientRequest/NodeClientRequest.js (evalmachine.<anonymous>:161:1)
    at __webpack_require__ (evalmachine.<anonymous>:37:33)
    at fn (evalmachine.<anonymous>:281:21)
    at eval (webpack-internal:///(middleware)/./node_modules/@mswjs/interceptors/lib/interceptors/ClientRequest/http.get.js:25:27)
    at Object.(middleware)/./node_modules/@mswjs/interceptors/lib/interceptors/ClientRequest/http.get.js (evalmachine.<anonymous>:172:1)
    at __webpack_require__ (evalmachine.<anonymous>:37:33)
error - Error [TypeError]: fetch failed
    at Object.processResponse (evalmachine.<anonymous>:3163:27)
    at <unknown> (evalmachine.<anonymous>:3507:42)
    at <unknown> (node:internal/process/task_queues:141:7)
    at AsyncResource.runInAsyncScope (node:async_hooks:199:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:138:8)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  middleware: true
}

[kristojorg]

This relates to #1404 probably and possible #1388

[kristojorg]

From a brief inspection, this looks like it would be supported by creating a third startup option: setupEdgeServer which only uses the FetchInterceptor, not the XMLHttpRequestInterceptor.

[kettanaito]

I wonder if we should use interceptors based on the availability of those modules.

setupServer([
  typeof XMLHttpRequest !== 'undefined' && XMLHttpRequestInterceptor
])

Will be hard to figure out Node.js built-ins like http, I guess.

Overall, edge support would be great but I don't see it as a priority anytime soon. If someone would like to make this happen without reworking much of the library, I'd be open to releasing it.

I think the conceptual issue here is that such Edge runtimes are still a custom thing. They do respect the standards but http is also a standard in Node.js. It comes down to the edge functions adhering to the web standards, not Node.js ones (feel free to correct me).

[kettanaito]

To make things a little more complex, there's no such thing as a standardized Edge environment. There's the browser and there's Node.js. I suppose edge frameworks try to emulate Node.js but may be implemented in different languages and the standard library is usually limited and incomplete. This makes it not an actual environment, really, but an infinite list of subsets of an environment.

Indeed, things may get a little easier with Edge frameworks adopting web standards, which some of the most popular ones already do, and MSW adopting them as well. But both those things need to happen for us to claim Edge support, for "Edge" is not really the same thing when referred to by different people.

I may be terribly wrong on these things so do not hesitate to correct me.

[kettanaito]

I will also move this to a discussion because, as I've said, there's no intention on working on this any time soon.

[kristojorg]

I think your read on this seems correct @kettanaito. I was hopeful that using createSetupServer with only the FetchInterceptor would make this work, but it seems that calling createSetupServer depends on the http module as well. So at this point what I'd like to do is setup the FetchInterceptor to intercept fetch requests without relying on createSetupServer. Does this seem like it would be a straightforward addition to the library? Before I take a stab at it, do you have any suggestions?

[kristojorg]

Yea this is definitely interesting! Would you recommend starting work on this API now or waiting for more progress in #1404 ? I will need to get these tests working sooner or later, and my only good options right now are either making a PR here or using https://github.com/mswjs/http-middleware to run my mocks as a separate server process I can actually fetch.

Let me know what you think the best route forward is and I will put it on my schedule to start working. Thank you for the help!

[kettanaito]

I'm actually making good progress in the Fetch API support but even if I finish it soon we probably won't release it just yet. I think it makes sense to start with the createSetup API in parallel, I don't think the changes will overlap.

Just don't refactor the entire codebase and we will be fine :)

[kristojorg]

Haha sounds great! I won't have time this week but will return to this next week : ). Thanks for the pointers

[Toxiapo]

Hi @kettanaito, I work with @kristojorg on the same project that would benefit from this new API. He's been caught up by some other responsibilities, but I can start working on this and hope to have a Pull Request ready by the end of the week!

[kettanaito]

Hey, @Toxiapo! That would be fantastic. I think createSetup (or a better name we discover along the way) is the API that would both make the internals more clear and allow for some edge cases like yours (pun intended).

[kristojorg]

Hi @kettanaito I want to briefly discuss what @Toxiapo and I are thinking for this. Our tentative plan is currently:

1. Create a SetupApi interface with the start/stop/printHandlers/etc.
2. Create a CreateSetup interface that is a function returning a function to setup
   a particular environment
3. In our own code, write a createSetupEdge that somewhat mimicks createSetupServer
   while avoiding the http module.

Does that sound about right? Do you imaging that a generic createSetup function can be created and exported, or do you think we will have to write our own createSetupEdge using the newly exported interfaces?

[kettanaito]

Hey. I think that strategy sound good.

The intention is to separate common API from both setupServer and setupWorker. Those functions may, in the end, extend this new generic SetupApi with their own methods, like the worker one would have .start() and .stop() instead of .listen() that the server has.

I'd summarize it like this:

Implement SetupApi base

// ./src/createSetupApi.ts
export class SetupApi {
  private interceptor: BatchInterceptor
  private handlers: Array<*>

  constructor(readonly interceptors: Array<*>, private readonly initialHandlers: Array<*>) {
    this.interceptor = new BatchInterceptor({
      name: '???',
      interceptors: interceptors.map((Interceptor) => new Interceptor())
    })
  }

  protected apply(): void {
    this.interceptor.apply()
  }

  protected dispose(): void {
    this.interceptor.dispose()
  }

  public use(...runtimeHandlers) {
    requestHandlersUtils.use(this.handlers, ...runtimeHandlers)
  }

  public resetHandlers(...nextHandlers) {
    // Reuse existing "requestHandlerUtils".
    this.handlers = requestHandlerUtils.resetHandlers(this.initialHandlers, ...nextHandlers)
  }
}

You can use existing setupServer or setupWorker implementations to extract the methods' logic. Just make sure you're not reusing environment-specific things, as createSetupApi needs to remain isomorphic.

One thing that will be particularly challenging is to define the interceptor.on('request') listener. It should live in the SetupApi base class but it should (I think) delegate the actual listener as a method each particular setup API can override. We can collaborate on this one closer together.

Reuse the new interface in setupServer and setupWorker

// src/node/setupServer.ts
import { ClientRequestInterceptor } from '@mswjs/interceptors/lib/interceptors/ClientRequest'
import { XMLHttpRequestInterceptor } from '@mswjs/interceptors/lib/interceptors/XMLHttpRequest'

export class SetupServerApi extends SetupApi {
  constructor(private readonly handlers: Array<*>) {
    // Configure interceptors for "setupServer". 
    super([ClientRequestInterceptor, XMLHttpRequestInterceptor])
  }

  // Create public enable/disable methods
  // relevant in the context of "setupServer".
  public listen() {
    super.apply()
  }

  public close() {
    super.dispose()
  }
}

export function setupServer(...handlers) {
  return new SetupServerApi(handlers)
}

[Toxiapo]

@kettanaito This is super useful, thank you!

I will start by separating the common API, and maybe we can work from there together.

[kettanaito]

I would love too! Let's ship this API together.

[Toxiapo]

Hi @kettanaito, thanks again for shipping this new API! I was able to successfully set up an MVP with the new API for the Edge environment, but I am running into an issue that I would love to hear your thoughts on.

1. While running the mock server in the Edge environment, I was getting the window is not defined error from this useFetch function call. This is because the function only grabs the fetch method from the node or browser environment, but the Edge's fetch method is a browser method inside the Edge server, which caused the window object is undefined error. One possible fix I can think of is to change the logic to

const useFetch: (input: RequestInfo, init?: RequestInit) => Promise<Response> =
  isNodeProcess()
    ? (input, init) =>
        import('node-fetch').then(({ default: nodeFetch }) =>
          (nodeFetch as unknown as typeof window.fetch)(input, init),
        )
    : globalThis.fetch
    ? globalThis.fetch
    : window.fetch

if this is something that you think is ok to do, I can create a quick PR for this. Happy to hear any other suggestions as well!

[kristojorg]

I believe that replacing window with globalThis will make this work. globalThis should resolve to window in browser environments, and to the global fetch in node and edge environments

[kettanaito]

Agree with @kristojorg: we need to reference the native fetch as globalThis.fetch to support Node environments where fetch is also a global. Please, open a pull request, let's ship this.

Funny that isNodeProcess() apparently doesn't return true on Edge environment. I suppose it's not technically Node, is it?

Maybe it'd be even safer to do:

// We can throw "isNodeProcess" check away and rely on
// either globally exposed fetch or use a polyfill. 
return globalThis.fetch || import('node-fetch').theRest

We can ship this change in two chunks:

1. window.fetch -> globalThis.fetch (non-breaking change);
2. Remove isNodeProcess (breaking change).

[Toxiapo]

Funny that isNodeProcess() apparently doesn't return true on Edge environment. I suppose it's not technically Node, is it?

I think you are correct. It's different depends on how you run it. It's Node and browser but doesn't contain all the APIs from either platform.

---

// We can throw "isNodeProcess" check away and rely on
// either globally exposed fetch or use a polyfill. 
return globalThis.fetch || import('node-fetch').theRest

I will open a PR to apply the globalThis change, thank you!

[nickbling]

Hey @kettanaito, do you have any news on this?