RFC: Performance-related API changes

[cyco130]

Currently, Hattip's performance can easily beat Express but it is quite a bit behind Hono's (especially on Node) and Fastify's. I propose a few changes and new APIs to get back in the game. I partially implemented some of these ideas along with a few router changes (I will create an RFC for that too later) and it looks like we can improve the performance by a lot, surpassing both frameworks by a healthy margin. And the margin gets much larger with the uWebSockets.js adapter (which is supported by neither).

Node adapter

The Node adapter is slow. I propose a few API changes to counter it.

new Response() is slow

The problem is not just new Response() being slow (it is!). Reading the body from a standard Response object always involves streams while the fast implementation can provide a quick path when the body is not a stream.

• 6.75ms (29617 req/sec) with standard new Response()
• 2.98ms (67089 req/sec) with fast implementation (see the next message)
• 2.27x faster

Unfortunately, overriding the global Response object (like Hono does) can cause unexpected behavior:

global.Response = FastResponse;
const response = await fetch("https://google.com");
console.log(response instanceof Response);
// Prints false because fetch always returns a "real" Response object

For this reason, I propose a new ctx.response() API which will take the exact same arguments as new Response() and return an instance of our faster implementation. We'll manually set its prototype so instanceof Response checks will still pass but the adapter will be able to provide a quick path for FastResponse instances (while still handling normal Response objects as before).

For environments that natively support the Fetch API, ctx.response will simply be implemented as new Response. For Node and uWebSockets on Node, it will use the FastResponse implementation.

This will also require replacements for Response.json(), Response.redirect() etc. So I also propose ctx.json(), ctx.redirect() etc.

new Request() is slow

• 4.15ms (48210 req/sec) with standard new Request()
• 3.03ms (65869 req/sec) with fast implementation (see the next message)
• 1.37x faster

This doesn't require a new API like FastResponse does since the adapter will be the only thing that will create a FastRequest object and it will still pass instanceof Request checks.

All adapters

new URL() is slow

• 3.19ms (62649 req/sec) with new URL()
• 3.06ms (65357 req/sec) with a simple parser
• 1.04x faster

The simple parser implementation:

function parseUrl(href: string): [path: string, search: string] {
  // Only supports http and https schemes
  const originStart = href[5] === "s" ? 8 : 7;
  const pathStart = href.indexOf("/", originStart);
  let pathEnd = href.indexOf("?");
  if (pathEnd === -1) {
    pathEnd = href.length;
  }

  return [href.slice(pathStart, pathEnd), href.slice(pathEnd + 1)];
}

search parsing is not strictly necessary but it's very cheap to compute (no noticeable slowdown on microbenchmarks).

Frankly, I'm a bit torn on this. ctx.url is useful in many cases and the difference is small. But it is noticeable. We can still keep the API but switch to a lazy implementation. A separate ctx.searchParams API might also be handy but, again, I'm not very sure.

await is slow

This is harder to measure in microbenchmarks (sync is around 1.03x faster). But it gets compounded with every middleware because ctx.next() has to be async. For this reason, I propose a change in the middleware API.

Instead of this:

async function myMiddleware(ctx) {
  if (!validate(ctx)) {
    return ctx.response("Invalid request", { status: 400 });
  }

  const response = await ctx.next();
  response.headers.append("X-Powered-By", "Hattip");

  return response;
}

I propose this useEffect-inspired API:

// No async unless absolutely necessary
function myMiddleware(ctx) {
  if (!validate(ctx)) {
    return ctx.response("Invalid request", { status: 400 });
  }

  // If you return a function, it will be called with the
  // response returned from the next handler/middleware.
  return (response) => {
    response.headers.append("X-Powered-By", "Hattip");
    // Or create a completely new Response an return that instead
  };
}

This way, the composer/router can call the returned function synchronously if everything in between was synchronous (i.e. it can switch from a synchronous fast path to the slower async path).

This will make even more sense when I, ehm, unveil my new router proposal 😃

[cyco130]

Each of these implementations basically avoid creating a native Response/Request object if it can handle things on its own.

Incomplete FastResponse implementation:

import { Buffer } from "node:buffer";
import { Readable, Transform, Writable } from "node:stream";
import type { ServerResponse } from "node:http";

export type FastBodyInit =
	| ArrayBuffer
	| AsyncIterable<Uint8Array>
	| Blob
	| FormData
	| Iterable<Uint8Array>
	| NodeJS.ArrayBufferView
	| URLSearchParams
	| null
	| string
        // This is specifically to support React streaming SSR with better perf
	| { pipe: (writable: Writable) => void };

type ResponseType =
	| "basic"
	| "cors"
	| "default"
	| "error"
	| "opaque"
	| "opaqueredirect";

// The adapter calls this to write the response to the Node response stream
export const writeToServerResponseKey = Symbol("writeToServerResponse");

export class FastResponse implements Response {
	#response: Response | null = null;

	#headers: Headers;
	#status: number;
	#statusText: string;
	#body: FastBodyInit | null;

	constructor(body?: FastBodyInit | null, init?: ResponseInit) {
		const status = Math.floor(init?.status ?? 200);
		if (status < 200 || status >= 600) {
			throw new RangeError(
				`Failed to construct 'Response': The status provided (${status}) is outside the range [200, 599]`,
			);
		}
		this.#status = status;

		// TODO: validate statusText
		this.#statusText = init?.statusText ?? "";

		this.#body = body ?? null;

		this.#headers = new Headers(init?.headers);
		if (this.#body !== null && !this.#headers.has("Content-Type")) {
			if (this.#body instanceof Blob) {
				this.#headers.set("Content-Type", this.#body.type);
			} else if (this.#body instanceof URLSearchParams) {
				this.#headers.set("Content-Type", "application/x-www-form-urlencoded");
			} else if (this.#body instanceof FormData) {
				this.#createResponse();
				this.#headers.set(
					"Content-Type",
					this.#response!.headers.get("Content-Type")!,
				);
			}
		}
	}

	#createResponse(): Response {
		if (this.#response) {
			return this.#response;
		}

		this.#response = new Response(this.#body as any, {
			headers: this.#headers,
			status: this.#status,
			statusText: this.#statusText,
		});

		return this.#response;
	}

	get headers(): Headers {
		return this.#headers;
	}

	get ok(): boolean {
		return this.#status >= 200 && this.#status < 300;
	}

	get redirected(): boolean {
		return false;
	}

	get status(): number {
		return this.#status;
	}

	get statusText(): string {
		return this.#statusText;
	}

	get type(): ResponseType {
		return "basic";
	}

	get url(): string {
		return "";
	}

	clone(): Response {
		throw new Error("TODO");
	}

	get body(): ReadableStream<Uint8Array> | null {
		const body = this.#body;

		if (body === null) {
			return null;
		}

		if (Buffer.isBuffer(body)) {
			return Readable.toWeb(Readable.from(body)) as ReadableStream<Uint8Array>;
		}

		if (body instanceof Readable) {
			return Readable.toWeb(body) as ReadableStream<Uint8Array>;
		}

		if (body instanceof Blob) {
			return body.stream();
		}

		if (body instanceof ReadableStream || body instanceof FormData) {
			const response = this.#createResponse();
			return response.body;
		}

		if (
			typeof body === "object" &&
			"pipe" in body &&
			typeof body.pipe === "function"
		) {
			const { pipe } = body;
			const transform = new Transform({
				transform(chunk, encoding, callback) {
					this.push(chunk, encoding);
					callback();
				},
			});
			pipe(transform);
			return Readable.toWeb(transform) as ReadableStream;
		}

		return new ReadableStream({
			start: async (controller) => {
				try {
					const output = await this.#readAll("buffer");
					await controller.enqueue(output);
				} catch (error) {
					controller.error(error);
				}

				controller.close();
			},
		});
	}

	get bodyUsed(): boolean {
		throw new Error("TODO");
	}

	async #readAll(as: "buffer"): Promise<Buffer>;
	async #readAll(as: "string"): Promise<string>;
	async #readAll(as: "either"): Promise<Buffer | string>;
	async #readAll(as: "buffer" | "string" | "either"): Promise<Buffer | string> {
		const body = this.#body;
		if (body === null) {
			return Buffer.alloc(0);
		}

		function fromBuffer(body: Buffer): string | Buffer {
			return as === "string" ? body.toString() : body;
		}

		if (Buffer.isBuffer(body)) {
			return fromBuffer(body);
		}

		if (body instanceof Readable) {
			const buffers: Buffer[] = [];
			for await (const chunk of body) {
				buffers.push(chunk);
			}

			const buffer = Buffer.concat(buffers);
			return fromBuffer(buffer);
		}

		if (body instanceof ArrayBuffer) {
			const buffer = Buffer.from(body);
			return fromBuffer(buffer);
		}

		if (typeof body === "object") {
			let buffer: Buffer;
			if ("pipe" in body) {
				const buffers: Buffer[] = [];
				const writable = new Writable({
					write(chunk, encoding, callback) {
						buffers.push(Buffer.from(chunk, encoding));
						callback();
					},
				});

				body.pipe(writable);

				buffer = await new Promise<Buffer>((resolve, reject) => {
					writable.on("finish", () => {
						resolve(Buffer.concat(buffers));
					});
					writable.on("error", reject);
				});

				return fromBuffer(buffer);
			} else if ("buffer" in body) {
				buffer = Buffer.from(body.buffer, body.byteOffset, body.byteLength);

				return fromBuffer(buffer);
			}
		}

		if (body instanceof Blob) {
			const buffer = await body.arrayBuffer();
			return fromBuffer(Buffer.from(buffer));
		}

		if (body instanceof ReadableStream || body instanceof FormData) {
			const response = this.#createResponse();
			const buffer = Buffer.from(await response.arrayBuffer());
			return fromBuffer(buffer);
		}

		const string = String(body);
		return as === "buffer" ? Buffer.from(string) : string;
	}

	async arrayBuffer(): Promise<ArrayBuffer> {
		if (this.#body instanceof ArrayBuffer) {
			return this.#body;
		}

		const buffer = await this.#readAll("buffer");
		return Uint8Array.from(buffer).buffer;
	}

	async blob(): Promise<Blob> {
		if (this.#body instanceof Blob) {
			return this.#body;
		}

		const buffer = await this.#readAll("buffer");
		return new Blob([buffer], {
			type: this.#headers.get("Content-Type") ?? undefined,
		});
	}

	async formData(): Promise<FormData> {
		const response = new Response(await this.#readAll("buffer"), {
			headers: this.#headers,
			status: this.#status,
			statusText: this.#statusText,
		});

		return response.formData();
	}

	async json(): Promise<any> {
		const text = await this.text();
		return JSON.parse(text);
	}

	text(): Promise<string> {
		return this.#readAll("string");
	}

	async [writeToServerResponseKey](response: ServerResponse): Promise<void> {
		response.statusCode = this.status;
		if (this.statusText) {
			response.statusMessage = this.statusText;
		}

		const uniqueHeaderNames = new Set(this.headers.keys());

		for (const key of uniqueHeaderNames) {
			if (key === "set-cookie") {
				const setCookie = this.headers.getSetCookie!();
				response.setHeader("set-cookie", setCookie);
			} else {
				response.setHeader(key, this.headers.get(key)!);
			}
		}

		const body = this.#body;
		if (body === null) {
			response.end();
			return;
		}

		if (Buffer.isBuffer(body) || body instanceof Blob) {
			response.end(body);
			return;
		}

		if (body instanceof Readable) {
			body.pipe(response, { end: true });
			return;
		}

		if (body instanceof ReadableStream || body instanceof FormData) {
			const r = this.#createResponse();
			const readable = Readable.fromWeb(r.body as any);
			readable.pipe(response, { end: true });
			return;
		}

		if (typeof body === "object" && "pipe" in body) {
			body.pipe(response);
			return;
		}

		const output = await this.#readAll("either");
		response.end(output, "utf8");
	}
}

Object.setPrototypeOf(FastResponse.prototype, Response.prototype);

Incomplete FastRequest implementation:

import { Buffer } from "node:buffer";
import type { IncomingMessage } from "node:http";
import { Readable } from "node:stream";

export interface FastRequestOptions {
	origin?: string;
	trustProxy?: boolean;
}

export class FastRequest implements Request {
	get duplex() {
		return "half" as const;
	}

	get cache() {
		return "default" as const;
	}

	get credentials() {
		return "same-origin" as const;
	}

	get destination() {
		return "" as const;
	}

	get integrity(): string {
		return "";
	}

	get keepalive(): boolean {
		return false;
	}

	get mode() {
		return "cors" as const;
	}

	get redirect() {
		return "follow" as const;
	}

	get referrer(): string {
		return "about:client";
	}

	get referrerPolicy() {
		return "";
	}

	#incomingMessage: IncomingMessage;
	#origin?: string;
	#trustProxy?: boolean;
	constructor(incomingMessage: IncomingMessage, options: FastRequestOptions) {
		this.#incomingMessage = incomingMessage;
		this.#origin = options.origin;
		this.#trustProxy = options.trustProxy;
	}

	#request: Request | null = null;
	#createRequest(): Request {
		if (this.#request) {
			return this.#request;
		}

		this.#request = new Request(this.url, {
			body: Readable.toWeb(this.#incomingMessage) as ReadableStream,
			method: this.method === "TRACE" ? "GET" : this.method,
			headers: this.headers,
			duplex: "half",
		});

		return this.#request;
	}

	formData(): Promise<FormData> {
		return this.#createRequest().formData();
	}

	#signal: AbortSignal | null = null;
	get signal(): AbortSignal {
		if (!this.#signal) {
			const controller = new AbortController();
			this.#signal = controller.signal;

			this.#incomingMessage.once("close", () => controller.abort());
		}

		return this.#signal;
	}

	#headers: Headers | null = null;
	get headers(): Headers {
		if (this.#request) {
			return this.#request.headers;
		}

		if (!this.#headers) {
			const headers = new Headers();
			for (let i = 0; i < this.#incomingMessage.rawHeaders.length; i += 2) {
				const key = this.#incomingMessage.rawHeaders[i]!;
				const value = this.#incomingMessage.rawHeaders[i + 1]!;

				if (key[0] === ":") {
					continue;
				}

				headers.append(key, value);
			}

			this.#headers = headers;
		}

		return this.#headers;
	}

	#url: string | null = null;
	get url(): string {
		if (this.#url) {
			return this.#url;
		}

		let url = this.#incomingMessage.url ?? "/";

		if (this.#origin) {
			url = this.#origin + url;
		} else {
			let proto: string | undefined;
			let host: string | undefined;
			if (this.#trustProxy) {
				const parseForwardedHeader = (name: string) => {
					return (this.headers.get("x-forwarded-" + name) || "")
						.split(",", 1)[0]!
						.trim();
				};

				proto = parseForwardedHeader("proto");
				host = parseForwardedHeader("host");
			}

			proto =
				proto ||
				(this.#incomingMessage as any).protocol ||
				(this.#incomingMessage.socket as any).encrypted
					? "https"
					: "http";
			host = host || this.headers.get("host")!;

			url = proto + "://" + host + url;
		}

		this.#url = url;

		return url;
	}

	clone(): Request {
		return this.#createRequest().clone();
	}

	get body(): ReadableStream | null {
		if (this.method === "GET" || this.method === "HEAD") {
			return null;
		}

		return Readable.toWeb(this.#incomingMessage) as ReadableStream;
	}

	get bodyUsed(): boolean {
		return this.#incomingMessage.readableEnded;
	}

	get method(): string {
		return this.#incomingMessage.method ?? "GET";
	}

	async arrayBuffer(): Promise<ArrayBuffer> {
		const buffer = await this.#readAll();
		return Uint8Array.from(buffer).buffer;
	}

	async blob(): Promise<Blob> {
		const buffer = await this.#readAll();
		return new Blob([buffer]);
	}

	async json(): Promise<any> {
		const text = await this.text();
		return JSON.parse(text);
	}

	async text(): Promise<string> {
		const buffer = await this.#readAll();
		return buffer.toString();
	}

	async #readAll(): Promise<Buffer> {
		const buffers: Buffer[] = [];
		for await (const chunk of this.#incomingMessage) {
			buffers.push(chunk);
		}

		return Buffer.concat(buffers);
	}
}

Object.setPrototypeOf(FastRequest.prototype, Request.prototype);