diff --git a/README.md b/README.md index 7b68bb304..c4ef5f93a 100644 --- a/README.md +++ b/README.md @@ -1,793 +1,29 @@ -
- Node Fetch -
-

A light-weight module that brings Fetch API to Node.js.

- Build status - Coverage status - Current version - Install size - Mentioned in Awesome Node.js - Discord -
-
- Consider supporting us on our Open Collective: -
-
- Open Collective -
+# @web-std/node-fetch ---- +[![ci][ci.icon]][ci.url] +[![package][version.icon] ![downloads][downloads.icon]][package.url] - +Web API compatible [fetch API][] for nodejs. -- [Motivation](#motivation) -- [Features](#features) -- [Difference from client-side fetch](#difference-from-client-side-fetch) -- [Installation](#installation) -- [Loading and configuring the module](#loading-and-configuring-the-module) -- [Upgrading](#upgrading) -- [Common Usage](#common-usage) - - [Plain text or HTML](#plain-text-or-html) - - [JSON](#json) - - [Simple Post](#simple-post) - - [Post with JSON](#post-with-json) - - [Post with form parameters](#post-with-form-parameters) - - [Handling exceptions](#handling-exceptions) - - [Handling client and server errors](#handling-client-and-server-errors) - - [Handling cookies](#handling-cookies) -- [Advanced Usage](#advanced-usage) - - [Streams](#streams) - - [Buffer](#buffer) - - [Accessing Headers and other Meta data](#accessing-headers-and-other-meta-data) - - [Extract Set-Cookie Header](#extract-set-cookie-header) - - [Post data using a file stream](#post-data-using-a-file-stream) - - [Post with form-data (detect multipart)](#post-with-form-data-detect-multipart) - - [Request cancellation with AbortSignal](#request-cancellation-with-abortsignal) -- [API](#api) - - [fetch(url[, options])](#fetchurl-options) - - [Options](#options) - - [Default Headers](#default-headers) - - [Custom Agent](#custom-agent) - - [Custom highWaterMark](#custom-highwatermark) - - [Insecure HTTP Parser](#insecure-http-parser) - - [Class: Request](#class-request) - - [new Request(input[, options])](#new-requestinput-options) - - [Class: Response](#class-response) - - [new Response([body[, options]])](#new-responsebody-options) - - [response.ok](#responseok) - - [response.redirected](#responseredirected) - - [Class: Headers](#class-headers) - - [new Headers([init])](#new-headersinit) - - [Interface: Body](#interface-body) - - [body.body](#bodybody) - - [body.bodyUsed](#bodybodyused) - - [body.arrayBuffer()](#bodyarraybuffer) - - [body.blob()](#bodyblob) - - [body.json()](#bodyjson) - - [body.text()](#bodytext) - - [body.buffer()](#bodybuffer) - - [Class: FetchError](#class-fetcherror) - - [Class: AbortError](#class-aborterror) -- [TypeScript](#typescript) -- [Acknowledgement](#acknowledgement) -- [Team](#team) - - [Former](#former) -- [License](#license) +## Comparison to Alternatives - +#### [node-fetch][] -## Motivation +The reason this fork exists is because [node-fetch][] chooses to compromise +Web API compatibility and by useing nodejs native [Readable][] stream. They way +they put it is: -Instead of implementing `XMLHttpRequest` in Node.js to run browser-specific [Fetch polyfill](https://github.com/github/fetch), why not go from native `http` to `fetch` API directly? Hence, `node-fetch`, minimal code for a `window.fetch` compatible API on Node.js runtime. +> +> - Make conscious trade-off when following [WHATWG fetch spec][whatwg-fetch] and [stream spec](https://streams.spec.whatwg.org/) implementation details, document known differences. +> - Use native Node streams for body, on both request and response. +> -See Jason Miller's [isomorphic-unfetch](https://www.npmjs.com/package/isomorphic-unfetch) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch) for isomorphic usage (exports `node-fetch` for server-side, `whatwg-fetch` for client-side). +We found these incompatibility to be really problematic when sharing code +across nodejs and browser rutimes. Insteadead of introducing such an +incompatibility this library exposes nodejs streams through `nodeStream` +property. This library introduces web compatibility by lazily wrapping +underlying node streams via [web-streams-polyfill][] on property access. -## Features - -- Stay consistent with `window.fetch` API. -- Make conscious trade-off when following [WHATWG fetch spec][whatwg-fetch] and [stream spec](https://streams.spec.whatwg.org/) implementation details, document known differences. -- Use native promise and async functions. -- Use native Node streams for body, on both request and response. -- Decode content encoding (gzip/deflate/brotli) properly, and convert string output (such as `res.text()` and `res.json()`) to UTF-8 automatically. -- Useful extensions such as redirect limit, response size limit, [explicit errors][error-handling.md] for troubleshooting. - -## Difference from client-side fetch - -- See known differences: - - [As of v3.x](docs/v3-LIMITS.md) - - [As of v2.x](docs/v2-LIMITS.md) -- If you happen to use a missing feature that `window.fetch` offers, feel free to open an issue. -- Pull requests are welcomed too! - -## Installation - -Current stable release (`3.x`) - -```sh -npm install node-fetch -``` - -## Loading and configuring the module - -```js -// CommonJS -const fetch = require('node-fetch'); - -// ES Module -import fetch from 'node-fetch'; -``` - -If you want to patch the global object in node: - -```js -const fetch = require('node-fetch'); - -if (!globalThis.fetch) { - globalThis.fetch = fetch; -} -``` - -For versions of Node earlier than 12, use this `globalThis` [polyfill](https://mathiasbynens.be/notes/globalthis). - -## Upgrading - -Using an old version of node-fetch? Check out the following files: - -- [2.x to 3.x upgrade guide](docs/v3-UPGRADE-GUIDE.md) -- [1.x to 2.x upgrade guide](docs/v2-UPGRADE-GUIDE.md) -- [Changelog](docs/CHANGELOG.md) - -## Common Usage - -NOTE: The documentation below is up-to-date with `3.x` releases, if you are using an older version, please check how to [upgrade](#upgrading). - -### Plain text or HTML - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://github.com/'); -const body = await response.text(); - -console.log(body); -``` - -### JSON - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://api.github.com/users/github'); -const data = await response.json(); - -console.log(data); -``` - -### Simple Post - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://httpbin.org/post', {method: 'POST', body: 'a=1'}); -const data = await response.json(); - -console.log(data); -``` - -### Post with JSON - -```js -const fetch = require('node-fetch'); - -const body = {a: 1}; - -const response = await fetch('https://httpbin.org/post', { - method: 'post', - body: JSON.stringify(body), - headers: {'Content-Type': 'application/json'} -}); -const data = await response.json(); - -console.log(data); -``` - -### Post with form parameters - -`URLSearchParams` is available on the global object in Node.js as of v10.0.0. See [official documentation](https://nodejs.org/api/url.html#url_class_urlsearchparams) for more usage methods. - -NOTE: The `Content-Type` header is only set automatically to `x-www-form-urlencoded` when an instance of `URLSearchParams` is given as such: - -```js -const fetch = require('node-fetch'); - -const params = new URLSearchParams(); -params.append('a', 1); - -const response = await fetch('https://httpbin.org/post', {method: 'POST', body: params}); -const data = await response.json(); - -console.log(data); -``` - -### Handling exceptions - -NOTE: 3xx-5xx responses are _NOT_ exceptions, and should be handled in `then()`, see the next section. - -Wrapping the fetch function into a `try/catch` block will catch _all_ exceptions, such as errors originating from node core libraries, like network errors, and operational errors which are instances of FetchError. See the [error handling document][error-handling.md] for more details. - -```js -const fetch = require('node-fetch'); - -try { - await fetch('https://domain.invalid/'); -} catch (error) { - console.log(error); -} -``` - -### Handling client and server errors - -It is common to create a helper function to check that the response contains no client (4xx) or server (5xx) error responses: - -```js -const fetch = require('node-fetch'); - -class HTTPResponseError extends Error { - constructor(response, ...args) { - this.response = response; - super(`HTTP Error Response: ${response.status} ${response.statusText}`, ...args); - } -} - -const checkStatus = response => { - if (response.ok) { - // response.status >= 200 && response.status < 300 - return response; - } else { - throw new HTTPResponseError(response); - } -} - -const response = await fetch('https://httpbin.org/status/400'); - -try { - checkStatus(response); -} catch (error) { - console.error(error); - - const errorBody = await error.response.text(); - console.error(`Error body: ${errorBody}`); -} -``` - -### Handling cookies - -Cookies are not stored by default. However, cookies can be extracted and passed by manipulating request and response headers. See [Extract Set-Cookie Header](#extract-set-cookie-header) for details. - -## Advanced Usage - -### Streams - -The "Node.js way" is to use streams when possible. You can pipe `res.body` to another stream. This example uses [stream.pipeline](https://nodejs.org/api/stream.html#stream_stream_pipeline_streams_callback) to attach stream error handlers and wait for the download to complete. - -```js -const {createWriteStream} = require('fs'); -const {pipeline} = require('stream'); -const {promisify} = require('util'); -const fetch = require('node-fetch'); - -const streamPipeline = promisify(pipeline); - -const response = await fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png'); - -if (!response.ok) throw new Error(`unexpected response ${response.statusText}`); - -await streamPipeline(response.body, createWriteStream('./octocat.png')); -``` - -In Node.js 14 you can also use async iterators to read `body`; however, be careful to catch -errors -- the longer a response runs, the more likely it is to encounter an error. - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://httpbin.org/stream/3'); - -try { - for await (const chunk of response.body) { - console.dir(JSON.parse(chunk.toString())); - } -} catch (err) { - console.error(err.stack); -} -``` - -In Node.js 12 you can also use async iterators to read `body`; however, async iterators with streams -did not mature until Node.js 14, so you need to do some extra work to ensure you handle errors -directly from the stream and wait on it response to fully close. - -```js -const fetch = require('node-fetch'); - -const read = async body => { - let error; - body.on('error', err => { - error = err; - }); - - for await (const chunk of body) { - console.dir(JSON.parse(chunk.toString())); - } - - return new Promise((resolve, reject) => { - body.on('close', () => { - error ? reject(error) : resolve(); - }); - }); -}; - -try { - const response = await fetch('https://httpbin.org/stream/3'); - await read(response.body); -} catch (err) { - console.error(err.stack); -} -``` - -### Buffer - -If you prefer to cache binary data in full, use buffer(). (NOTE: buffer() is a `node-fetch` only API) - -```js -const fetch = require('node-fetch'); -const fileType = require('file-type'); - -const response = await fetch('https://octodex.github.com/images/Fintechtocat.png'); -const buffer = await response.buffer(); -const type = await fileType.fromBuffer(buffer) - -console.log(type); -``` - -### Accessing Headers and other Meta data - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://github.com/'); - -console.log(response.ok); -console.log(response.status); -console.log(response.statusText); -console.log(response.headers.raw()); -console.log(response.headers.get('content-type')); -``` - -### Extract Set-Cookie Header - -Unlike browsers, you can access raw `Set-Cookie` headers manually using `Headers.raw()`. This is a `node-fetch` only API. - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://example.com'); - -// Returns an array of values, instead of a string of comma-separated values -console.log(response.headers.raw()['set-cookie']); -``` - -### Post data using a file stream - -```js -const {createReadStream} = require('fs'); -const fetch = require('node-fetch'); - -const stream = createReadStream('input.txt'); - -const response = await fetch('https://httpbin.org/post', {method: 'POST', body: stream}); -const data = await response.json(); - -console.log(data) -``` - -### Post with form-data (detect multipart) - -```js -const fetch = require('node-fetch'); -const FormData = require('form-data'); - -const form = new FormData(); -form.append('a', 1); - -const response = await fetch('https://httpbin.org/post', {method: 'POST', body: form}); -const data = await response.json(); - -console.log(data) - -// OR, using custom headers -// NOTE: getHeaders() is non-standard API - -const options = { - method: 'POST', - body: form, - headers: form.getHeaders() -}; - -const response = await fetch('https://httpbin.org/post', options); -const data = await response.json(); - -console.log(data) -``` - -node-fetch also supports spec-compliant FormData implementations such as [form-data](https://github.com/form-data/form-data) and [formdata-node](https://github.com/octet-stream/form-data): - -```js -const fetch = require('node-fetch'); -const FormData = require('formdata-node'); - -const form = new FormData(); -form.set('greeting', 'Hello, world!'); - -const response = await fetch('https://httpbin.org/post', {method: 'POST', body: form}); -const data = await response.json(); - -console.log(data); -``` - -### Request cancellation with AbortSignal - -You may cancel requests with `AbortController`. A suggested implementation is [`abort-controller`](https://www.npmjs.com/package/abort-controller). - -An example of timing out a request after 150ms could be achieved as the following: - -```js -const fetch = require('node-fetch'); -const AbortController = require('abort-controller'); - -const controller = new AbortController(); -const timeout = setTimeout(() => { - controller.abort(); -}, 150); - -try { - const response = await fetch('https://example.com', {signal: controller.signal}); - const data = await response.json(); -} catch (error) { - if (error instanceof fetch.AbortError) { - console.log('request was aborted'); - } -} finally { - clearTimeout(timeout); -} -``` - -See [test cases](https://github.com/node-fetch/node-fetch/blob/master/test/) for more examples. - -## API - -### fetch(url[, options]) - -- `url` A string representing the URL for fetching -- `options` [Options](#fetch-options) for the HTTP(S) request -- Returns: Promise<[Response](#class-response)> - -Perform an HTTP(S) fetch. - -`url` should be an absolute url, such as `https://example.com/`. A path-relative URL (`/file/under/root`) or protocol-relative URL (`//can-be-http-or-https.com/`) will result in a rejected `Promise`. - - - -### Options - -The default values are shown after each option key. - -```js -{ - // These properties are part of the Fetch Standard - method: 'GET', - headers: {}, // Request headers. format is the identical to that accepted by the Headers constructor (see below) - body: null, // Request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream - redirect: 'follow', // Set to `manual` to extract redirect headers, `error` to reject redirect - signal: null, // Pass an instance of AbortSignal to optionally abort requests - - // The following properties are node-fetch extensions - follow: 20, // maximum redirect count. 0 to not follow redirect - compress: true, // support gzip/deflate content encoding. false to disable - size: 0, // maximum response body size in bytes. 0 to disable - agent: null, // http(s).Agent instance or function that returns an instance (see below) - highWaterMark: 16384, // the maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource. - insecureHTTPParser: false // Use an insecure HTTP parser that accepts invalid HTTP headers when `true`. -} -``` - -#### Default Headers - -If no values are set, the following request headers will be sent automatically: - -| Header | Value | -| ------------------- | ------------------------------------------------------ | -| `Accept-Encoding` | `gzip,deflate,br` _(when `options.compress === true`)_ | -| `Accept` | `*/*` | -| `Connection` | `close` _(when no `options.agent` is present)_ | -| `Content-Length` | _(automatically calculated, if possible)_ | -| `Transfer-Encoding` | `chunked` _(when `req.body` is a stream)_ | -| `User-Agent` | `node-fetch` | - - -Note: when `body` is a `Stream`, `Content-Length` is not set automatically. - -#### Custom Agent - -The `agent` option allows you to specify networking related options which are out of the scope of Fetch, including and not limited to the following: - -- Support self-signed certificate -- Use only IPv4 or IPv6 -- Custom DNS Lookup - -See [`http.Agent`](https://nodejs.org/api/http.html#http_new_agent_options) for more information. - -In addition, the `agent` option accepts a function that returns `http`(s)`.Agent` instance given current [URL](https://nodejs.org/api/url.html), this is useful during a redirection chain across HTTP and HTTPS protocol. - -```js -const http = require('http'); -const https = require('https'); - -const httpAgent = new http.Agent({ - keepAlive: true -}); -const httpsAgent = new https.Agent({ - keepAlive: true -}); - -const options = { - agent: function(_parsedURL) { - if (_parsedURL.protocol == 'http:') { - return httpAgent; - } else { - return httpsAgent; - } - } -}; -``` - - - -#### Custom highWaterMark - -Stream on Node.js have a smaller internal buffer size (16kB, aka `highWaterMark`) from client-side browsers (>1MB, not consistent across browsers). Because of that, when you are writing an isomorphic app and using `res.clone()`, it will hang with large response in Node. - -The recommended way to fix this problem is to resolve cloned response in parallel: - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://example.com'); -const r1 = await response.clone(); - -const results = await Promise.all([response.json(), r1.text()]); - -console.log(results[0]); -console.log(results[1]); -``` - -If for some reason you don't like the solution above, since `3.x` you are able to modify the `highWaterMark` option: - -```js -const fetch = require('node-fetch'); - -const response = await fetch('https://example.com', { - // About 1MB - highWaterMark: 1024 * 1024 -}); - -const result = await res.clone().buffer(); -console.dir(result); -``` - -#### Insecure HTTP Parser - -Passed through to the `insecureHTTPParser` option on http(s).request. See [`http.request`](https://nodejs.org/api/http.html#http_http_request_url_options_callback) for more information. - - - - -### Class: Request - -An HTTP(S) request containing information about URL, method, headers, and the body. This class implements the [Body](#iface-body) interface. - -Due to the nature of Node.js, the following properties are not implemented at this moment: - -- `type` -- `destination` -- `referrer` -- `referrerPolicy` -- `mode` -- `credentials` -- `cache` -- `integrity` -- `keepalive` - -The following node-fetch extension properties are provided: - -- `follow` -- `compress` -- `counter` -- `agent` -- `highWaterMark` - -See [options](#fetch-options) for exact meaning of these extensions. - -#### new Request(input[, options]) - -_(spec-compliant)_ - -- `input` A string representing a URL, or another `Request` (which will be cloned) -- `options` [Options][#fetch-options] for the HTTP(S) request - -Constructs a new `Request` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request). - -In most cases, directly `fetch(url, options)` is simpler than creating a `Request` object. - - - -### Class: Response - -An HTTP(S) response. This class implements the [Body](#iface-body) interface. - -The following properties are not implemented in node-fetch at this moment: - -- `Response.error()` -- `Response.redirect()` -- `type` -- `trailer` - -#### new Response([body[, options]]) - -_(spec-compliant)_ - -- `body` A `String` or [`Readable` stream][node-readable] -- `options` A [`ResponseInit`][response-init] options dictionary - -Constructs a new `Response` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response). - -Because Node.js does not implement service workers (for which this class was designed), one rarely has to construct a `Response` directly. - -#### response.ok - -_(spec-compliant)_ - -Convenience property representing if the request ended normally. Will evaluate to true if the response status was greater than or equal to 200 but smaller than 300. - -#### response.redirected - -_(spec-compliant)_ - -Convenience property representing if the request has been redirected at least once. Will evaluate to true if the internal redirect counter is greater than 0. - - - -### Class: Headers - -This class allows manipulating and iterating over a set of HTTP headers. All methods specified in the [Fetch Standard][whatwg-fetch] are implemented. - -#### new Headers([init]) - -_(spec-compliant)_ - -- `init` Optional argument to pre-fill the `Headers` object - -Construct a new `Headers` object. `init` can be either `null`, a `Headers` object, an key-value map object or any iterable object. - -```js -// Example adapted from https://fetch.spec.whatwg.org/#example-headers-class -const {Headers} = require('node-fetch'); - -const meta = { - 'Content-Type': 'text/xml', - 'Breaking-Bad': '<3' -}; -const headers = new Headers(meta); - -// The above is equivalent to -const meta = [['Content-Type', 'text/xml'], ['Breaking-Bad', '<3']]; -const headers = new Headers(meta); - -// You can in fact use any iterable objects, like a Map or even another Headers -const meta = new Map(); -meta.set('Content-Type', 'text/xml'); -meta.set('Breaking-Bad', '<3'); -const headers = new Headers(meta); -const copyOfHeaders = new Headers(headers); -``` - - - -### Interface: Body - -`Body` is an abstract interface with methods that are applicable to both `Request` and `Response` classes. - -The following methods are not yet implemented in node-fetch at this moment: - -- `formData()` - -#### body.body - -_(deviation from spec)_ - -- Node.js [`Readable` stream][node-readable] - -Data are encapsulated in the `Body` object. Note that while the [Fetch Standard][whatwg-fetch] requires the property to always be a WHATWG `ReadableStream`, in node-fetch it is a Node.js [`Readable` stream][node-readable]. - -#### body.bodyUsed - -_(spec-compliant)_ - -- `Boolean` - -A boolean property for if this body has been consumed. Per the specs, a consumed body cannot be used again. - -#### body.arrayBuffer() - -#### body.blob() - -#### body.json() - -#### body.text() - -_(spec-compliant)_ - -- Returns: `Promise` - -Consume the body and return a promise that will resolve to one of these formats. - -#### body.buffer() - -_(node-fetch extension)_ - -- Returns: `Promise` - -Consume the body and return a promise that will resolve to a Buffer. - - - -### Class: FetchError - -_(node-fetch extension)_ - -An operational error in the fetching process. See [ERROR-HANDLING.md][] for more info. - - - -### Class: AbortError - -_(node-fetch extension)_ - -An Error thrown when the request is aborted in response to an `AbortSignal`'s `abort` event. It has a `name` property of `AbortError`. See [ERROR-HANDLING.MD][] for more info. - -## TypeScript - -**Since `3.x` types are bundled with `node-fetch`, so you don't need to install any additional packages.** - -For older versions please use the type definitions from [DefinitelyTyped](https://github.com/DefinitelyTyped/DefinitelyTyped): - -```sh -npm install --save-dev @types/node-fetch -``` - -## Acknowledgement - -Thanks to [github/fetch](https://github.com/github/fetch) for providing a solid implementation reference. - -## Team - -| [![David Frank](https://github.com/bitinn.png?size=100)](https://github.com/bitinn) | [![Jimmy Wärting](https://github.com/jimmywarting.png?size=100)](https://github.com/jimmywarting) | [![Antoni Kepinski](https://github.com/xxczaki.png?size=100)](https://github.com/xxczaki) | [![Richie Bendall](https://github.com/Richienb.png?size=100)](https://github.com/Richienb) | [![Gregor Martynus](https://github.com/gr2m.png?size=100)](https://github.com/gr2m) | -| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | -| [David Frank](https://bitinn.net/) | [Jimmy Wärting](https://jimmy.warting.se/) | [Antoni Kepinski](https://kepinski.me) | [Richie Bendall](https://www.richie-bendall.ml/) | [Gregor Martynus](https://twitter.com/gr2m) | - -###### Former - -- [Timothy Gu](https://github.com/timothygu) -- [Jared Kantrowitz](https://github.com/jkantr) ## License @@ -798,3 +34,27 @@ Thanks to [github/fetch](https://github.com/github/fetch) for providing a solid [node-readable]: https://nodejs.org/api/stream.html#stream_readable_streams [mdn-headers]: https://developer.mozilla.org/en-US/docs/Web/API/Headers [error-handling.md]: https://github.com/node-fetch/node-fetch/blob/master/docs/ERROR-HANDLING.md + +[ci.icon]: https://github.com/web-std/node-fetch/workflows/CI/badge.svg +[ci.url]: https://github.com/web-std/node-fetch/actions +[version.icon]: https://img.shields.io/npm/v/@web-std/node-fetch.svg +[downloads.icon]: https://img.shields.io/npm/dm/@web-std/node-fetch.svg +[package.url]: https://npmjs.org/package/@web-std/node-fetch +[downloads.image]: https://img.shields.io/npm/dm/@web-std/node-fetch.svg +[downloads.url]: https://npmjs.org/package/@web-std/node-fetch +[prettier.icon]: https://img.shields.io/badge/styled_with-prettier-ff69b4.svg +[prettier.url]: https://github.com/prettier/prettier +[blob]: https://developer.mozilla.org/en-US/docs/Web/API/Blob/Blob +[fetch-blob]: https://github.com/node-fetch/fetch-blob +[readablestream]: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream +[readable]: https://nodejs.org/api/stream.html#stream_readable_streams +[w3c blob.stream]: https://w3c.github.io/FileAPI/#dom-blob-stream +[web-streams-polyfill]:https://www.npmjs.com/package/web-streams-polyfill +[for await]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of +[buffer]: https://nodejs.org/api/buffer.html +[weakmap]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakMap +[ts-jsdoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html +[Uint8Array]:https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array +[node-fetch]:https://github.com/node-fetch/ +[fetch api]:https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API +[readable]: https://nodejs.org/api/stream.html#stream_readable_streams diff --git a/package.json b/package.json index efb081a62..16ffd5fa9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { - "name": "node-fetch", - "version": "3.0.0-beta.9", + "name": "@web-std/node-fetch", + "version": "1.0.0", "description": "A light-weight module that brings Fetch API to node.js", "main": "./dist/index.cjs", "module": "./src/index.js", diff --git a/src/body.js b/src/body.js index f1233034d..21d5436bc 100644 --- a/src/body.js +++ b/src/body.js @@ -14,19 +14,26 @@ import {FetchError} from './errors/fetch-error.js'; import {FetchBaseError} from './errors/base.js'; import {formDataIterator, getBoundary, getFormDataLength} from './utils/form-data.js'; import {isBlob, isURLSearchParameters, isFormData} from './utils/is.js'; +import {blobToNodeStream} from './utils/blob-to-stream.js'; const INTERNALS = Symbol('Body internals'); +export const BODY = Symbol('Body content'); /** * Body mixin * * Ref: https://fetch.spec.whatwg.org/#body * - * @param Stream body Readable stream + * @param Stream stream Readable stream * @param Object opts Response options * @return Void */ export default class Body { + /** + * + * @param {BodyInit|Stream} body + * @param {{size?:number}} options + */ constructor(body, { size = 0 } = {}) { @@ -61,6 +68,7 @@ export default class Body { } this[INTERNALS] = { + /** @type {Stream|Buffer|Blob|null} */ body, boundary, disturbed: false, @@ -78,8 +86,8 @@ export default class Body { } } - get body() { - return this[INTERNALS].body; + get [BODY]() { + return this[INTERNALS].body } get bodyUsed() { @@ -155,7 +163,8 @@ Object.defineProperties(Body.prototype, { * * Ref: https://fetch.spec.whatwg.org/#concept-body-consume-body * - * @return Promise + * @param {Body} data + * @return {Promise} */ async function consumeBody(data) { if (data[INTERNALS].disturbed) { @@ -168,7 +177,7 @@ async function consumeBody(data) { throw data[INTERNALS].error; } - let {body} = data; + let {body} = data[INTERNALS]; // Body is null if (body === null) { @@ -177,7 +186,7 @@ async function consumeBody(data) { // Body is blob if (isBlob(body)) { - body = body.stream(); + body = blobToNodeStream(body); } // Body is buffer @@ -197,14 +206,15 @@ async function consumeBody(data) { try { for await (const chunk of body) { - if (data.size > 0 && accumBytes + chunk.length > data.size) { + const bytes = typeof chunk === 'string' ? Buffer.from(chunk) : chunk; + if (data.size > 0 && accumBytes + bytes.byteLength > data.size) { const err = new FetchError(`content size at ${data.url} over limit: ${data.size}`, 'max-size'); body.destroy(err); throw err; } - accumBytes += chunk.length; - accum.push(chunk); + accumBytes += bytes.byteLength; + accum.push(bytes); } } catch (error) { if (error instanceof FetchBaseError) { @@ -217,10 +227,6 @@ async function consumeBody(data) { if (body.readableEnded === true || body._readableState.ended === true) { try { - if (accum.every(c => typeof c === 'string')) { - return Buffer.from(accum.join('')); - } - return Buffer.concat(accum, accumBytes); } catch (error) { throw new FetchError(`Could not create Buffer from response body for ${data.url}: ${error.message}`, 'system', error); @@ -240,7 +246,7 @@ async function consumeBody(data) { export const clone = (instance, highWaterMark) => { let p1; let p2; - let {body} = instance; + let {body} = instance[INTERNALS]; // Don't allow cloning a used body if (instance.bodyUsed) { @@ -323,11 +329,11 @@ export const extractContentType = (body, request) => { * * ref: https://fetch.spec.whatwg.org/#concept-body-total-bytes * - * @param {any} obj.body Body object from the Body instance. + * @param {Body} request Body object from the Body instance. * @returns {number | null} */ export const getTotalBytes = request => { - const {body} = request; + const body = request[BODY]; // Body is null or undefined if (body === null) { @@ -362,16 +368,16 @@ export const getTotalBytes = request => { * Write a Body to a Node.js WritableStream (e.g. http.Request) object. * * @param {Stream.Writable} dest The stream to write to. - * @param obj.body Body object from the Body instance. + * @param {null|Buffer|Blob|Stream} body Body object from the Body instance. * @returns {void} */ -export const writeToStream = (dest, {body}) => { +export const writeToStream = (dest, body) => { if (body === null) { // Body is null dest.end(); } else if (isBlob(body)) { // Body is Blob - body.stream().pipe(dest); + blobToNodeStream(body).pipe(dest); } else if (Buffer.isBuffer(body)) { // Body is buffer dest.write(body); diff --git a/src/index.js b/src/index.js index a46e65f1e..195bcc96f 100644 --- a/src/index.js +++ b/src/index.js @@ -12,7 +12,7 @@ import zlib from 'zlib'; import Stream, {PassThrough, pipeline as pump} from 'stream'; import dataUriToBuffer from 'data-uri-to-buffer'; -import {writeToStream} from './body.js'; +import {writeToStream, BODY} from './body.js'; import Response from './response.js'; import Headers, {fromRawHeaders} from './headers.js'; import Request, {getNodeRequestOptions} from './request.js'; @@ -55,15 +55,15 @@ export default async function fetch(url, options_) { const abort = () => { const error = new AbortError('The operation was aborted.'); reject(error); - if (request.body && request.body instanceof Stream.Readable) { - request.body.destroy(error); + if (request[BODY] && request[BODY] instanceof Stream.Readable) { + request[BODY].destroy(error); } - if (!response || !response.body) { + if (!response || !response[BODY]) { return; } - response.body.emit('error', error); + response[BODY].emit('error', error); }; if (signal && signal.aborted) { @@ -96,7 +96,7 @@ export default async function fetch(url, options_) { }); fixResponseChunkedTransferBadEnding(request_, err => { - response.body.destroy(err); + response[BODY].destroy(err); }); /* c8 ignore next 18 */ @@ -113,7 +113,7 @@ export default async function fetch(url, options_) { if (response && endedWithEventsCount < s._eventsCount && !hadError) { const err = new Error('Premature close'); err.code = 'ERR_STREAM_PREMATURE_CLOSE'; - response.body.emit('error', err); + response[BODY].emit('error', err); } }); }); @@ -166,13 +166,13 @@ export default async function fetch(url, options_) { agent: request.agent, compress: request.compress, method: request.method, - body: request.body, + body: request[BODY], signal: request.signal, size: request.size }; // HTTP-redirect fetch step 9 - if (response_.statusCode !== 303 && request.body && options_.body instanceof Stream.Readable) { + if (response_.statusCode !== 303 && request[BODY] && options_.body instanceof Stream.Readable) { reject(new FetchError('Cannot follow redirect with body being a readable stream', 'unsupported-redirect')); finalize(); return; @@ -286,7 +286,7 @@ export default async function fetch(url, options_) { resolve(response); }); - writeToStream(request_, request); + writeToStream(request_, request[BODY]); }); } diff --git a/src/request.js b/src/request.js index 05999fa9d..5b50cda7b 100644 --- a/src/request.js +++ b/src/request.js @@ -9,7 +9,7 @@ import {format as formatUrl} from 'url'; import Headers from './headers.js'; -import Body, {clone, extractContentType, getTotalBytes} from './body.js'; +import Body, {clone, extractContentType, getTotalBytes, BODY} from './body.js'; import {isAbortSignal} from './utils/is.js'; import {getSearch} from './utils/get-search.js'; @@ -18,8 +18,8 @@ const INTERNALS = Symbol('Request internals'); /** * Check if `obj` is an instance of Request. * - * @param {*} obj - * @return {boolean} + * @param {any} object + * @return {object is Request} */ const isRequest = object => { return ( @@ -36,6 +36,10 @@ const isRequest = object => { * @return Void */ export default class Request extends Body { + /** + * @param {RequestInfo} input Url or Request instance + * @param {RequestInit} init Custom options + */ constructor(input, init = {}) { let parsedURL; @@ -51,14 +55,14 @@ export default class Request extends Body { method = method.toUpperCase(); // eslint-disable-next-line no-eq-null, eqeqeq - if (((init.body != null || isRequest(input)) && input.body !== null) && + if (((isRequest(input) || init.body != null) && input[BODY] !== null) && (method === 'GET' || method === 'HEAD')) { throw new TypeError('Request with GET/HEAD method cannot have body'); } const inputBody = init.body ? init.body : - (isRequest(input) && input.body !== null ? + (isRequest(input) && input[BODY] !== null ? clone(input) : null); @@ -150,7 +154,7 @@ Object.defineProperties(Request.prototype, { /** * Convert a Request to Node.js http request options. * - * @param Request A Request instance + * @param {Request} A Request instance * @return Object The options object to be passed to http.request */ export const getNodeRequestOptions = request => { @@ -164,11 +168,11 @@ export const getNodeRequestOptions = request => { // HTTP-network-or-cache fetch steps 2.4-2.7 let contentLengthValue = null; - if (request.body === null && /^(post|put)$/i.test(request.method)) { + if (request[BODY] === null && /^(post|put)$/i.test(request.method)) { contentLengthValue = '0'; } - if (request.body !== null) { + if (request[BODY] !== null) { const totalBytes = getTotalBytes(request); // Set Content-Length if totalBytes is a number (that is not NaN) if (typeof totalBytes === 'number' && !Number.isNaN(totalBytes)) { diff --git a/src/response.js b/src/response.js index 3c69d5e9d..d1d6a6e8d 100644 --- a/src/response.js +++ b/src/response.js @@ -25,7 +25,7 @@ export default class Response extends Body { const headers = new Headers(options.headers); if (body !== null && !headers.has('Content-Type')) { - const contentType = extractContentType(body); + const contentType = extractContentType(body, this); if (contentType) { headers.append('Content-Type', contentType); } diff --git a/src/utils/blob-to-stream.js b/src/utils/blob-to-stream.js new file mode 100644 index 000000000..824956252 --- /dev/null +++ b/src/utils/blob-to-stream.js @@ -0,0 +1,23 @@ +import {Readable} from 'stream'; + +// 64 KiB (same size chrome slice theirs blob into Uint8array's) +const POOL_SIZE = 65536; + +/* c8 ignore start */ +async function * read(blob) { + let position = 0; + while (position !== blob.size) { + const chunk = blob.slice(position, Math.min(blob.size, position + POOL_SIZE)); + // eslint-disable-next-line no-await-in-loop + const buffer = await chunk.arrayBuffer(); + position += buffer.byteLength; + yield new Uint8Array(buffer); + } +} +/* c8 ignore end */ + +export function blobToNodeStream(blob) { + return Readable.from(blob.stream ? blob.stream() : read(blob), { + objectMode: false + }); +} diff --git a/src/utils/is.js b/src/utils/is.js index fa8d15922..fb401309e 100644 --- a/src/utils/is.js +++ b/src/utils/is.js @@ -11,7 +11,7 @@ const NAME = Symbol.toStringTag; * ref: https://github.com/node-fetch/node-fetch/issues/296#issuecomment-307598143 * * @param {*} obj - * @return {boolean} + * @return {object is URLSearchParams} */ export const isURLSearchParameters = object => { return ( @@ -31,16 +31,20 @@ export const isURLSearchParameters = object => { * Check if `object` is a W3C `Blob` object (which `File` inherits from) * * @param {*} obj - * @return {boolean} + * @return {object is Blob} */ export const isBlob = object => { return ( typeof object === 'object' && typeof object.arrayBuffer === 'function' && typeof object.type === 'string' && - typeof object.stream === 'function' && + // typeof object.stream === 'function' && typeof object.constructor === 'function' && - /^(Blob|File)$/.test(object[NAME]) + ( + /* c8 ignore next 2 */ + /^(Blob|File)$/.test(object[NAME]) || + /^(Blob|File)$/.test(object.constructor.name) + ) ); }; @@ -48,7 +52,7 @@ export const isBlob = object => { * Check if `obj` is a spec-compliant `FormData` object * * @param {*} object - * @return {boolean} + * @return {object is FormData} */ export function isFormData(object) { return ( @@ -70,7 +74,7 @@ export function isFormData(object) { * Check if `obj` is an instance of AbortSignal. * * @param {*} obj - * @return {boolean} + * @return {object is AbortSignal} */ export const isAbortSignal = object => { return ( diff --git a/test/form-data.js b/test/form-data.js index fe08fe4c6..d0003aea0 100644 --- a/test/form-data.js +++ b/test/form-data.js @@ -1,5 +1,7 @@ import FormData from 'formdata-node'; import Blob from 'fetch-blob'; +import {Response, Request} from '../src/index.js'; +import {getTotalBytes, BODY} from '../src/body.js'; import chai from 'chai'; @@ -101,4 +103,30 @@ describe('FormData', () => { expect(String(await read(formDataIterator(form, boundary)))).to.be.equal(expected); }); + + it('Response supports FormData body', async () => { + const form = new FormData(); + form.set('blob', new Blob(['Hello, World!'], {type: 'text/plain'})); + + const response = new Response(form); + const type = response.headers.get('content-type') || ''; + expect(type).to.match(/multipart\/form-data;\s*boundary=/); + expect(await response.text()).to.have.string('Hello, World!'); + // Note: getTotalBytes assumes body could be form data but it never is + // because it gets normalized into a stream. + expect(getTotalBytes({...response, [BODY]: form})).to.be.greaterThan(20); + }); + + it('Request supports FormData body', async () => { + const form = new FormData(); + form.set('blob', new Blob(['Hello, World!'], {type: 'text/plain'})); + + const request = new Request('https://github.com/node-fetch/', { + body: form, + method: 'POST' + }); + const type = request.headers.get('content-type') || ''; + expect(type).to.match(/multipart\/form-data;\s*boundary=/); + expect(await request.text()).to.have.string('Hello, World!'); + }); }); diff --git a/test/main.js b/test/main.js index 49e6e7eb6..92a4125ab 100644 --- a/test/main.js +++ b/test/main.js @@ -32,7 +32,7 @@ import {FetchError as FetchErrorOrig} from '../src/errors/fetch-error.js'; import HeadersOrig, {fromRawHeaders} from '../src/headers.js'; import RequestOrig from '../src/request.js'; import ResponseOrig from '../src/response.js'; -import Body, {getTotalBytes, extractContentType} from '../src/body.js'; +import Body, {getTotalBytes, extractContentType, BODY} from '../src/body.js'; import TestServer from './utils/server.js'; const { @@ -138,7 +138,7 @@ describe('node-fetch', () => { return fetch(url).then(res => { expect(res).to.be.an.instanceof(Response); expect(res.headers).to.be.an.instanceof(Headers); - expect(res.body).to.be.an.instanceof(stream.Transform); + expect(res[BODY]).to.be.an.instanceof(stream.Transform); expect(res.bodyUsed).to.be.false; expect(res.url).to.equal(url); @@ -619,8 +619,8 @@ describe('node-fetch', () => { expect(res.ok).to.be.true; return expect(new Promise((resolve, reject) => { - res.body.on('error', reject); - res.body.on('close', resolve); + res[BODY].on('error', reject); + res[BODY].on('close', resolve); })).to.eventually.be.rejectedWith(Error, 'Premature close') .and.have.property('code', 'ERR_STREAM_PREMATURE_CLOSE'); }); @@ -663,7 +663,7 @@ describe('node-fetch', () => { return chunks; }; - return expect(read(res.body)) + return expect(read(res[BODY])) .to.eventually.be.rejectedWith(Error, 'Premature close') .and.have.property('code', 'ERR_STREAM_PREMATURE_CLOSE'); }); @@ -1071,7 +1071,7 @@ describe('node-fetch', () => { )) .to.eventually.be.fulfilled .then(res => { - res.body.once('error', err => { + res[BODY].once('error', err => { expect(err) .to.be.an.instanceof(Error) .and.have.property('name', 'AbortError'); @@ -1704,7 +1704,7 @@ describe('node-fetch', () => { expect(res.status).to.equal(200); expect(res.statusText).to.equal('OK'); expect(res.headers.get('content-type')).to.equal('text/plain'); - expect(res.body).to.be.an.instanceof(stream.Transform); + expect(res[BODY]).to.be.an.instanceof(stream.Transform); return res.text(); }).then(text => { expect(text).to.equal(''); @@ -1734,7 +1734,7 @@ describe('node-fetch', () => { expect(res.status).to.equal(200); expect(res.statusText).to.equal('OK'); expect(res.headers.get('allow')).to.equal('GET, HEAD, OPTIONS'); - expect(res.body).to.be.an.instanceof(stream.Transform); + expect(res[BODY]).to.be.an.instanceof(stream.Transform); }); }); @@ -1780,8 +1780,8 @@ describe('node-fetch', () => { it('should allow piping response body as stream', () => { const url = `${base}hello`; return fetch(url).then(res => { - expect(res.body).to.be.an.instanceof(stream.Transform); - return streamToPromise(res.body, chunk => { + expect(res[BODY]).to.be.an.instanceof(stream.Transform); + return streamToPromise(res[BODY], chunk => { if (chunk === null) { return; } @@ -1795,8 +1795,8 @@ describe('node-fetch', () => { const url = `${base}hello`; return fetch(url).then(res => { const r1 = res.clone(); - expect(res.body).to.be.an.instanceof(stream.Transform); - expect(r1.body).to.be.an.instanceof(stream.Transform); + expect(res[BODY]).to.be.an.instanceof(stream.Transform); + expect(r1[BODY]).to.be.an.instanceof(stream.Transform); const dataHandler = chunk => { if (chunk === null) { return; @@ -1806,8 +1806,8 @@ describe('node-fetch', () => { }; return Promise.all([ - streamToPromise(res.body, dataHandler), - streamToPromise(r1.body, dataHandler) + streamToPromise(res[BODY], dataHandler), + streamToPromise(r1[BODY], dataHandler) ]); }); }); diff --git a/test/request.js b/test/request.js index 19fb8af3b..4fbe7cf78 100644 --- a/test/request.js +++ b/test/request.js @@ -10,6 +10,7 @@ import Blob from 'fetch-blob'; import TestServer from './utils/server.js'; import {Request} from '../src/index.js'; +import {BODY} from '../src/body.js' const {expect} = chai; @@ -80,7 +81,7 @@ describe('Request', () => { expect(r2.method).to.equal('POST'); expect(r2.signal).to.equal(signal); // Note that we didn't clone the body - expect(r2.body).to.equal(form); + expect(r2[BODY]).to.equal(form); expect(r1.follow).to.equal(1); expect(r2.follow).to.equal(2); expect(r1.counter).to.equal(0); @@ -129,7 +130,7 @@ describe('Request', () => { it('should default to null as body', () => { const request = new Request(base); - expect(request.body).to.equal(null); + expect(request[BODY]).to.equal(null); return request.text().then(result => expect(result).to.equal('')); }); @@ -237,7 +238,7 @@ describe('Request', () => { expect(cl.agent).to.equal(agent); expect(cl.signal).to.equal(signal); // Clone body shouldn't be the same body - expect(cl.body).to.not.equal(body); + expect(cl[BODY]).to.not.equal(body); return Promise.all([cl.text(), request.text()]).then(results => { expect(results[0]).to.equal('a=1'); expect(results[1]).to.equal('a=1'); diff --git a/test/response.js b/test/response.js index f02b67f4d..6454e42cd 100644 --- a/test/response.js +++ b/test/response.js @@ -3,8 +3,10 @@ import * as stream from 'stream'; import {TextEncoder} from 'util'; import chai from 'chai'; import Blob from 'fetch-blob'; +import buffer from 'buffer'; import {Response} from '../src/index.js'; import TestServer from './utils/server.js'; +import {BODY} from '../src/body.js' const {expect} = chai; @@ -130,7 +132,7 @@ describe('Response', () => { expect(cl.statusText).to.equal('production'); expect(cl.ok).to.be.false; // Clone body shouldn't be the same body - expect(cl.body).to.not.equal(body); + expect(cl[BODY]).to.not.equal(body); return cl.text().then(result => { expect(result).to.equal('a=1'); }); @@ -173,6 +175,15 @@ describe('Response', () => { }); }); + if (buffer.Blob) { + it('should support Buffer.Blob as body', () => { + const res = new Response(new buffer.Blob(['a=1'])); + return res.text().then(result => { + expect(result).to.equal('a=1'); + }); + }); + } + it('should support Uint8Array as body', () => { const encoder = new TextEncoder(); const res = new Response(encoder.encode('a=1')); @@ -191,7 +202,7 @@ describe('Response', () => { it('should default to null as body', () => { const res = new Response(); - expect(res.body).to.equal(null); + expect(res[BODY]).to.equal(null); return res.text().then(result => expect(result).to.equal('')); });