diff --git a/public/img/manuscript-on-wooden-desk.webp b/public/img/manuscript-on-wooden-desk.webp new file mode 100644 index 0000000..7eafd2d Binary files /dev/null and b/public/img/manuscript-on-wooden-desk.webp differ diff --git a/src/content/drafts/nodejs-custom-streams.md b/src/content/drafts/nodejs-custom-streams.md new file mode 100644 index 0000000..0767e37 --- /dev/null +++ b/src/content/drafts/nodejs-custom-streams.md @@ -0,0 +1,191 @@ +--- +title: Node.js Custom Streams +date: 2024-06-14 +author: Nathan +desc: | + How to create custom Node.JS readable, writable and transformer stream + components. +img: /img/manuscript-on-wooden-desk.webp +--- + + +
+ +
+
This is a draft!
+
+
+ + +
+ +
+
This is part of a series on handling streams with Node.js
+ Part 1: Node.js Stream Composition
+ Part 2: Node.js Custom Streams
+ Part 3: [[Streaming to AWS S3 with Node.js]]
+ Part 4: [[Streaming JSON Lines with Node.js]]
+
+
+ + + +## Overview + +Node.js `stream.Readable`, `stream.Writable` and `stream.Transform` are base +classes that can be extended to create custom streams. They have internal +methods that are intended to be overridden by implementors. The internal +methods are prefixed with an underscore. The internal methods should not be +invoked directly. + +Each of the base classes has a constructor which takes options to control the +stream behavior. The base constructor must be invoked even when no options are +supplied.  + +## Object Mode + +The “objectMode” option allows a stream to handle objects instead of buffers. +Transform implementations can also specify object mode for just the readable or +writable sides of the stream by using “readableObjectMode” or +“writableObjectMode” options respectively. + +When composing stream pipelines, the modes for each interfacing stream +component must match. For example, piping a readable in object mode into a +non-object mode writable like +[zlib.Gzip]([https://nodejs.org/api/zlib.html#zlibcreategzipoptions](https://nodejs.org/api/zlib.html#zlibcreategzipoptions)) +would produce an error. + +See: [https://nodejs.org/api/stream.html#object-mode](https://nodejs.org/api/stream.html#object-mode) + +## Shorthand Syntax + +The stream.Readable, stream.Writable and stream.Transform base classes can be +instantiated directly and the overridable methods can be provided as +constructor options. This pattern avoids some boilerplate but makes the code +harder to understand. Defining a new class that extends Readable, Writable or +Transform will improve clarity. + +## Custom Readable + +Readable is implemented by extending stream.Readable and overriding the +`_read()` method. The `_read()` implementation should fetch underlying data and +call `this.push()` until pushing returns false. The value `null` is pushed to +indicate the end of the stream. + +### Official Guide for Implementing Readable +[https://nodejs.org/api/stream.html#implementing-a-readable-stream](https://nodejs.org/api/stream.html#implementing-a-readable-stream) + +### Tips for Implementing Readable + +- Readable is the more complicated stream type to implement because back + pressure needs to be respected. +- The null reference pushed to end a readable stream is not propagated to the + connected writable streams. + +### Respecting Readable Back Pressure + +The push function returns false to indicate back pressure is being applied. +This will happen when the stream’s buffer high-water mark is met. Respecting +the back pressure signal prevents buffers from growing faster than they can be +consumed. Respecting back pressure significantly improves memory efficiency and +allows a streaming application to handle arbitrarily large streams. + +### Readable Example Using Item Emitter + +Delegating to a stateful [EventEmitter](https://nodejs.dev/en/api/v19/events/#eventemitter) that can be started and stopped can help simplify the implementation of a custom readable. + +The `_destroy()` method also needs to be overridden to clean up the item emitter when a stream pipeline encounters an error. + +```javascript +class MyReadable extends stream.Readable { + + constructor({ itemEmitter }) { + super({ objectMode: true }); + this.itemEmitter = itemEmitter; + this.itemEmitter.on('item', item => { + if(!this.push(item)) { + this.itemEmitter.stop(); + } + }); + this.itemEmitter.on('done', () => { + this.push(null); + }); + this.itemEmitter.on('error', (err) => { + this.destroy(err); + }); + + } + + _destroy(err, callback) { + this.itemEmitter.stop(); + callback(err); + } + + _read() { + this.itemEmitter.start(); + } + +} +``` + + + +## Custom Writable + +General pattern: implement `_write()` by consuming a chunk and then notifying the callback.  + + +### Writable Example +TODO example + + +### Official guide for implementing Writable +[https://nodejs.org/api/stream.html#implementing-a-writable-stream](https://nodejs.org/api/stream.html#implementing-a-writable-stream) + +### Tips for Implementing Writable + +* The `_write()` function can be async. + +## Custom Transform + +Transform is implemented by overriding the `_transform()` method. The `_transform()` implementation should call `this.push()` with transformed data and then notify the callback.  + +### Transform Example + +```javascript +class MyObjectTransform extends stream.Transform { + constructor() { + super({ objectMode: true }); + } + _transform(item, encoding, callback) { + this.push(transformItem(item)); + callback(); + } +} +``` + + +### Official guide for implementing Transform +[https://nodejs.org/api/stream.html#implementing-a-transform-stream](https://nodejs.org/api/stream.html#implementing-a-transform-stream) + +### Tips for Implementing Transform + +- The \_transform() method can be async. When a transform depends on + asynchronous I/O it can simply await the operation before notifying the + callback. +- A transformer can filter out objects from the stream by making the push + conditional. +- Back pressure doesn't need to be considered when implementing Transform. Back + pressure is automatically propagated back to the Readable at the start of a + stream pipeline. +- The `_flush()` method should be overridden when a Transform is buffering + chunks such as when parsing items from a raw data stream. + + + +TODO What happens when a transformer pushes null? + +Todo what other methods can be overridden? + + + diff --git a/tailwind.config.js b/tailwind.config.js index 17d982a..b14fcc6 100644 --- a/tailwind.config.js +++ b/tailwind.config.js @@ -1,5 +1,5 @@ module.exports = { - content: ['./src/**/*.{astro,html,svelte,vue,js,ts,jsx,tsx}'], + content: ['./src/**/*.{astro,html,svelte,vue,js,ts,jsx,tsx,md}'], plugins: [ require('@tailwindcss/typography'), require('daisyui'),