Feat/rs webworker (#431)

* chore: Update example, using reed solomon * chore: Update Rollup config * chore: Benchmark * docs: Update README * feat: Support WebWorker * feat: Add types * chore: Add benchmark * Merge branch 'alpha' into feat/rs_webworker
bnb-chain · Dec 19, 2023 · 2e9fce0 · 2e9fce0
1 parent dce67c4
commit 2e9fce0
Show file tree

Hide file tree

Showing 11 changed files with 190 additions and 102 deletions.
diff --git a/.changeset/many-cars-jump.md b/.changeset/many-cars-jump.md
@@ -0,0 +1,5 @@
+---
+'@bnb-chain/reed-solomon': patch
+---
+
+feat: Add Types
diff --git a/README.md b/README.md
@@ -9,7 +9,7 @@ You can find some package documentation below:
 | Package | Description | Version | 
 | --- | --- | --- |
 | [@bnb-chain/greenfield-js-sdk](./packages/js-sdk/README.md) | A client library for Greenfield Chain | [![npm](https://img.shields.io/npm/v/%40bnb-chain%2Fgreenfield-js-sdk?color=blue)](https://www.npmjs.com/package/@bnb-chain/greenfield-js-sdk) |
-| [@bnb-chain/greenfiled-file-handle](./packages/file-handle/README.md) | WASM module that handle file, such as `checksums` | [![npm](https://img.shields.io/npm/v/%40bnb-chain%2Fgreenfiled-file-handle?color=blue)](https://www.npmjs.com/package/@bnb-chain/greenfiled-file-handle) |
+| [@bnb-chain/reed-solomon](./packages/reed-solomon/README.md) | calculate file's `checksums` | [![npm](https://img.shields.io/npm/v/%40bnb-chain%2Freed-solomon?color=blue)](https://www.npmjs.com/package/@bnb-chain/reed-solomon) |
 | [@bnb-chain/greenfield-zk-crypto](./packages/zk-crypto/README.md) | WASM module about sign crypto | [![npm](https://img.shields.io/npm/v/%40bnb-chain%2Fgreenfield-zk-crypto?color=blue)](https://www.npmjs.com/package/@bnb-chain/greenfield-zk-crypto) |
 | [@bnb-chain/create-gnfd-app](./packages/create-gnfd-app/README.md) | Create Greenfield App Quickly | [![npm](https://img.shields.io/npm/v/%40bnb-chain%2Fcreate-gnfd-app?color=blue)](https://www.npmjs.com/package/@bnb-chain/create-gnfd-app) |
 

diff --git a/packages/reed-solomon/README.md b/packages/reed-solomon/README.md
@@ -12,6 +12,17 @@ Compatible with [greenfield-common](https://github.com/bnb-chain/greenfield-comm
 
 ## Usage Examples
 
+### ESM
+
+If you use module bundler such as [Webpack](https://webpack.js.org/), [Rollup](https://rollupjs.org/guide/en/), etc:
+
+```js
+import {ReedSolomon} from '@bnb-chain/reed-solomon'
+
+const rs = new RS.ReedSolomon();
+const res = rs.encode(new Uint8Array(fileBuffer))
+```
+
 ### Browser
 
 Use directly in the browser via script tag:
@@ -22,6 +33,7 @@ Use directly in the browser via script tag:
   get reed solomon
 </button>
 
+<script src="https://cdn.jsdelivr.net/npm/@bnb-chain/reed-solomon/dist/index.aio.js"></script>
 <script src="https://cdn.jsdelivr.net/npm/@bnb-chain/reed-solomon/dist/index.aio.js"></script>
 <script>
   const fileInput = document.getElementById('file');
@@ -32,25 +44,77 @@ Use directly in the browser via script tag:
 
     if (!arrBuffer) alert('no file selected');
 
-    const rs = new RS.ReedSolomon();
-    const res = rs.encode(new Uint8Array(arrBuffer))
+    const rs = new RS.ReedSolomon()
+    const res = await rs.encode(sourceData)
   }
 </script>
 ```
 
-### ESM
+[Code](./examples/web.html)
 
-If you use module bundler such as [Webpack](https://webpack.js.org/), [Rollup](https://rollupjs.org/guide/en/), etc:
+### Browser(WebWorker)
 
-```js
-import {ReedSolomon} from '@bnb-chain/reed-solomon'
+```html
+<script>
+  document.getElementById('worker-btn').onclick = async function() {
+    const selectFile = fileInput.files[0];
+    const arrBuffer = await selectFile.arrayBuffer()
+    if (!arrBuffer) alert('no file selected');
 
-const rs = new RS.ReedSolomon();
-const res = rs.encode(new Uint8Array(fileBuffer))
+    const sourceData = new Uint8Array(arrBuffer)
+    const rs = new WebAdapter.WebAdapterReedSolomon()
+    const res = await rs.encodeInWorker(injectWorker, sourceData)
+  }
+
+  // inject worker
+  function injectWorker() {
+    // replace your CDN url
+    importScripts('http://localhost:9002/dist/web.adapter.aio.js');
+    importScripts('http://localhost:9002/dist/utils.aio.js');
+
+    const rs = new WebAdapter.WebAdapterReedSolomon();
+
+    onmessage = function (event) {
+      const { index, chunk } = event.data;
+      const encodeShards = rs.encodeSegment(chunk);
+      let encodeDataHash = [];
+
+      for (let i = 0; i < encodeShards.length; i++) {
+        const priceHash = RSUtils.sha256(encodeShards[i]);
+        encodeDataHash.push(priceHash);
+      }
+
+      postMessage({
+        index,
+        segChecksum: RSUtils.sha256(chunk),
+        encodeDataHash,
+      });
+
+      self.close();
+    };
+  }
+</script>
 ```
 
+[Code](./examples/web-worker.html)
+
 ### Nodejs
 
+```js
+const { ReedSolomon } = require('@bnb-chain/reed-solomon')
+
+const rs = new ReedSolomon();
+const res = await rs.encode(Uint8Array.from(fileBuffer));
+```
+
+[Code](./examples/node.js)
+
+More examples:
+
+* [calcute several file in a folder](./examples/folder.js)
+
+### Nodejs(`worker_threads`)
+
 Using in Nodejs:
 
 ```js
@@ -62,5 +126,8 @@ const rs = new NodeAdapterReedSolomon();
 const res = await rs.encodeInWorker(__filename, Uint8Array.from(fileBuffer))
 ```
 
-* [calcute single file](./examples/singlefile.js)
-* [calcute several file in a folder](./examples/folder.js)
+[Code](./examples/node-worker.js)
+
+## Benchark
+
+[benchmark](./benchmark.md)
diff --git a/packages/reed-solomon/benchmark.md b/packages/reed-solomon/benchmark.md
@@ -1,16 +1,9 @@
 # benchmarks
 
-## 20M
-
-* Nodejs: 1s
-* Nodejs(worker): 1s
-
-## 200M
-
-* Go: 300ms
-* Nodejs: 10s
-* Nodejs(worker): 2s
-* Browser: 15s
+| file size | [Nodejs](./examples/node.js) | [Nodejs(worker_threads)](./examples/node-worker.js) | [Browser](./examples/web.html) | [Browser(webworker)](./examples/web-worker.html) |
+| - | - | - | - | - |
+| 20M | 1s | 1s | 1.6s | 1.3s |
+| 200M | 10s | 2s | 15s | 2.2s |
 
 ## conclusion
 

diff --git a/packages/reed-solomon/examples/node-worker.js b/packages/reed-solomon/examples/node-worker.js
@@ -0,0 +1,16 @@
+/* eslint-disable */
+const fs = require('node:fs');
+const path = require('node:path');
+const { NodeAdapterReedSolomon } = require('../dist/node.adapter');
+
+const fileBuffer = fs.readFileSync('./README.md');
+
+(async () => {
+  const rs = new NodeAdapterReedSolomon();
+
+  console.time('cost worker_threads');
+  console.log('file size', sourceData.length / 1024 / 1024, 'm');
+  const res = await rs.encodeInWorker(__filename, Uint8Array.from(fileBuffer));
+  console.log('res', res);
+  console.timeEnd('cost worker_threads');
+})();
diff --git a/packages/reed-solomon/examples/node.js b/packages/reed-solomon/examples/node.js
@@ -0,0 +1,15 @@
+/* eslint-disable */
+const fs = require('node:fs');
+const path = require('node:path');
+const { ReedSolomon } = require('../dist/index');
+
+const fileBuffer = fs.readFileSync('./README.md');
+
+(async () => {
+  const rs = new ReedSolomon();
+  console.log('file size', sourceData.length / 1024 / 1024, 'm');
+  console.time('cost');
+  const res = await rs.encode(Uint8Array.from(fileBuffer));
+  console.log('res', res);
+  console.timeEnd('cost');
+})();
diff --git a/packages/reed-solomon/examples/web-worker.html b/packages/reed-solomon/examples/web-worker.html
@@ -0,0 +1,63 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>RS webworker</title>
+</head>
+<body>
+  <input type="file" id="file" />
+
+  <button id="worker-btn">
+    get reed solomon (webworker)
+  </button>
+
+  <script src="../dist/web.adapter.aio.js"></script>
+  <script src="../dist/utils.aio.js"></script>
+  <script type="module">
+    const fileInput = document.getElementById('file');
+
+    // use webworker
+    document.getElementById('worker-btn').onclick = async function() {
+      const selectFile = fileInput.files[0];
+      const arrBuffer = await selectFile.arrayBuffer()
+      if (!arrBuffer) alert('no file selected');
+
+      const sourceData = new Uint8Array(arrBuffer)
+      console.time('webworker cost')
+      console.log('file size', sourceData.length / 1024 / 1024, 'm')
+      const rs = new WebAdapter.WebAdapterReedSolomon()
+      const res = await rs.encodeInWorker(injectWorker, sourceData)
+      console.log('res', res)
+      console.timeEnd('webworker cost')
+    }
+
+    function injectWorker() {
+      importScripts('http://localhost:9002/dist/web.adapter.aio.js');
+      importScripts('http://localhost:9002/dist/utils.aio.js');
+
+      const rs = new WebAdapter.WebAdapterReedSolomon();
+
+      onmessage = function (event) {
+        const { index, chunk } = event.data;
+        const encodeShards = rs.encodeSegment(chunk);
+        let encodeDataHash = [];
+
+        for (let i = 0; i < encodeShards.length; i++) {
+          const priceHash = RSUtils.sha256(encodeShards[i]);
+          encodeDataHash.push(priceHash);
+        }
+
+        postMessage({
+          index,
+          segChecksum: RSUtils.sha256(chunk),
+          encodeDataHash,
+        });
+
+        self.close();
+      };
+    }
+
+  </script>
+</body>
+</html>
diff --git a/packages/reed-solomon/examples/web.html b/packages/reed-solomon/examples/web.html
@@ -3,7 +3,7 @@
 <head>
   <meta charset="UTF-8">
   <meta name="viewport" content="width=device-width, initial-scale=1.0">
-  <title>Document</title>
+  <title>RS</title>
 </head>
 <body>
   <input type="file" id="file" />
@@ -12,13 +12,7 @@
     get reed solomon
   </button>
 
-  <button id="worker-btn">
-    get reed solomon (webworker)
-  </button>
-
   <script src="../dist/index.aio.js"></script>
-  <script src="../dist/web.adapter.aio.js"></script>
-  <script src="../dist/utils.aio.js"></script>
   <script type="module">
     const fileInput = document.getElementById('file');
 
@@ -37,21 +31,6 @@
       console.timeEnd('cost')
     }
 
-    // use webworker
-    document.getElementById('worker-btn').onclick = async function() {
-      const selectFile = fileInput.files[0];
-      const arrBuffer = await selectFile.arrayBuffer()
-      if (!arrBuffer) alert('no file selected');
-
-      const sourceData = new Uint8Array(arrBuffer)
-      console.time('webworker cost')
-      console.log('file size', sourceData.length / 1024 / 1024, 'm')
-      const rs = new WebAdapter.WebAdapterReedSolomon()
-      const res = await rs.encodeInWorker(WebAdapter.injectWorker, sourceData)
-      console.log('res', res)
-      console.timeEnd('webworker cost')
-    }
-
   </script>
 </body>
 </html>
diff --git a/packages/reed-solomon/package.json b/packages/reed-solomon/package.json
@@ -34,15 +34,9 @@
   },
   "typesVersions": {
     "*": {
-      "node.adapter": [
-        "./types/node.adapter.d.ts"
-      ],
-      "web.adapter": [
-        "./types/web.adapter.d.ts"
-      ],
-      "utils": [
-        "./types/utils.d.ts"
-      ]
+      "node.adapter": ["./types/node.adapter.d.ts"],
+      "web.adapter": ["./types/web.adapter.d.ts"],
+      "utils": ["./types/utils.d.ts"]
     }
   },
   "scripts": {

diff --git a/packages/reed-solomon/src/web.adapter.js b/packages/reed-solomon/src/web.adapter.js
@@ -2,17 +2,14 @@ import { ReedSolomon } from '.';
 import { sha256, getIntegrityUint8Array, toBase64, splitPrice } from './utils';
 
 export class WebAdapterReedSolomon extends ReedSolomon {
-  async encodeInWorker(workerFn, sourceData, { webAdapterUrl, utilsUrl }) {
+  async encodeInWorker(workerFn, sourceData) {
     const chunkList = splitPrice(sourceData, this.segmentSize);
 
     const workers = [];
 
     for (let i = 0; i < chunkList.length; i++) {
       // const worker = new Worker('worker.js');
-      const worker = createWorker(workerFn, {
-        webAdapterUrl,
-        utilsUrl,
-      });
+      const worker = createWorker(workerFn);
       workers.push(worker);
       worker.postMessage({
         index: i,
@@ -60,41 +57,9 @@ export class WebAdapterReedSolomon extends ReedSolomon {
   }
 }
 
-function createWorker(f, { webAdapterUrl, utilsUrl }) {
-  var blob = new Blob([
-    '(' + f.toString() + ')(' + `'${webAdapterUrl}'` + ',' + `'${utilsUrl}'` + ')',
-  ]);
+function createWorker(f) {
+  var blob = new Blob(['(' + f.toString() + ')()']);
   var url = window.URL.createObjectURL(blob);
   var worker = new Worker(url);
   return worker;
 }
-
-// inject worker script
-export function injectWorker(cdnsUrls) {
-  importScripts(
-    cdnsUrls.webAdapterUrl ||
-      'https://cdn.jsdelivr.net/npm/@bnb-chain/reed-solomon/dist/index.aio.js',
-  );
-  importScripts('https://cdn.jsdelivr.net/npm/@bnb-chain/reed-solomon/dist/utils.aio.js');
-
-  const rs = new WebAdapter.WebAdapterReedSolomon();
-
-  onmessage = function (event) {
-    const { index, chunk } = event.data;
-    const encodeShards = rs.encodeSegment(chunk);
-    let encodeDataHash = [];
-
-    for (let i = 0; i < encodeShards.length; i++) {
-      const priceHash = RSUtils.sha256(encodeShards[i]);
-      encodeDataHash.push(priceHash);
-    }
-
-    postMessage({
-      index,
-      segChecksum: RSUtils.sha256(chunk),
-      encodeDataHash,
-    });
-
-    self.close();
-  };
-}
diff --git a/packages/reed-solomon/types/web.adapter.d.ts b/packages/reed-solomon/types/web.adapter.d.ts
@@ -1,14 +1,5 @@
 declare module '@bnb-chain/reed-solomon/web.adapter' {
   export class WebAdapterReedSolomon {
-    encodeInWorker(
-      workerFn: () => void,
-      data: Uint8Array,
-      cdnUrls: {
-        webAdapterUrl: string;
-        utilsUrl: string;
-      },
-    ): Promise<string[]>;
+    encodeInWorker(workerFn: () => void, data: Uint8Array): Promise<string[]>;
   }
-
-  export function injectWorker(cdnsUrls?: { webAdapterUrl: string; utilsUrl: string }): void;
 }