build(typescript): project-wide fixes to allow us to use ESM-only deps

Apologies for the huge diff, this can't be broken down to smaller changes that would still compile because of cross-package dependencies. I realize that this change is not exactly the optimal solution, but it is probably a step in the right direction. If I somehow found the time to submit pull requests to the libraries that I needed to fork and re-publish (see details below) and then get the changes onto the upstream and get them released as the official packages, then we could (in theory) arrive at a solution that is the recommended way of fixing these problems (apart from going full ESM-only) This work stands on the shoulders of the previous commits from @outSH and takes a slightly different direction compared to what we've been talking about earlier on account of the problem that the eval-import workaround causes crashes in Jest. Based on the above I went through the following adventures: 1. I migrated the build system of kubo-rpc-client myself so that it correctly exports CJS and ESM and typings for both of those as well, I put that code on my fork [1] and then published it onto npm as well [2] After this, I was hoping that now we could just import the package in our CJS code without issues, but what really happened is that instead of crashing at the require call that pull in kubo itself, it started crashing deeper in the require stack where kubo itself was requiring it's own ESM only dependencies (of which there seem to be at least 10 or 15). At this point I realized that me migrating and self-publishing all of these additional packages might not be worth the effort and started looking for something easier. 2. I gave dynamic imports + moduleResultion=Node16 as my next attempt to get our build back to working order. With this, the kubo-rpc-client can now be imported dynamically without issues in packages that declare themselves as resolving modules as "Node16" in their tsconfig.json Other issues here were encountered because not all of our ESM only packages are used in a way that they can be imported dynamically (for example if their types are part of our own types or are being re-exported). The two libraries with this problem were `run-time-error` and `socket.io-client` for both of which I ended up going through the same treatment as for kubo-rpc-client above (but this time my effort wasn't) in vain. They work and so I did some search and replace in the entire codebase to use these re-published packages with the correct types: [3] [4] [5] [6] 3. After this the project build was working, but Jest was still failing with compiler errors which I determined to happen because it uses the root tsconfig.json file for it's internal TS compilation and that root tsconfig.json file was not setting module resolution to Node16. 4. After fixing that the final hurdle (hopefully) was to ensure that jest gets execued with the custom node option as below: NODE_OPTIONS=--experimental-vm-modules yarn jest [1] https://github.com/petermetz/js-kubo-rpc-client-esm-cjs [2] https://www.npmjs.com/package/kubo-rpc-client-esm-cjs [3] https://github.com/petermetz/socket.io-client [4] https://www.npmjs.com/package/socket.io-client-fixed-types [5] https://github.com/petermetz/RuntimeError [6] https://www.npmjs.com/package/run-time-error-cjs Huge thanks for https://arethetypeswrong.github.io/ a tool I used extensively to create the fixes for the libraries above. One more thing that I tried just to collect more data points was to set the moduleResultion project-wide to Node16 via setting it in the root tsconfig.base.json but this broke the compiler itself, as in, there is a bug in the Typescript compiler in v4.x as seen here: microsoft/TypeScript#51221 So this is one more reason for us to upgrade to 5.x as soon as possible. Signed-off-by: Peter Somogyvari <[email protected]>
hyperledger-cacti · Nov 1, 2023 · 7e80682 · 7e80682
1 parent bcad3be
commit 7e80682
Show file tree

Hide file tree

Showing 125 changed files with 265 additions and 371 deletions.
diff --git a/examples/cactus-example-cbdc-bridging-backend/package.json b/examples/cactus-example-cbdc-bridging-backend/package.json
@@ -75,8 +75,8 @@
     "dotenv": "^16.0.1",
     "fabric-network": "2.2.19",
     "fs-extra": "10.1.0",
-    "ipfs-http-client": "60.0.1",
     "knex": "2.5.1",
+    "kubo-rpc-client": "3.0.1",
     "nyc": "^13.1.0",
     "openapi-types": "9.1.0",
     "sqlite3": "^5.0.8",

diff --git a/...ging-backend/src/main/typescript/infrastructure/cbdc-bridging-app-dummy-infrastructure.ts b/...ging-backend/src/main/typescript/infrastructure/cbdc-bridging-app-dummy-infrastructure.ts
@@ -1,7 +1,6 @@
 import path from "path";
 import { v4 as uuidv4 } from "uuid";
 import fs from "fs-extra";
-import { create } from "ipfs-http-client";
 import {
   Logger,
   Checks,
@@ -289,7 +288,8 @@ export class CbdcBridgingAppDummyInfrastructure {
   public async createIPFSConnector(): Promise<PluginObjectStoreIpfs> {
     this.log.info(`Creating Besu Connector...`);
 
-    const ipfsClientOrOptions = create({
+    const kuboRpcClientModule = await import("kubo-rpc-client");
+    const ipfsClientOrOptions = kuboRpcClientModule.create({
       url: await this.ipfs.getApiUrl(),
     });
 

diff --git a/examples/cactus-example-cbdc-bridging-backend/tsconfig.json b/examples/cactus-example-cbdc-bridging-backend/tsconfig.json
@@ -5,6 +5,7 @@
     "outDir": "./dist/lib/",
     "declarationDir": "dist/lib",
     "rootDir": "./src",
+    "moduleResolution": "Node16",
     "tsBuildInfoFile": "../../.build-cache/cactus-example-cbdc-bridging-backend.tsbuildinfo",
   },
   "references": [

diff --git a/examples/cactus-example-discounted-asset-trade/business-logic-asset-trade.ts b/examples/cactus-example-discounted-asset-trade/business-logic-asset-trade.ts
@@ -38,7 +38,7 @@ import {
 } from "./fabric-connector";
 import { sendEthereumTransaction } from "./transaction-ethereum";
 import { hasKey } from "@hyperledger/cactus-common";
-import { RuntimeError } from "run-time-error";
+import { RuntimeError } from "run-time-error-cjs";
 
 const moduleName = "BusinessLogicAssetTrade";
 const logger = getLogger(`${moduleName}`);

diff --git a/examples/cactus-example-electricity-trade/tools/transferNumericAsset/transferNumericAsset.ts b/examples/cactus-example-electricity-trade/tools/transferNumericAsset/transferNumericAsset.ts
@@ -5,7 +5,7 @@
  * transferNumericAsset.ts
  */
 
-import { io } from "socket.io-client";
+import { io } from "socket.io-client-fixed-types";
 
 {
   // Validator test program.(socket.io client)

diff --git a/...ain-business-logic-plugin/src/main/typescript/model/converter/bamboo-harvest-converter.ts b/...ain-business-logic-plugin/src/main/typescript/model/converter/bamboo-harvest-converter.ts
@@ -1,5 +1,5 @@
 import { BambooHarvest } from "../../generated/openapi/typescript-axios";
-import { RuntimeError } from "run-time-error";
+import { RuntimeError } from "run-time-error-cjs";
 
 /**
  * Responsible for converting model entities such as the `BambooHarvest` to and

diff --git a/...ly-chain-business-logic-plugin/src/main/typescript/model/converter/bookshelf-converter.ts b/...ly-chain-business-logic-plugin/src/main/typescript/model/converter/bookshelf-converter.ts
@@ -1,5 +1,5 @@
 import { Bookshelf } from "../../generated/openapi/typescript-axios/index";
-import { RuntimeError } from "run-time-error";
+import { RuntimeError } from "run-time-error-cjs";
 
 /**
  * Responsible for converting model entities such as the `Bookshelf` to and

diff --git a/...chain-frontend/src/app/bamboo-harvest/bamboo-harvest-detail/bamboo-harvest-detail.page.ts b/...chain-frontend/src/app/bamboo-harvest/bamboo-harvest-detail/bamboo-harvest-detail.page.ts
@@ -1,5 +1,5 @@
 import { v4 as uuidv4 } from "uuid";
-import { RuntimeError } from "run-time-error";
+import { RuntimeError } from "run-time-error-cjs";
 
 import { Component, Inject, Input, OnInit } from "@angular/core";
 import {

diff --git a/examples/cactus-example-tcs-huawei/tools/transferNumericAsset/transferNumericAsset.ts b/examples/cactus-example-tcs-huawei/tools/transferNumericAsset/transferNumericAsset.ts
@@ -5,7 +5,7 @@
  * transferNumericAsset.ts
  */
 
-import { io } from "socket.io-client";
+import { io } from "socket.io-client-fixed-types";
 
 {
   // Validator test program.(socket.io client)

diff --git a/extensions/cactus-plugin-htlc-coordinator-besu/package.json b/extensions/cactus-plugin-htlc-coordinator-besu/package.json
@@ -70,8 +70,8 @@
     "joi": "14.3.1",
     "openapi-types": "7.0.1",
     "prom-client": "13.1.0",
-    "run-time-error": "1.4.0",
-    "socket.io-client": "4.5.4",
+    "run-time-error-cjs": "1.4.0",
+    "socket.io-client-fixed-types": "4.5.4",
     "typescript-optional": "2.0.1"
   },
   "devDependencies": {

diff --git a/...s/cactus-plugin-htlc-coordinator-besu/src/main/typescript/plugin-htlc-coordinator-besu.ts b/...s/cactus-plugin-htlc-coordinator-besu/src/main/typescript/plugin-htlc-coordinator-besu.ts
@@ -4,7 +4,7 @@ import { v4 as uuidv4 } from "uuid";
 import { Express } from "express";
 import { promisify } from "util";
 import { Optional } from "typescript-optional";
-import { RuntimeError } from "run-time-error";
+import { RuntimeError } from "run-time-error-cjs";
 import fastSafeStringify from "fast-safe-stringify";
 import OAS from "../json/openapi.json";
 import {

diff --git a/extensions/cactus-plugin-object-store-ipfs/package.json b/extensions/cactus-plugin-object-store-ipfs/package.json
@@ -55,12 +55,11 @@
     "webpack:dev:web": "webpack --env=dev --target=web --config ../../webpack.config.js"
   },
   "dependencies": {
-    "@hyperledger/cacti-esm-compat-hacks": "2.0.0-alpha.2",
     "@hyperledger/cactus-common": "2.0.0-alpha.2",
     "@hyperledger/cactus-core": "2.0.0-alpha.2",
     "@hyperledger/cactus-core-api": "2.0.0-alpha.2",
     "axios": "1.5.1",
-    "run-time-error": "1.4.0",
+    "run-time-error-cjs": "1.4.0",
     "typescript-optional": "2.0.1",
     "uuid": "8.3.2"
   },
@@ -71,6 +70,8 @@
     "express": "4.18.2",
     "ipfs-core-types": "0.14.1",
     "ipfs-unixfs": "9.0.1",
+    "kubo-rpc-client": "3.0.1",
+    "kubo-rpc-client-esm-cjs": "3.0.1",
     "multiformats": "11.0.2"
   },
   "engines": {

diff --git a/extensions/cactus-plugin-object-store-ipfs/src/main/typescript/kubo-rpc-client-types.ts b/extensions/cactus-plugin-object-store-ipfs/src/main/typescript/kubo-rpc-client-types.ts
@@ -3,11 +3,15 @@
  * To fix this we define required types here, based on their counterparts in kubo-rpc-client.
  */
 
+// @ts-ignore
 import type { Multiaddr } from "@multiformats/multiaddr";
+// @ts-ignore
 import type { MultihashHasher } from "multiformats/hashes/interface";
 import type { Agent as HttpAgent } from "http";
 import type { Agent as HttpsAgent } from "https";
+// @ts-ignore
 import type { CID } from "multiformats/cid";
+// @ts-ignore
 import type { Mtime } from "ipfs-unixfs";
 
 /////////////////////////////////////
@@ -16,10 +20,12 @@ import type { Mtime } from "ipfs-unixfs";
 // Some are simplified when details are not needed
 
 export type MultibaseCodec<Prefix extends string = any> =
-  import("multiformats/bases/interface").MultibaseCodec<Prefix>;
+// @ts-ignore
+import("multiformats/bases/interface").MultibaseCodec<Prefix>;
 export type BlockCodec<
-  T1 = any,
-  T2 = any,
+T1 = any,
+T2 = any,
+// @ts-ignore
 > = import("multiformats/codecs/interface").BlockCodec<T1, T2>;
 
 export interface LoadBaseFn {

diff --git a/extensions/cactus-plugin-object-store-ipfs/src/main/typescript/plugin-object-store-ipfs.ts b/extensions/cactus-plugin-object-store-ipfs/src/main/typescript/plugin-object-store-ipfs.ts
@@ -1,6 +1,7 @@
 import path from "path";
 import type { Express } from "express";
-import { RuntimeError } from "run-time-error";
+// @ts-ignore
+import { RuntimeError } from "run-time-error-cjs";
 import {
   Logger,
   Checks,
@@ -18,7 +19,6 @@ import type {
   SetObjectRequestV1,
   SetObjectResponseV1,
 } from "@hyperledger/cactus-core-api";
-import { dynamicImportKuboRpcClientESMWorkaround } from "@hyperledger/cacti-esm-compat-hacks";
 
 import OAS from "../json/openapi.json";
 
@@ -60,8 +60,8 @@ export class PluginObjectStoreIpfs implements IPluginObjectStore {
     if (isLikeIpfsHttpClient(this.opts.ipfsClientOrOptions)) {
       this.ipfs = this.opts.ipfsClientOrOptions;
     } else if (this.opts.ipfsClientOrOptions) {
-      const kuboRpcModule = await dynamicImportKuboRpcClientESMWorkaround();
-      this.ipfs = kuboRpcModule.create(this.opts.ipfsClientOrOptions);
+      const { create } = await import("kubo-rpc-client");
+      this.ipfs = create(this.opts.ipfsClientOrOptions);
     } else {
       const errorMessage = `initIpfs Need either "ipfsClient" or "ipfsClientOptions" to construct ${this.className} Neither was provided.`;
       throw new RuntimeError(errorMessage);

diff --git a/...plugin-object-store-ipfs/src/test/typescript/integration/plugin-object-store-ipfs.test.ts b/...plugin-object-store-ipfs/src/test/typescript/integration/plugin-object-store-ipfs.test.ts
@@ -16,7 +16,6 @@ import {
   GoIpfsTestContainer,
   Containers,
 } from "@hyperledger/cactus-test-tooling";
-import { dynamicImportKuboRpcClientESMWorkaround } from "@hyperledger/cacti-esm-compat-hacks";
 
 import { PluginObjectStoreIpfs } from "../../../main/typescript";
 import { DefaultApi as ObjectStoreIpfsApi } from "../../../main/typescript/public-api";
@@ -61,8 +60,10 @@ test(testCase, async (t: Test) => {
   t.comment(`Go IPFS Test Container API URL: ${ipfsApiUrl}`);
   t.comment(`Go IPFS Test Container Gateway URL: ${ipfsGatewayUrl}`);
 
-  const kuboRpcModule = await dynamicImportKuboRpcClientESMWorkaround();
-  const ipfsClientOrOptions = kuboRpcModule.create({
+
+  const { create } = await import("kubo-rpc-client");
+
+  const ipfsClientOrOptions = create({
     url: ipfsApiUrl,
   });
   const instanceId = uuidv4();

diff --git a/...cactus-plugin-object-store-ipfs/src/test/typescript/unit/plugin-object-store-ipfs.test.ts b/...cactus-plugin-object-store-ipfs/src/test/typescript/unit/plugin-object-store-ipfs.test.ts
@@ -8,15 +8,14 @@ import bodyParser from "body-parser";
 import { Servers } from "@hyperledger/cactus-common";
 import type { IListenOptions, LogLevelDesc } from "@hyperledger/cactus-common";
 import { Configuration } from "@hyperledger/cactus-core-api";
-import { dynamicImportKuboRpcClientESMWorkaround } from "@hyperledger/cacti-esm-compat-hacks";
 
 import { PluginObjectStoreIpfs } from "../../../main/typescript";
 import type { IPluginObjectStoreIpfsOptions } from "../../../main/typescript";
 import { DefaultApi as ObjectStoreIpfsApi } from "../../../main/typescript/public-api";
 
 test("PluginObjectStoreIpfs", async (t1: Test) => {
   const logLevel: LogLevelDesc = "TRACE";
-  const kuboRpcModule = await dynamicImportKuboRpcClientESMWorkaround();
+  const kuboRpcModule = await import("kubo-rpc-client");
   const ipfsClientOrOptions = kuboRpcModule.create();
   t1.doesNotThrow(
     () =>
@@ -49,7 +48,7 @@ test("PluginObjectStoreIpfs", async (t1: Test) => {
   });
 
   test.skip("get,set,has,delete alters state as expected", async (t: Test) => {
-    const kuboRpcModule = await dynamicImportKuboRpcClientESMWorkaround();
+    const kuboRpcModule = await import("kubo-rpc-client");
     const options: IPluginObjectStoreIpfsOptions = {
       ipfsClientOrOptions: kuboRpcModule.create(), // FIXME: use an actual mock IPFS client
       instanceId: uuidv4(),

diff --git a/extensions/cactus-plugin-object-store-ipfs/tsconfig.json b/extensions/cactus-plugin-object-store-ipfs/tsconfig.json
@@ -7,6 +7,7 @@
     "tsBuildInfoFile": "../../.build-cache/cactus-plugin-object-store-ipfs.tsbuildinfo",
     "skipLibCheck": true,
     "composite": true,
+    "moduleResolution": "Node16"
   },
   "include": [
     "./src",
@@ -22,9 +23,6 @@
     {
       "path": "../../packages/cactus-core-api/tsconfig.json"
     },
-    {
-      "path": "../../packages/cacti-esm-compat-hacks/tsconfig.json"
-    },
     {
       "path": "../../packages/cactus-test-tooling/tsconfig.json"
     }

diff --git a/jest.config.js b/jest.config.js
diff --git a/package.json b/package.json
@@ -68,9 +68,9 @@
     "webpack": "lerna run webpack:dev",
     "webpack:dev:web": "lerna run webpack:dev:web",
     "webpack:dev:node": "lerna run webpack:dev:node",
-    "test:jest:all": "NODE_OPTIONS=--max_old_space_size=3072 jest",
-    "test:tap:all": "tap",
-    "test:all": "yarn test:jest:all && yarn test:tap:all",
+    "test:jest:all": "NODE_OPTIONS=\"--max_old_space_size=3072 --experimental-vm-modules\" jest",
+    "test:tap:all": "NODE_OPTIONS=\"--experimental-vm-modules\" tap",
+    "test:all": "NODE_OPTIONS=\"--experimental-vm-modules\" yarn test:jest:all && yarn test:tap:all",
     "prettier": "prettier --write --config .prettierrc.js \"./**/src/main/json/openapi.json\"",
     "lerna-publish-canary": "lerna publish --canary --force-publish --dist-tag $(git branch --show-current) --preid $(git branch --show-current) --loglevel=silly",
     "prepare": "husky install",
@@ -146,7 +146,7 @@
     "openapi-types": "12.1.3",
     "prettier": "3.0.3",
     "protoc-gen-ts": "0.8.6",
-    "run-time-error": "1.4.0",
+    "run-time-error-cjs": "1.4.0",
     "secp256k1": "4.0.3",
     "semver-parser": "4.1.4",
     "shebang-loader": "0.0.1",

diff --git a/packages-python/cactus_validator_socketio_indy/testcli/package.json b/packages-python/cactus_validator_socketio_indy/testcli/package.json
@@ -5,6 +5,6 @@
   "private": true,
   "dependencies": {
     "jsonwebtoken": "8.5.1",
-    "socket.io-client": "4.5.4"
+    "socket.io-client-fixed-types": "4.5.4"
   }
 }
diff --git a/packages-python/cactus_validator_socketio_indy/testcli/testsock.js b/packages-python/cactus_validator_socketio_indy/testcli/testsock.js
@@ -1,7 +1,7 @@
 const fs = require("fs");
 const jwt = require("jsonwebtoken");
 const { exit } = require("process");
-const io = require("socket.io-client");
+const io = require("socket.io-client-fixed-types");
 
 const url = "http://localhost:10080";
 const socket = io(url, {

diff --git a/packages/cacti-esm-compat-hacks/README.md b/packages/cacti-esm-compat-hacks/README.md
diff --git a/packages/cacti-esm-compat-hacks/package.json b/packages/cacti-esm-compat-hacks/package.json