feat(connector-polkadot): add connector pkg, openapi specs, test suite

Primary Changes
---------------
1. Created openapi specs for get-prometheus-exporter-metrics, get-transaction-info,
   get-raw-transaction, sign-raw-transaction, run-transaction, deploy-contract-ink,
   and invoke-contract endpoints
2. Created relevant types for the request as well as response for the above endpoints
3. Added generated code for these endpoints
4. Created connector class with functions to interact with the polkadot ledger
5. Created webservices to interact with the endpoint and the relevant class method
6. Created unit and integration testcases in jest to test base functionality of the connector

Secondary Changes
-----------------
1. Added an ink! contract for running the testcases
2. Added the polkadot connector to ci workflow
3. Created substrate.md to docs-cactus
4. Added the polkadot connector to tsconfig.json

Signed-off-by: Anmol Bansal <[email protected]>

=======================================================================

feat(polkadot): creation of polkadot AIO image

Primary Changes
---------------
1. Updated docker image to include multi-stage build
2. Added healthcheck to the image
3. Added Supervisord and removed unneccessary start script
4. Updated the Readme with relevant commands

Signed-off-by: Anmol Bansal <[email protected]>

=======================================================================

feat(polkadot): update substrate test tooling

Primary Changes
---------------
1. Added correct healthcheck for ledger container
2. Update the Substrate test ledger testcases

Signed-off-by: Anmol Bansal <[email protected]>

=======================================================================

feat(polkadot): creation of readme and architecture reference diagrams

Primary Changes
---------------
1. Added README.md for the connector
2. Added Architecture diagrams for the plugin

Secondary Changes
-----------------
1. Added the docker image for the polkadot connector

Co-authored-by: Peter Somogyvari <[email protected]>

Signed-off-by: Anmol Bansal <[email protected]>
Signed-off-by: Peter Somogyvari <[email protected]>

.cspell.json

@@ -170,6 +170,7 @@
         "vscc",
         "vuln",
         "wasm",
+        "WSPROVIDER",
         "Xdai"
     ],
     "dictionaries": [

.github/workflows/ci.yaml

@@ -19,6 +19,7 @@ jobs:
   compute_changed_packages:
     outputs:
       cmd-api-server-changed: ${{ steps.changes.outputs.cmd-api-server-changed }}
+      plugin-ledger-connector-polkadot-changed: ${{ steps.changes.outputs.plugin-ledger-connector-polkadot-changed }}
       plugin-ledger-connector-aries-changed: ${{ steps.changes.outputs.plugin-ledger-connector-aries-changed }}
       plugin-ledger-connector-besu-changed: ${{ steps.changes.outputs.plugin-ledger-connector-besu-changed }}
       plugin-ledger-connector-corda-changed: ${{ steps.changes.outputs.plugin-ledger-connector-corda-changed }}
@@ -49,6 +50,14 @@ jobs:
               - './packages/cactus-plugin-keychain-vault/**'
               # - './.github/workflows/ci.yaml'
 
+            plugin-ledger-connector-polkadot-changed:
+              - './packages/cactus-plugin-ledger-connector-polkadot/**!(*.md|*.css|*.html|*.jpg|*.jpeg|*.png)'
+              - './packages/cactus-common/**!(*.md|*.css|*.html|*.jpg|*.jpeg|*.png)'
+              - './packages/cactus-core/**!(*.md|*.css|*.html|*.jpg|*.jpeg|*.png)'
+              - './packages/cactus-core-api/**!(*.md|*.css|*.html|*.jpg|*.jpeg|*.png)'
+              - './packages/cactus-test-tooling/**!(*.md|*.css|*.html|*.jpg|*.jpeg|*.png)'
+              # - './.github/workflows/ci.yaml'
+            
             plugin-ledger-connector-aries-changed:
               - './packages/cactus-plugin-ledger-connector-aries/**'
               - './packages/cactus-common/**'
@@ -982,6 +991,34 @@ jobs:
           node-version: ${{ env.NODEJS_VERSION }}
       - uses: actions/[email protected]
 
+      - id: yarn-cache
+        name: Restore Yarn Cache
+        uses: actions/[email protected]
+        with:
+          key: ${{ runner.os }}-yarn-${{ hashFiles('./yarn.lock') }}
+          path: ./.yarn/
+          restore-keys: |
+            ${{ runner.os }}-yarn-${{ hashFiles('./yarn.lock') }}
+      - run: ./tools/ci.sh
+  cactus-plugin-ledger-connector-polkadot:
+    continue-on-error: false
+    env:
+      FULL_BUILD_DISABLED: true
+      JEST_TEST_PATTERN: packages/cactus-plugin-ledger-connector-polkadot/src/test/typescript/(unit|integration|benchmark)/.*/*.test.ts
+      JEST_TEST_RUNNER_DISABLED: false
+      TAPE_TEST_RUNNER_DISABLED: true
+    needs:
+      - build-dev
+      - compute_changed_packages
+    if: needs.compute_changed_packages.outputs.plugin-ledger-connector-polkadot-changed == 'true'
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Use Node.js ${{ env.NODEJS_VERSION }}
+        uses: actions/[email protected]
+        with:
+          node-version: ${{ env.NODEJS_VERSION }}
+      - uses: actions/[email protected]
+
       - id: yarn-cache
         name: Restore Yarn Cache
         uses: actions/[email protected]

docs-cactus/source/support.rst

@@ -11,6 +11,6 @@ This section contains the ledger supported versions for connectors in Hyperledge
    Fabric <support/fabric.md>
    Iroha <support/iroha.md>
    Quorum <support/quorum.md>
-   xDai <support/xdai.md>
-
+   xDai <support/xdai.md>  
+   Substrate <support/substrate.md>  
 

docs-cactus/source/support/substrate.md

@@ -0,0 +1,16 @@
+Substrate Support 
+-----------------
+
+
+```{note}
+Substrate chains include Polkadot, Kusama, Rococco, etc. The deployContract feature is for development and test case authoring only, not recommended to be used in production environments for managing smart contracts.
+```
+
+<details>
+  <summary>Hyperledger Cactus v1.0.0-rc3</summary>
+
+  | Substrate API version | deployContract* | invokeContract | runTransaction |
+  | --- | :---: | :---: | :---: |
+  | @polkadot/api 10.9.1 | ✅ [test]() | ✅ [test]() | ✅ [test]() |
+
+</details>

packages/cactus-plugin-ledger-connector-polkadot/Dockerfile

@@ -0,0 +1,10 @@
+FROM ghcr.io/hyperledger/cactus-cmd-api-server:2024-01-02-1fb2551
+
+RUN npm install -g yarn \
+    && yarn set version 3.6.3 \
+    && yarn config set nodeLinker node-modules
+
+ENV NODE_ENV=production
+ARG NPM_PKG_VERSION=latest
+
+RUN yarn add @hyperledger/cactus-plugin-ledger-connector-polkadot@${NPM_PKG_VERSION}

packages/cactus-plugin-ledger-connector-polkadot/README.md

@@ -0,0 +1,240 @@
+# `@hyperledger/cactus-plugin-ledger-connector-polkadot` <!-- omit in toc -->
+
+## Table of Contents <!-- omit in toc -->
+
+- [1. Usage](#1-usage)
+  - [1.1. Installation](#11-installation)
+  - [1.2. Using as a Library](#12-using-as-a-library)
+  - [1.3. Using Via The API Client](#13-using-via-the-api-client)
+- [2. Architecture](#2-architecture)
+  - [2.1. run-transaction-endpoint](#21-run-transaction-endpoint)
+- [3. Containerization](#3-containerization)
+  - [3.1. Building/running the container image locally](#31-buildingrunning-the-container-image-locally)
+  - [3.2. Running the container](#32-running-the-container)
+- [4. Prometheus Exporter](#4-prometheus-exporter)
+  - [4.1. Usage Prometheus](#41-usage-prometheus)
+  - [4.2. Prometheus Integration](#42-prometheus-integration)
+  - [4.3. Helper code](#43-helper-code)
+    - [4.3.1. response.type.ts](#431-responsetypets)
+    - [4.3.2. data-fetcher.ts](#432-data-fetcherts)
+    - [4.3.3. metrics.ts](#433-metricsts)
+- [5. Contributing](#5-contributing)
+- [6. License](#6-license)
+- [7. Acknowledgments](#7-acknowledgments)
+
+
+## 1. Usage
+
+This plugin provides a way to interact with Substrate networks.
+Using this one can perform:
+* Deploy smart contracts (ink! contract).
+* Execute transactions on the ledger.
+* Invoke ink! contract functions.
+
+The above functionality can either be accessed by importing the plugin directly as a library (embedding) or by hosting it as a REST API through the [Cactus API server](https://www.npmjs.com/package/@hyperledger/cactus-cmd-api-server)
+
+We also publish the [Cactus API server as a container image](https://github.com/hyperledger/cactus/pkgs/container/cactus-cmd-api-server) to the GitHub Container Registry that you can run easily with a one liner.
+The API server is also embeddable in your own NodeJS project if you choose to do so.
+
+### 1.1. Installation
+
+**npm**
+
+```sh
+npm install @hyperledger/cactus-plugin-ledger-connector-polkadot
+```
+
+**yarn**
+
+```sh
+yarn add @hyperledger/cactus-plugin-ledger-connector-polkadot
+```
+
+### 1.2. Using as a Library
+
+```typescript
+import {
+  PluginLedgerConnectorPolkadot,
+} from "@hyperledger/cactus-plugin-ledger-connector-polkadot";
+
+const plugin = new PluginLedgerConnectorPolkadot({
+  // See test cases for exact details on what parameters are needed
+});
+
+const req: RunTransactionRequest = {
+  // See tests for specific examples on request properties
+};
+
+try {
+  const res = await plugin.transact(req);
+} catch (ex: Error) {
+  // Make sure to handle errors gracefully (which is dependent on your use-case)
+  console.error(ex);
+  throw ex;
+}
+```
+
+### 1.3. Using Via The API Client
+
+**Prerequisites**
+- A running Substrate ledger (network)
+- You have a running Cactus API server on `$HOST:$PORT` with the Polkadot connector plugin installed on it (and the latter configured to have access to the Substrate ledger from point 1)
+
+```typescript
+import {
+  PluginLedgerConnectorPolkadot,
+  DefaultApi as PolkadotApi,
+} from "@hyperledger/cactus-plugin-ledger-connector-polkadot";
+
+// Step zero is to deploy your Substrate ledger and the Cactus API server
+
+const apiHost = `http://${address}:${port}`;
+
+const apiConfig = new Configuration({ basePath: apiHost });
+
+const apiClient = new PolkadotApi(apiConfig);
+
+const req: RunTransactionRequest = {
+  // See tests for specific examples on request properties
+};
+
+try {
+  const res = await apiClient.runTransaction(req);
+} catch (ex: Error) {
+  // Make sure to handle errors gracefully (which is dependent on your use-case)
+  console.error(ex);
+  throw ex;
+}
+```
+## 2. Architecture
+The sequence diagrams for various endpoints are mentioned below
+
+### 2.1. run-transaction-endpoint
+
+![run-transaction-endpoint sequence diagram](docs/architecture/images/run-transaction-endpoint.png)  
+The above diagram shows the sequence diagram of run-transaction-endpoint. User A (One of the many Users) interacts with the API Client which in turn, calls the API server. API server then executes transact() method which is explained in detailed in the subsequent diagrams.  
+![run-transaction-endpoint transact() method](docs/architecture/images/run-transaction-endpoint-transact.png)  
+The above diagram shows the sequence diagram of transact() method of the PluginLedgerConnectorPolkadot class. The caller to this function, which in reference to the above sequence diagram is API server, sends RunTransactionRequest object as an argument to the transact() method. Based on the type of Web3SigningCredentialType, corresponding responses are sent back to the caller.  
+![run-transaction-endpoint transactCactusKeychainRef() method](docs/architecture/images/run-transaction-endpoint-transact-cactuskeychainref.png)  
+The above diagram shows transactCactusKeychainReference() method being called by the transact() method of the PluginLedgerConnector class when the Web3SigningCredentialType is CACTUSKEYCHAINREF. This method inturn calls transactMnemonicString() which calls the signAndSend() method of the Polkadot library.  
+![runtransaction-endpoint transactMnemonicString() method](docs/architecture/images/run-transaction-endpoint-transact-mnemonicstring.png)  
+The above diagram shows transactMnemonicString() method being called by the transact() method of the PluginLedgerConnector class when the Web3SigningCredentialType is MNEMONICSTRING. This method then calls the signAndSend() method of the Polkadot library.  
+![run-transaction-endpoint transactSigned() method](docs/architecture/images/run-transaction-endpoint-transact-signed.png)  
+The above diagram shows transactSigned() method being called by the transact() method of the PluginLedgerConnector class when the Web3SigningCredentialType is NONE. This method calls the api.rpc.author.submitAndWatchExtrinsic() of the Polkadot library.  
+
+
+## 3. Containerization
+### 3.1. Building/running the container image locally
+
+In the Cactus project root say:
+
+```sh
+DOCKER_BUILDKIT=1 docker build -f ./packages/cactus-plugin-ledger-connector-polkadot/Dockerfile . -t cplcb
+```
+
+Build with a specific version of the npm package:
+```sh
+DOCKER_BUILDKIT=1 docker build --build-arg NPM_PKG_VERSION=latest -f ./packages/cactus-plugin-ledger-connector-polkadot/Dockerfile . -t cplcb
+```
+
+### 3.2. Running the container
+
+Launch container with plugin configuration as an **environment variable**:
+
+```sh
+docker run \
+  --rm \
+  --publish 3000:3000 \
+  --publish 4000:4000 \
+  --env AUTHORIZATION_PROTOCOL='NONE' \
+  --env AUTHORIZATION_CONFIG_JSON='{}' \
+  --env GRPC_TLS_ENABLED=false \
+  cplcb \
+  node_modules/@hyperledger/cactus-cmd-api-server/dist/lib/main/typescript/cmd/cactus-api.js \
+  --env PLUGINS='[{"packageName": "cactus-plugin-ledger-connector-polkadot", "type": "org.hyperledger.cactus.plugin_import_type.LOCAL", "action": "org.hyperledger.cactus.plugin_import_action.INSTALL",  "options": {"wsProviderUrl":"ws://127.0.0.1:9944", "instanceId": "some-unique-polkadot-connector-instance-id"}}]'
+```
+
+Launch container with plugin configuration as a **CLI argument**:
+```sh
+docker run \
+  --rm \
+  --publish 3000:3000 \
+  --publish 4000:4000 \
+  --publish 5000:5000 \
+  --env AUTHORIZATION_PROTOCOL='NONE' \
+  --env AUTHORIZATION_CONFIG_JSON='{}' \
+  --env GRPC_TLS_ENABLED=false \
+  cplcb \
+    node_modules/@hyperledger/cactus-cmd-api-server/dist/lib/main/typescript/cmd/cactus-api.js \
+    --plugins='[{"packageName": "cactus-plugin-ledger-connector-polkadot", "type": "org.hyperledger.cactus.plugin_import_type.LOCAL", "action": "org.hyperledger.cactus.plugin_import_action.INSTALL",  "options": {"wsProviderUrl":"ws://127.0.0.1:9944", "instanceId": "some-unique-polkadot-connector-instance-id"}}]'
+```
+
+Launch container with **configuration file** mounted from host machine:
+```sh
+
+echo '[{"packageName": "cactus-plugin-ledger-connector-polkadot", "type": "org.hyperledger.cactus.plugin_import_type.LOCAL", "action": "org.hyperledger.cactus.plugin_import_action.INSTALL",  "options": {"wsProviderUrl":"ws://127.0.0.1:9944", "instanceId": "some-unique-polkadot-connector-instance-id"}}]' > cactus.json
+
+docker run \
+  --rm \
+  --publish 3000:3000 \
+  --publish 4000:4000 \
+  --env AUTHORIZATION_PROTOCOL='NONE' \
+  --env AUTHORIZATION_CONFIG_JSON='{}' \
+  --env GRPC_TLS_ENABLED=false \
+  --mount type=bind,source="$(pwd)"/cactus.json,target=/cactus.json \
+  cplcb \
+    node_modules/@hyperledger/cactus-cmd-api-server/dist/lib/main/typescript/cmd/cactus-api.js \
+    --config-file=/cactus.json
+```
+
+## 4. Prometheus Exporter
+
+This class creates a Prometheus exporter, which scraps the transactions (total transaction count) for the use cases incorporating the use of Fabric connector plugin.
+
+
+### 4.1. Usage Prometheus
+The Prometheus exporter object is initialized in the `PluginLedgerConnectorPolkadot` class constructor itself, so instantiating the object of the `PluginLedgerConnectorPolkadot` class, gives access to the exporter object.
+You can also initialize the Prometheus exporter object separately and then pass it to the `IPluginLedgerConnectorPolkadotOptions` interface for `PluginLedgerConnectorPolkadot` constructor.
+
+`getPrometheusExporterMetricsEndpoint` function returns the Prometheus exporter metrics, currently displaying the total transaction count, which currently increments every time the `transact()` method of the `PluginLedgerConnectoPolkadot` class is called.
+
+### 4.2. Prometheus Integration
+To use Prometheus with this exporter make sure to install [Prometheus main component](https://prometheus.io/download/).
+Once Prometheus is setup, the corresponding scrape_config needs to be added to the prometheus.yml
+
+```(yaml)
+- job_name: 'polkadot_ledger_connector_exporter'
+  metrics_path: api/v1/plugins/@hyperledger/cactus-plugin-ledger-connector-polkadot/get-prometheus-exporter-metrics
+  scrape_interval: 5s
+  static_configs:
+    - targets: ['{host}:{port}']
+```
+
+Here the `host:port` is where the Prometheus exporter metrics are exposed. The test cases (For example, packages/cactus-plugin-ledger-connector-polkadot/src/test/typescript/integration/run-transaction.test.ts) exposes it over `0.0.0.0` and a random port(). The random port can be found in the running logs of the test case and looks like (42379 in the below mentioned URL)
+`Metrics URL: http://0.0.0.0:42379/api/v1/plugins/@hyperledger/cactus-plugin-ledger-connector-polkadot/get-prometheus-exporter-metrics`
+
+Once edited, you can start the Prometheus service by referencing the above edited prometheus.yml file.
+On the Prometheus graphical interface (defaulted to http://localhost:9090), choose **Graph** from the menu bar, then select the **Console** tab. From the **Insert metric at cursor** drop down, select **cactus_Polkadot_total_tx_count** and click **execute**
+
+### 4.3. Helper code
+
+#### 4.3.1. response.type.ts
+This file contains the various responses of the metrics.
+
+#### 4.3.2. data-fetcher.ts
+This file contains functions encasing the logic to process the data points
+
+#### 4.3.3. metrics.ts
+This file lists all the Prometheus metrics and what they are used for.
+
+## 5. Contributing
+
+We welcome contributions to Hyperledger Cactus in many forms, and there’s always plenty to do!
+
+Please review [CONTIRBUTING.md](../../CONTRIBUTING.md) to get started.
+
+## 6. License
+
+This distribution is published under the Apache License Version 2.0 found in the [LICENSE](../../LICENSE) file.
+
+## 7. Acknowledgments

packages/cactus-plugin-ledger-connector-polkadot/docs/architecture/run-transaction-endpoint-transact-cactuskeychainref.puml

@@ -0,0 +1,29 @@
+@startuml
+title Hyperledger Cactus\nSequence Diagram\nRun Transaction Endpoint\ntransactCactusKeychainRef() method
+
+skinparam sequenceArrowThickness 2
+skinparam roundcorner 20
+skinparam maxmessagesize 120
+skinparam sequenceParticipant underline
+
+actor "Caller" as caller
+participant "transactCactusKeychainRef()" as tckr
+participant "transactMnemonicString()" as tms
+participant ".signAndSend()" as sas
+
+caller-> tckr: req
+activate caller
+group #LightBlue if web3SigningCredential == CACTUSKEYCHAINREF
+    activate tckr
+    tckr -> tms: [transactionConfig, web3SigningCredential]
+    activate tms
+    tms -> sas: [transactionConfig, mnemonicString]
+    activate sas
+    sas --> tms: return [success, blockhash, transactionHash]
+    deactivate sas
+        tms --> tckr: return [success, blockhash, transactionHash]
+        tckr --> caller : return {success, blockhash, transactionHash} as resBody
+        deactivate tckr
+    deactivate tms
+end
+@enduml