Development (#1387)

* integrate assets cdn * fetch token price from mx-assets priceSource.url (#1290) * fetch token price from mx-assets priceSource.url * update extractData methods * use apiService.get to fetch result data * update list of providers * update key * review & adjustments * removed v2 from allEsdtTokens --------- Co-authored-by: tanghel <[email protected]> * Nodes and tokens fetch from external api (#1306) * updated devnet config * Signed commit test * Signed commit restore changes * Signed commit test * Signed message restore changes * Added tokens fetch from external API * Added nodes fetch from external API * Add ApiService as provider in tests * Add nodes/tokens fetch features in config files * Add unit tests for external api * Divide requests for nodes and tokens in smaller requests containing maximum 1000 elements and bug fixes * Wrong function call when fetching nodes fixed * refactoring * simplified node / token fetching * Fixed tests bug --------- Co-authored-by: GuticaStefan <[email protected]> Co-authored-by: Nicolae Mogage <[email protected]> Co-authored-by: Gabriel Matei <[email protected]> * default features disabled (#1323) * use assets cdn url * use destructuring syntax * add applications unit tests (#1321) * Create applications.spec.ts * Update schema.gql * Update applications.spec.ts * fetch also collection ranks * update API to latest sdk-nestjs version 3.7.4 (#1322) * Change rabbitmq docker image version from 3.5 to 3.9 (#1325) * Integration of K6 into API Github workflow (#1309) * Add workflow load-tests * Add initialization of the project * Fix port when waiting for API * Add config for API * Add folder content for debug * Add folder content for debug before api start * Add debug for config file * Remove config.yaml from gitignore * Add config.yaml file in dist * Move config copy after build * Add docker services * Remove wrong added - * Change rabbitmq version * Add background API start * Add docker installation in workflow * Changed docker compose version use to start services * Changed docker compose version * Add docker compose global * Change docker compose plugin version * Change docker compose plugin version * Update trend names * Add cache preload * Add 1 minute duration * Add preload for all tokens/nodes * Add preload for all tokens/nodes * test * Change table * Change wrong closing tag for table * Refactor generateTable function * Add more endpoints * Fix wrong name for Trends * Add empty line at end of file * Fix wrong trend calls * Changed actions/upload-artifacts from v2 to v3 * changed actions/download-artifact from v2 to v3 * Clear docker images caching in workflow * removed unncessary clear docker images cache * Update load-tests.yml * Update load-test.yml * updated preload.js path * update path for preload.js * test * disable preload cache * revert ref base * run action for PRs to main or development only * update load-tests.yml --------- Co-authored-by: tanghel <[email protected]> Co-authored-by: GuticaStefan <[email protected]> Co-authored-by: cfaur09 <[email protected]> * Fetch-providers-from-external-API (#1326) * added feature to fetch providers from external api instead of computing them everytime the api-service starts * removed unnecessary return type * added unit test for fetch providers from external api * linter fix * add more unit tests for providers service --------- Co-authored-by: cfaur09 <[email protected]> * Update nodes.spec.ts (#1330) * fixes after review * tokens fetch from external api unit tests (#1331) * upgrade tokens unit tests WIP * add unit test for tokens * fix lint * remove console log * Filter NFTs by full tag (#1332) * removed erl_crash.dump file (#1337) * Handle activation epoch where staking v4 flag is checked (#1338) * handle activation epoch where staking v4 flag is checked * update specs --------- Co-authored-by: cfaur09 <[email protected]> * remove support for indexer v 3 (#1324) * remove indexerV3 + specs * fix specs * remove support for indexerV5 * remove token.keyword --------- Co-authored-by: tanghel <[email protected]> * applications assets (#1340) * add applications assets * code formatting --------- Co-authored-by: tanghel <[email protected]> * add support for token priceSource filter (#1342) * add mex token charts (#1334) * add mex token charts * Create mex.token.charts.spec.ts * check if given token has pair * Update mex.token.charts.spec.ts * user after instead of start query param * add caching * Update mex.token.charts.spec.ts * fixes after review * Update mex.token.charts.spec.ts * fix format * fixes after review * small adjustments --------- Co-authored-by: tanghel <[email protected]> * update deprecated pairs query (#1335) * update deprecated pairs query * add tradesCount24h * add mex pair status and apply filter by state * integrate-latest-sdk-nestjs-version (#1343) * fix nft filters (#1344) * fix nft filters * fix failing unit tests * relayed v3 features (#1299) * add innerTransactions * add result status field * add relayer field + filter * remove relayerAddr from query * update tests * filter by relayer address * add account transfer relayer * remove relayerAddr * fixes after review * extract relayed version from ES * revert changes * add receiverUsername in innerTransaction * fix specs * Update mex.token.charts.spec.ts --------- Co-authored-by: tanghel <[email protected]> * contracts deploys endpoints (#1336) * replace account contracts with account deploys * add account contracts * add AccountContract Entity instead of any * Update mex.token.charts.spec.ts --------- Co-authored-by: tanghel <[email protected]> * add support for relayer v3 (#1346) * add support for isRelayed in global trasfers endpoint * small identation fix * apply relayer field for transaction details * add support for isRelayed filter * add support for wildSearchcard accounts (#1347) * Basic support for innerTx in transfers endpoint (#1351) * add feature flag * feat/sovereign (#1341) * add cacheDuration * add support for NftSubType * Update elastic.indexer.helper.ts * Update nft.filter.ts * Update nft.controller.ts * add filter.type on collection elastic * add subType even for SFT * fix mex.token.charts.spec.ts * Update nft.controller.ts * revert changes * add subType filter * add collections subType filter * add address collection roles subType * add subType filter for roles/collections count * add accounts/nfts subType filter * add accounts/collection subType Filter * improved support for subType filtering * commented out field decoration for inner transactions in context of graphgql * fix array subType filter * Update elastic.indexer.helper.ts * remove empty line --------- Co-authored-by: tanghel <[email protected]> * add token owners history info * use double quotes * Update tokens.spec.ts * add entrypoint * remove support for relayed inner transactions ( relayedV3 ) (#1361) * fix nfts count + upgrade deps (#1362) * update package name (#1363) * add round filter * add address keys proxy (#1366) * add events support (#1367) * add events support * fixes and improvements * remove postgres support * add NFT balance * remove if condition for ESDT types * Reset transferLast24h to 0 for accounts that have no transactions in the last 24h (#1355) * reset transferLast24h to 0 for accounts that have no transactions in the last 24h * fix lint errors --------- Co-authored-by: cfaur09 <[email protected]> * remove support for graphql & update swagger version (#1368) * remove support for graphql * SERVICES-2711: update Swagger required fields * fix: update property name for locked accounts in TokenSupplyResult * fix: correct type definition for lockedAccounts in TokenSupplyResult * fix: update type definition for lockedAccounts in TokenSupplyResult * fixes * further fixes * fixes after review * update swagger version * remove unnecessary GraphQL field from Token entity --------- Co-authored-by: bogdan-rosianu <[email protected]> Co-authored-by: bogdan-rosianu <[email protected]> * API-70: improve tx pool endpoint * API-70: remove module import * API-70: senderShard + receiverShard filters * added missing filters to tx pool count endpoint * API query: normalize token identifier case (#1375) * Merge main into dev nov15 (#1383) * added dockerfile, workflow and entrypoint (#1364) * added dockerfile, workflow and entrypoint * add entrypoint --------- Co-authored-by: liviuancas-elrond <[email protected]> Co-authored-by: cfaur09 <[email protected]> * Add guardianData to GatewayComponentRequest in GatewayService (#1377) --------- Co-authored-by: Rebegea Dragos-Alexandru <[email protected]> Co-authored-by: liviuancas-elrond <[email protected]> Co-authored-by: cfaur09 <[email protected]> Co-authored-by: Catalin Faur <[email protected]> * Merge main development (#1386) --------- Co-authored-by: cfaur09 <[email protected]> Co-authored-by: Catalin Faur <[email protected]> Co-authored-by: tanghel <[email protected]> Co-authored-by: GuticaStefan <[email protected]> Co-authored-by: Nicolae Mogage <[email protected]> Co-authored-by: Gabriel Matei <[email protected]> Co-authored-by: GuticaStefan <[email protected]> Co-authored-by: bogdan-rosianu <[email protected]> Co-authored-by: Liviu Ancas <[email protected]> Co-authored-by: bogdan-rosianu <[email protected]> Co-authored-by: liviuancas-elrond <[email protected]>
multiversx · Nov 18, 2024 · 9037a37 · 9037a37
1 parent 92ce317
commit 9037a37
Show file tree

Hide file tree

Showing 341 changed files with 4,869 additions and 74,203 deletions.
diff --git a/.github/workflows/load-tests.yml b/.github/workflows/load-tests.yml
@@ -0,0 +1,186 @@
+name: Load Tests
+
+on:
+  push:
+    branches: [main, development]
+  pull_request:
+    branches: [main, development]
+
+jobs:
+  test-base:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        ref: ${{ github.event.pull_request.base.sha }}
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v2
+      with:
+        node-version: '20'
+
+    - name: Install dependencies
+      run: npm ci
+
+    - name: Initialize the project
+      run: npm run init
+
+    - name: Build
+      run: npm run build
+
+    - name: Copy devnet config file from src to dist
+      run: cp ./config/config.devnet.yaml ./dist/config/config.yaml
+
+    - name: Start docker services
+      run: docker compose up -d
+
+    - name: Start Node.js API
+      run: node ./dist/src/main.js &
+
+    - name: Install k6
+      run: |
+        sudo gpg -k
+        sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
+        echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
+        sudo apt-get update
+        sudo apt-get install k6
+
+    - name: Wait for API to be ready
+      run: |
+        until curl --output /dev/null --silent --fail http://localhost:4001/hello; do
+          echo 'Waiting for API...'
+          sleep 1
+        done
+
+    - name: Preload cache
+      run: k6 run ./k6/preload.js
+
+    - name: Run k6 Load Test
+      run: k6 run ./k6/script.js
+
+    - name: Upload result file for base branch
+      uses: actions/upload-artifact@v3
+      with:
+        name: base-results
+        path: k6/output/summary.json
+
+    - name: Stop docker services
+      run: docker compose down
+
+  test-head:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        ref: ${{ github.event.pull_request.head.sha }}
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v2
+      with:
+        node-version: '20'
+
+    - name: Install dependencies
+      run: npm ci
+
+    - name: Initialize the project
+      run: npm run init
+
+    - name: Build
+      run: npm run build
+
+    - name: Copy devnet config file from src to dist
+      run: cp ./config/config.devnet.yaml ./dist/config/config.yaml
+
+    - name: Start docker services
+      run: docker compose up -d
+
+    - name: Start Node.js API
+      run: node ./dist/src/main.js &
+
+    - name: Install k6
+      run: |
+        sudo gpg -k
+        sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
+        echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
+        sudo apt-get update
+        sudo apt-get install k6
+
+    - name: Wait for API to be ready
+      run: |
+        until curl --output /dev/null --silent --fail http://localhost:4001/hello; do
+          echo 'Waiting for API...'
+          sleep 1
+        done
+
+    - name: Preload cache
+      run: k6 run ./k6/preload.js
+
+    - name: Run k6 Load Test
+      run: k6 run ./k6/script.js
+
+    - name: Upload result file for head branch
+      uses: actions/upload-artifact@v3
+      with:
+        name: head-results
+        path: k6/output/summary.json
+
+    - name: Stop docker services
+      run: docker compose down
+
+  compare-results:
+    runs-on: ubuntu-latest
+
+    needs: [test-base, test-head]
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Download all artifacts
+        uses: actions/download-artifact@v3
+        with:
+          path: artifacts
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v2
+        with:
+          node-version: '20'
+
+      - name: Compare test results
+        run: |
+          node ./k6/compare-results.js ${{ github.event.pull_request.base.sha }} artifacts/base-results/summary.json ${{ github.event.pull_request.head.sha }} artifacts/head-results/summary.json report.md
+
+      - name: Render the report from the template
+        id: template
+        uses: chuhlomin/render-template@v1
+        if: github.event_name == 'pull_request'
+        with:
+          template: report.md
+          vars: |
+            base: ${{ github.event.pull_request.base.sha }}
+            head: ${{ github.event.pull_request.head.sha }}
+
+      - name: Upload the report markdown
+        uses: actions/upload-artifact@v3
+        if: github.event_name == 'pull_request'
+        with:
+          name: report-markdown
+          path: report.md
+
+      - name: Find the comment containing the report
+        id: fc
+        uses: peter-evans/find-comment@v2
+        if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
+        with:
+          issue-number: ${{ github.event.pull_request.number }}
+          comment-author: 'github-actions[bot]'
+          body-includes: 'k6 load testing comparison'
+
+      - name: Create or update the report comment
+        uses: peter-evans/create-or-update-comment@v2
+        if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
+        with:
+          comment-id: ${{ steps.fc.outputs.comment-id }}
+          issue-number: ${{ github.event.pull_request.number }}
+          body: ${{ steps.template.outputs.result }}
+          edit-mode: replace
diff --git a/config/config.devnet-old.yaml b/config/config.devnet-old.yaml
@@ -22,7 +22,6 @@ flags:
   useRequestLogging: false
   useVmQueryTracing: false
   processNfts: true
-  indexer-v3: true
   collectionPropertiesFromGateway: false
 features:
   eventsNotifier:
@@ -88,6 +87,7 @@ urls:
   ipfs: 'https://ipfs.io/ipfs'
   socket: 'devnet-socket-api.multiversx.com'
   maiarId: 'https://devnet-old-id-api.multiversx.com'
+  assetsCdn: 'https://tools.multiversx.com/assets-cdn'
 indexer:
   type: 'elastic'
   maxPagination: 10000

diff --git a/config/config.devnet.yaml b/config/config.devnet.yaml
@@ -18,7 +18,6 @@ flags:
   useRequestLogging: false
   useVmQueryTracing: false
   processNfts: true
-  indexer-v3: true
   collectionPropertiesFromGateway: false
 features:
   eventsNotifier:
@@ -32,9 +31,9 @@ features:
     hitsThreshold: 100
     ttl: 12
   transactionPool:
-    enabled: true
+    enabled: false
   transactionPoolWarmer:
-    enabled: true
+    enabled: false
     cronExpression: '*/5 * * * * *'
     ttlInSeconds: 60
   updateCollectionExtraDetails:
@@ -50,6 +49,9 @@ features:
   dataApi:
     enabled: false
     serviceUrl: 'https://devnet-data-api.multiversx.com'
+  assetsFetch:
+    enabled: true
+    assetesUrl: 'https://tools.multiversx.com/assets-cdn'
   auth:
     enabled: false
     maxExpirySeconds: 86400
@@ -92,8 +94,17 @@ features:
     nftQueueName: 'api-process-nfts'
     deadLetterQueueName: 'api-process-nfts-dlq'
   tps:
-    enabled: true
+    enabled: false
     maxLookBehindNonces: 100
+  nodesFetch:
+    enabled: true
+    serviceUrl: 'https://devnet-api.multiversx.com'
+  tokensFetch:
+    enabled: true
+    serviceUrl: 'https://devnet-api.multiversx.com'
+  providersFetch:
+    enabled: true
+    serviceUrl: 'https://devnet-api.multiversx.com'
 image:
   width: 600
   height: 600
@@ -136,6 +147,7 @@ caching:
   cacheTtl: 6
   processTtl: 600
   poolLimit: 50
+  cacheDuration: 3
 keepAliveTimeout:
   downstream: 61000
   upstream: 60000
@@ -156,4 +168,4 @@ inflation:
   - 719203
 nftProcess:
   parallelism: 1
-  maxRetries: 3
+  maxRetries: 3
diff --git a/config/config.e2e-mocked.mainnet.yaml b/config/config.e2e-mocked.mainnet.yaml
@@ -21,7 +21,6 @@ flags:
   useRequestCaching: true
   useKeepAliveAgent: true
   useTracing: false
-  indexer-v3: true
   collectionPropertiesFromGateway: false
 urls:
   self: 'https://api.multiversx.com'

diff --git a/config/config.e2e.mainnet.yaml b/config/config.e2e.mainnet.yaml
@@ -24,7 +24,6 @@ flags:
   useRequestCaching: true
   useKeepAliveAgent: true
   useTracing: false
-  indexer-v3: true
   collectionPropertiesFromGateway: false
 urls:
   self: 'https://api.multiversx.com'

diff --git a/config/config.mainnet.yaml b/config/config.mainnet.yaml
@@ -18,7 +18,6 @@ flags:
   useRequestLogging: false
   useVmQueryTracing: false
   processNfts: true
-  indexer-v3: false
   collectionPropertiesFromGateway: false
 features:
   eventsNotifier:
@@ -32,9 +31,9 @@ features:
     hitsThreshold: 100
     ttl: 12
   transactionPool:
-    enabled: true
+    enabled: false
   transactionPoolWarmer:
-    enabled: true
+    enabled: false
     cronExpression: '*/5 * * * * *'
     ttlInSeconds: 60 
   updateCollectionExtraDetails:
@@ -96,8 +95,20 @@ features:
     nftQueueName: 'api-process-nfts'
     deadLetterQueueName: 'api-process-nfts-dlq'
   tps:
+    enabled: false
+    maxLookBehindNonces: 100
+  nodesFetch:
+    enabled: true
+    serviceUrl: 'https://api.multiversx.com'
+  tokensFetch:
+    enabled: true
+    serviceUrl: 'https://api.multiversx.com'
+  providersFetch:
+    enabled: true
+    serviceUrl: 'https://api.multiversx.com'
+  assetsFetch:
     enabled: true
-    maxLookBehindNonces: 100  
+    assetesUrl: 'https://tools.multiversx.com/assets-cdn'
 image:
   width: 600
   height: 600
@@ -140,6 +151,7 @@ caching:
   cacheTtl: 6
   processTtl: 600
   poolLimit: 50
+  cacheDuration: 3
 keepAliveTimeout:
   downstream: 61000
   upstream: 60000
@@ -160,4 +172,4 @@ inflation:
   - 719203
 nftProcess:
   parallelism: 1
-  maxRetries: 3
+  maxRetries: 3
diff --git a/config/config.testnet.yaml b/config/config.testnet.yaml
@@ -18,7 +18,6 @@ flags:
   useRequestLogging: false
   useVmQueryTracing: false
   processNfts: true
-  indexer-v3: true
   collectionPropertiesFromGateway: false
 features:
   eventsNotifier:
@@ -32,9 +31,9 @@ features:
     hitsThreshold: 100
     ttl: 12
   transactionPool:
-    enabled: true
+    enabled: false
   transactionPoolWarmer:
-    enabled: true
+    enabled: false
     cronExpression: '*/5 * * * * *'
     ttlInSeconds: 60
   updateCollectionExtraDetails:
@@ -95,8 +94,20 @@ features:
     nftQueueName: 'api-process-nfts'
     deadLetterQueueName: 'api-process-nfts-dlq'
   tps:
-    enabled: true
+    enabled: false
     maxLookBehindNonces: 100
+  nodesFetch:
+    enabled: true
+    serviceUrl: 'https://testnet-api.multiversx.com'
+  tokensFetch:
+    enabled: true
+    serviceUrl: 'https://testnet-api.multiversx.com'
+  providersFetch:
+    enabled: true
+    serviceUrl: 'https://testnet-api.multiversx.com'
+  assetsFetch:
+    enabled: true
+    assetesUrl: 'https://tools.multiversx.com/assets-cdn'
 image:
   width: 600
   height: 600
@@ -139,6 +150,7 @@ caching:
   cacheTtl: 6
   processTtl: 600
   poolLimit: 50
+  cacheDuration: 3
 keepAliveTimeout:
   downstream: 61000
   upstream: 60000