Open source PPaaS Scripting (#150)

* Removed references to SystemTest (#134)

* Added README instructions for building and link to examples (#136)

* Zip all map (#138)

* add ZipAllMap, and some doc comments and tests

* add doctests to pr.sh

---------

Co-authored-by: Zachery Olson <[email protected]>

* Adds skipBody CLI argument - Skips Request and Response Body in Try Output (#140)

* Cargo version upgrade

* Updated the Guide to include the new available flag options for instructional purposes

* Added the ability to include skipBody argument in TryConfig and modified output to honor new skip flag

* included the new argument in the TmpTryConfig for testing purposes

* change version upgrade back to original as requested

* Removed unnessesary matches as requested

* Changes made to output format as requested

* Format changes as requested

* format changes as requested by Rustfmt in the github checks

* Manually fixed format errors

* Fixed additional fmt failure after updating rust to 1.72

---------

Co-authored-by: Trevor McMaster <[email protected]>

* Updated dependencies and fixed deprecations (#143)

* Updated dependencies and fixed deprecations

* Updated Wasm-pack to match the version in the 0.6.0 scripting branch

* Example wasm tests (#147)

* Added additional tests for all the example yaml files

* Updated the README for the current changes

* Added initial common directory

* Added open source agent code

* Added initial controller check-in

- Not fully working. Need to get monorepo set up

* Removed references to familysearch.org

* Added code to the pr script to build the hdr-histogram for the controller

* Attempt to fix the common install/build issues in the controller

* Added code to make a monorepo out of the common, agent, and controller

* Added scripting changes from controller and agent

* Fixed merge duplicate test

* Added new PPaaS PR flow

- Renamed the old flows to show what they are testing
- Changed the guide PR to only monitor the guide files
- Added new flow for the PPaaS common, agent, and controller

* Updated the PR flow and added husky

* Added the .env.test files we're missing for the github action

* Changed all catch blocks to catch unknown

- Added type checking around the S3 errors

* Basic working build

* Working version of next buildgit status!

* Fixed storybook by removing reference to deprecated zlib

* Removed references to fslocal.org examples

* Removed husky since it's not needed on all check-ins

* Removed duplication in pr script

* Re-enabled the react build now that it's working

* Fixed the Application Name for the controller for unit tests

* Added code to generage a .env.local for the build

* Changed the .env.local file to environment variables

* Cleaned up the global script to improve performance and nyc output

* Fixed the nyc merge

* Changed the build environment files back to .env so it can be overridden

- environment variables override all .env files.

* Brought over scripting changes to the yamlparser from common

* Removed unneeded totalTime var

* Open source PPaaS (#149)

* Added initial common directory

* Added open source agent code

* Added initial controller check-in

- Not fully working. Need to get monorepo set up

* Removed references to familysearch.org

* Added code to the pr script to build the hdr-histogram for the controller

* Attempt to fix the common install/build issues in the controller

* Added code to make a monorepo out of the common, agent, and controller

* Added new PPaaS PR flow

- Renamed the old flows to show what they are testing
- Changed the guide PR to only monitor the guide files
- Added new flow for the PPaaS common, agent, and controller

* Updated the PR flow and added husky

* Added the .env.test files we're missing for the github action

* Changed all catch blocks to catch unknown

- Added type checking around the S3 errors

* Basic working build

* Working version of next buildgit status!

* Fixed storybook by removing reference to deprecated zlib

* Removed references to fslocal.org examples

* Removed husky since it's not needed on all check-ins

* Removed duplication in pr script

* Re-enabled the react build now that it's working

* Fixed the Application Name for the controller for unit tests

* Added code to generage a .env.local for the build

* Changed the .env.local file to environment variables

* Cleaned up the global script to improve performance and nyc output

* Fixed the nyc merge

* Changed the build environment files back to .env so it can be overridden

- environment variables override all .env files.

---------

Co-authored-by: Zachery Olson <[email protected]>
Co-authored-by: Idel <[email protected]>

.eslintrc

@@ -0,0 +1,74 @@
+{
+  "root": true,
+  "parser": "@typescript-eslint/parser",
+  "parserOptions": {
+    "project": [
+      "./common/tsconfig.json",
+      "./agent/tsconfig.json",
+      "./controller/tsconfig.json",
+      "./controller/tsconfig.test.json"
+    ]
+  },
+  "plugins": [
+    "@typescript-eslint"
+  ],
+  "extends": [
+    "eslint:recommended",
+    "plugin:@typescript-eslint/stylistic",
+    "plugin:@typescript-eslint/recommended"
+  ],
+  "ignorePatterns": [
+    "guide",
+    "lib",
+    "dist",
+    "next-env.d.ts",
+    "next.config.js",
+    "setup.js",
+    "controller/lib/hdr-histogram-wasm"
+  ],
+  "rules": {
+    "@typescript-eslint/no-explicit-any": 0,
+    "@typescript-eslint/no-non-null-assertion": 0,
+    "@typescript-eslint/explicit-module-boundary-types": 0,
+    "@typescript-eslint/ban-types": 1,
+    "@typescript-eslint/no-inferrable-types": 0,
+    "@typescript-eslint/no-unused-vars": [1, { "argsIgnorePattern": "^_" }],
+    "no-prototype-builtins": 1,
+    "require-await": 1,
+    "class-name": 0,
+    "curly": 1,
+    "eqeqeq": ["error", "smart"],
+    "linebreak-style": 1,
+    "object-literal-sort-keys": 0,
+    "only-arrow-functions": 0,
+    "max-classes-per-file": 1,
+    "max-line-length": 0,
+    "member-ordering": 0,
+    "no-angle-bracket-type-assertion": 0,
+    "no-bitwise": 1,
+    "no-console": 1,
+    "no-multiple-empty-lines": ["error", { "max": 2, "maxEOF": 0, "maxBOF": 0 }],
+    "no-empty": [1, { "allowEmptyCatch": true }],
+    "no-empty-interface": 0,
+    "no-reference": 0,
+    "no-string-literal": 0,
+    "no-trailing-spaces": 1,
+    "no-unused-expressions": 1,
+    "no-useless-catch": 0,
+    "prefer-const": 1,
+    "semi": 1,
+    "sort-imports": 1,
+    "space-before-function-paren": 1,
+    "spaced-comment": ["error", "always", { "block": { "balanced": true } }],
+    "space-infix-ops":"warn",
+    "strict": 1,
+    "comma-dangle": 1,
+    "triple-equals": 0,
+    "unified-signatures": 0,
+    "camelcase": 1,
+    "no-irregular-whitespace": 1,
+    "object-shorthand": 1,
+    "@typescript-eslint/await-thenable": 1,
+    "quotes": ["warn", "double"]
+  }
+}

.github/workflows/pr-js.yml → .github/workflows/pr-guide.yml

@@ -1,14 +1,14 @@
 on:
   pull_request:
     paths:
-      - '**.js*'
-      - '**.ts*'
-      - '**.html'
+      - 'guide/**.js*'
+      - 'guide/**.ts*'
+      - 'guide/**.html'
       - 'guide/src/**/*.md'
-      - '**/package.json'
-      - '**/package-lock.json'
+      - 'guide/**/package.json'
+      - 'guide/**/package-lock.json'
 
-name: Pull Request Javascript
+name: Pull Request Guide
 jobs:
   create-release:
     name: Build guide

.github/workflows/pr-ppaas.yml

@@ -0,0 +1,92 @@
+on:
+  pull_request:
+    paths:
+      - './package.json'
+      - './package-lock.json'
+      - 'common/**.js*'
+      - 'common/**.ts*'
+      - 'common/**/package.json'
+      - 'agent/**.js*'
+      - 'agent/**.ts*'
+      - 'agent/**/package.json'
+      - 'controller/**.html'
+      - 'controller/**.js*'
+      - 'controller/**.ts*'
+      - 'controller/**/package.json'
+
+name: Pull Request PPaaS
+jobs:
+  test-release:
+    name: Build project
+    strategy:
+      matrix:
+        node-version: [18.x, 20.x]
+      fail-fast: false
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Add Rust toolchain
+        uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          target: wasm32-unknown-unknown
+          toolchain: stable
+          override: true
+      - name: Add Node.js toolchain ${{ matrix.node-version }}
+        uses: actions/setup-node@v3
+        with:
+          node-version: ${{ matrix.node-version }}
+
+      - name: Build Webassemblies ${{ matrix.node-version }}
+        run: |
+          set -x
+          # install mdbook and wasm-pack
+          mkdir ~/bin
+          PATH=$PATH:~/bin
+          curl -sSL https://github.com/rustwasm/wasm-pack/releases/download/v0.12.1/wasm-pack-v0.12.1-x86_64-unknown-linux-musl.tar.gz \
+            | tar -xz --strip-components=1 -C ~/bin --no-anchored wasm-pack
+
+          # setup some envs to various paths for convenience
+          PROJECT_ROOT=$PWD
+          CONTROLLER_DIR=$(realpath $PROJECT_ROOT/controller)
+          CONFIG_WASM_LIB_DIR=$(realpath $PROJECT_ROOT/lib/config-wasm)
+          HDR_WASM_LIB_DIR=$(realpath $PROJECT_ROOT/lib/hdr-histogram-wasm)
+          HDR_WASM_OUTPUT_REACT_DIR=$CONTROLLER_DIR/lib/hdr-histogram-wasm
+
+          # build the hdr-histogram-wasm for the results viewer
+          cd $HDR_WASM_LIB_DIR
+          wasm-pack build --release -t bundler -d $HDR_WASM_OUTPUT_REACT_DIR --scope fs
+
+          # build the config-wasm for the yaml parser
+          cd $CONFIG_WASM_LIB_DIR
+          wasm-pack build --release -t nodejs --scope fs
+
+      - name: Create .env file for controller that can be overridden by a .env.production file
+        run: |
+          set -x
+          # setup some envs to various paths that re required for build
+          PROJECT_ROOT=$PWD
+          CONTROLLER_DIR=$(realpath $PROJECT_ROOT/controller)
+          # .env.production will override .env, but NOT .env.local or environment variables passed in
+          ENV_FILE=$CONTROLLER_DIR/.env
+          touch "$ENV_FILE"
+          echo PEWPEWCONTROLLER_UNITTESTS_S3_BUCKET_NAME="unit-test-bucket" >> "$ENV_FILE"
+          echo PEWPEWCONTROLLER_UNITTESTS_S3_BUCKET_URL="https://unit-test-bucket.s3.amazonaws.com" >> "$ENV_FILE"
+          echo PEWPEWCONTROLLER_UNITTESTS_S3_KEYSPACE_PREFIX="unittests/" >> "$ENV_FILE"
+          echo PEWPEWCONTROLLER_UNITTESTS_S3_REGION_ENDPOINT="s3-us-east-1.amazonaws.com" >> "$ENV_FILE"
+          echo APPLICATION_NAME=pewpewcontroller >> "$ENV_FILE"
+          echo AGENT_ENV="unittests" >> "$ENV_FILE"
+          echo AGENT_DESC="c5n.large" >> "$ENV_FILE"
+          echo PEWPEWAGENT_UNITTESTS_SQS_SCALE_OUT_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/unittests/sqs-scale-out" >> "$ENV_FILE"
+          echo PEWPEWAGENT_UNITTESTS_SQS_SCALE_IN_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/unittests/sqs-scale-in" >> "$ENV_FILE"
+          echo PEWPEWCONTROLLER_UNITTESTS_SQS_COMMUNICATION_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/unittests/sqs-communication" >> "$ENV_FILE"
+
+      - name: Install NPM Dependencies
+        run: npm ci
+      - name: Run Lint
+        run: npm run linterror
+      - name: Build Controller
+        run: npm run build:react
+      - name: Run Tests
+        run: NODE_ENV=test npm test

.github/workflows/pr.yml → .github/workflows/pr-rust.yml

@@ -5,7 +5,7 @@ on:
       - '**/Cargo.toml'
       - '**/Cargo.lock'
 
-name: Pull Request
+name: Pull Request Rust
 jobs:
   test:
     name: Test Suite

.gitignore

@@ -7,5 +7,15 @@
 /*.txt
 /*.dump
 /*.log
-/*.json
+/stats-*.json
+/test-*.json
+/integration.json
 node_modules
+.nyc_output/
+coverage/
+testmerge.json
+.env.local*
+.env.development.local*
+.env.test.local*
+.env.production.local*
+

agent/.ebextensions/15pewpewagent-security.eb.config

@@ -0,0 +1,13 @@
+Resources:
+  AWSEBSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      SecurityGroupIngress:
+      - IpProtocol: tcp
+        FromPort: 8080
+        ToPort: 8081
+        CidrIp: 10.0.0.0/8
+      - IpProtocol: tcp
+        FromPort: 80
+        ToPort: 80
+        CidrIp: 10.0.0.0/8

agent/.ebextensions/20pewpewagent-performance.config

@@ -0,0 +1,5 @@
+commands:
+  10set-file-max:
+    command: "echo fs.file-max = 999999 >> /etc/sysctl.conf"
+  11set-file-limit:
+    command: "echo '*               -       nofile         999999' >> /etc/security/limits.conf"

agent/.ebextensions/30pewpewagent-autoscale.eb.config

@@ -0,0 +1,5 @@
+Resources:
+  AWSEBAutoScalingGroup:
+    Type: AWS::AutoScaling::AutoScalingGroup
+    Properties:
+      DesiredCapacity: 1

agent/.ebextensions/40pewpewagent-change-npm-permissions.config

@@ -0,0 +1,12 @@
+files:
+  "/opt/elasticbeanstalk/hooks/appdeploy/post/00_set_tmp_permissions.sh":
+    mode: "000755"
+    owner: root
+    group: root
+    content: |
+      #!/usr/bin/env bash
+      echo "set /tmp permissions start"
+      sudo chown -R nodejs:nodejs /tmp/.npm
+      sudo chown -R nodejs:nodejs /tmp/.config
+      sudo chmod -R 755 /tmp/.config
+      echo "set /tmp permissions done"

agent/.env.test

@@ -0,0 +1,15 @@
+# NODE_ENV=test ignores .env and .env.local
+PEWPEWCONTROLLER_UNITTESTS_S3_BUCKET_NAME="unit-test-bucket"
+PEWPEWCONTROLLER_UNITTESTS_S3_BUCKET_URL="https://unit-test-bucket.s3.amazonaws.com"
+PEWPEWCONTROLLER_UNITTESTS_S3_KEYSPACE_PREFIX="unittests/"
+PEWPEWCONTROLLER_UNITTESTS_S3_REGION_ENDPOINT="s3-us-east-1.amazonaws.com"
+ADDITIONAL_TAGS_ON_ALL="application=pewpewagent"
+
+APPLICATION_NAME=pewpewagent
+CONTROLLER_ENV="unittests"
+AGENT_DESC="c5n.large"
+PEWPEWAGENT_UNITTESTS_SQS_SCALE_OUT_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/unittests/sqs-scale-out"
+PEWPEWAGENT_UNITTESTS_SQS_SCALE_IN_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/unittests/sqs-scale-in"
+PEWPEWCONTROLLER_UNITTESTS_SQS_COMMUNICATION_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/unittests/sqs-communication"
+
+ENV_KEY=".env.test"

agent/.github/workflows/pr.yml

@@ -0,0 +1,36 @@
+on:
+  pull_request:
+
+name: Pull Request Javascript
+jobs:
+  test-release:
+    name: Build project
+    strategy:
+      matrix:
+        node-version: [18.x, 20.x]
+    runs-on: ubuntu-latest
+    # env:
+    #   USE_XVFB: true
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Add Node.js toolchain ${{ matrix.node-version }}
+        uses: actions/setup-node@v2
+        with:
+          node-version: ${{ matrix.node-version }}
+
+      - name: Setup Artifactory
+        env:
+          CI_USER_TOKEN: ${{ secrets.CI_USER_TOKEN }}
+          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
+        run: |
+          echo -e "machine github.com\n  login $CI_USER_TOKEN" > ~/.netrc
+          echo "//familysearch.jfrog.io/artifactory/api/npm/fs-npm-prod-virtual/:_authToken=${NPM_TOKEN}" >> ~/.npmrc
+          echo "@fs:registry=https://familysearch.jfrog.io/artifactory/api/npm/fs-npm-prod-virtual/" >> ~/.npmrc
+          echo git config --global --add url."https://[email protected]/".insteadOf "https://github.com/"
+
+      - name: Install NPM Dependencies
+        run: npm ci
+      - name: Run Tests
+        run: NODE_ENV=test npm test

agent/.gitignore

@@ -0,0 +1,34 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+setaws.sh
+/.nyc_output/
+
+# production
+/build
+
+# development
+/dist
+
+# misc
+.DS_Store
+.env
+.env.local*
+.env.development.local*
+.env.test.local*
+.env.production.local*
+.idea
+.vscode/
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+agent.log*
+app-pp*.json*

agent/.platform/hooks/predeploy/200_npm_rebuild.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+set -ex
+
+echo "npm rebuild start"
+
+APP_STAGING_DIR=$( /opt/elasticbeanstalk/bin/get-config  platformconfig -k AppStagingDir )
+
+# Add NPM-installed executables to the PATH
+NPM_LIB=$( npm list -g | head -1 )
+NPM_HOME=$( dirname "${NPM_LIB}" )
+export PATH="${NPM_HOME}/bin:${PATH}"
+
+# rebuild to fix the node_modules/.bin/ folder
+cd "${APP_STAGING_DIR}"
+npm rebuild
+chmod a+x node_modules/.bin/*
+
+echo "npm rebuild done"

agent/.sample-env

@@ -0,0 +1,17 @@
+# https://nextjs.org/docs/pages/building-your-application/configuring/environment-variables
+# Next.js will load these automatically. We use dotenv-flow to load them for mocha
+# Copy this file to .env.local and modify these to your services
+
+# AWS_PROFILE=default
+PEWPEWCONTROLLER_UNITTESTS_S3_BUCKET_NAME="my-test-service"
+PEWPEWCONTROLLER_UNITTESTS_S3_BUCKET_URL="https://my-test-service.s3.amazonaws.com"
+PEWPEWCONTROLLER_UNITTESTS_S3_KEYSPACE_PREFIX="pewpewcontroller-unittests-s3/"
+PEWPEWCONTROLLER_UNITTESTS_S3_REGION_ENDPOINT="s3-us-east-1.amazonaws.com"
+ADDITIONAL_TAGS_ON_ALL="application=pewpewcontroller"
+
+APPLICATION_NAME=pewpewagent
+CONTROLLER_ENV="unittests"
+AGENT_DESC="c5n.large"
+PEWPEWAGENT_UNITTESTS_SQS_SCALE_OUT_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/my-account/pewpewagent-unittests-sqs-scale-out"
+PEWPEWAGENT_UNITTESTS_SQS_SCALE_IN_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/my-account/pewpewagent-unittests-sqs-scale-in"
+PEWPEWCONTROLLER_UNITTESTS_SQS_COMMUNICATION_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/my-account/pewpewcontroller-unittests-sqs-communication"