UMCCR Orchestration Bus that leverage AWS EventBridge as Event Bus to automate the BioInformatics Workflows Pipeline.
At the top level, the Git repository root is the CDK TypeScript project. It is bootstrapped with cdk init orcabus --language typescript
. Therefore, the outer level codebase is the single CDK infrastructure application.
Please note; this is the INVERSE of some typical standalone project setup such that the repo root of the project is your app toolchain codebase and the deployment code are arranged under some arbitrary subdirectory like ./deploy/<cdk app root>
. We do not do this in this repo as we anticipate that we are going to deploy multiple of closely related micro applications.
In this repo, we flip this view such that the Git repo root is the TypeScript CDK project; that wraps our applications into ./lib/
directory. You may sparse checkout or directly open subdirectory to set up the application project alone if you wish; e.g. webstorm lib/workload/stateless/metadata_manager
or code lib/workload/stateless/metadata_manager
or pycharm lib/workload/stateless/sequence_run_manager
or rustrover lib/workload/stateless/filemanager
. However, code .
is a CDK TypeScript project.
This root level CDK app contains 4 major stacks: stateful-pipeline
,stateless-pipeline
, stateful
and stateless
. Pipeline stack is the CI/CD automation with CodePipeline setup. The stateful
stack holds and manages some long-running AWS infrastructure resources. The stateless
stack manages self-mutating CodePipeline reusable CDK Constructs for the MicroService Applications. In terms of CDK deployment point-of-view, the microservice application will be "stateless" application such that it will be changing/mutating over time; whereas "the data" its holds like PostgreSQL server infrastructure won't be changing that frequent. When updating "stateful" resources, there involves additional cares, steps and ops-procedures such as backing up database, downtime planning and so on; hence stateful. We use configuration constants to decouple the reference between stateful
and stateless
AWS resources.
In most cases, we deploy with automation across operational target environments or AWS accounts: beta
, gamma
, prod
. For some particular purpose (such as onboarding procedure, isolated experimentation), we can spin up the whole infrastructure into some unique isolated AWS account. These key CDK entrypoints are documented in the following sections: Automation and Manual.
CI/CD through CodePipeline automation from AWS toolchain account
There are 2 pipeline stacks in this project, one for the stateful and one for the stateless stack deployment. There is a
script to access the cdk
command for each pipeline:
-cdk-stateless-pipeline
- for stateless pipeline
-cdk-stateful-pipeline
- for stateful pipeline
make install
make check
make test
yarn cdk-stateless-pipeline synth
yarn cdk-stateless-pipeline diff
yarn cdk-stateless-pipeline deploy
manual deploy to an isolated specific AWS account
export AWS_PROFILE=dev
make install
make check
make test
yarn orcabus --help
yarn cdk-orcabus list
yarn cdk-orcabus synth OrcaBusStatefulStack
yarn cdk-orcabus diff OrcaBusStatefulStack
yarn cdk-orcabus deploy OrcaBusStatefulStack
yarn cdk-orcabus deploy --all
yarn cdk-orcabus destroy --all
Heads up: Polyglot programming environment. We shorten some trivial steps into Makefile
target. You may deduce step-by-step from Makefile
, if any.
When possible, please use either OrcaBus
(camel case) or orcabus
(all lower case).
Setting up baseline toolchain
docker --version
Docker version 24.0.7, build afdd53b
node -v
v18.19.0
npm i -g yarn
yarn -v
3.5.1
Additionally, we expect the following common tools be installed and available in your system shell PATH. We provide Brewfile as an example. You may manage these common tools in any other way as see fit for your local setup.
brew bundle
- We use docker compose as a mock stack for application local dev and running test suite purpose.
- Typically, you will have your own application compose stack defined at your app project root, if any.
- You can also reuse common docker compose stack, if applicable. See shared/README.md
- Run lint:
yarn lint
- Fix lint issue:
yarn lint-fix
- Opt-out lint: See .eslintignore
TypeScript
- Run prettier:
yarn prettier
- Fix prettier issue:
yarn prettier-fix
- Opt-out prettier: See .prettierignore
- Run
yarn audit
for package security vulnerabilities - Recommend fixing/updating any package with direct dependencies
- If vulnerabilities found in transitive dependency, but it has yet to resolve, then list them in
package.json > resolutions
node as Selective Dependency Resolutions condition explained here.
NOTE: We use pre-commit. It will guard and enforce static code analysis such as
lint
and any securityaudit
via pre-commit hook. You are encouraged to fix those. If you wish to skip this for good reason, you can by-pass Git pre-commit hooks by usinggit commit --no-verify
flag.
git config --unset core.hooksPath
pre-commit install
pre-commit run --all-files
We have GitHub Action workflow to reinforce Lint, Code Formatting and Pre-commit Hook check as Pull Request Build pipeline before main CI/CD automation run at CodePipeline. This is to protect any accidental secrets leak and/or pre-flight check for CI/CD automation.