The article Angular Universal: a Complete Practical Guide offers an excellent introduction to +Angular Universal, server-side rendering (SSR) and search engine optimization (SEO).
+++The Angular AOT compiler converts your Angular HTML and TS code into efficient JavaScript code" +(angular.io/guide/aot-compiler). That code is still run on client-side to render the first and +every other view. SSR is not an alternative but an additional technique to further increase +performance for first-time visitors and SEO, as the first view is rendered on server side and the +client receives that assembled and styled HTML, so there is no need to dynamically render anything +at the beginning.
+
Martin Schneider Aug 26, 2021 at 14:15 (source)
+We are going to go start with an existing Angular application, and we will progressively turn it +into an Angular Universal application while explaining every step along the way!
+The second part of the instructions describe how to tweak the app to meet the need of this project. +The previous version of the OpenChallenges app generated in early 2022 is referenced as the "legacy" +app.
+nx generate @nx/angular:app openchallenges --routing=true --style=scss
+++Note Add the option
+--dry-runto preview the output of Nx commands.Note The legacy OpenChallenges app generated early 2022 was build with the generator +
+@nx/angular:webpack-browser. The newly generated app is now built with +@angular-devkit/build-angular:browser(seeproject.json).
openchallenges application by running the following in the terminal:nx generate @schematics/angular:universal --project=openchallenges
+Follow the instructions listed in the article Server-side rendering (SSR) with Angular for Nx
+ workspaces to add the project target serve-ssr.
Delete the folder src/assets and src/environments. Import the folder src/assets and
+ src/config from the legacy OpenChallenges app.
Update project.json
Add this line to the top of project.json is missing.
"$schema": "../../node_modules/nx/schemas/project-schema.json",
+Update the value of prefix.
"prefix": "openchallenges",
+Copy-paste the following targets from the legacy app.
+preparelint-fixbuild-imageUpdate the value of assets specified in the build target.
"assets": [
+ "apps/openchallenges/src/assets",
+ "apps/openchallenges/src/config",
+ {
+ "input": "libs/shared/typescript/assets/src/assets",
+ "glob": "**/*",
+ "output": "assets"
+ },
+ {
+ "input": "libs/openchallenges/assets/src/assets",
+ "glob": "**/*",
+ "output": "openchallenges-assets"
+ },
+ {
+ "input": "libs/shared/typescript/assets/src",
+ "glob": "favicon.ico",
+ "output": ""
+ }
+],
+Update the value of budgets in the configurations object of the build target.
"budgets": [
+ {
+ "type": "initial",
+ "maximumWarning": "500kb",
+ "maximumError": "1mb"
+ },
+ {
+ "type": "anyComponentStyle",
+ "maximumWarning": "2kb",
+ "maximumError": "8kb"
+ }
+],
+Empty the array value of fileReplacements in the configurations object of the
+ build target.
Set the project tags.
"tags": [
+ "type:app",
+ "scope:client"
+],
+Set the project implicitDependencies.
+
"implicitDependencies": [
+ "openchallenges-styles",
+ "openchallenges-themes",
+ "shared-typescript-assets",
+ "openchallenges-api-gateway",
+ "openchallenges-keycloak"
+]
+Replace the content of the src/main.ts with the following code to make use of the
+ configuration defined in the folder src/config to enable Build once, deploy many.
import { enableProdMode } from '@angular/core';
+import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';
+
+import { AppModule } from './app/app.module';
+import {
+ AppConfig,
+ APP_CONFIG,
+ Environment,
+} from '@sagebionetworks/openchallenges/config';
+
+function bootstrap() {
+ fetch('/config/config.json')
+ .then((response) => response.json() as Promise<AppConfig>)
+ .then((config: AppConfig) => {
+ if (
+ [Environment.Production, Environment.Staging].includes(
+ config.environment
+ )
+ ) {
+ enableProdMode();
+ }
+
+ console.log('openchallenges config', config);
+
+ platformBrowserDynamic([{ provide: APP_CONFIG, useValue: config }])
+ .bootstrapModule(AppModule)
+ .catch((err) => console.error(err));
+ });
+}
+
+if (document.readyState === 'complete') {
+ bootstrap();
+} else {
+ document.addEventListener('DOMContentLoaded', bootstrap);
+}
+Update the src files based on the legacy code. These files include but are not limited to:
src/app/*
_app-theme.scsssrc/index.htmlsrc/proxy.conf.jsonstyles.scss
Update the files in the project folder based on their legacy version. These files include but + are not limited to:
+docker/.eslintrc.jsonDockerfilenx server openchallenges --prod
+
+$ ls -alh dist/apps/openchallenges/server/
+total 5.3M
+drwxr-xr-x 2 vscode vscode 4.0K Jul 28 15:39 .
+drwxr-xr-x 3 vscode vscode 4.0K Jul 28 15:39 ..
+-rw-r--r-- 1 vscode vscode 174K Jul 28 15:39 297.js
+-rw-r--r-- 1 vscode vscode 112K Jul 28 15:39 3rdpartylicenses.txt
+-rw-r--r-- 1 vscode vscode 6.9K Jul 28 15:39 513.js
+-rw-r--r-- 1 vscode vscode 95K Jul 28 15:39 603.js
+-rw-r--r-- 1 vscode vscode 6.7K Jul 28 15:39 654.js
+-rw-r--r-- 1 vscode vscode 299K Jul 28 15:39 715.js
+-rw-r--r-- 1 vscode vscode 174K Jul 28 15:39 731.js
+-rw-r--r-- 1 vscode vscode 248K Jul 28 15:39 748.js
+-rw-r--r-- 1 vscode vscode 13K Jul 28 15:39 762.js
+-rw-r--r-- 1 vscode vscode 7.1K Jul 28 15:39 822.js
+-rw-r--r-- 1 vscode vscode 102K Jul 28 15:39 989.js
+-rw-r--r-- 1 vscode vscode 4.0M Jul 28 15:39 main.js
+From Angular Universal: a Complete Practical Guide:
+++We will need the production version of the Universal bundle, as the development bundle will not +work.
+
Really?
+To review and adapt:
+# Development
+npm run start
+http://localhost:4200/
+
+# Tests
+npm run lint
+npm run test
+npm run e2e
+
+# AOT Compilation
+npm run build
+
+# SSR Compilation
+npm run build:ssr
+npm run serve:ssr
+http://localhost:4000/
+This document describes how SCSS styles, themes and assets are organized across a web app and its +libraries.
+Each Angular component has its own style file (css, scss, less, etc). When building the app, Angular +includes the component style files into the app style bundle. Additional styling files are used in +our applications. These files are organized as shown below.
+apps/
+└─ web-app/
+ └─ src/
+ ├─ _app-themes.scss <---- app styles (2)
+ ├─ index.html
+ └─ styles.scss <---- app styles (1)
+dist/
+└─ apps/
+ └─ web-app/
+ ├─ index.html
+ └─ styles.<hash>.css <---- app style bundle (7)
+libs/
+├─ shared/
+│ ├─ assets/ <---- cross-app assets
+│ ├─ styles/ <---- cross-app styles (5)
+│ └─ themes/ <---- cross-app assets (6)
+└─ web/
+ ├─ assets/ <---- app-specific assets
+ ├─ styles/ <---- app-specific styles (3)
+ └─ themes/ <---- app-specific assets (4)
+libs/web/styles and
+ the app themes defined in apps/web-app/src/_app-themes.scss. This file is referenced in the
+ build options defined in the project.json of the app.libs/web/themes.libs/shared/typescript/styles.libs/web.Assets like images are used in HTML and SCSS files. Assets that are shared across applications are
+stored in libs/shared/typescript/assets. Assets that are specific to an app are located in
+libs/<app>/assets (e.g., where libs/web/assets). In order to use these assets in the app and
+libraries of the web-app application, the path to the asset folders are referenced in the build
+options of the app (see project.json). This is also where a prefix is defined to distinguish the
+asset files from these libraries. This example shows how to import an image from the cross-app and
+app-specific asset libraries.
<img src="/shared-typescript-assets/images/github.png">
+<img src="/openchallenges-assets/images/challenge-view-header-background.png">
+Create a file named _<component>-theme.scss that will be used to apply a theme to a component.
+While the style file of a component is responsible for the layout of the component, the theme file
+is responsible for defining the colors and typography of the component.
$ ls -1 libs/web/about/src/lib/
+about.component.html
+about.component.scss <---- style file
+about.component.spec.ts
+about.component.ts
+about.module.ts
+_about-theme.scss <---- theme file
+Seed the theme file with the following template:
+++Note +
+@mixin color()collects the color-related styles,colorand*-color(i.ebackground-color), while the +typography styles,line-heightandfont-*(i.efont-size), should be added into@mixin typography().
@use 'sass:map';
+@use '@angular/material' as mat;
+
+@mixin color($theme) {
+ $config: mat.get-color-config($theme);
+ $primary: map.get($config, primary);
+ $accent: map.get($config, accent);
+ $warn: map.get($config, warn);
+ $background: map.get($config, background);
+ $foreground: map.get($config, foreground);
+
+ // Specify the colors of the elements of the component here.
+ // Example:
+ .awesome-class {
+ color: mat.get-color-from-palette($primary, default);
+ }
+}
+
+@mixin typography($theme) {
+ $config: mat.get-typography-config($theme);
+
+ // Specify the typography of the elements of the component here.
+ // Example:
+ .awesome-class {
+ font-family: mat.font-family($config);
+ font-size: mat.font-size($config);
+ font-weight: mat.font-weight($config);
+ }
+}
+
+@mixin theme($theme) {
+ $color-config: mat.get-color-config($theme);
+ @if $color-config != null {
+ @include color($theme);
+ }
+
+ $typography-config: mat.get-typography-config($theme);
+ @if $typography-config != null {
+ @include typography($theme);
+ }
+}
+Reference the theme file created in the index theme file _lib-theme.scss of the library you are
+working on.
@use './lib/<component>-theme'; <---- import theme file
+@use './lib/<component-2>-theme';
+
+@mixin theme($theme) {
+ @include <component>-theme.theme($theme); <---- reference theme file
+ @include <component-2>-theme.theme($theme);
+}
+If you are creating a new library, you need to reference the library index theme file in the index
+file of the themes library of the application. The themes library of the application web-app
+is located in libs/web/themes. Open its index file libs/web/themes/src/_index.scss and add the
+reference to the index theme file of the new library.
@use 'libs/web/<lib>/src/lib-theme' as web-<lib>;
+@use 'libs/web/<lib-2>/src/lib-theme' as web-<lib-2>;
+
+@mixin theme($theme) {
+ @include web-<lib>.theme($theme);
+ @include web-<lib-2>.theme($theme);
+}
+The theme defined for the application should now be applied to the component.
+ + + + + + + + + + + + + +This cheat sheet provides an overview of the commands needed when developing in this monorepo.
+The workspace of this monorepo was generated with:
+yarn create nx-workspace openchallenges --preset=empty --packageManager=yarn
+Nx projects are created using a "generator". A generator must be installed before being able to use +it. The generators listed in this document are automatically installed when the npm dependencies of +this workspace have been installed.
+yarn add -D <generator>
+Create a new project:
+nx g <generator>:<project-type> <project> [--dry-run]
+The option --dry-run simulates the generation of the project and shows the location of the files
+that would be generated.
Generators used in this workspace:
+| Project Type | +Generator Command | +
|---|---|
| Angular app | +nx g @nx/angular:app <project> |
+
| React app | +nx g @nrwl/react:app <project> |
+
| TypeScript CLI | +nx generate @nx/js:app <project> |
+
| Angular lib | +nx g @nx/angular:lib <project> |
+
Newly created projects are added to workspace.json.
+nx g @nx/angular:component <component> --project <project>
+Example:
+Add a component named header to the library web-ui:
nx g @nx/angular:component header --project web-ui --dry-run
+
+CREATE libs/web/ui/src/lib/header/header.component.scss
+CREATE libs/web/ui/src/lib/header/header.component.html
+CREATE libs/web/ui/src/lib/header/header.component.spec.ts
+CREATE libs/web/ui/src/lib/header/header.component.ts
+UPDATE libs/web/ui/src/lib/web-ui.module.ts
+nx g @nx/angular:service page-title --project web-data-access
+Example:
+Add a service named page-title to the library web-data-access:
nx g @nx/angular:service page-title --project web-data-access --dry-run
+
+CREATE libs/web/data-access/src/lib/page-title.service.spec.ts
+CREATE libs/web/data-access/src/lib/page-title.service.ts
+Each project includes the file project.json that defines one or more "targets".
Run a single target:
+nx <target> <project> // OR
+nx run <project>:<target>
+Run a given target for multiple projects:
+[nx run-many] --target=test --all
+nx run-many --target=test --all --parallel // executes in parallel (default: 3)
+nx run-many --target=test --all --parallel=2
+nx run-many --target=test --projects=proj1,proj2
+List the apps and libs that are affected by uncommitted changes.
+nx affected:apps
+nx affected:libs
+Run a given target for all the projects affected:
+nx affected:<target>
+nx affected:<target> --parallel
+Explore the project dependency graph:
+nx graph
+
+> NX Affected criteria defaulted to --base=main --head=HEAD
+> NX Project graph started at http://127.0.0.1:4211
+
+nx affected:graph // select all the projects affected
+This document describes how to centralize and explore the logs and system metrics generated by a +fleet of hosts (EC2 instances) and Docker containers. The instructions are provided in the context +of monitoring hosts and containers during a scientific challenge organized by Sage Bionetworks.
+The solution described here offers a single, user-friendly interface to monitor logs and visualize +system metrics generated by a fleet of hosts and containers. This solution contributes to
+Set alerts.
+Improved stability
+Set alerts.
+Better engineer experience
+A single, user-friendly dashboard enables the engineer in charge of monitoring the system to + quickly access logs and system health information.
+Enhanced reporting
+In the context of the organization of a scientific challenge, the collected data aim to answer the +following questions:
+
The components of the stack are:
+challenge-elk: EC2 instance that runs the ELK stackchallenge-node-*: EC2 instances that run Beat agents (Metricbeat, Filebeat).++Note +Instructions on how to deploy and use this monitoring solution are given below. The instructions are +given in the context of monitoring a stack used to process submissions received during a scientific +challenge. However, this solution can be used to monitor hosts and containers deployed for other +applications.
+
The only additional resource introduced by this architecture is the EC2 instance where the ELK stack
+is deployed. The instance must have at least 4 GB and 2 vCPUs, which is why a t2.medium is
+recommended. The storage capacity needed depends on the number of instances monitored, the size and
+frequency of the data collected. It is currently recommended to allocate 100 GB of storage (under
+evaluation).
For example, here is a cost estimate for the recommended EC2 instance (valid on 2022-07-24).
+t2.medium Instance HourTotal: 30 * 24 * 0.0464 + 100 * 0.10 = $43.4/month
+Please give credit to the following persons if you are using or building on top of this solution.
+ +Create an EC2 instance in the AWS account CnbAccount with the following specifications.
++Note +
+CnbAccountrepresents the AWS account of the Challenge & Benchmarking group at Sage.
<challenge name>-elk (e.g. ptb-challenge-elk)Tags:
+Department: CNBProject: challenge (selected from this
+ list)CostCenter: select value from these
+ listsApplication and OS Images (Amazon Machine Image)
+Ubuntu Server 22.04 LTS (HVM), SSD Volume TypeArchitecture: 64-bit (x86)
Instance type
+Instace type: t2.medium
+ > Note
+ > The instance needs at least 2 vCPUs and 4 GB of memory.
Key pair
+Select your SSH key pair or create a new one.
+Network settings
+Editvpc-0a70996f3e816e067subnet-01898b708714fa3b6Select existing security group.sg-0807a0e374542affd (cnbvpc-VpnSecurityGroup-JBXT9AUVDXCA).++Note These values will be different if you are creating the EC2 instance from an AWS account +different from
+CnbAccount.
Once the instance has been created, configure it to use the following IAM:
+EC2 > Instances.Actions > Security > Modify IAM role.AmazonSSMRoleForInstancesQuickSetup.Use the AWS Systems Manager Agent (SSM Agent) to connect to the EC2 instance. If this is your +first time ever connecting to an instance created by the CnbAccount AWS account, complete the +following step first. Otherwise, skip to steps 2 and 3.
+Create an ~/.aws/config file if you do not already have one. Add the following:
+
[profile cnb]
+region = us-east-1
+sso_start_url = https://d-906769aa66.awsapps.com/start
+sso_region = us-east-1
+sso_account_id = 216152803258
+sso_role_name = Administrator
+output = json
+cnb.
+Before connecting to any cnb instances, first login to AWS SSO:
+
aws --profile cnb sso login
+Allow. You are now
+ authenticated for remote access for the next 8-10 hours. Re-login as necessary.
+Connect to the instance as ubuntu, passing the Instance ID to --target:
+
aws ssm start-session \
+ --profile cnb \
+ --document-name AWS-StartInteractiveCommand \
+ --parameters command="sudo su - ubuntu" \
+ --target <instance id>
+Follow these instructions to connect to the instance via SSH using the command ssh <profile name>.
Add a new profile to the ~/.ssh/config file.
+
Host <challenge name>-elk
+ HostName i-0fad4cb3e6543283e
+ User ubuntu
+ IdentityFile ~/.ssh/<your pem file>
+ ProxyCommand sh -c "AWS_PROFILE=cnb aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
+Copy the public portion of your SSH key (on your local computer) to the instance’s
+ ~/.ssh/authorizedkeys file.
Set the permission of the authorizedkeys file to 600. +
chmod 600 ~/.ssh/authorizedkeys
+Connect to the instance. +
ssh <challenge name>-elk
+Start by updating the system packages on the instance. System packages should be regularly + updated for enhanced security. On Ubuntu, +
sudo apt update
+sudo apt upgrade -y
+sudo systemctl enable docker
+We will use the hostname of the EC2 instances to filter logs and metrics in Kibana. Therefore, it + is recommended to use short and descriptive hostnames. To update the hostname of an EC2 instance:
+/etc/hostname.<challenge name>-elk. For example, ptb-challenge-elk for the instance
+ running the ELK stack for the Preterm Birth Prediction DREAM Challenge, and
+ ptb-challenge-node-<uuid> for instances that process challenge submissions.Clone the GH repository of the ELK stack. +
git clone --depth 1 https://github.com/Sage-Bionetworks/docker-elk.git
+Configure the ELK stack.
+.env.++Note Create new passwords using the password generator that should come with your +favorite password manager.
+
Save the new passwords in Sage password manager.
+Start the ELK stack services using Docker Compose:
+docker compose up -d
+++Note +The stack can be stopped with
+docker compose stopand restarted withdocker compose start.
Kibana (the "K" in ELK) is a free and open user interface that lets you visualize your +Elasticsearch data and navigate the Elastic Stack.
+To access Kibana, start by forwarding the Kibana port (5601) to your localhost using AWS SSM.
+$ AWS_PROFILE=cnb aws ssm start-session --target <instance id> --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["5601"], "localPortNumber":["5601"]}'
+
+Starting session with SessionId: ...
+Port 5601 opened for sessionId ...
+Waiting for connections...
+++Note For developers of the OpenChallenges, VS Code is already forwarding the port 5601 to +localhost when running the project dev container. Either make sure that the dev container is not +running when accessing Kibana or forward the Kibana port to another port on localhost, e.g. with +
+--parameters '{"portNumber":["5601"], "localPortNumber":["56010"]}'(notice the trailing0).
Give Kibana about 2-3 minutes to initialize after having started it, then access the Kibana web UI
+by opening http://localhost:5601 in a web browser and use the following (default) credentials
+to log in:
elastic<elastic password>Congrats! You have successfully logged into Kibana. You should be welcomed with a dialog that +invites you to install integration components. You can skip this step.
+Metricbeat is a Elastic Beat agent that we will deploy on all the EC2 instances, including the +instance that runs the ELK stack, to monitor the following metrics: CPU usage, memory usage, +filesystem usage, and network usage. Metricbeat will send this information at the host and Docker +container levels.
+Install Metricbeat (select the DEB tab).
+ > Note The version of Metricbeat must match the version of the ELK stack. The version of the
+ > ELK stack is specified in the configuration file .env of the ELK stack.
Enable the Metricbeat docker module.
+
sudo metricbeat modules enable docker
+Check that the following modules are enable: system, docker.
+
sudo metricbeat modules list
+Specify the IP address and credentials that Metricbeat must use to connect to the EKL stack.
+/etc/metricbeat/metricbeat.yml.output.elasticsearch.
+ output.elasticsearch:
+ hosts: ["<elasticsearch ip>:9200"]
+ username: "elastic"
+ password: "<elastic password>"
+Update the metrics that the docker module should capture.
docker module configuration file /etc/metricbeat/modules.d/docker.yml.- module: docker
+ metricsets:
+ - container
+ - cpu
+ - diskio
+ - event
+ - healthcheck
+ - image
+ - info
+ - memory
+ - network
+ # - network_summary
+ period: 10s
+ hosts: ["unix:///var/run/docker.sock"]
+Enable Metricbeat to start at startup. +
sudo systemctl enable metricbeat
+Start Metricbeat. +
sudo systemctl start metricbeat
+Check that Metricbeat has successfully started. +
sudo systemctl status metricbeat
+sudo systemctl restart metricbeat.
+Metricbeat should now be sending data to the ELK Stack!
+Filebeat is a Elastic Beat agent that we will deploy on all the EC2 instances, including the +instance that runs the ELK stack, to monitor system log files.
+Install Filebeat (select the DEB tab).
+ > Note The version of Filebeat must match the version of the ELK stack. The version of the
+ > ELK stack is specified in the configuration file .env of the ELK stack.
Enable the Filebeat system module.
+
sudo filebeat modules enable system
+Check that the following modules are enable: system.
+
sudo filebeat modules list
+Specify the IP address and credentials that Filebeat must use to connect to the EKL stack.
+/etc/filebeat/filebeat.yml.Update the configuration for the section output.elasticsearch.
+
output.elasticsearch:
+ hosts: ["<elasticsearch ip>:9200"]
+ username: "elastic"
+ password: "<elastic password>"
+Update the metrics that the system module should capture.
system module configuration file /etc/filebeat/modules.d/system.yml.- module: system
+ syslog:
+ enabled: true
+Enable Filebeat to start at startup. +
sudo systemctl enable filebeat
+Start Filebeat. +
sudo systemctl start filebeat
+Check that Filebeat has successfully started. +
sudo systemctl status filebeat
+sudo systemctl restart filebeat.
+Filebeat should now be sending data to the ELK Stack!
+Beats such as Metricbeat and Filebeat comes with predefined assets for parsing, indexing, and +visualizing your data. To load these assets, run the following commands from the EC2 instance +that runs the ELK stack.
+sudo metricbeat setup -e --dashboards
+sudo filebeat setup -e --dashboards
+These commands may take 1-2 minutes to complete.
+After logging in Kibana, try to access the following sections to start exploring and monitoring the +data sent to the ELK stack.
+This page shows the compute resources used by all the hosts (EC2 instances) and Docker containers. +To open the Inventory,
+Observability > Metrics.Each host is represented by a tile in the Inventory. Click on one of the hosts to quickly access +information about CPU, memory and network usage. The same dialog gives access to the following +information:
+Logs: access the logs generated by the host (sent by Filebeat).Processes: see the list of processes running on the host.Metadata: see host metadata such as OS distrubution name and version, architecture, etc.
To display similar information for the running Docker containers, set the value of the menu Show
+to Docker Containers. The tiles shown now represent the Docker containers running across all the
+hosts. Use the search bar to filter hosts and containers. For example, enter host.hostname:
+ptb-challenge-gpu to show only the containers running on the host ptb-challenge-gpu.
This page shows the logs generated by all the hosts and sent by Filebeat to the ELK stack. To open +the Logs Stream,
+Observability > Logs.
Earlier, we initialized the ELK Stack with Beats assets, which include several dashboards that we +can use to monitor the fleet of hosts and containers. Here is a list of dashboards that are useful +for monitoring the challenge hosts and containers.
+To access the complete list of dashboards,
+Analytics > Dashboard.This dashboard provides an overview of system metrics such as CPU, memory, network and filesystem
+usage. By default, the dashboard displays information aggregated across all the hosts and Docker
+containers. A filter can be specified to display data for a single host. For example, by specifying
+the filter host.name:"<instance hostname>".
The Discover page enables us to quickly explore all the data indexed in the ELK stack.
+To access the Discover page,
+Analytics > Discover.For example, we want to visualize the evolution of the available filesystem of all the hosts. This +information is important when troubleshooting the crash of an host as it is possible that the crash +occurred because the filesystem was filled up to its capacity.
+On the Discovery page,
+metricbeat-*.Search field names input, enter system.filesystem.available.In the list of available fields, click on system.filesystem.available, then click on the
+ button Visualize.
In the right panel, set the following properties:
+@timestampMedian of system.filesystem.availablehost.nameGenerate a line plot by selecting Line after clicking on one of the menu button above the plot.
You should now be able to see a view similar to the one shown below.
+
From there, the following actions are available:
+Download as CSV.Save.aws module of Metricbeat to collect CloudWatch data, including billing
+ information.Create a file .coveralls.yml which at minimum should contain this one line:
repo_token: "coveralls-token-for-your-repo"
+Generate the coverage report for a project.
+nx test openchallenges-about
+Push a coverage report to Coveralls.
+npx coveralls < coverage/libs/openchallenges/about/lcov.info
+This doc describes how to create a new library + component in the OpenChallenges app, though +the steps can be applied to any app in this project. This doc will also include information on +where/how to copy-paste code from the [Figma-to-code export] into the app (starting at [Step 5]).
+To create a UI library within openchallenges, run:
nx g @nx/angular:lib <new library name> --directory openchallenges
+(Optional but recommended) Use --dryrun to first see what and where the entities will be created;
+this will help visualize and validate the intended directory structure, e.g.
$ nx g @nx/angular:lib awesome-lib --directory openchallenges --dry-run
+
+> NX Generating @nx/angular:library
+
+...
+UPDATE workspace.json
+CREATE libs/openchallenges/awesome-lib/README.md
+CREATE libs/openchallenges/awesome-lib/tsconfig.lib.json
+CREATE libs/openchallenges/awesome-lib/tsconfig.spec.json
+CREATE libs/openchallenges/awesome-lib/src/index.ts
+CREATE libs/openchallenges/awesome-lib/src/lib/openchallenges-awesome-lib.module.ts
+CREATE libs/openchallenges/awesome-lib/tsconfig.json
+CREATE libs/openchallenges/awesome-lib/project.json
+UPDATE tsconfig.base.json
+CREATE libs/openchallenges/awesome-lib/jest.config.ts
+CREATE libs/openchallenges/awesome-lib/src/test-setup.ts
+CREATE libs/openchallenges/awesome-lib/.eslintrc.json
+
+NOTE: The "dryRun" flag means no changes were made.
+Due to how the OpenChallenges app is currently structured, some additional +steps are required:
+Remove openchallenges- from the filename of the module TypeScript in src/lib/, e.g.
openchallenges-awesome-lib.module.ts → awesome-lib.module.ts
Simiarly, in src/index.ts, remove openchallenges- from the import filepath.
<new library name>.module.ts), remove ChallengeRegistry from
+ the class name, e.g.
+ export class ChallengeRegistryAwesomeLibModule {} → export class AwesomeLibModule {}++Note: still have questions about libraries? See [Libraries] for more details.
+
To create the component, use:
+nx g @nx/angular:component <new component name> --project <project-name>
+where <project name> is the name of the project defined in project.json of the newly-created
+Angular library.
For example, to create an Angular component for the awesome-lib library, the project name would be
+openchallenges-awesome-lib, as defined in libs/openchallenges/awesome-lib/project.json:
{
+ "name": "openchallenges-awesome-lib",
+ "$schema": "../../../node_modules/nx/schemas/project-schema.json",
+ "projectType": "library",
+ ...
+}
+The resulting command would then be:
+$ nx g @nx/angular:component awesome-lib --project openchallenges-awesome-lib --dry-run
+
+> NX Generating @nx/angular:component
+
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib/awesome-lib.component.scss
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib/awesome-lib.component.html
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib/awesome-lib.component.spec.ts
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib/awesome-lib.component.ts
+UPDATE libs/openchallenges/awesome-lib/src/lib/awesome-lib.module.ts
+
+NOTE: The "dryRun" flag means no changes were made.
+++Note: notice that the command above is using
+dry-run. Again, this is just to ensure +that the new entities will be created in the right folder. If everything looks correct, +remove the flag to actually create the new component.
To directly create the files into the parent folder, use --flat in the command:
$ nx g @nx/angular:component awesome-lib --project openchallenges-awesome-lib --flat -dry-run
+
+> NX Generating @nx/angular:component
+
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib.component.scss
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib.component.html
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib.component.spec.ts
+CREATE libs/openchallenges/awesome-lib/src/lib/awesome-lib.component.ts
+UPDATE libs/openchallenges/awesome-lib/src/lib/awesome-lib.module.ts
+
+NOTE: The "dryRun" flag means no changes were made.
+Notice how the component files would be created directly in awesome-lib/src/lib/, compared
+with awesome-lib/src/lib/awesome-lib/ when --flat is not used.
Before moving on, some additional edits are required:
+<new component name>.component.spec.ts file -- it is not needed.In the <new component name>.component.ts file:
OnInit from the import and all instances of it from the classConfigService from @sagebionetworks/openchallenges/configconfigService into the constructor as private readonlyThe final result should look something like this:
+import { Component } from '@angular/core';
+import { ConfigService } from '@sagebionetworks/openchallenges/config';
+
+@Component({
+ selector: 'sagebionetworks-awesome-lib',
+ templateUrl: './awesome-lib.component.html',
+ styleUrls: ['./awesome-lib.component.scss'],
+})
+export class AwesomeLibComponent {
+ constructor(private readonly configService: ConfigService) {}
+}
+Revisit the library module (<library name>.module.ts) and import the UI and
+ routing modules:
```ts
+...
+import { RouterModule, Routes } from '@angular/router';
+
+const routes: Routes = [{ path: '', component: <component> }];
+
+@NgModule({
+ imports: [CommonModule, RouterModule.forChild(routes)],
+ ...
+```
+
+where `<component>` is the newly-created Angular component. You will also need
+to export the newly-created Angular component, e.g.
+
+```ts
+@NgModule({
+ ...
+ exports: [AwesomeLibComponent],
+})
+```
+
+The final result should look something like this:
+
+ ```ts
+ import { NgModule } from '@angular/core';
+ import { CommonModule } from '@angular/common';
+ import { RouterModule, Routes } from '@angular/router';
+ import { AwesomeLibComponent } from './awesome-lib.component';
+
+ const routes: Routes = [{ path: '', component: AwesomeLibComponent }];
+
+ @NgModule({
+ imports: [CommonModule, RouterModule.forChild(routes)],
+ declarations: [AwesomeLibComponent],
+ exports: [AwesomeLibComponent],
+ })
+ export class AwesomeLibModule {}
+ ```
+In apps/openchallenges/src/app/app-routing.module.ts, add a router for the new component, e.g.
{
+ path: <new path name>,
+ loadChildren: () =>
+ import('<index>').then(
+ (m) => m.<module>
+ ),
+ },
+where <index> is the path defined tsconfig.base.json. For example, the base for AwesomeLib is:
{
+ ...
+ "paths": {
+ "@sagebionetworks/openchallenges/awesome-lib": [
+ "libs/openchallenges/awesome-lib/src/index.ts"
+ ],
+ ...
+ }
+}
+So, the resulting router would look something like this:
+ {
+ path: 'awesome-path',
+ loadChildren: () =>
+ import('@sagebionetworks/openchallenges/awesome-lib').then(
+ (m) => m.AwesomeLibModule
+ ),
+ },
+++⚠️ IMPORTANT: we follow [Angular's standards for routing order], that is:
+List routes with a static path first, followed by an empty path route, which matches the default route. The wildcard route comes last because it matches every URL and the Router selects it only if no other routes match first.
+Please adhere to this recommended order when adding a new route.
+
If you haven't already, start a local server to test the newly-created component:
+nx serve openchallenges
+If everything is setup correctly, http://localhost:4200/<new path name> will open a web page that
+displays something like this:
<component name> works!
+If the Figma-to-code export was downloaded as Angular, the directory structure should be comparative +to our current app structure, that is:
+<project name>-angular
+├── angular.json
+├── browserslist
+├── package.json
+├── src
+│ ├── app
+│ │ ├── app.component.css
+│ │ ├── app.component.html
+│ │ ├── app.component.ts
+│ │ ├── app.module.ts
+│ │ ├── components
+│ │ │ └── components.module.ts
+│ │ └── pages
+│ │ ├── *
+│ │ │ ├── *.component.css
+│ │ │ ├── *.component.html
+│ │ │ ├── *.component.ts
+│ │ │ └── *.module.ts
+│ │ └── **
+│ │ ├── **.component.css
+│ │ ├── **.component.html
+│ │ ├── **.component.ts
+│ │ └── **.module.ts
+│ ├── environments
+│ │ ├── environment.prod.ts
+│ │ └── environment.ts
+│ ├── index.html
+│ ├── main.ts
+│ ├── polyfills.ts
+│ └── styles.css
+├── tsconfig.app.json
+├── tsconfig.json
+└── tslint.json
+Locate the HTML and CSS files within src/app/pages/, then copy-paste the code as needed.
+Alternatively, you can move the files into the app, then re-direct the paths in *.component.ts
+so that it uses these files:
@Component({
+ selector: 'openchallenges-team',
+ templateUrl: './<new HTML file>',
+ styleUrls: ['./<new CSS/SCSS file>'],
+})
+If the Figma-to-code export was downloaded as HTML, the directory structure will be very simple:
+<project name>-html
+├── *.css
+├── *.html
+├── **.css
+├── **.html
+├── package.json
+└── style.css
+Copy-paste the HTML and CSS as needed, or move the files into the app then re-direct the paths
+in *.component.ts.
Once the exported styles files are copy-pasted to the app, some additional steps are required to configurate the themes of exported pages. Take awesome-lib as an example:
+├── awesome-lib
+│ ├── src
+│ │ ├── lib
+│ │ │ ├── awesome-lib.component.css
+│ │ │ ├── awesome-lib.component.html
+│ │ │ ├── awesome-lib.component.ts
+│ │ │ ├── awesome-lib.module.ts
+│ │ │ └── sub-awesome-lib
+│ │ │ ├── sub-awesome-lib.component.css
+│ │ │ ├── sub-awesome-lib.component.html
+│ │ │ ├── sub-awesome-lib.component.ts
+│ │ │ └── sub-awesome-lib.module.ts
+...
+_<component-name>-theme.scss in each component, i.e. libs/openchallenges/awesome-lib/_awesome-lib-theme.scss and libs/openchallenges/awesome-lib/src/lib/sub-awesome-lib/_sub-awesome-lib-theme.css (repeat the step for each sub-component if any). Copy-paste below essential codes into all _<component-name>-theme.scss files:@use 'sass:map';
+@use '@angular/material' as mat;
+
+@mixin color($theme) {
+ $config: mat.get-color-config($theme);
+ $primary: map.get($config, 'primary');
+ $accent: map.get($config, 'accent');
+ $warn: map.get($config, 'warn');
+ $figma: map.get($config, 'figma');
+}
+
+@mixin typography($theme) {
+}
+
+@mixin theme($theme) {
+ $color-config: mat.get-color-config($theme);
+ @if $color-config != null {
+ @include color($theme);
+ }
+
+ $typography-config: mat.get-typography-config($theme);
+ @if $typography-config != null {
+ @include typography($theme);
+ }
+}
+awesome-lib/src/_lib_themes.scss and import all the themes of components:@use './lib/awesome-lib-theme' as awesome-lib;
+@use './lib/sub-awesome-lib/sub-awesome-lib-theme' as sub-awesome-lib;
+@mixin theme($theme) {
+ @include awesome-lib.theme($theme);
+ @include sub-awesome-lib.theme($theme);
+}
+libs/openchallenges/themes/src/_index.scss with below snippet:@use 'libs/openchallenges/awesome-lib/src/lib-theme' as openchallenges-awesome-lib;
+@mixin theme($theme) {
+ @include openchallenges-awesome-lib.theme($theme);
+}
+At this point, all theme files have been generated:
+├── awesome-lib
+│ ├── src
+│ │ ├── _lib-theme.scss
+│ │ ├── lib
+│ │ │ ├── _awesome-lib-theme.scss
+│ │ │ ├── awesome-lib.component.css
+│ │ │ ├── awesome-lib.component.html
+│ │ │ ├── awesome-lib.component.ts
+│ │ │ ├── awesome-lib.module.ts
+│ │ │ └── sub-awesome-lib
+│ │ │ ├── _sub-awesome-lib-theme.css
+│ │ │ ├── sub-awesome-lib.component.css
+│ │ │ ├── sub-awesome-lib.component.html
+│ │ │ ├── sub-awesome-lib.component.ts
+│ │ │ └── sub-awesome-lib.module.ts
+...
+In each component including sub-components, move all the colors and fonts from <component-name>.component.scss to _<component-name>-themes.scss.
Now, the themes of new exported page has been configurated. However, some manually fine-tunes will still require to make page look like what figma's design. It's recommended to adjust the styles with the app's local server
+nx serve openchallenges
+Most palettes used in the figma has been already configured in the app and defined in libs/themes/src/_palettes.scss. The palettes can be always retrieved via map.get(<theme-object-name>, <color-variable-name>), i.e. map.get($figma, dl-color-default-hover1). It's similar with the constant variables defined in libs/styles/src/lib/_constants.scss, i.e. border-radius: constants.$dl-radius-radius-radius16;
++ + + + + + + + + + + + + +Note
+
All color/font styles needs to be defined in_<component-name>-theme.scss.
This document describes how to configure and develop in the dev container specified by this +repository. The goal is to provide developers with the same development environment that is mainly +isolated from their host. This method promotes reproducibility and remove the need to install +dependencies on the host such as Node.js or Python. For additional information on VS Code +development containers, see Developing inside a Container.
+Open the workspace folder in VS Code after cloning it. VS Code should invite you to open the folder +in the dev container specified by this repository. Alternatively:
+F1 or Command+Shift+P (macOS) or
+ Ctrl+Shift+P (Windows) or click on Help > Show All CommandsRemote-Containers: Open Folder in Container...That's it! VS Code should now have open the workspace folder in the dev container as indicated by +the following green button displayed in the bottom-left corner of VS Code.
+
To reopen the workspace folder locally, click on the above green button and select Reopen Folder
+Locally or Reopen Folder in WSL when developing on Windows with WSL.
To run git commands that require authentication inside the dev container, please follow the
+instructions described in Sharing Git credentials with your container.
If this repository has been closed using an SSH key, follow the instructions to share your host SSH +keys with the dev container. The command below should list the same keys when executed on the host +and inside the container.
+ssh-add -l
+Signing commits requires a GPG key. Follow the above instructions to share the host GPG keys with +the dev container. The command below should now list the same keys when executed on the host and +inside the container.
+gpg --list-keys
+Team members who develop locally may not benefit from the same compute resources. The most notable +resources that can impact the productivity of developers are the number and frequency of the CPU +cores, the memory available and internet speed. The worse case is when a machine does not have the +resources to run the apps that the team develops, for example when not enough memory is available. +On other times, the time required to complete a task may be many times slower on a computer with +lower CPU resources.
+Working remotely means that developers no longer benefit from the same internet speed, either +because of the quality of the internet connection available at their location or because the speed +is shared among the members of a household. As a result, tasks that involve downloading or uploading +artifacts, like pulling or pushing Docker images, may take significantly longer to complete.
+This page describes how to setup a development environment that enables developers to use VS Code +while using the compute resources of a remote host. The developers start by creating identical EC2 +instances before connecting to them with VS +Code. This SOP enables +developers to continue working inside the devcontainer provided with this project, +hence further contributing to the standardization of the development envrionment.
+++Note 2023-01-28: Added documentation to connect to a GitHub Codespace.
+
This table summarizes the local compute resources available to the developers of the challenge +registry. The same information is displayed for two types of Amazon EC2 instances and one type of +GitHub Codespace instance that were selected as candidate alternative development environments for +the team members. The table also includes the runtimes in seconds of different tasks such as linting +or testing all the projects included in the monorepo (the method used to generate these results is +described in the next section).
+| + | Shirou | +Rin | +Sakura | +m5.2xlarge | +t3a.xlarge | +4-core Codespace | +8-core Codespace | +
|---|---|---|---|---|---|---|---|
| Computer Type | +Desktop PC | +MacBook Pro | +MacBook Pro | +Amazon EC2 | +Amazon EC2 | +GitHub Codespace | +GitHub Codespace | +
| Architecture | +64-bit (x86) | +64-bit (x86) | +64-bit (x86) | +64-bit (x86) | +64-bit (x86) | +64-bit (x86) | +64-bit (x86) | +
| CPU Count | +8 | +4 | +4 | +8 | +4 | +4 | +8 | +
| CPU Frequency (GHz) | +3.6 | +2.4 | +1.7 | +2.5 | +2.2 | +2.7 | +2.8 | +
| Memory (GB) | +32 | +16 | +16 | +32 | +16 | +8 | +16 | +
| Runtime: Lint All Projects (s) | +15.4 | +208.9 | +183.8 | +18.6 | +33.4 | +24.6 | +16.9 | +
| Runtime: Build All Projects (s) | +19.4 | +196.2 | +162.2 | +26.7 | +44.9 | +32.3 | +14.1 | +
| Runtime: Test All Projects (s) | +12.4 | +117.1 | +82.8 | +15.3 | +29.2 | +31.6 | +24.5 | +
| Runtime: Test api (s) | +6.2 | +29.6 | +21.3 | +7.2 | +10.4 | +6.5 | +6.5 | +
| Runtime: Test web-app (s) | +5.3 | +43.0 | +35.0 | +6.5 | +9.2 | +6.7 | +6.0 | +
| Download speed (Mbit/s) | +395.9 | +52.1 | +160.1 | +2165.0 | +1606.7 | +8571 | +8603 | +
| Upload speed (Mbit/s) | +183.3 | +15.6 | +10.3 | +1861.0 | +1030.2 | +4893 | +5125 | +
| On-Demand Cost ($/day) | +n/a | +n/a | +n/a | +9.2 | +3.6 | +8.64 (1,2) | +17.28 (1,2) | +
| On-Demand Cost ($/year) | +n/a | +n/a | +n/a | +3363.8 | +1317.5 | +3153.6 (1,2) | +6307.2 (1,2) | +
(1) GitHub codespaces stop automatically after 1h of inactivity. A codespace used by an engineer +with 100 %FTE and 8 working hours per day - without taking into account vacation for the sake of +simplicity - would cost 8 hours/day * 5 days/week * 52 weeks * $0.36/hour (4-core) = $748/year (see +Codespaces pricing). Similarly, the cost for an 8-core codespace would become $1496/year. In +addition, GitHub bills $0.07 of GB of storage.
+(2) GitHub offers core hours and storage. For example, a Free user can use a 2-core instance for 60 +hours per month for free or an 8-core instance for 15 hours. You will be notified by email when you +have used 75%, 90%, and 100% of your included quotas. + - Free users: 120 core hours/month and 15 GB month of storage + - Pro users: 180 core hours/month and 20 GB month of storage
+Note that developers have been asked to measure runtimes and internet speeds while keeping open the +applications that are usually running when they develop (e.g. Spotify, several instances of VS Code, +browser with many tabs open). This could be one reason why runtimes reported by a developer are +larger that those reported by another developer who has less compute resources available.
+The table below shows the number of times a task is faster than the slowest runtime (denoted by +"1.0").
+| + | Shirou | +Rin | +Sakura | +m5.2xlarge | +t3a.xlarge | +
|---|---|---|---|---|---|
| Runtime: Lint All Projects | +13.6 | +1.0 | +1.1 | +11.2 | +6.3 | +
| Runtime: Build All Projects | +10.1 | +1.0 | +1.2 | +7.3 | +4.4 | +
| Runtime: Test All Projects | +9.4 | +1.0 | +1.4 | +7.6 | +4.0 | +
| Runtime: Test api | +4.8 | +1.0 | +1.4 | +4.1 | +2.8 | +
| Runtime: Test web-app | +8.0 | +1.0 | +1.2 | +6.6 | +4.6 | +
| Download speed | +7.6 | +1.0 | +3.1 | +41.5 | +30.8 | +
| Upload speed | +17.8 | +1.5 | +1.0 | +180.5 | +99.9 | +
For example, linting all the projects of this monorepo is 13.6 times faster on Shirou's computer +than on Rin's. Moreover, all the developers can benefit from improved download speeds (up to 41.5 +faster for Rin) and upload speeds (up to 180.5 times faster for Sakura) when developing on an EC2 +instance. This table illustrates well the diversity in compute resources available locally to +developers, and how relying on remote hosts like EC2 instances can provide a better working +environment to developers.
+$ nproc
+$ cat /proc/cpuinfo
+$ cat /proc/meminfo
+$ hyperfine --warmup 1 --runs 10 'nx run-many --all --target=lint --skip-nx-cache'
+$ hyperfine --warmup 1 --runs 10 'nx run-many --all --target=build --skip-nx-cache'
+$ hyperfine --warmup 1 --runs 10 'nx run-many --all --target=test --skip-nx-cache'
+$ hyperfine --warmup 1 --runs 10 'nx test api --skip-nx-cache'
+$ hyperfine --warmup 1 --runs 10 'nx test web-ui --skip-nx-cache'
+$ speedtest
+This section describes how to instantiate an AWS EC2 as the remote host. Steps outlined below will +assume you have access to the Sage AWS Service Catalog.
+<GitHub username>-devcontainerst3.2xlargeAmazonLinuxDocker (leave default)Department: IBC or CNB (selected from this
+ list)Project: challenge (selected from this
+ list)CostCenter: NIH-ITCR / 101600 (selected from these
+ lists)++Note:¶
+If this is your first time ever connecting to an instance from your machine, you will first +need to set up EC2 access with the AWS Systems Manager (SSM). Follow the instructions below to +complete the setup: + - Create a Synapse personal access + token + - SSM access to an + Instance
+(Don't worry, you will only need to do this once for your local machine!)
+
EC2InstancePrivateIpAddress~/.ssh/config:
+ Host devcontainers
+ HostName <private_ip>
+ User ec2-user
+ IdentityFile ~/.ssh/id_rsa
+~/.ssh/config was setup correctly.
+ ssh devcontainers
+sudo yum update -y
+docker --version
+Remote - SSH and Remote - Containers.Remote-SSH: Connect to Host... > Select the host.SSH: <host name> upon successfully
+ connecting to the remote instance.
Remote-Containers: Open Folder in Container...OK.Dev Container: OpenChallenges @
+ ssh://<host name>.
Congratulations, you are now ready to develop in the devcontainer that runs on the EC2 instance! 🚀
+Repository: Select your fork of the monorepoBranch: Select the default branchDev container configuration: Select the dev container definitionRegion: Select your preferred regionMachine type: Select the machine type
+ > Note 4-core is preferred for the OpenChallenges project as a trade-off between
+ > performance and cost.If your codespace is open in your browser, you can stop it with the following step. Note that a +codespace stops automatically after one hour of inactivity.
+If you prefer to develop with VS Code rather than inside your browser:
+The type of machine used by a codespace can be changed at any time, for example when a beefier +codespace instance is needed. To change the machine type of an existing codespace.
+The devcontainer provided with this project uses the VS Code devcontainer feature
+docker-in-docker. In addition to isolating the Docker engine running in the devcontainer from the
+engine running on the host, this feature enables VS Code to forward the ports defined in
+devcontainer.json to the local envrionment of the developer. Therefore, apps and services can be
+accessed using the address http://localhost even though they are running on the remote host!
Accessing the apps and services using the IP address of the remote host won't work, unless you
+replace the feature docker-in-docker by docker-from-docker. In this case, http://localhost can
+no longer be used to access the apps and services.
Simply drag and drop files to the VS Code explorer to upload files from your local environment to +the remote host.
+Click on the button in the bottom-left corner of VS Code and select one of these options:
+Close Remote Connection to close the connection with the remote host.Reopen Folder in SSH if you want to stop the devcontainer but stay connected to the remote host.The devcontainer used to contribute to this monorepo comes with one Python environment. The version +of Python provided by this environment is as of May 19, 2023:
+$ python --version
+Python 3.9.2
+By default, creating a new Python project in this monorepo with Poetry will use the Python version +provided by the devcontainer. However, for various reasons, this Python version may not be +compatible with the Python requirement of a given project.
+Let's take the project schematic-api as an example. The dependencies of this project are managed
+with Poetry. The project is currently configured to use Python 3.9.2. However, we now need to update
+the version of Python used by this project to 3.11.0 to solve a recent issue.
First, we need to move to the project folder:
+mv apps/schematic/api
+Install the Python env 3.11.0 with pyenv:
pyenv install 3.11.0
+Activate the new environment:
+pyenv local 3.11.0
+Check that the new env is active:
+$ pyenv versions
+ system
+* 3.11.0 (set by /workspaces/sage-monorepo/apps/schematic/api/.python-version)
+Update the Python version specified in the project files pyproject.toml:
poetry env use 3.11.0
+Update the Python version specified in the lock file:
+poetry lock --no-update
+Build the project with the new Python env:
+nx build schematic-api
+New Project.openchallenges-keycloak-googlesagebase.orgsagebase.orgCreate.Credentials in the left menu.Configure Consent Screen.Internal (suitable for testing)Create.Challenge Keycloak Test App<support email>http://localhost:4200 (OpenChallenges web app)openchallenges.org (Optional)<contact addresses>Save and Continue.Add or Remove Scopes..../auth/userinfo.email.../auth/userinfo.profileopenidUpdate.Save and Continue.Back to Dashboard.Create Credentials.OAuth client ID.Web applicationChallenge Keycloak Test Clienthttp://localhost:8080/realms/test/broker/google/endpointCreate.Download JSON to download the Client ID and Client Secret.OK.Identify ProvidersGoogleRedirect URI value which will be used in the next steps.Add Google provider.Add.Log In.Google.The solution is to temporarily disable the git config commit.gpgsign.
git config --global commit.gpgsign false
+yarn create nx-workspace openchallenges --preset=empty --packageManager=yarn
+git config --global commit.gpgsign true
+api:lint fails¶✖ nx run api:lint
+ Loading .env environment variables...
+ Error: the command black could not be found within PATH or Pipfile's [scripts].
+ ERROR: Something went wrong in @nrwl/run-commands - Command failed: pipenv run black ./openapi_server --check --exclude '(models|test)'
+Run nx python api to create the Python virtualenv and install the tools needed.
This error likely happens because the root filesystem is full. Verify whether that is the case by +running the following command on the host (not from within the dev container).
+$ df -h
+Filesystem Size Used Avail Use% Mounted on
+devtmpfs 7.8G 0 7.8G 0% /dev
+tmpfs 7.8G 0 7.8G 0% /dev/shm
+tmpfs 7.8G 692K 7.8G 1% /run
+tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
+/dev/nvme0n1p1 50G 50G 679M 99% /
+tmpfs 1.6G 0 1.6G 0% /run/user/1000
+The root filesystem (/) is indeed full. The command below lists the folders starting from the +largest one. Make sure to run this command as root, otherwise only the folders visible to the user +will be taken into account.
+[ec2-user@ip-10-41-30-136 ~]$ sudo du -aBM / 2>/dev/null | sort -nr | head -n 50 | more
+53903M /
+41411M /var
+40578M /var/lib
+40455M /var/lib/docker
+30810M /var/lib/docker/overlay2
+10171M /home/ec2-user
+10171M /home
+9609M /var/lib/docker/volumes
+8585M /var/lib/docker/volumes/dind-var-lib-docker/_data
+8585M /var/lib/docker/volumes/dind-var-lib-docker
+7976M /var/lib/docker/volumes/dind-var-lib-docker/_data/overlay2
+5184M /var/lib/docker/overlay2/a4675f542ab8ee863953360af4be4d3ab46fa9743bbb4a0ec56a56a887c8ef52
+...
+Regularly building Docker images filled up the root filesystem. Consider performing the following +actions to free up disk space.
+Remove unused images:
+docker container prune --force && docker rmi $(docker images -aq) --force
+++Note When using VS Code dev container with the feature
+docker-in-docker, removing the images +on the host won't remove the images in the dev container. If that is the case, you can also run +the above command inside the dev container.
Prune Docker system:
+docker system prune
+Prune Docker volumes:
+# remove selected volume
+docker volume ls
+docker volume rm <volume name>
+
+# remove all volumes
+docker volume prune
+++ + + + + + + + + + + + + +Note The volume
+dind-var-lib-dockeris created by the dev container feature +docker-in-docker. The volumevscodeis created when starting a dev container with VS Code. +These two volumes should not be removed if a dev container is running.
Often times, it can be quite time-consuming to go from design concepts to code. With the Challenge +Registry, we hope to alleviate this hurdle by utilizing TeleportHQ's Figma-to-code capability. +TeleportHQ is an online collaborative platform meant for front-end development, in which a static +website can either be built from scratch, or, in our case, from pre-defined components and frames +created in Figma.
+++Note Paid plans are provided for both tools at Sage upon request. Please submit a Jira ticket +to Sage IT to request a Figma and/or TeleportHQ license (tag Jake Albrecht for approval). +Otherwise, you are allowed up to 3 Figma files and 3 TeleportHQ projects when using the basic +(free).
+Note You will need Edit access to a Figma document in order to use TeleportHQ plugin (see +below). If you don't have Edit access to a Figma document that you want to export, make a copy of +the Figma document first, then export the copy to TeleportHQ.
+
This guide should work with both the web and desktop Figma app.
+Download the TeleportHQ plug-in in Figma (this step is only needed once).
+Create components and frames in Figma as needed. If a component or frame uses any color and/or +text styles, these will also be included in the export. Learn more about creating color and text +styles here.
+When ready, select a frame or component to include in the export. This is easiest to do from the
+Layers panel. The export can include multiple components and frames -- hold the cmd key on a Mac
+or shift key on a Windows to select your desired layers.
Right-click on the selected layer(s) from the Layers panel > Plugins > TeleportHQ - Figma to + Code.
+A pop-up window will be displayed, where you can review the layers, styles, and config that will +be included in the export. Used styles will be highlighted in blue.
+If you already have a TeleportHQ project defined, you can select Copy Layers to Clipboard to +manually copy over the layers to the existing project. Otherwise, click on +Export as a new project. Depending on how many layers have been selected, this can take +anywhere from a couple of seconds to 1-2 minutes; you will then be taken to the TeleportHQ platform.
+++Note This section is inspired by TeleportHQ's docs. Some parts are copied over verbatim.
+
When you enter a project in TeleportHQ, your workspace will consist of three panels.
+By default, the left panel will display Layers & Files. This is further divided into two +sections, where the top half will display the pages and components of the project, and the bottom +half will display the layers of the selected file/component.
+Other options for the left panel include: +* Elements: the basic building blocks with which you can start building your user interfaces. +Most of them correspond to native HTML elements, e.g. Row, Column. You can drag and drop an element +from this panel into the Canvas, or use their keyboard shortcuts. See their lesson on Elements to +learn more. +* CSS Classes: sets of styles that can be reused on multiple elements. TeleportHQ will include +some by default. You can create, search, or edit classes as needed. See their lesson on +CSS Classes to learn more. +* Asset manager: allows you to upload your own assets, access directly tens of thousands of +images from the Unsplash, and choose icons from several popular libraries.
+This is your Canvas, aka where all of the visual building ✨ magic ✨ happens.
+When an element is not selected, the right panel will display the Design Language Panel. This is +where you can explore and define style tokens used throughout the project. TeleportHQ includes some +styles by default, e.g. "Primary" for colors. If your Figma export included color palettes, they +will be listed under "Default" in the Color tab. Similarly, if your Figma export included text +styles, they will be listed individually in the Text tab.
+When an element is selected, the right panel will change to the Style Inspector Panel, where +you can add styles to the elements and/or edit the.
+++Note All tokens defined in the Design Language Panel will be included in the final code +export. Feel free to add/modify/remove tokens as needed, so that the code export only has the +definitions you need.
+
Before exporting to code, we recommend reviewing the Figma-exported elements first, as the +Figma-to-TeleportHQ export is not always perfect.
+For example, let's say you created a set of layers in Figma that is intended to be a button, that
+is, the layer group includes a shape, a color fill, and some text. When this gets exported to
+TeleportHQ, it becomes as a group of layers that are an image (the color fill) and a text block.
+This subsequently gets exported as <img> and <span> in the code, which may not be what you
+intended.
Instead, remove the image layer, then apply a CSS class to the parent group. If you are using one +of TeleportHQ's pre-defined CSS classes, you can edit it from the CSS Classes panel. You can +alternatively use TeleportHQ's Button element from their Elements panel, and replace the +"button" layers with it.
+If there is time or if you are not familiar with editing HTML & CSS, we recommend applying a layout +to the Figma-exported project, e.g. applying the Row and Column elements. Additionally, if +responsive design is important in the final product, we recommend using relative sizing and media +queries. See their lesson on Responsive design to learn more.
+Otherwise, you are more than welcome to add/edit the layout and responsiveness in the exported code.
+In the top-right corner, select Export Project (the icon next to the Publish button).
+Select Angular or HTML as the library, whichever you are more comfortable with. Note +that exporting as Angular will download the files as a standalone app!
+Click on the file icon at the bottom of the dropdown to download the project as a zip.
+After creating a new library and component within the app, copy-paste code from the exported project +into the app. See create-a-new-angular-component for more details on how to create the library and +component, and where to copy-paste the code.
+ + + + + + + + + + + + + +Option 1: Click on the button shown below to open Sage Monorepo in its development container +with VS Code. This option is suitable if you want to explore the content of Sage Monorepo.
+Option 2: If you plan to contribute to this project, please create a fork and use its URL for +cloning. For more information on contributing and/or our Forking Workflow, see +CONTRIBUTING.md.
+Then open your fork repo inside our dev container using these instructions:
+Love Sage Monorepo and its projects? Give our repo a star ⭐.
+ + + + + + + + + + + + + + +
./gradlew bootRunYou can force Gradle to execute all tasks ignoring up-to-date checks using the --rerun-tasks
+option:
./gradlew test --rerun-tasks
+To disable the progress logging we can set the option --console=plain the environment variable
+TERM to the value dumb. To make it a default behavior, add org.gradle.console=plain to
+gradle.properties file.
./gradlew test --console=plain
+TERM=dumb ./gradlew test
+This table lists the repositories used to develop the individual components of the Challenge +Registry. Since XXX, the codebase of these components is maintained in this monorepo.
+| Repository | +Monorepo Project | +
|---|---|
| Sage-Bionetworks/rocc-app | +apps/web-app | +
| Sage-Bionetworks/rocc-service | +apps/api | +
| Sage-Bionetworks/rocc-client-angular | +libs/api-client-angular | +
| Sage-Bionetworks/rocc-client-r | +libs/api-r | +
| Sage-Bionetworks/rocc-schemas | +libs/openchallenges/api-description | +
This document explains how to create Angular and Web Components libraries.
+nx g @nx/angular:lib awesome-lib [--dry-run]
+The library is added to tsconfig.base.json, which will make it available to other projects.
Creation of the UI library for the web-app in libs/web/ui.
nx g @nx/angular:lib ui --directory web
+nx g @nx/angular:component footer --project=web-ui
+nx g @nx/web:lib ui-footer [--dry-run]
+The library is added to tsconfig.base.json, which will make it available to other projects.
no-undef¶ESLint is expected to show errors like the ones shown below. The reason is because it does not see
+their definition, which are provided by the DOM. The solution is to disable the ESLint rule
+no-undef in the .eslintrc.json of the library. TypeScript already provides this check.
'HTMLElement' is not defined (eslint `no-undef`)
+'customElements' is not defined (eslint `no-undef`)
+We need need to register CUSTOM_ELEMENTS_SCHEMA in our app or component module. this will tell the
+Angular compiler to refrain from throwing errors when seeing non-standard element tags in our
+templates.
import { CUSTOM_ELEMENTS_SCHEMA, NgModule } from '@angular/core';
+import { BrowserModule } from '@angular/platform-browser';
+
+import { AppComponent } from './app.component';
+import { NxWelcomeComponent } from './nx-welcome.component';
+import { RouterModule } from '@angular/router';
+
+@NgModule({
+ declarations: [AppComponent, NxWelcomeComponent],
+ imports: [BrowserModule, RouterModule.forRoot({ initialNavigation: 'enabledBlocking' })],
+ providers: [],
+ schemas: [CUSTOM_ELEMENTS_SCHEMA],
+ bootstrap: [AppComponent],
+})
+export class AppModule {}
+Import the UI library in an Angular component.
+import '@sagebionetworks/ui-footer';
+The component ui-welcome provided by the library can now be used in the HTML file of our Angular
+component.
<ui-welcome title="Awesome App"></ui-welcome>
+We need to tell React to allow components that are not defined within it. To allow this, we will
+need to create a type file on the root of the src folder or the React project. Name it
+intrinsic.d.ts and add inside the following:
declare namespace JSX {
+ interface IntrinsicElements {
+ [elemName: string]: any;
+ }
+}
+Use the UI component ui-welcome in React.
export function App() {
+ return (
+ <>
+ <ui-welcome title="web-app-react" />
+ </>
+ );
+}
+
+export default App;
+| File type | +File extension | +Linter | +
|---|---|---|
| Dockerfile | +Dockerfile |
+hadolint | +
| HTML | +*.html |
+Webhint | +
| Java | +*.java |
+Checkstyle | +
| SCSS | +*.scss |
+VS Code (SCSS) | +
| TypeScript | +*.ts |
+ESLint | +
| XML | +*.xml |
+- | +
Linter:
+Linter configuration:
+.hintrc| File type | +File extension | +Formatter | +Package type | +
|---|---|---|---|
| Dockerfile | +Dockerfile |
+- | ++ |
| HTML | +*.html |
+Prettier | ++ |
| Java | +*.java |
+google-java-format | +npm | +
| SCSS | +*.scss |
+VS Code (SCSS) | ++ |
| TypeScript | +*.ts |
+ESLint | ++ |
| XML | +*.xml |
+Prettier | ++ |
Formatter:
+Formatter configuration:
+.prettierrc.prettierignorenpmpackage.json${workspaceRoot}/node_modules/.bin/google-java-format.vscode/settings.json*.java.vscode/settings.json)format for the affected projects (since last commit) identified by Nx.format of Java projects runs @nxrocks/nx-spring-boot:format.build.gradle).*.ts.eslintrc.json. The ESLint project config file references the ESLint workspace config file..prettierrc are read from ESLint workspace
+ config file .eslintrc.json.Notes:
+"editor.formatOnSave": false. To turn off the auto formatting for this
+ extension:
+ "editor.codeActionsOnSave": {
+ // "source.fixAll": false,
+ "source.fixAll.eslint": false
+},
+This project was generated using Nx.
+
🔎 Smart, Fast and Extensible Build System
+Nx supports many plugins which add capabilities for developing different types of applications and +different tools.
+These capabilities include generating applications, libraries, etc as well as the devtools to test, +and build projects as well.
+Below are our core plugins:
+npm install --save-dev @nrwl/reactnpm install --save-dev @nx/webnpm install --save-dev @nx/angularnpm install --save-dev @nrwl/nestnpm install --save-dev @nrwl/expressnpm install --save-dev @nx/nodeThere are also many community plugins you could add.
+Run nx g @nrwl/react:app my-app to generate an application.
++You can use any of the plugins above to generate applications as well.
+
When using Nx, you can create multiple applications and libraries in the same workspace.
+Run nx g @nrwl/react:lib my-lib to generate a library.
++You can also use any of the plugins above to generate libraries as well.
+
Libraries are shareable across libraries and applications. They can be imported from
+@sagebionetworks/mylib.
Run nx serve my-app for a dev server. Navigate to http://localhost:4200/. The app will
+automatically reload if you change any of the source files.
Run nx g @nrwl/react:component my-component --project=my-app to generate a new component.
Run nx build my-app to build the project. The build artifacts will be stored in the dist/
+directory. Use the --prod flag for a production build.
Run nx test my-app to execute the unit tests via Jest.
Run nx affected:test to execute the unit tests affected by a change.
Run nx e2e my-app to execute the end-to-end tests via Cypress.
Run nx affected:e2e to execute the end-to-end tests affected by a change.
Run nx graph to see a diagram of the dependencies of your projects.
Visit the Nx Documentation to learn more.
+
Nx Cloud pairs with Nx in order to enable you to build and test code more rapidly, by up to 10 +times. Even teams that are new to Nx can connect to Nx Cloud and start saving time instantly.
+Teams using Nx gain the advantage of building full-stack applications with their preferred framework +alongside Nx’s advanced code generation and project dependency graph, plus a unified experience for +both frontend and backend developers.
+Visit Nx Cloud to learn more.
+ + + + + + + + + + + + + +type¶Values:
+type:apptype:servicetype:dbtype:featuretype:utiltype:assetstype:stylestype:themesCandidate values:
+type:angulartype:reacttype:modelscope¶Values:
+scope:clientscope:adminscope:backendscope:shared| Project | +Description | +
|---|---|
| challenge-api-gateway | +API gateway | +
| challenge-elasticsearch | +Elasticsearch | +
| challenge-keycloak | +Access and Identity Management (IAM) | +
| challenge-kibana | +Visualization dashboard for Elasticsearch | +
| challenge-logstash | +Data processing pipeline for Elasticsearch | +
| challenge-mariadb | +MariaDB database | +
| challenge-mongodb | +MongoDB database | +
| challenge-postgresql | +PostgreSQL database | +
| openchallenges | +OpenChallenges application | +
| openchallenges-e2e | +OpenChallenges E2E test | +
| challenge-service-registry | +Service registration and discovery | +
| Project | +Description | +
|---|---|
| openchallenges-about | +OpenChallenges About page | +
| openchallenges-assets | +OpenChallenges assets | +
| openchallenges-auth | +OpenChallenges AuthN/AuthZ | +
| openchallenges-challenge | +OpenChallenges Challenge page | +
| openchallenges-challenge-search | +OpenChallenges Challenge search page | +
| openchallenges-config | +OpenChallenges configuration | +
| openchallenges-home | +OpenChallenges Home page | +
| openchallenges-not-found | +OpenChallenges Not found page | +
| openchallenges-org-profile | +OpenChallenges Organization profile page | +
| openchallenges-org-search | +OpenChallenges Organization search page | +
| openchallenges-signup | +OpenChallenges Signup page | +
| openchallenges-styles | +OpenChallenges SCSS styles | +
| openchallenges-themes | +OpenChallenges SCSS themes | +
| openchallenges-ui | +OpenChallenges reusable UI components | +
| openchallenges-user-profile | +OpenChallenges User profile page | +
| shared-assets | +Assets shared by multiple TypeScript projects | +
| shared-java-util | +Util library shared by multiple Java projects | +
| shared-styles | +SCSS styles shared by multiple TypeScript projects | +
| shared-theme | +SCSS themes shared by multiple TypeScript projects | +
| shared-util | +Util library shared by multiple TypeScript projects | +
| shared-web-components | +Web Components shared by multiple web projects | +
ngrok config add-authtoken <token>
+ngrok http --host-header=rewrite 4200
+We can send the name of the agent - here Twitterbot - to the server. This information could be
+used by the server during dynamic rendering.
curl -A Twitterbot http://localhost:4200
+Run this command to start the ELK stack in your local environment.
+challenge-elk-serve-detach
+Open Kibana in your browser at http://localhost:5601.
+++Note Only the logs of dockerized apps are captured at this time.
+
elasticchangemedocker.name:"/challenge-logstash".This workspace manages dependency updates using Renovate, which is controlled by a GitHub +workflow. The main benefit of Renovate is that its behavior can be fully customized, unlike +Dependabot's. For example, the following strategies are used to minimize the number of email/in-app +notifications:
+++Note +This strategy is probably viable only for projects that have a high level of test coverage.
+
Renovate provides this dashboard to manage dependency updates. The following section describes the +workflow currently applied to update the dependencies.
+++Note +This workflow is a work in progress.
+
The following tools must be updated manually as described. Continuing to ask Renovate to report on +updates available for these dependencies is still benefitial. One improvement could be to prevent +Renovate from automatically opening PRs for these tools. Once dependencies have been manually +updated, Renovate will automatically remove the corresponding items from the dashboard.
+migrate tool]. Nx will take care
+ of updating its version number as well as the version number of supported tools like Angular and
+ Jest. Nx will also modify files like the project.json files.The workflow used to update the dev container is described in this +ticket.
+yarn in the Dockerfile of the devcontainer.yarn used in the workspace.
+ yarn set version <version>
+Identify whether a new version is available for a package.
+yarn outdated <package>
+Update the package.
+yarn upgrade <package> --latest
+Example:
+$ yarn outdated @nxrocks/nx-spring-boot
+yarn outdated v1.22.19
+info Color legend :
+ "<red>" : Major Update backward-incompatible updates
+ "<yellow>" : Minor Update backward-compatible features
+ "<green>" : Patch Update backward-compatible bug fixes
+Package Current Wanted Latest Package Type URL
+@nxrocks/nx-spring-boot 4.0.2 4.1.0 4.1.0 devDependencies https://github.com/tinesoft/nxrocks/blob/master/packages/nx-spring-boot#readme
+Done in 4.26s.
+
+$ yarn upgrade @nxrocks/nx-spring-boot --latest
+...
+success Saved 2 new dependencies.
+info Direct dependencies
+└─ @nxrocks/nx-spring-boot@4.1.0
+info All dependencies
+├─ @nxrocks/common@1.1.0
+└─ @nxrocks/nx-spring-boot@4.1.0
+Done in 162.06s.
+Embracing the monorepo-style development often requires some changes to the development workflow.
+Our CI should run the following checks:
+nx format:check).Note all the projects affected by a PR/commit. This is very important. Monorepo-style +development only works if we rebuild, retest, and relint only the projects that can be affected by +our changes. If we instead retest everything, we will get the the following problems:
+We should utilize affected:* commands to build and test projects. Read more about them
+here.
Monorepo-style development works best when used with trunk-based development.
+When using trunk-based development, we have a single main branch (say main) where every team
+submits their code. And they do it as soon as possible. So if someone works on a large feature, they
+split it into a few small changes that can be integrated into main in a week. In other words, when
+using trunk-based development, teams can create branches, but they are short-lived and focus on a
+specific user story.
One issue folks often raise in regards to trunk-based development is "things change under you while
+you are trying to create a release". This can definitely happen, especially when manual testing is
+involved. To mitigate we can create a release branch where we would cherry-pick commits from main
+to. With this, we can still frequently merge code into main and have our release isolated from
+changes made by other teams.
A typical Nx workspace has many more libs than apps, so pay especially careful attention to the +organization of the libs directory.
+It's a good convention to put applications-specific libraries into the directory matching the +application name. This provides enough organization for small to mid-size applications.
+For example:
+apps/web-app <---- app
+libs/web-app <---- app-specific libraries
+If more than one web app was added to the monorepo, say booking and check-in for an airline, the
+file structure would look something like this:
apps/booking <---- app
+apps/check-in <---- app
+libs/booking <---- app-specific libraries
+libs/check-in <---- app-specific libraries
+With Nx, we can partition our code into small libraries with well-defined public API. So we can +categorize our libraries based on what they contain.
+This categorization is a good starting point, but other library types are quite common too (e.g., +mock libraries). It's a good idea to establish naming conventions (e.g., utilities-testing, +components-buttons). Having them helps developers explore the code and feel comfortable no matter +where they are in the repository.
+For a large organization it's crucial to establish how projects can depend on each other. For +instance:
+shared/ui) should not depend on the libraries with
+ narrower scope (e.g., happynrwlapp/search/utils-testing).Nx provides a feature called tags that can be used to codify and statically-enforce these rules. +Read more about tags here.
+It's crucial for a large company with multiple teams contributing to the same repository to +establish clear code ownership.
+Since Nx allows us to group apps and libs in directories, those directories can become +code-ownership boundaries. That's why the structure of an Nx workspace often reflects the structure +of an organization. GitHub users can use the CODEOWNERS file for that.
+/libs/happynrwlapp julie
+/apps/happynrwlapp julie
+/libs/shared/ui hank
+/libs/shared/utils-testing julie,hank
+If you want to know more about code ownership on Github, please check the documentation on the +CODEOWNERS +file.
+This cheat sheet provides an overview of the commands needed when developing in +this monorepo.
+The content of this document was initially copied-pasted from:
+ + + + + + + + + + + + + + +