diff --git a/docs/infra/README.md b/docs/infra/README.md deleted file mode 100644 index cc6bb53..0000000 --- a/docs/infra/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# PASS Infrastructure - -At a high level, PASS is decomposed into the following parts: - -![PASS Architecture V1](../assets/architecture/pass-architecture-simple-v2.jpg) - -### Infrastructure - -The following describes the current PASS Infrastructure. - -#### EC2 - -EC2 virtual machines provide access to internal resources within the AWS VPC. - -Learn more about [deploying pass-docker via EC2](ec2.md). - -#### ECS - -PASS was developed using Docker containers and running within Amazon's ECS infrastructure running as a single task (multiple containers) under 1 service. Future development is expected to transition to using Kubernetes and utilizing Amazon's EKS. - -#### RDS - -PASS uses a PostgreSQL database instance under RDS to store metadata and system information. - -#### Amazon MQ - -PASS uses Amazon MQ as a queue to capture publication events that are read and executed by the deposit service. - -#### S3 - -The object storage is within AWS's S3 infrastructure. In particular, - -* Used to store binary data that is managed by OCFL -* Stores deployment configuration - -#### Application Load Balancer - -SSL termination and front end for the entire infrastructure. No special rules, just forwards to the ECS service on port 80. - -#### Shibboleth - -We run a number of Shibboleth service providers, one for each environment. - -### Build Artifacts: - -| Component | Artifact | Source repo | Notes | -| --- | --- | --- | --- | -| UI | Docker image, Nginx w/web app | Source code from [`pass-ui`](https://github.com/eclipse-pass/pass-ui), artifact created in [`pass-docker`](https://github.com/eclipse-pass/pass-docker) | Includes NPM package we publish from [`pass-emnber-adapter`](https://github.com/eclipse-pass/pass-ember-adapter) | -| Ember assets | Docker image, Nginx | Source code from [`pass-ui-public`](https://github.com/eclipse-pass/pass-ui-public), artifact created in [`pass-docker`](https://github.com/eclipse-pass/pass-docker) | | -| Apache HTTPd reverse proxy | Docker image, Apache | [`pass-docker`](https://github.com/eclipse-pass/pass-docker) | | -| Async Service: deposit services | Docker image, multiple intermediate Docker images (not deployed) | [`pass-deposit-services`](https://github.com/eclipse-pass/pass-deposit-services), [`pass-package-providers`](https://github.com/eclipse-pass/pass-package-providers) (intermediate) | Maven builds intermediate Docker images, [`pass-docker`](https://github.com/eclipse-pass/pass-docker) builds final image | -| Async service: notification services | Docker image | [`pass-notification-services`](https://github.com/eclipse-pass/pass-notification-services) | Docker image created in [`pass-docker`](https://github.com/eclipse-pass/pass-docker) from Spring Boot JAR created by Maven build in [`pass-notification-services`](https://github.com/eclipse-pass/pass-notification-services), released to Sonatype | -| Sync service: REST API | Docker image (`ghcr.io/eclipse-pass/pass-core-main`) | [`pass-core`](https://github.com/eclipse-pass/pass-core) | Generates an executable JAR | -| Batch service: COEUS | JAR | [`pass-grant-loader`](https://github.com/eclipse-pass/pass-grant-loader) | Task run manually or cron job | -| Batch service: NIHMS loader (formerly NIHMS ETL) | JAR | [`pass-nihms-loader`](https://github.com/eclipse-pass/pass-nihms-loader) | Task run manually or cron job | -| Batch service: Journal loader | JAR | [`pass-journal-loader`](https://github.com/eclipse-pass/pass-journal-loader) | Task run manually or cron job | -| Support: Java client | JAR | [`pass-data-client`](https://github.com/eclipse-pass/pass-support/pass-data-client) | JAR for interacting with objects in the PASS data model over JSON API | - -### Production Deployment (Kubernetes) - -We will soon be upgrading our initial production instance (which is based on Fedora) to the new PASS architecture described above. Part of this transition will also be to move from ECS to EKS, in order to deploy in a more standardized Kubernetes environment. This will also allow us to explore support for other cloud providers (such as Azure) in the future. We welcome questions, discussion, and pull requests in this area. - -## Going Forward - -As we move towards an Eclipse Foundation hosted Open-Access PASS there will be changes to the PASS architecture, changes to the infrastructure, and changes to the deployment process. This will be documented here, as well as within our [PASS Development Pipeline](pipeline.md). - -## References - -* [Deployment Pipeline](pipeline.md) -* [Our attempt to use Komposer to migrate Docker Compose to k8s manifest](docker-composer-to-k8s-manifest.md) -* [Working with GitHub secrets](github-secrets.md) -* [Self-Hosted GitHub Runners](self_hosted_github_runners.md) -* [Eclipse Foundation ops](eclipseops.md) -* [Ansible](ansible.md) diff --git a/docs/infra/ansible.md b/docs/infra/ansible.md deleted file mode 100644 index 0f0a7dd..0000000 --- a/docs/infra/ansible.md +++ /dev/null @@ -1,32 +0,0 @@ -# Ansible - -## Installation - -Ansible can be installed using PIP on any platform but some platforms (e.g., Ubuntu) have packages that make installation easier. - -Checkout the [Ansible installation guide](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installation-guide) for instructions. - -## Setup - -Ansible requires a control node (which must have Ansible installed) and any number of remote clients which can be modified by Ansible. Each remote system must be ssh-accessible from the control node but has no other prerequisites. - -### Managed nodes - -Remote systems in Ansible are known as "managed nodes". The control node must be able to ssh to each managed node that it wants to modify. To do this, the `.ssh/authorized_keys` file on each managed node must have the public key for an ssh key available on the control node. - -The list of nodes managed by Ansible can be managed multiple ways: - -1. In the `/etc/ansible/hosts` file on the control node. -2. In an `~/inventory.yaml` file. The file can be named or located elsewhere but then the path must be provided in each inventory command. - -### The Playbook - -To execute tasks on a managed node, you'll need to create a "playbook". This is a yaml file containing a reference to the group of managed nodes that it applies to and the list of tasks to be performed. - -### Running locally - -To run Ansible playbooks against the control node, set the `hosts` property in the playbook to `all` and then when running the playbook, specify `localhost,` (the comma is intentional) as the inventory: - -```bash -ansible-playbook playbook.yaml -i localhost -``` diff --git a/docs/infra/bitwarden.md b/docs/infra/bitwarden.md deleted file mode 100644 index 8355ba8..0000000 --- a/docs/infra/bitwarden.md +++ /dev/null @@ -1,107 +0,0 @@ -# Bitwarden - -A [password tool](https://bitwarden.com) to help us manage our [organizations secrets](https://vault.bitwarden.com/#/vault?organizationId=e429b264-5b49-4794-be50-b0660125456a). -Another potential tool is the [pass CLI](https://www.passwordstore.org) where `pass` is for password not _our_ Open Access PASS. - -Here is an outline of passwords we are tracking -(duplicating here as can better document the integrations with other services -like [Otterdog](/docs/infra/otterdog.md) and Github Actions). - -![Bitwarden Secrets User Interface](/docs/assets/bitwarden/ui.png) - -| Item Name | Item Id | Field | Description | -| --- | --- | --- | --- | -| HELLO_WORLD | 23801ca4-fd27-446c-b5af-b07b0108f443 | quest | A test secret of `holy_grail` for demontsrating how secrets are managed | -| HELLO_WORLD | 23801ca4-fd27-446c-b5af-b07b0108f443 | color | Another test secret of `blue_no_ahh` | - -## Installing CLI - -Some instructions for installing the [bitwarden cli are here](https://github.com/bitwarden/clients). -The tools command line name is `bw` and you can see which version you are running with - -```bash -bw -v -``` - -To install it via [homebrew](https://brew.sh) you can run - -```bash -brew install bitwarden-cli -``` - -You can log into the account with - -```bash -bw login -``` - -And then you can list the passwords (in plain text, be careful). - -```bash -bw list items --search HELLO_WORLD -``` - -Note these are NOT sensitive passwords so it's OK to be shown - -```bash -[ - { - "passwordHistory": - [ - { - "lastUsedDate": "2023-09-26T17:32:20.947Z", - "password": "HELLO_WORLD" - } - ], - "revisionDate": "2023-09-26T17:39:02.833Z", - "creationDate": "2023-09-12T16:04:39.793Z", - "deletedDate": null, - "object": "item", - "id": "23801ca4-fd27-446c-b5af-b07b0108f443", - "organizationId": "e429b264-5b49-4794-be50-b0660125456a", - "folderId": null, - "type": 1, - "reprompt": 0, - "name": "HELLO_WORLD", - "notes": null, - "favorite": false, - "fields": - [ - { - "name": "quest", - "value": "holy_grail", - "type": 1, - "linkedId": null - }, - { - "name": "color", - "value": "blue_no_ahh", - "type": 1, - "linkedId": null - } - ], - "login": - { - "username": null, - "password": "WORLD_HELLO", - "totp": null, - "passwordRevisionDate": "2023-09-26T17:32:20.947Z" - }, - "collectionIds": - [ - "af5a75d8-8762-4cec-8d96-b06601263cf1" - ] - } -] -``` - - -## Troubleshooting - -### How do I get the ID of an item from the UI - -You will need to **Inspect** the page and watch for the network traffic -after you edit the item. - -![Bitwarden Secrets User Interface](/docs/assets/bitwarden/ui_get_id.png) - diff --git a/docs/infra/deploy_demo.md b/docs/infra/deploy_demo.md deleted file mode 100644 index 90ddc6f..0000000 --- a/docs/infra/deploy_demo.md +++ /dev/null @@ -1,113 +0,0 @@ -# Deploying a demo PASS system - -What follows are developer-oriented instructions about our -[GitHub Self-Hosted Runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) -connected to [GitHub actions](https://github.com/features/actions) -for automatable deploys of PASS demo system. - -For information on actually deploying updates to a _running_ demo system -such as [demo.eclipse-pass.org](https://demo.eclipse-pass.org) -please refer to [Eclipse Foundation infrastructure](eclipseops.md) - -## Self-Hosted Demo Code - -On a self-hosted server, the code is located in an (unfortunately) deeply nested structure at - -```bash -/opt/githubrunner/pass-docker/pass-docker/pass-docker -``` - -On a [GitHub actions](https://github.com/features/actions) deploy, -the latest code will be available at that location, as shown below. - -```bash -a2forward@nightly-pass:/opt/githubrunner/pass-docker/pass-docker/pass-docker$ git log -1 -commit 3766a8db3faabe4e28d0eb1ac9549d5048ec65b2 (grafted, HEAD -> main, origin/main) -Author: Grant McSheffrey -Date: Wed Sep 21 10:49:42 2022 -0400 - - Merge pull request #271 from eclipse-pass/k8s-separation - - Separate Kubernetes changes into new files -``` - -## Action Triggers - -The [GitHub actions](https://github.com/features/actions) are defined in -[pass-docker/.github/workflows]((https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/) -such as our [nightly action](https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/deploy_passnightly.yml). - -The [nightly action](https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/deploy_passnightly.yml) -will run every night (and on every pull-request merge). - -## Debugging On The Server - -Please ensure the issue is actually related to the deployed environment -and IS NOT reproducible locally. - -If you are troubleshooting strange behaviour on a -[GitHub Self-Hosted Runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) -then you can directly manipulate the code outside of an official deploy. - - -For that, first SSH onto that server, and then switch to the `githubrunner` user. - -```bash -# SSH into the server -cd /opt/githubrunner/pass-docker/pass-docker/pass-docker -sudo su githubrunner -``` - -From here, you can run the application (any version of it) - -```bash -docker-compose up -``` - -Please refer to the specifics of the [Eclipse Foundation infrastructure](eclipseops.md) -for the actual commands to run on a specific environment. For example, this -is the specific script that (currently) runs [demo.eclipse-pass.org](https://demo.eclipse-pass.org) - -``` -docker-compose -f eclipse-pass.base.yml -f eclipse-pass.demo.yml up -``` - -For debugging purposes you can also change the branch and run a -specific build (what follows is for demonstration purposes not the -specific commands you want to run) - -```bash -git fetch -git checkout demo-spike -docker-compose -f demo.yml pull -docker-compose -f demo.yml --env-file .demo_env up -``` - -If the application did launch as expected, you should be able to -see the working site (for example using `curl`). - -```bash -curl -k https://localhost -``` - -Which will return something similar to - -```html - - - - PASS docker - - - - PASS Docker! - -

Apps:

- - - - - -``` diff --git a/docs/infra/digitalocean.md b/docs/infra/digitalocean.md deleted file mode 100644 index ffce536..0000000 --- a/docs/infra/digitalocean.md +++ /dev/null @@ -1,89 +0,0 @@ -# PASS Digital Ocean Deployment - -This will document how to run PASS in [Digital Ocean](https://www.digitalocean.com) -as [Self-Hosted GitHub Runners](self_hosted_github_runners.md). - -## Create Server - -You will first need a -[linux server as your base OS](https://docs.digitalocean.com/products/droplets/how-to/create/) -with [sudo privileges](https://en.wikipedia.org/wiki/Sudo). - -Going forward, the following scripts will be RUN ON THE SERVER, for example -after you SSH into the box. - -```bash -ssh -``` - -## Install PASS Dependencies - -On the server, you will need [docker / docker-compose](https://docs.docker.com/compose/) and [pass-docker](https://github.com/eclipse-pass/pass-docker) - -The following script will install those dependencies (these should be run as [sudo](https://en.wikipedia.org/wiki/Sudo)) - -```bash -apt-get -y update -apt-get install -y gnupg2 pass docker-compose -mkdir -p /src -cd /src -git clone git@github.com:eclipse-pass/pass-docker.git -cd pass-docker -git checkout minimal-assets -``` - -Or, you can run this directly on your server - -```bash -/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/eclipse-pass/main/main/tools/github_runner/deps.sh)" -``` - -If the installation worked, then you can pull and run the application - -```bash -cd /src/pass-docker && \ - docker-compose pull && \ - docker-compose up -``` - -## Install GitHub Runner Scripts - -If you also wanted to use this server as a -[self-Hosted GitHub Runners software](self_hosted_github_runners.md) -then run - -```bash -/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/eclipse-pass/main/main/tools/github_runner/install.sh)" -``` - -## Connect Server to GitHub Runner - -You can then connect the server to GitHub using the following variables - -```bash -PASSORG=eclipse-pass -GITHUB_RUNNER_TOKEN=ghp_abc123def456abc123def456abc123def456 -```` - -To execute the following script. - -```bash -(cd /opt/githubrunner && \ - sudo -u githubrunner ./config.sh \ - --unattended \ - --url https://github.com/${PASSORG}/pass-docker \ - --token $GITHUB_RUNNER_TOKEN \ - --name $NAME \ - --runnergroup default \ - --labels $NAME \ - --work pass-docker \ - --replace && \ - ./svc.sh install githubrunner && \ - ./svc.sh start) -``` - -These scripts to [add.sh](/tools/github_runner/add.sh) -and [remove.sh](/tools/github_runner/remove.sh) -your server from GitHub Runner -are available in [/tools/github_runner](/tools/github_runner). - diff --git a/docs/infra/ec2.md b/docs/infra/ec2.md deleted file mode 100644 index d51e7e0..0000000 --- a/docs/infra/ec2.md +++ /dev/null @@ -1,19 +0,0 @@ -# PASS EC2 Deployment - -This will document how to run PASS in EC2 by reusing the existing pass-docker setup. - -## Background - -The existing pass-docker setup is intended for local deployment. It uses a hard-coded hostname of `localhost`. It includes a customized Identity Provider for authentication as well as DSpace for testing deposits. All of the other images are production images just configured for `localhost`. See the [pass-docker project](https://github.com/eclipse-pass/pass-docker) for more information. - -## Configuration - -First the server must have a hostname and certificate. Then the pass-core image which is a Spring Boot application needs to be modified to use this hostname instead of `localhost`. -See [https://docs.spring.io/spring-boot/reference/features/ssl.html] for instructions. - -Variables that that mention `localhost` in the `.env` need to be changed. - -## Running - -Follow the [pass-docker project](https://github.com/eclipse-pass/pass-docker) instructions to run. - diff --git a/docs/infra/eclipseops.md b/docs/infra/eclipseops.md deleted file mode 100644 index 3497f2b..0000000 --- a/docs/infra/eclipseops.md +++ /dev/null @@ -1,248 +0,0 @@ -# Eclipse Foundation Operations - -The following site are managed by the Eclipse Foundation. - -| URL | Notes | Status -| --- | --- | --- | -| eclipse-pass.org | Github Pages HTML / CSS | running-locally (DNS not configured) -| demo.eclipse-pass.org | GitHub Self-Hosted Runner / Docker Compose | running -| nightly.eclipse-pass.org | GitHub Self-Hosted Runner / Docker Compose | running-locally (DNS not configured) - -The underlying infrastructure for each is documented here - -| Infra | User | Details -| --- | --- | --- | -| Github Pages HTML / CSS | N/A | [Automatic via GitHub Actions](https://pages.github.com) -| GitHub Self-Hosted Runner / Docker Compose | githubrunner | Run from /opt/githubrunner/pass-docker/pass-docker/pass-docker - -If you are looking for more detailed developer-oriented instructions -to help debug an issue with the above, please refer to -[managing our demo servers](deploy_demo.md) documentation. - -## Configurations - -The [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) project allows our team to -[self-service several aspect of our organization](https://www.eclipse.org/projects/handbook/#resources-github-self-service) -via a tool called [Otterdog](/docs/infra/otterdog.md) and secrets in [Bitwarden](/docs/infra/bitwarden.md). - -These [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) repo / tools gives access to: - -* Organization Settings -* Organization Webhooks -* Repositories and their settings -* Branch Protection Rules - -Learn more about [Otterdog here](/docs/infra/otterdog.md). - - -## Deployment - -### eclipse-pass.org - -This is managed with [GitHub Pages](https://pages.github.com) -from the [eclipse-pass.github.io](https://github.com/eclipse-pass/eclipse-pass.github.io) repo. - -#### Sync Demo To Marketing - -The [demo.eclipse-pass.org](https://demo.eclipse-pass.org) site includes marketing -information about PASS that we currently synchronize with the main [eclipse-pass.org](https://pages.github.com) site. - -1. Download the publicly accessible site. - -This can be done with a tool like [SiteSucker](https://apps.apple.com/us/app/sitesucker/id442168834) on a Mac. - -![Download site with SiteSucker](../assets/eclipsesync/sitesucker.png) - -2. Update the [eclipse-pass.github.io](https://github.com/eclipse-pass/eclipse-pass.github.io) repo. - -In a new branch, copy all files from the download site. - -3. Fix references to the demo site - -Any link to "idp" for login purposes, such as - -```html -Use the Demo -``` - -Should be updated to - -```html -Use the Demo -``` - -3. PR branch into main - -Once the code is in main, the site will automatically be deployed to [eclipse-pass.org](https://eclipse-pass.org). - -![Updates deployed on merge](../assets/eclipsesync/github_pages_actions_deploy.png) - - -### demo.eclipse-pass.org - -The [demo.eclipse-pass.org](https://demo.eclipse-pass.org) site is deployed -on demand using [GitHub actions](https://github.com/features/actions). -Note that this site is not yet publically available. - -The demo system is based on the [pass-docker](https://github.com/eclipse-pass/pass-docker) project -and you can view the available [actions](https://github.com/eclipse-pass/pass-docker/actions) -including the [Deploy passdemo action](https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/deploy_passdemo.yml) - -#### Publish Via GitHub.com - -To publish an update, access the actions page. - -![pass-docker actions](../assets/demo/passdocker_actions.png) - -You can then [run the workflow](https://github.com/eclipse-pass/pass-docker/actions/workflows/deploy_passdemo.yml) - -![run workflow](../assets/demo/run_workflow.png) - -And watch the deploy. - -![deployed actions](../assets/demo/deploy_actions.png) - -#### Locally Run Via Docker Compose - -The [demo.eclipse-pass.org](https://demo.eclipse-pass.org) site is -managed using [Docker Compose](https://docs.docker.com/compose/). - -To run locally first get a local copy of [pass-docker](https://github.com/eclipse-pass/pass-docker) -then run it with the following commands - -``` -cd pass-docker && \ - docker-compose -f eclipse-pass.base.yml -f eclipse-pass.demo.yml pull && \ - docker-compose -f eclipse-pass.base.yml -f eclipse-pass.demo.yml up -``` - -For more information debugging the deployment to our demo servers -please refer to our [demo deploy documentation](deploy_demo.md). - -### nightly.eclipse-pass.org - -The [nightly.eclipse-pass.org](https://demo.eclipse-pass.org) site is deployed -automatically on PR merged (as well as nightly) using [GitHub actions](https://github.com/features/actions). -Note that this site is not yet publically available. - -Much like [demo.eclipse-pass.org](https://demo.eclipse-pass.org) the nightly -system is based on the [pass-docker](https://github.com/eclipse-pass/pass-docker) project -and you can view the available [actions](https://github.com/eclipse-pass/pass-docker/actions) -including the [Deploy passdemo action](https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/deploy_passdemo.yml) - -#### Locally Run Via Docker Compose - -The [nightly.eclipse-pass.org](https://nightly.eclipse-pass.org) site is -managed using [Docker Compose](https://docs.docker.com/compose/). - -To run locally first get a local copy of [pass-docker](https://github.com/eclipse-pass/pass-docker) -then run it with the following commands - -``` -cd pass-docker && \ - docker-compose -f eclipse-pass.base.yml -f eclipse-pass.nightly.yml pull && \ - docker-compose -f eclipse-pass.base.yml -f eclipse-pass.nightly.yml up -``` - -For more information debugging the deployment to our demo servers -(such as [nightly.eclipse-pass.org](https://nightly.eclipse-pass.org)) -please refer to our [demo deploy documentation](deploy_demo.md). - - -## Infrastructure - -A bootstrapped server [installer](/tools/eclipse_ops/bootstrap) -to run a stand-alone PASS application via the [pass-docker](https://github.com/eclipse-pass/pass-docker) -project - -Which will do the following - -* Install docker -* Install [github self-hosted runners](self_hosted_github_runners.md) -* Intall [pass-docker](https://github.com/eclipse-pass/pass-docker) -* Configure [pass docker for a GH runner](self_hosted_github_runners.md) - -To configure the self-hosted runner you will need the GITHUB token, -replacing the `XXX` with the actual token value. - -``` -GITHUB_RUNNER_TOKEN=XXX -``` - -Learn more from the [installer script itself](/tools/eclipse_ops/bootstrap). - -## Run PASS - -To manually run the application, execute - -```bash -cd /src/pass-docker && \ - sudo docker-compose pull && \ - sudo docker-compose up -``` - -If you want to run in the background then use `-d`. - -```bash -sudo docker-compose up -d -``` - -## Troubleshooting - -### Access Eclipse Servers - -To access the Eclipse infrastructure you will need a `bastion.eclipse.org` login. -If you do not have a `bastion` login, then you will need to reach out to -a [team member](https://github.com/orgs/eclipse-pass/teams/technology-pass-committers/members) -for instructions (and permission) to get such a login. - -| Server | IP | -| --- | --- | -| nightly.eclipse-pass.org | `ssh ${ME}@bastion.eclipse.org -t ssh ${ME}@172.30.206.14` -| demo.eclipse-pass.org | `ssh ${ME}@bastion.eclipse.org -t ssh ${ME}@172.30.206.15` - -You will need to change `ME` to your username - -```bash -ME=myusername -``` - -Once on the server, then CD into the directory and change to the `githubrunner` user - -```bash -cd /opt/githubrunner/pass-docker/pass-docker/pass-docker -sudo su githubrunner -``` - -Now you have full access to the application and can run the application as shown below - -### Run Remote Apps - -From our [GitHub actions](https://github.com/features/actions) you will see -the specifics on running the servers - -| Server | Configs | Runner | -| --- | --- | --- | -| nightly.eclipse-pass.org | `docker-compose -f eclipse-pass.base.yml -f eclipse-pass.nightly.yml` | [nightly script](https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/deploy_passnightly.yml) -| demo.eclipse-pass.org | `docker-compose -f eclipse-pass.base.yml -f eclipse-pass.demo.yml` | [demo script](https://github.com/eclipse-pass/pass-docker/blob/main/.github/workflows/deploy_passdemo.yml) - -For example, to stop all `demo` services run, - -``` -docker-compose -f eclipse-pass.base.yml -f eclipse-pass.demo.yml down -``` - -### Remove Rogue Processes - -If the docker-compose file has drastically changed, then there migth be _rogue_ processes -still running. You can view them with - -```bash -docker ps -``` - -Then grab the container IDs and manually stop them. - -```bash -docker stop 03159019094f -``` diff --git a/docs/infra/github-secrets.md b/docs/infra/github-secrets.md deleted file mode 100644 index a60648d..0000000 --- a/docs/infra/github-secrets.md +++ /dev/null @@ -1,77 +0,0 @@ -## GitHub Secrets - -GitHub secrets are intended for use by GitHub Actions workflows. If you want a workflow to interact with something that requires authentication, you will likely need to create a secret which will allow authentication to happen securely without making any private information available to anybody having access to the repositories. - -There are three levels that GitHub secrets can exist at: organization, repo, and environment. - -An organization secret belongs to the `eclipse-pass` organization and is available to workflows that are part of any repository that exists under the `eclipse-pass` organization. Secrets that will be used across many repositories should be placed here to prevent duplication. - -A repository secret exists within that repository and can only be used by workflows belonging to that repository. - -An environment secret exists within a specific environment within a repository. It can only be used by workflows belonging to that repository and the workflow must specify the environment that it's using. - -### Creating Secrets - -GitHub offers two mechanisms for creating secrets: UI and API. - -The GitHub UI requires that a person creating a secret have admin privileges for that entity (either org or repo). Since members of the PASS project don't have admin privileges at any level, we need to open a ticket with Eclipse to add secrets through the UI. - -Oddly, the GitHub API allows the creation of repository secrets with `repo` privileges. All members of the PASS project should have sufficient privileges to do so. - -To create a repository or environment secret, use the [github_secrets.py script](/tools/github_secrets.py) in this repo like so: -``` -python github_secrets.py -u -t -r -n -v [-e ] -``` -* username: Your GitHub username -* token: A GitHub personal access token with 'repo' access -* repo: The name of the GitHub repository containing the secret -* name: Name of the secret being created or updated -* value: The value to be set for the secret. The value will be encrypted before being sent to GitHub. - -To create an organization secret, a ticket must be opened with the [Eclipse Help Desk](https://gitlab.eclipse.org/eclipsefdn/helpdesk). - -### Reading Secrets - -The value of a secret can only be read by GitHub Actions workflows. - -The metadata associated with a secret can be retrieved with the `github_secrets.py` script by omitting the `value`. The returned data includes the secret's `name`, `created_at`, and `updated_at`. This is primarily useful for verifying that a secret exists where you expect it to be. - -### Secrets in GitHub Actions workflows - -To access a GitHub secret from a top-level workflow, you can reference it from within the secrets context like this: -``` -${{ secrets.KUBE_CONFIG }} -``` - -If you want to use a secret in a reusable or "called" workflow, you must pass the secrets that will be used. To do this, when you reference the workflow you additionally provide a `secrets` property containing a list of the secrets that must be passed. - -In this example two secrets, KUBE_CONFIG and DIGITALOCEAN_ACCESS_TOKEN are being passed to the `html-app-publish.yml` workflow: - -``` -call-publish-docker: - name: Build and Publish Docker image - uses: eclipse-pass/playground/.github/workflows/html-app-publish.yml@main - secrets: - KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }} - DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }} -``` - -Within the called workflow, you must declare the secrets that can be expected like this: - -``` -on: - workflow_call: - secrets: - KUBE_CONFIG: - required: true - description: Configuration info for test k8s cluster - DIGITALOCEAN_ACCESS_TOKEN: - required: true - description: Access token for Digital Ocean -``` - -The secret can now be referenced within the workflow as it would be from a top-level workflow: - -``` -${{ secrets.KUBE_CONFIG }} -``` diff --git a/docs/infra/k8s-migration.md b/docs/infra/k8s-migration.md deleted file mode 100644 index 76b692a..0000000 --- a/docs/infra/k8s-migration.md +++ /dev/null @@ -1,92 +0,0 @@ -# Kubernetes Manual Migration - -See our [initial attempt](docker-composer-to-k8s-manifest.md) to migrate our Docker compose configuration automatically to a Kubernetes configuration. This automated conversion was used as a starting basis for -the configuration that we now have. - -## Test Cluster - -We are using the Kubernetes hosting service at [Digital Ocean](https://www.digitalocean.com). This is a -solution to get our k8s configuration figured out with an environment that we have full control over. -Once we have completed the configuration, we will move the configuration to a k8s cluster hosted -by the Eclipse Foundation. - -## Creating and Configuring the Cluster - -### Prerequisites - -1. [Install doctl](https://docs.digitalocean.com/reference/doctl/how-to/install/). -2. [Generate a token](https://cloud.digitalocean.com/account/api/tokens/new) with Digital Ocean. -3. Authenticate with Digital Ocean using the token: `doctl auth init`. -4. [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) (this is the Kubernetes command line tool) - -### Cluster Creation & Configuration - -These are the steps to creating and configuring the test cluster: - -1. Create the cluster: - - ```bash - doctl kubernetes cluster create pass-docker \ - --auto-upgrade=true \ - --node-pool "name=passnp;count=2;auto-scale=true;min-nodes=1;max-nodes=3;size=s-2vcpu-4gb" \ - --region tor1 - ``` - - The available list of node sizes can be obtained by running 'doctl compute size list' - -2. Create a secret for accessing the GitHub Container Registry: - - ```bash - export GITHUB_USERNAME=MyUsername - export GITHUB_TOKEN=MyGitHubPersonalAccessToken - kubectl create secret docker-registry regcred \ - --docker-server=https://ghcr.io \ - --docker-username=GITHUB_USERNAME \ - --docker-password=GITHUB_TOKEN \ - ``` - - This secret is referenced in k8s-deployment.yaml. - - Use a personal access token with registry read access. - -3. Apply the Kubernetes configuration files found in the [pass-docker](https://github.com/eclipse-pass/pass-docker) - repo. The list of files and the order they should be run in can be found in the - [Kubernetes README file](https://github.com/eclipse-pass/pass-docker/tree/main/k8s/README.md). - -4. Find the external IP address of the cluster: - - ```bash - doctl compute load-balancer list \ - --format ID,Name,Created,IP,Status - ``` - - It can take some time before the IP address becomes available, run this command until Status is 'active' - -### Deployment Configuration - -The yaml files found in the pass-docker `k8s` folder specify the configuration for the PASS Eclipse -kubernetes setup. Each of the services within the PASS ecosystem have a `deployment` which contains -a `pod` which is monitored and will ensure that if the pod fails, it will be restarted. Each pod -is a wrapper for a `container` that is a configured Docker image running a PASS Eclipse service. - -The images specified correspond to images pushed to either the Docker Hub oapass repository or the -eclipse-pass GitHub Container Registry (currently done manually). - -Another important part of pod configuration is specifying any `persistent volumes` that will be used -by the container. The volumes themselves are configured in their own yaml files. - -Most deployments have a corresponding Kubernetes `service` defined that configures access to the -pod through a load balancer. Most of the configuration in the services is specifying the ports to -be used. This will route traffic to those ports to the correct container. If there is any conflict -where multiple containers try to use the same port, an error will occur on container startup and the -container will fail. - -### Cluster Deletion - -To destroy all resources belonging to the test cluster, run this command: - -```bash -doctl kubernetes cluster delete pass-docker --dangerous -``` - -The `--dangerous` flag tells Kubernetes to delete all resources associated with the cluster diff --git a/docs/infra/otterdog.md b/docs/infra/otterdog.md deleted file mode 100644 index a766d9a..0000000 --- a/docs/infra/otterdog.md +++ /dev/null @@ -1,157 +0,0 @@ -# Otterdog - -Our organization configs are managed on [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) -using the [Otterdog tool](https://gitlab.eclipse.org/eclipsefdn/security/otterdog) - -Our first project will be to manage secrets via [bitwarden](/docs/infra/bitwarden.md). - -## Pushing GitHub Infra Changes - -To make changes, you must push a [forked PR like this one](https://github.com/eclipse-pass/.eclipsefdn/pull/1). - -![Sample PR](/docs/assets/otterdog/otterdog_sample_pr.png) - - -## Configurations - -The base configuration is [eclipse-pass.jsonnet](https://github.com/eclipse-pass/.eclipsefdn/blob/main/otterdog/eclipse-pass.jsonnet), -and can be monitored at [eclipse-pass.org/.eclipsefdn](https://eclipse-pass.org/.eclipsefdn/) - -![Otterdog Config Monitoring](/docs/assets/otterdog/dashboard.png) - -The base entry for jsonnett configs is via - -```javascript -local orgs = import 'otterdog-defaults.libsonnet'; -``` - -### Bitwarden Configs - -To integrate [bitwarden into otterdog configs](https://gitlab.eclipse.org/eclipsefdn/security/otterdog#bitwarden) -the request is to add an _"organization"_ directly to the JSON, but instead lets use the jsonnett based -on the outputs from the [otterdog playground](http://eclipse-pass.org/.eclipsefdn/playground/). - -```javascript -orgs.newOrg('eclipse-pass') { - credentials+: [{ - "provider": "bitwarden", - "item_id" : "23801ca4-fd27-446c-b5af-b07b0108f443" - }, - ], -} -``` - -And then we can specify secrets based on the structure of `bitwarden:@`. - -### Organizational Secrets - -Here is documentation on managing [organization secrets](https://otterdog.readthedocs.io/en/latest/reference/organization/secret/) - -```javascript -orgs.newOrg('eclipse-pass') { - secrets+: [ - orgs.newOrgSecret('HELLO_WORLD_QUEST') { - value: "bitwarden:23801ca4-fd27-446c-b5af-b07b0108f443@quest", - }, - orgs.newOrgSecret('HELLO_WORLD_COLOR') { - value: "bitwarden:23801ca4-fd27-446c-b5af-b07b0108f443@color", - }, - ], -} -``` - -Please refer to [bitwarden for specifics on password management](/docs/infra/bitwarden.md) - -## Playground (Online Editor) - -From the [monitoring application]((https://eclipse-pass.org/.eclipsefdn/)) you can access a [otterdog playground](http://eclipse-pass.org/.eclipsefdn/playground/) - -![Otterdog Playground](/docs/assets/otterdog/playground.png) - -## Installing Locally - -If you are testing github integrations, you will need to [install Otterdog locally](https://otterdog.readthedocs.io/en/latest/install/) - -Below is a summary of the [installation guide above](https://otterdog.readthedocs.io/en/latest/install/). -If you run into problems, then please read the source installation guide and update these docs. - -You will need the following tools - -```bash -python3 --version # 3.10+, tested on 3.11.4 -pip --version # any, tested on 23.2 -poetry --version # any, tested on 1.4.2 -go version # 1.13+, tested on 1.20.5 -jb --version # any, tested on 'dev' (aka 0.5.1 as shown in troubleshooting below) -bw -v # any, tested on 2023.9.0 -``` - -Now you can clone the [Otterdog repo](https://gitlab.eclipse.org/eclipsefdn/security/otterdog/) -and build it locally - -```bash -# Somewhere outside of the eclipse-pass repos -git clone git@gitlab.eclipse.org:eclipsefdn/security/otterdog.git -cd otterdog -make init -``` - -You now have `otterdog` installed locally. - -```bash -otterdog --version -# otterdog.sh, version 0.3.0.dev0 -``` - -### Troubleshooting - -#### jb / jsonnet-bundler not found - -Here's how you install [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler) (aka `jb`) - -``` -go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@v0.5.1 -``` - -Note that [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler) was installed to a `$HOME/go/bin` which was NOT -on our path, so I had to add that to ensure `jb` was accessible. - -```bash -# in your ~/.bash_profile or similar -PATH="$HOME/go/bin:$PATH" -``` - -#### bw / bitwarden not found - -Here's how you install [bitwarden](https://github.com/bitwarden/clients) (aka `bw`). - -The installation uses `snap`, but (home)`brew` also works - -```bash -brew install bitwarden-cli -``` - -#### .local/bin/otterdog: No such file or directory - -If you see something like - -```bash -test -f ~/.local/bin/otterdog || ln -s /Users/aforward/sin/projects/eclipse-pass/otterdog/otterdog.sh ~/.local/bin/otterdog -ln: /Users/aforward/.local/bin/otterdog: No such file or directory -make: *** [init] Error 1 -``` - -Then ensure that you have a `~/.local/bin` to receive the file, and that is also on your path - -```bash -mkdir -p ~/.local/bin - -# in your ~/.bash_profile or similar -PATH="$HOME/.local/bin:$PATH" -``` - -#### How do I get the github organization ID from the UI - -You will need to **Inpsect** the org page and look for `organization:` - -![GitHub Organization ID](/docs/assets/github/ui_get_org_id.png) diff --git a/docs/infra/pipeline.md b/docs/infra/pipeline.md deleted file mode 100644 index c15c4ce..0000000 --- a/docs/infra/pipeline.md +++ /dev/null @@ -1,268 +0,0 @@ -# PASS Pipeline - -## Orchestration - -Below is an outline of how the various pieces of the PASS infrastructure will be coordinated into a release. -We will start with non-production-ready version using [pass-docker](https://github.com/eclipse-pass/pass-docker) and in particular one -based on [GitHub Actions via Self-Hosted Runner](https://github.com/eclipse-pass/pass-docker/tree/178-docker-compose-gh-actions). - -Once branch `178-docker-compose-gh-actions` is merged, this documentation should be updated to -point back to the `main` branch. - -As different environments (cd, nightly, dev, demo, pre-production, production) are -introduced this documentation will be updated to reflect those differences (for -example cd, nightly, dev, demo will all use a _fake_ nihms FTP server). - -The [docker containers should be migrated based on eclipse-pass dependencies](../dev/integration-test-docker-dependencies.md). - -### Application Reposibilities - -The PASS application will be responsible for - -* Building core artefacts (e.g. customizations to the fedora images) -* Updating environment manifests (e.g a new container image to upgrade the deposit service) -* Deoploying to a setup environment (based on the manifest) - -![Application pipeline actions](../assets/pipeline/pipeline_app.png) - -Note that the `pass-app` currently refers to the [pass-docker repo](https://github.com/eclipse-pass/pass-docker). -The recommendation is that `pass-docker` be renamed (and enhanced) to -capture using [infrastructure as code](https://en.wikipedia.org/wiki/Infrastructure_as_code)) all configurable environments. - -Here is a summary of the actions and their current triggers / implementation status (note not all _actions_ have been fully automated). - -| Action | Trigger | Note -| --- | --- | --- | -| deploy_demo | GH Action Button | In development -| update_core_versions | Manual | Manually build / publish containers, update pass-docker manifests -| update_datastores | Manual | Same as `update_core_versions` -| update_project | Manual | Update pass-docker manifests - -### Project Reposibilities - -The individual projects will be responsible for - -* Running automated quality control (aka unit and integration tests) -* Building project artefacts (e.g. Java JARs, Docker images, Golang binaries) -* Notifying the application of the update - -![Project pipeline actions](../assets/pipeline/pipeline_project.png) - -The artefacts will be stored in a consistent manner so the application -can correct grab the appropriate versions when orchestrating a release. - -| Action | Trigger | Note -| --- | --- | --- | -| test | GH PRs | Mostly in place for all projects -| build | Manual | Working towards GH actions -| notify | Manual | Notify via email/slack/carrier pigeon of update - - -## Artefact Decomposition - -The PASS project can be viewed as a decomposition of - -* core infrastrcture (e.g. database, web servers, indexes) -* access to third party data stores (e.g. DSpace, Nihms) -* identify management (e.g. login, authentication, authorization) -* PASS projects (e.g. ui, deposit services, loaders) - -Each decomposition will have different levels of automation and -different release cycles. - -### Core Infrastructure - -The following core pieces of infrastructure are required for the PASS Application. -These pieces are more stable and will not be updated frequently. - -| Name | Based On | Notes | -| --- | --- | --- | -| fcrepo | [tomcat:8.5.40-jre8-alpine docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/fcrepo/4.7.5/Dockerfile) | Old version (4.7.5) and relies heavily on `OAPASS` instead of `eclipse-pass` -| postgres | [postgres:10.3-alpine docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/postgres/10.3/Dockerfile) | Underlying DB for fcrepo -| activemq | [openjdk:8-jre-alpine docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/activemq/Dockerfile) | Version 5.15.11 -| httpd-proxy | [centos docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/httpd-proxy/Dockerfile) | Orchestration between services -| elasticsearch | [elasticsearch/elasticsearch-oss 6.2.3 docker](https://www.docker.elastic.co/r/elasticsearch/elasticsearch-oss) | [Security vulnerability to patch](https://github.com/eclipse-pass/main/issues/274) -| mail | oapass/docker-mailserver | Should be updated to [eclipse-pass/pass-docker-mailserver](https://github.com/eclipse-pass/pass-docker-mailserver/blob/main/Dockerfile) - -### Third Party Data Stores - -The following third party data stores are configured / configurable within PASS. -These pieces are more stable and will not be updated frequently. - -| Name | Based On | Notes | -| --- | --- | --- | -| nihms ftpserver | [stilliard/pure-ftpd docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/ftpserver/0.0.1-demo/Dockerfile) | Sample [Nimhs FTP Server](https://www.nihms.nih.gov/login/?next=/submission/) -| dspace | [jetty:9.4.7-jre8-alpine docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/dspace/6.2/Dockerfile) | Sample [DSpace database](https://www.dspace.com/en/inc/home.cfm) - -### Identity Management - -The following core pieces manage authentication. -These pieces are more stable and will not be updated frequently. - -| Name | Based On | Notes | -| --- | --- | --- | -| idp | [unicon/shibboleth-idp docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/idp/Dockerfile) | Based on shibboleth and configurable for common, harvard and jhu -| ldap | [centos docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/ldap/Dockerfile) | Configured for fakek accounts on common, harvard and jhu -| shibboleth-sp | [unicon/shibboleth-sp docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/sp/2.6.1/Dockerfile) | Core shibboleth service provider -| authz | [openjdk:8u212-jre-alpine3.9 docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/authz/Dockerfile) | Managed via `oapass.org` - -### PASS Projects - -The following projects will manage building their own containers. - -| Name | Installation | Notes | -| --- | --- | --- | -| pass-ui | [docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/ember/Dockerfile) | Relies on `OA-PASS/pass-ember` and should be updated to `eclipse-pass/pass-ui` -| pass-ui-static | [docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/static-html/Dockerfile) | Relies on `OA-PASS` should be updated to `eclipse-pass` -| indexer | [openjdk:8u212-jre-alpine3.9 docker](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/indexer/0.0.18-SNAPSHOT/Dockerfile) | Relies on `OA-PASS` should be updated to `eclipse-pass` -| assets | [alpine:3.7](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/assets/Dockerfile) | Based on [minimal assets data.tar](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/assets/data.tar) within [pass-docker](https://github.com/eclipse-pass/pass-docker) -| deposit-services | [pass-package-providers docker](https://github.com/eclipse-pass/pass-package-providers) | Based on `oapass/deposit-services-providers` and should be moved to `eclipse-pass/pass-package-providers` Configurable for [JHU](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/deposit-services/repositories-jhu.json) or [Harvard](https://github.com/eclipse-pass/pass-docker/blob/178-docker-compose-gh-actions/deposit-services/repositories-harvard.json) -| notification-services | notification-services/0.1.0-3.4 | Update to to [eclipse-pass/pass-notification-services](https://github.com/eclipse-pass/pass-notification-services) -| schemaservice | oapass/schema-service:v0.6.1-3 | Update to [eclipse-pass/pass-metadata-schemas](https://github.com/eclipse-pass/pass-metadata-schemas) -| policyservice | oapass/policy-service:v0.1.2 | Update to [eclipse-pass/pass-policy-service](https://github.com/eclipse-pass/pass-policy-service) -| doiservice | oapass/doi-service:version1.0.0 | Update to [eclipse-pass/pass-doi-service](https://github.com/eclipse-pass/pass-doi-service) -| downloadservice | oapass/download-service:v1.0.2 | Update to [eclipse-pass/pass-download-service](https://github.com/eclipse-pass/pass-download-service) - -## Component CI/CD Capabilities - -The table below will document the progression from source code commits to -a production running application for the many [PASS components](../dev#components). - -| Project | Build | UTs |ITs | Deploy | -| --- | --- | --- | --- | --- | -| [pass-authz](https://github.com/eclipse-pass/pass-authz) | ✅ | ✅ | ✅ | ❓ | -| [pass-data-model](https://github.com/eclipse-pass/pass-data-model) | ❌ | ❌ | ❌ | ❓ | -| [pass-ui](https://github.com/eclipse-pass/pass-ui) | ✅ | ✅ | ❌ | ❓ | -| [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public) | ❌ | ❌ | ❌ | ❓ | -| [pass-ember-adapter](https://github.com/eclipse-pass/pass-ember-adapter) | ✅ | ✅ | ❎ | ❓ | -| [pass-indexer](https://github.com/eclipse-pass/pass-indexer) | ✅ | ✅ | ✅ | ❓ | -| [pass-policy-service](https://github.com/eclipse-pass/pass-policy-service) | ✅ | ✅ | ✅ | ❓ | -| [pass-metadata-schemas](https://github.com/eclipse-pass/pass-metadata-schemas) | ✅ | ✅ | ❌ | ❓ | -| [pass-download-service](https://github.com/eclipse-pass/pass-download-service) | ✅ | ❌ | ❎ | ❓ | -| [pass-notification-services](https://github.com/eclipse-pass/pass-notification-services) | ✅ | ✅ | ✅ | ❓ | -| [pass-doi-service](https://github.com/eclipse-pass/pass-doi-service) | ✅ | ✅ | ❎ | ❓ | -| [pass-deposit-services](https://github.com/eclipse-pass/pass-deposit-services) | ✅ | ✅ | ✅ | ❓ | -| [pass-indexer-checker](https://github.com/eclipse-pass/pass-indexer-checker) | ✅ | ✅ | ✅ | ❓ | -| [pass-java-client](https://github.com/eclipse-pass/pass-java-client) | ✅ | ✅ | ✅ | ❓ | -| [pass-messaging-support](https://github.com/eclipse-pass/pass-messaging-support) | ✅ | ✅ | ✅ | ❓ | -| [pass-package-providers](https://github.com/eclipse-pass/pass-package-providers) | ✅ | ✅ | ✅ | ❓ | - -There are a few additional project that help support Eclipse PASS but are not -part of the core application. - -| Project | Build | UTs |ITs | Deploy | -| --- | --- | --- | --- | --- | -| [pass-docker](https://github.com/eclipse-pass/pass-docker) | ❌ | ❌ | ❌ | ❓ | -| [playground](https://github.com/eclipse-pass/playground) | ❌ | ❌ | ❌ | ❓ | -| [pass-dupe-checker](https://github.com/eclipse-pass/pass-dupe-checker) | ✅ | ✅ | ❌ | ❓ | -| [pass-docker-mailserver](https://github.com/eclipse-pass/pass-docker-mailserver) | ❎ | ❌ | ❌ | ❓ | - -The following application help load data into PASS - -| Project | Build | UTs |ITs | Deploy | -| --- | --- | --- | --- | --- | -| [pass-nihms-loader](https://github.com/eclipse-pass/pass-nihms-loader) | ✅ | ✅ | ✅ | ❓ | -| [pass-grant-loader](https://github.com/eclipse-pass/pass-grant-loader) | ✅ | ✅ | ✅ | ❓ | -| [pass-journal-loader](https://github.com/eclipse-pass/pass-journal-loader) | ✅ | ✅ | ✅ | ❓ | - -The following repositories have been forked from others - -| Project | Forked From | -| --- | --- | -| [modeshape](https://github.com/eclipse-pass/modeshape) | [ModeShape](https://github.com/ModeShape/modeshape) | -| [pass-fcrepo-module-auth-rbacl](https://github.com/eclipse-pass/pass-fcrepo-module-auth-rbacl) | [birkland](https://github.com/birkland/fcrepo-module-auth-rbacl) | - -## Unit Testing - -| Project | Command | Dependencies | -| --- | --- | --- | -| pass-ember-adapter | export FEDORA_ADAPTER_INTEGRATION_TEST=0
yarn install
ember test | Ember.js Ember -| pass-ui | npm config set spin false
yarn install
ember test --test-port=4200 | NPM, Yarn, Ember.js -| pass-deposit-services | mvn test | Maven, Java -| pass-authz | mvn -pl -pass-authz-integration test | Maven, Java 8 -| pass-java-client | mvn test -DskipITs=true | Java, mvn test works if mvn package run first. -| pass-indexer | mvn test -DskipITs=true | Java -| pass-policy-service | go test ./... | Go -| pass-metadata-schemas | go test ./... | Go -| pass-download-service | go test ./... | Go -| pass-notification-services | mvn test | Java 8 -| pass-doi-service | mvn test | Java -| pass-indexer-checker | mvn test | Java -| pass-dupe-checker | go test | Go -| pass-package-providers | mvn test | Java, Mvn 3.6.3 (depending on platform, later release might work) -| pass-messaging-support | mvn test | Java -| pass-nihms-loader | mvn test | Java -| pass-grant-loader | mvn test | Java -| pass-journal-loader | mvn test | Java -| pass-fcrepo-module-auth-rbacl | mvn test | Java 8, Mvn 3.6.3 (depending on platform, later release might work) - -## Integration Testing - -| Project | Command | Dependencies | -| --- | --- | --- | -| pass-ember-adapter | docker-compose up -d
yarn install
ember test | Ember.js Ember Docker -| pass-deposit-services | mvn verify | Maven, Java, Docker -| pass-authz | mvn verify | Maven, Java 8, Docker -| pass-java-client | mvn verify | Java, Docker, mvn verify works if mvn package run first. -| pass-indexer | docker-compose up -d
mvn verify | Java, Docker -| pass-policy-service | docker-compose up -d
go test -tags=integration ./... | Go -| pass-download-service | docker-compose up -d
go test -tags=integration ./... | Go -| pass-notification-services | mvn verify | Java 8, Docker -| pass-doi-service | mvn verify | Java, Docker -| pass-indexer-checker | mvn verify | Java, Docker -| pass-package-providers | mvn verify | Java, Docker, Mvn 3.6.3 (depending on platform, later release might work) -| pass-messaging-support | mvn verify | Java, Docker -| pass-nihms-loader | mvn verify | Java, Docker -| pass-grant-loader | mvn verify | Java, Docker -| pass-journal-loader | mvn verify | Java, Docker -| pass-fcrepo-module-auth-rbacl | mvn test | Java 8, Mvn 3.6.3 (depending on platform, later release might work) - -## Building - -| Project | Command | Dependencies | -| --- | --- | --- | -| pass-ember-adapter | yarn install
ember build | Ember.js Ember -| pass-ui | yarn install
ember build | Ember.js Ember -| pass-deposit-services | mvn install | Maven, Java, Docker -| pass-authz | mvn install | Maven, Java 8, Docker -| pass-java-client | mvn install | Java, Docker -| pass-indexer | docker-compose up -d
mvn install | Java, Docker -| pass-policy-service | go generate ./...
go build ./cmd/pass-policy-service | Go -| pass-docker-mailserver | docker build -t oapass/docker-mailserver:DATE . | Docker -| pass-metadata-schemas | go build ./cmd/pass-schema-service | Go -| pass-download-service | go build | Go -| pass-notification-services | mvn install | Java, Docker -| pass-doi-service | mvn install | Java, Docker -| pass-indexer-checker | mvn install | Java, Docker -| pass-dupe-checker | go build | Go -| pass-package-providers | mvn install | Java, Docker -| pass-messaging-support | mvn install | Java, Docker -| pass-nihms-loader | mvn install | Java, Docker -| pass-grant-loader | mvn install | Java, Docker -| pass-journal-loader | mvn install | Java, Docker - -## Dependencies - -| Element | Current Version | Latest Version | Notes -| --- | --- | --- | --- | -| Go | 1.12 | [1.18](https://go.dev/dl/) | Programming language -| Java | 17 | [22](https://www.java.com/releases/) | Programming language -| Fedora | 4.7.5 | [6.1.1](https://github.com/fcrepo/fcrepo/releases) | Document store (database) for PASS -| Postgres | 13.3 | [14.2](https://www.postgresql.org/docs/release/) | Underlying storage of Fedora -| Node | 14.x | [18.x](https://nodejs.org/en/about/releases/) | JavaScript runtime, unconfirmed on desired version -| Npm | 6.14.x | [6.14.8](https://github.com/npm/npm/releases) | Package manager for node | -| Maven | 3.6.3 | [3.8.5](https://maven.apache.org/docs/history.html) | Java Package manager -| ElasticSearch| 7.13.3 | [8.1.3](https://github.com/elastic/elasticsearch/releases) | Search / indexing -| Shibboleth | 2 | [3.3.0](https://shibboleth.atlassian.net/wiki/spaces/SP3/pages/2065335693/ReleaseNotes) | Authentication -| Ember.js | 3.8 | [3.28.8](https://github.com/emberjs/ember.js/releases) | The user interface -| Ember CLI | 2.13 | [4.3.0](https://github.com/ember-cli/ember-cli/releases) | CLI for managing an ember application | -| Docker Compose | 1.29.2 | [2.5.0](https://github.com/docker/compose/releases) | CLI for managing docker containers | - -## References - -* [Deploying pass-docker via EC2](ec2.md) -* [Deploying pass-docker via Digital Ocean](digitalocean.md) -* [Deploying pass-docker via Eclipse Foundation (EF)](eclipseops.md) -* [Our attempt to use Komposer to migrate Docker Compose to k8s manifest](docker-composer-to-k8s-manifest.md) -* [Travis configs for pass-ui](https://travis-ci.org/github/OA-PASS/pass-ember/jobs/770188235/config) -* [Coveralls for pass-ui (pass-ember URL no longer valid)](https://coveralls.io/) diff --git a/docs/infra/self_hosted_github_runners.md b/docs/infra/self_hosted_github_runners.md deleted file mode 100644 index 4c8fd22..0000000 --- a/docs/infra/self_hosted_github_runners.md +++ /dev/null @@ -1,150 +0,0 @@ -# Self Hosted GitHub Runners - -A self-hosted runner is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com. -This can be useful for - -* running automation scripts -* deploying production / pre-production servers - -To use self-hosted runners, you will need to be an administrator of the GitHub project. -If you can access `/settings/actions/runners` then you have the right access permissions. - -![GitHub Settings Runners](../assets/github/github_settings_runners.png) - -## Create Self-Hosted Runner - -You can get instructions on how to add a self-hosted actions runner from the settings page (change the organization/project information in the -URL below to your organization/project). - -``` -https://github.com/eclipse-pass/pass-docker/settings/actions/runners/new -``` - -![GitHub Setup Runner Code](../assets/github/create_self_runner_code.png) - -Please refer to the latest self-hosted action runner setup instructions directly -from GitHub, what follows is a (slightly modified) copy of the -those scripts on the `/settings/actions/runners/new` page. - -### Download the tools - -Download the actions-runner code from GitHub, please -refer to `/settings/actions/runners/new` for the latest URLs and checksum value. - -```bash -mkdir -p /opt/actions-runner -(cd /opt/actions-runner && \ - curl -o actions-runner-linux-x64-2.293.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.293.0/actions-runner-linux-x64-2.293.0.tar.gz && \ - echo "06d62d551b686239a47d73e99a557d87e0e4fa62bdddcf1d74d4e6b2521f8c10 actions-runner-linux-x64-2.293.0.tar.gz" | shasum -a 256 -c && \ - tar xzf ./actions-runner-linux-x64-2.293.0.tar.gz) -``` - -Note that any service that you want to run on the self-hosted runner -must be accessible via `githubrunner` group. For example, to -run docker commands, you will need to modify the `docker` user as follows - -```bash -usermod -aG docker githubrunner -``` - -A helper script has been created (and tested on a Linux x64 server) to -help you setup your self-hosted runner. Copy and execute -[/tools/github_runner/install.sh](/tools/github_runner/install.sh) -on your self-hosted server. - -Or, you can run this directly on your server - -```bash -/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/eclipse-pass/main/main/tools/github_runner/install.sh)" -``` - -If the installation above worked, you should be able to run - -```bash -cd /opt/githubrunner && sudo -u githubrunner ./config.sh --help -``` - -And see something similar to - -```bash -Commands: - ./config.sh Configures the runner - ./config.sh remove Unconfigures the runner - ./run.sh Runs the runner interactively. Does not require any options. -``` - -### Configure Runner - -Then we create the runner and start the configuration experience. -The token information will be different. GitHub self-hosted runners -should NOT be run as root. In our example below we are using a -`githubrunner` user (to create that user, you can run `useradd githubrunner`). -If you are running in a docker environment, you probably also want to -add them to the docker group (to add that group, you can run `usermod -aG docker githubrunner`). - -```bash -sudo -u githubrunner ./config.sh \ - --unattended \ - --url https://github.com/eclipse-pass/pass-docker \ - --token $GITHUB_RUNNER_TOKEN \ - --name passdemo \ - --runnergroup default \ - --labels passdemo \ - --work pass-docker \ - --replace -``` - -And you will want the runner as a service, which is configured with - -```bash -./svc.sh install githubrunner -``` - -If you want to run it locally, then just run - -```bash -./run.sh -``` - -If you already configured a runner (i.e. the above config.sh fails), you can remove it with - -```bash -sudo -u githubrunner ./config.sh remove --token $GITHUB_RUNNER_TOKEN -``` - -## View Runner via UI - -If everything is configured correctly, then you should see the runners (assumes admin user) - -``` -https://github.com/eclipse-pass/pass-docker/settings/actions/runners -``` - -![Passdemo runner on GitHub.com](../assets/github/passdemo_runner.png) - - -## View Runners via API - -You can query this information via [GitHub's Self-Hosted Runner APIs](https://docs.github.com/en/rest/actions/self-hosted-runners) - -To run the API calls you will need a [Personal Access Token PAT](https://github.com/settings/tokens) -stored in an environment variable - -```bash -export GITHUB_TOKEN=ghp_abc123def456abc123def456abc123def456 -```` - -If you configured the above correctly, you should now see the self-hosted runner with via the API. - -```bash -curl \ - -H "Accept: application/vnd.github.v3+json" \ - -H "Authorization: token $GITHUB_TOKEN" \ - https://api.github.com/repos/eclipse-pass/pass-docker/actions/runners -``` - -## References - -* [Self-Hosted Runner Documentation](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) -* [Adding Self-Hosted Runner](https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners) -* [Self-Hosted Runner APIs](https://docs.github.com/en/rest/actions/self-hosted-runners) \ No newline at end of file