Skip to content

Commit

Permalink
Merge changes form main refs #15907
Browse files Browse the repository at this point in the history
  • Loading branch information
AshviniKabadi committed Aug 14, 2024
2 parents 13b56ad + a184cc8 commit 1eac9fd
Show file tree
Hide file tree
Showing 17 changed files with 103 additions and 664 deletions.
16 changes: 1 addition & 15 deletions .env.dist
Original file line number Diff line number Diff line change
@@ -1,16 +1,2 @@
RUBY_VERSION=3.2.2
RAILS_VERSION=7.1.3.4
VIEW_COMPONENT_VERSION=2.6.6
SOLR_VERSION=9.3.0
ZK_VERSION=3.8.2
NODE_MAJOR_VERSION=16
SOLR_PASS=UALibrariesRawk
SOLR_USER=solr
SOLR_PORT=8984
GOB_PORT=3000
APP_USER=gob
APP_PASS=gobb13
DEPLOY_SSH_KEY_PATH="/home/gob/.ssh/ed25519"
DGID=1000
DUID=1000
DEPLOY_SSH_KEY_PATH="/user/.ssh/ed25519" # In lando, you're home directory is mounted to `/user` inside the container by default
SLACK_WEB_HOOK="" # Get from stache
37 changes: 37 additions & 0 deletions .lando.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: ual-geoblacklight
proxy:
appserver:
- geo.lndo.site:3000
solr:
- solr.lndo.site:8983
services:
appserver:
type: compose
services:
build:
context: .
dockerfile: ./docker/app/Dockerfile
command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
environment:
SOLR_URL: "http://solr:8983/solr/blacklight-core/"
build:
- bin/rails db:migrate RAILS_ENV=development
solr:
type: solr:9
core: blacklight-core
config:
dir: ./solr/conf

tooling:
rails:
service: appserver
cmd: bundle exec rails
bundle:
service: appserver
rake:
service: appserver
cap:
service: appserver
ual_data_load:
service: appserver
cmd: rake "ual_docs:load" "ual_docs:migrate"
93 changes: 40 additions & 53 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,21 @@

## Overview

This Docker orchestration runs a fully decoupled GeoBlacklight search service that uses a Blacklight (Ruby on Rails) application querying the Solr & ZooKeeper middleware "ensemble".
This Docker orchestration runs a fully decoupled GeoBlacklight search service that uses a Blacklight (Ruby on Rails) application querying the Solr.

**How does it work? 30K ft view...**

Here's a little diagram of the GBL and Solr Cloud interaction (Mermaid syntax):
Here's a little diagram of the GBL and Solr interaction (Mermaid syntax):

```mermaid
graph LR;
A[Geoblacklight]==>|QueryRequest|B{CloudSolr};
B{CloudSolr}-->D(Zookeeper);
D(Zookeeper)-->B{CloudSolr};
B{CloudSolr}==>|DataResponse|A[Geoblacklight];
A[Geoblacklight]==>|QueryRequest|B{Solr};
B{Solr}==>|DataResponse|A[Geoblacklight];
```

The GBL app container queries the Solr instance directly. "Sharding" information is managed by the ZooKeeper middleware. Using the "blacklight-core" metadata, Solr sends back its data response in a format that can be ingested by RoR models.
The GBL app container queries the Solr instance directly. Using the "blacklight-core" metadata, Solr sends back its data response in a format that can be ingested by RoR models.

## Setup

Expand All @@ -28,33 +26,31 @@ Here's a little diagram of the GBL and Solr Cloud interaction (Mermaid syntax):
- Terminal access
- Git
- Docker, either Desktop or just the straight engine if deploying on any of the Linux distros
- Lando

**Preliminary Steps:**

Start by cloning this repository to pretty much anywhere on most filesystems that the Docker daemon has access to. There may be filesystem permission issues for your user. Either find your user's UID and GID or ask another IT compadre if they know what the heck I'm talking about. Put those ids in the `.env` file at the root of the project, replacing the UID and GID, respectively. Then proceed to ...
Start by cloning this repository to pretty much anywhere on most filesystems that the Docker daemon has access to.

**1. Build the UAL-GBL Docker images (for GBL and Solr containers):**
**Build the UAL-GBL Docker images (for GBL and Solr containers):**

```shell
$ ./dbuild.sh
$ lando start
```

**2. Start or rebuild the Docker network:**
**Re-build image and containers while preserving volumes**

```shell
$ ./start-me-up.sh
$ lando rebuild
```

The GBL app is installed automatically if it does not already exist. This will take a little while and the server is still not started. A list of dependencies should print out as the GBL is installed.
**Ingest test docs and UAL OGM records from Github (runs both "ual_docs:load" and "ual_docs:migrate" rake tasks)**

Build scripts set up Apache ZooKeeper and Solr decoupled to propagate search configuration and data in "cloud mode". Search data is located in Docker volumes on startup.

**3. View the application and Solr admin:**

See the following URLs:
```shell
$ lando ual_data_load
```

* `localhost:3000` for the GeoBlacklight application
* `localhost:8984` for the Solr admin (admin is locked down; see the .env file for creds)
**Note**: If you're using Docker Desktop on a Mac, and encounter the following error: `ERROR ==> connect ENOENT /var/run/docker.sock`, try toggling the checkbox for `Allow the default Docker socket to be used (requires password)` and re-running `lando start`.

## Deployment

Expand All @@ -63,27 +59,15 @@ We use [Capistrano](https://capistranorb.com/) for deployments. Deployments are
Here are the steps to deploy to production:

0. Make sure you're on the Library network (on site or using VPN)
1. SSH into the `ual_gob_app` service
1. Run the Capistrano deployment command

```shell
docker compose exec ual_gob_app bash
```

2. Change directory to `/geoblacklight/app` inside the container

```shell
cd /geoblacklight/app
```

3. Run the Capistrano deployment command

```shell
cap production deploy
lando cap production deploy
```
This will deploy the `main` branch to production. Alternatively you can provide a branch name to deploy:

```shell
cap production deploy BRANCH=some_branch_name
lando cap production deploy BRANCH=some_branch_name
```

You can check the status of the deployment in the `#tess-dev-deployer` Slack channel in the UAL Slack workspace.
Expand All @@ -95,22 +79,25 @@ You can check the status of the deployment in the `#tess-dev-deployer` Slack cha
This is non-destructive. All containers remain stateful, as well as volumes and network.

```shell
$ ./start-me-up.sh pause
$ lando stop
```

**Run Rake commands in the containerized application directory:**

```shell
$ docker exec -it gob-app bash -c -l './rake_command.sh "<command-to-run>"'
$ lando rake "<command-to-run>"
# View all available rake tasks
$ lando rake --tasks
# Populate default Solr test fixtures:
$ docker exec -it gob-app bash -c -l './rake_command.sh "geoblacklight:index:seed[:remote]"'
$ lando rake "geoblacklight:index:seed[:remote]"
# Ingest UAL test docs:
$ docker exec -it gob-app bash -c -l './rake_command.sh "ual_docs:load"'
$ lando rake "ual_docs:load"
# Ingest all UAL OGM records from Github:
$ docker exec -it gob-app bash -c -l './rake_command.sh "ual_docs:migrate"'
$ lando rake "ual_docs:migrate"
```

See Geoblacklight tasks [here](https://github.com/geoblacklight/geoblacklight/blob/main/lib/tasks/geoblacklight.rake).
Expand All @@ -134,8 +121,8 @@ See Geoblacklight tasks [here](https://github.com/geoblacklight/geoblacklight/bl
**Inspect GBL application logs for development**
```shell
# number of lines to watch can be set, defaults to 10
$ ./inspect-app-logs.sh 50
# number of lines to watch can be set
$ lando logs
```
**Run Solr queries for development and testing**
Expand All @@ -151,10 +138,18 @@ Also see the Solr query screen in the Solr admin: https://solr.apache.org/guide/
**Tear-down**
WARNING: This destroys _all_ data, meaning containers and volumes. (It does not remove Docker images, however.) The dialogue will ask if you want to delete all files in the tmp directory, log directory, and sqlite files.
WARNING: This destroys _all_ data, meaning containers and volumes. (It does not remove Docker images, however.)
```shell
$ lando destroy
```
This does not remove images. If you want to remove images you can run `docker image prune -a` wich will **remove all unused images**.
To also remove any app related files that are not code (tmp directory files, SQLite file, etc) you can run:
```shell
$ ./destroy.sh
./rm_app_files.sh
```
## Notes
Expand All @@ -163,7 +158,6 @@ $ ./destroy.sh
* https://github.com/geoblacklight/geoblacklight
* https://github.com/geobtaa/geoportal-solr-config
* https://github.com/projectblacklight/blacklight
* https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-ensemble.html
* https://github.com/docker-solr/docker-solr/tree/master/scripts
* https://github.com/geoblacklight/geoblacklight/tree/main/spec/fixtures/solr_documents
* https://github.com/OpenGeoMetadata/edu.berkeley/blob/master/ark28722/s7/059k/geoblacklight.json (example geoblacklight.json file in GBL ver.1 schema)
Expand All @@ -173,17 +167,10 @@ $ ./destroy.sh
## Helpful hints
* Software versions are controlled in `.env`, as are a few other important environment variables.
* Blacklight-core metadata and config are stored in the `solr/conf` directory, which is mounted into the Solr container.
## Rails Console
If you need to access the Rails Console, you can do so by attaching the `ual-goblight:latest` container (if you are using VSCode and Docker plugin).
After attaching the container:
1. `cd` into `/geoblacklight/app`
2. Run `rails c` to fire up the console
To access rails console run `lando rails c` to fire up the console
### Helpful Rails Console Commands
Expand Down
23 changes: 0 additions & 23 deletions dbuild.sh

This file was deleted.

66 changes: 0 additions & 66 deletions docker-compose.yml

This file was deleted.

Loading

0 comments on commit 1eac9fd

Please sign in to comment.