Skip to content

Commit

Permalink
Merge pull request #6 from bmeg/feature/tests
Browse files Browse the repository at this point in the history 
Adds local tests with mock auth and updates readme docs
  • Loading branch information
@matthewpeterkort
matthewpeterkort authored Nov 15, 2024
2 parents 150ee06 + 5f14339 commit e2f74c2
Show file tree
Hide file tree
Showing 25 changed files with 772 additions and 370 deletions.
38 changes: 38 additions & 0 deletions .github/workflows/build.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Build and publish grip graphql plugin server image

on:
push:
workflow_dispatch:

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v4

- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_ROBOT_TOKEN }}

- name: Build and push image
run: |
# Set Image tag to the branch name
BRANCH=$(echo ${GITHUB_REF#refs/*/} | tr / _)
REPO=quay.io/ohsu-comp-bio/grip-caliper
echo "Setting image tag to $REPO:$BRANCH"
# Login to Quay.io and build image
docker login quay.io
docker build -t $REPO:$BRANCH .
# Add 'latest' tag to 'main' image
if [[ $BRANCH == 'main' ]]; then
docker image tag $REPO:main $REPO:latest
fi
# Push the tagged image to Quay.io
docker push --all-tags $REPO
135 changes: 135 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# IDE
.idea/

# scratch
tmp/
85 changes: 6 additions & 79 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,84 +1,11 @@
### Grip Graphql Plugins

# GRIP GraphQL Endpoint
Configurable GraphQL endpoint for the GRaph Integration Platform
grip-graphql is a collection of go plugins designed and implemented for connecting a [Grip](https://github.com/bmeg/grip) server to other microservices in a modified [Gen3](https://gen3.org/) software stack.

gen3_writer directory contains a [Gin](https://github.com/gin-gonic/gin) go server plugin that is used for Writing / Deleting data from Graphs on a grip server

## Build
gripgraphql directory contains a graphql based read query plugin that uses a [goja](https://github.com/dop251/goja) engine to read from a static schema defined as a config file to create custom graphql queries that can be used to abstract Grip's complex query language into a more digestible query format for the frontend to use.

Run `make` to build both the plugin (grip-graphql-endpoint.so) and proxy server (grip-graphql-proxy).
graphql_gen3 is a legacy implementation of a reader reader plugin using a more traditional graphql schema builder.


## Plugin
Run as a shared object within the GRIP server
```
grip server -w graphql=grip-graphql-endpoint.so -l graphql:config=./config/config.js -l graphql:graph=test-db
```

## Proxy
Run the server as a proxy endpoint connected to an external GRIP service
```
./grip-graphql-proxy <grip server> <server port> <config> <database>
```

## Example configuration file

```javascript

endpoint.add({
name: "projects",
schema: [
"String"
],
handler: (G, args) => {
return G.V().hasLabel("Project").render("_gid").toList()
}
})

endpoint.add({
name: "cases",
schema: [
"String"
],
args: {
offset: "Int",
limit: "Int",
project_id : "String"
},
defaults: {
offset: 0,
limit: 100
},
handler: (G, args) => {
if (args.project_id === undefined) {
return G.V().hasLabel("Case").skip(args.offset).limit(args.limit).render("_gid").toList()
} else {
return G.V().hasLabel("Case").has(gripql.eq("project_id", args.project_id)).skip(args.offset).limit(args.limit).render("_gid").toList()
}
}
})

endpoint.add({
name: "caseCounts",
schema: {
cases: "Int",
samples: "Int",
aliquots: "Int"
},
args: {
project_id: "String"
},
handler: (G, args) => {
return {
"cases": G.V().hasLabel("Case").has(gripql.eq("project_id", args.project_id)).count().toList()[0],
"samples": G.V().hasLabel("Case").has(gripql.eq("project_id", args.project_id)).out("samples").count().toList()[0],
"aliquots": G.V().hasLabel("Case").has(gripql.eq("project_id", args.project_id)).out("samples").out("aliquots").count().toList()[0],
}
}
})
```


## GRIP setup with frontend framework

1. setup frontendframework see docs
2. grip server -c mongo.yml -w graphql=grip-graphql-endpoint.so -l graphql:config=config/gen3.js -l graphql:graph=gdc
See ./gen3_writer for tests and additional documentation
101 changes: 72 additions & 29 deletions gen3_writer/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,59 +1,102 @@
# GRIP FHIR RESTFUL API

## Setup

Create a grip executable or build one locally by cloning grip main branch and
do a:

```
export PATH=$PATH:$HOME/go/bin
go build .
go install .
```
## Local Setup

```
go install github.com/bmeg/grip
go build --buildmode=plugin ./gen3_writer
grip server -w api/writer=gen3_writer.so
grip server -w graphql=gen3_writer.so
```

Note: the -w api/writer=gen3_writer.so option can be stacked to include multiple endpoints and long as you have
built the .so file to the corresponding endplint as shown above.
Note: the -w graphql=gen3_writer.so option can be stacked to include multiple endpoints and long as you have
built the .so file to the corresponding endplint as shown above. Ex:

`grip server -c mongo.yml -w graphql=gen3_writer.so -w reader=grip-graphql-endpoint.so -l reader:config=./config/gen3.js -l reader:graph=CALIPER`

## Example queries:

Note: ENV var ACCESS_TOKEN is a valid Gen3 jwt token. An access token is needed for all queries except GET \_status and GET list-graphs

## Example queries:
### Delete an edge and then grep for it to see if it has been deleted or not. Format:

_http://localhost:8201/graphql/[graph-name]/del-edge/[edge-id]/[gen3-project-id]_

Delete an edge and then grep for it to see if it has been deleted or not
```
curl -X DELETE -H "Content-Type: applicationjson" http://localhost:8201/api/writer/test/del-edge/2XM1hqErQvIw9s0cUWDuRpKPbQR
grip query test "E()" | grep 2XM1hqErQvIw9s0cUWDuRpKPbQR
curl -X DELETE http://localhost:8201/graphql/CALIPER/del-edge/fb60e763-e799-4d59-82a3-66977cc6696c/ohsu-test
-H "Content-Type: applicationjson" \
-H "Authorization: bearer $ACCESS_TOKEN"
grip query CALIPER "E()" | grep fb60e763-e799-4d59-82a3-66977cc6696c
```

Bulk load some edges from an edge file:
### Bulk load some edges from a file:

_http://localhost:8201/graphql/[graph-name]/bulk-load/[gen3-project-id]_

```
curl -X POST -H "Content-Type: applicationjson" -d '{"edge": "../../aced-data/grip-aced-data/edge.ndjson"}' http://localhost:8201/api/writer/test/bulk-load
curl -X POST "http://localhost:8201/graphql/CALIPER/bulk-load/ohsu-test" \
-H "Authorization: bearer $ACCESS_TOKEN" \
-F "types=edge" \
-F "[email protected]"
```

Note: Each edge in edge file above will be of format:
Newline delimited edges should be of form:

```
{
"label": "custodian",
"gid": "bee5bd86-4f06-5eb2-b71a-f62110cf5aa9",
"label": "specimen_observation",
"from": "9bc10566-5d7e-4a53-bbc0-6fe9700584a5",
"to": "Organization/ea4ea5e7-2780-46cf-8cc4-fbb40ad63928"
"to": "ea4ea5e7-2780-46cf-8cc4-fbb40ad63928"
}
```
With required keys "label", "from" and "to"

Get the value of the vertex with id 302324d5-1d92-5425-80d5-ac6c63af84b6
With required keys "label", "from", "to", and "gid" and optional key "data" with value of type dict.

### Get the data from a vertex given a known vertex id

_http://localhost:8201/graphql/[graph-name]/get-vertex/[vertex-id]/[gen3-project-id]_

```
curl -X GET http://localhost:8201/graphql/CALIPER/get-vertex/875ddaf8-42da-5d72-b5c5-39c2b16151cd/ohsu-test \
-H "Authorization: bearer $ACCESS_TOKEN"
```
curl -X GET http://localhost:8201/api/graphql/test/get-vertex/302324d5-1d92-5425-80d5-ac6c63af84b6

### Get the list of graphs present

```
curl http://localhost:8201/graphql/list-graphs
```

### Revproxy Setup --

The above curl commands assume that you are acessing this grip plugin from within the cluster. equivalent queries can be used from outside the cluster by changing the nginx paths to the form:

`https://[your_instance_endpoint]/grip/writer/graphql/list-graphs` for the writer or
`https://[your_instance_endpoint]/grip/reader` for the reader api

These paths assume you have checked out to the grip branch of helm and reployed

## Tests:

Tests can be run locally by specifying that you want to turn on the plugin in testing mode using the `TEST` Graph. For example:

Get the list of graphs present
```
curl -X GET http://localhost:8201/api/graphql/list-graphs
grip server -w graphql=gen3_writer.so \
-w reader=grip-graphql-endpoint.so \
-l reader:config=./config/gen3.js \
-l reader:graph=TEST \
-l graphql:test=true \
-l reader:test=true
```

then cd to gen3_writer directory and run:

`go test` or `go test -v` for logs or `go test -run [specific_test_name]` to run only a specific test

If the graph name is `TEST` and the config is setup for test=true, mock auth will be used, and these tests can be run locally outside of a gen3 instance.

## Version
go version go1.21.3

## Tests: not currently functional.
go version go1.22.6
Loading

0 comments on commit e2f74c2

Please sign in to comment.