diff --git a/.gradient/available_ipus.py b/.gradient/available_ipus.py deleted file mode 100644 index 0ad41ca..0000000 --- a/.gradient/available_ipus.py +++ /dev/null @@ -1,12 +0,0 @@ -# Copyright (c) 2022 Graphcore Ltd. All rights reserved. - -import json -import subprocess - -j = subprocess.check_output(["gc-monitor", "-j"]) -data = json.loads(j) -num_ipuMs = len(data["cards"]) -num_ipus = 4 * num_ipuMs - -# to be captured as a variable in the bash script that calls this python script -print(num_ipus) diff --git a/.gradient/launch_vscode_server.sh b/.gradient/launch_vscode_server.sh deleted file mode 100644 index 0558b74..0000000 --- a/.gradient/launch_vscode_server.sh +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash -# Copyright (c) 2023 Graphcore Ltd. All rights reserved. - -cd /notebooks - -CODE_FILE=/storage/.vscode/code -if [ ! -f "$CODE_FILE" ]; then - mkdir -p /storage/.vscode/server - curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output /storage/vscode_cli.tar.gz - tar -xf /storage/vscode_cli.tar.gz -C /storage/.vscode -fi - -ln -s /storage/.vscode/server $HOME/.vscode-server - -pip install -r requirements-dev.txt - -$CODE_FILE tunnel --accept-server-license-terms --name=${1:-ipu-paperspace} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cc82fc0..6a0f1fb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,36 +4,6 @@ You can contribute to the development of the BESS-KGE project, even if you don't have access to IPUs (you can use the [IPUModel](https://docs.graphcore.ai/projects/poptorch-user-guide/en/3.2.0/reference.html#poptorch.Options.useIpuModel) to emulate most functionalities of the physical hardware). -## VS Code server on Paperspace - -Setting up a VS Code server on [Paperspace](https://www.paperspace.com/graphcore) will allow you to tunnel into a machine with IPUs from the VS Code web editor or the desktop app. This requires minimum effort and is an excellent solution for developing and testing code directly on IPU hardware. Here's how to do it. - -1. Fork the [BESS-KGE repository](https://github.com/graphcore-research/bess-kge). - -2. You can launch a 6-hours session on a Paperspace machine with access to 4 IPUs **for free** by using a link of the form: - ``` - https://console.paperspace.com/github/{USERID}/{REPONAME}?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4 - ``` - - where `{USERID}/{REPOPNAME}` is the github address of the forked repository (e.g. `graphcore-research/bess-kge` for the original repo). - -3. Start the machine (this will also clone the repo for you) and open up a terminal from the left pane. - - ![terminal_pane](docs/source/images/Terminal3.png) - -4. In the terminal, run the command - ```shell - bash .gradient/launch_vscode_server.sh {tunnel-name} - ``` - - where `tunnel-name` is an optional argument that you can use to define the name of the remote tunnel (if not set, it will default to `ipu-paperspace`). The script will download and install all dependencies and start the tunnel. - -5. When asked, authorize the tunnel through GitHub (with an account having writing privileges to the forked repository). You will be then provided with the tunnel link. Please refer to [this notebook](https://ipu.dev/fmo4AZ) for additional details on these steps and to connect the VS Code desktop app to the remote tunnel. - -6. Once VS Code is connected to the Paperspace machine, run `./dev build` to build all custom ops. You are now ready to start developing! - -When closing a session and stopping the Paperspace machine, remember to unregister the tunnel in VS Code as explained in the "Common Issues" paragraph of the [notebook](https://ipu.dev/fmo4AZ). To resume your work, just access the clone of the BESS-KGE repo in the "Projects" section of your Paperspace profile, start a new machine and repeat the operations above. All code changes to the local repo, as well as VS Code settings and extensions installed, will persist across sessions. - ## Setup on local machine To develop on a local machine, first install the Poplar SDK following the instructions in the [Getting Started guide for your IPU system](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started). diff --git a/README.md b/README.md index 20cd1ff..5cba63f 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ ![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square) [**Installation guide**](#usage) -| [**Tutorials**](#paperspace-notebook-tutorials) +| [**Tutorials**](#notebook-tutorials) | [**Documentation**](https://graphcore-research.github.io/bess-kge/) @@ -82,7 +82,7 @@ All APIs are documented in the [BESS-KGE API documentation](https://graphcore-re BESS-KGE provides built-in dataloaders for the following datasets. Notice that the use of these datasets is at own risk and Graphcore offers no warranties of any kind. It is the user's responsibility to comply with all license requirements for datasets downloaded with dataloaders in this repository. -| Dataset | Builder method | Entities | Entity types | Relation types | Triples | License | +| Dataset | Builder method | Entities | Entity types | Relation types | Triples | License | | --- | --- | --- | --- | --- | --- | --- | | [ogbl-biokg](https://ogb.stanford.edu/docs/linkprop/#ogbl-biokg) | [KGDataset.build_ogbl_biokg](https://graphcore-research.github.io/bess-kge/generated/besskge.dataset.KGDataset.html#besskge.dataset.KGDataset.build_ogbl_biokg) | 93,773 | 5 | 51 | 5,088,434 | CC-0 | | [ogbl-wikikg2](https://ogb.stanford.edu/docs/linkprop/#ogbl-wikikg2) | [KGDataset.build_ogbl_wikikg2](https://graphcore-research.github.io/bess-kge/generated/besskge.dataset.KGDataset.html#besskge.dataset.KGDataset.build_ogbl_wikikg2) | 2,500,604 | 1 | 535 | 16,968,094 | CC-0 | @@ -183,14 +183,14 @@ pip install git+https://github.com/graphcore-research/bess-kge.git import besskge ``` -## Paperspace notebook tutorials +## Notebook tutorials For a walkthrough of the `besskge` library functionalities, see our Jupyter notebooks. We recommend the following sequence: -1. [KGE training and inference on the OGBL-BioKG dataset](notebooks/1_biokg_training_inference.ipynb) [![Run on Gradient](docs/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F1_biokg_training_inference.ipynb) -2. [Link prediction on the YAGO3-10 dataset](notebooks/2_yago_topk_prediction.ipynb) [![Run on Gradient](docs/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F2_yago_topk_prediction.ipynb) -3. [FP16 weights and compute on the OGBL-WikiKG2 dataset](notebooks/3_wikikg2_fp16.ipynb) [![Run on Gradient](docs/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F3_wikikg2_fp16.ipynb) +1. [KGE training and inference on the OGBL-BioKG dataset](notebooks/1_biokg_training_inference.ipynb) +2. [Link prediction on the YAGO3-10 dataset](notebooks/2_yago_topk_prediction.ipynb) +3. [FP16 weights and compute on the OGBL-WikiKG2 dataset](notebooks/3_wikikg2_fp16.ipynb) -For pointers on how to run BESS-KGE on a custom Knowledge Graph dataset, see the notebook [Using BESS-KGE with your own data](notebooks/0_custom_KG_dataset.ipynb) [![Run on Gradient](docs/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F0_custom_KG_dataset.ipynb) +For pointers on how to run BESS-KGE on a custom Knowledge Graph dataset, see the notebook [Using BESS-KGE with your own data](notebooks/0_custom_KG_dataset.ipynb) ## Contributing @@ -205,4 +205,4 @@ Copyright (c) 2023 Graphcore Ltd. Licensed under the MIT License. The included code is released under the MIT license (see [details of the license](LICENSE)). -See [notices](NOTICE.md) for dependencies, credits, derived work and further details. \ No newline at end of file +See [notices](NOTICE.md) for dependencies, credits, derived work and further details. diff --git a/dev b/dev index b1d14c1..b275c71 100755 --- a/dev +++ b/dev @@ -104,7 +104,7 @@ def copyright() -> None: """check for Graphcore copyright headers on relevant files""" command = ( "find Makefile " - + " ".join(PYTHON_ROOTS + ["besskge/custom_ops", ".gradient/"]) + + " ".join(PYTHON_ROOTS + ["besskge/custom_ops"]) + " -type f -not -name *.pyc" " | xargs grep -L 'Copyright (c) 202. Graphcore Ltd[.] All rights reserved[.]'" ) diff --git a/docs/gradient-badge.svg b/docs/gradient-badge.svg deleted file mode 100644 index 51036d0..0000000 --- a/docs/gradient-badge.svg +++ /dev/null @@ -1 +0,0 @@ -gradient-badge-gradient-05-d-05 diff --git a/docs/source/user_guide.rst b/docs/source/user_guide.rst index 012a8c9..17f3401 100644 --- a/docs/source/user_guide.rst +++ b/docs/source/user_guide.rst @@ -47,11 +47,6 @@ We recommend the following sequence: 2. `Link prediction on the YAGO3-10 dataset `_. 3. `FP16 weights and compute on the OGBL-WikiKG2 dataset `_. -.. |run_on_gradient| image:: ../gradient-badge.svg - -Click on the |run_on_gradient| button inside the notebooks to run them **for free** on physical IPUs -available on `Paperspace `_. - Limitations ------------------------ diff --git a/notebooks/0_custom_KG_dataset.ipynb b/notebooks/0_custom_KG_dataset.ipynb index c88405f..5aa0edd 100644 --- a/notebooks/0_custom_KG_dataset.ipynb +++ b/notebooks/0_custom_KG_dataset.ipynb @@ -16,18 +16,6 @@ "As an example, we will download and build the [OGBL-BioKG](https://ogb.stanford.edu/docs/linkprop/#ogbl-biokg) biomedical knowledge graph. While BESS-KGE provides a built-in dataloader for this dataset (see [besskge.dataset.KGDataset.build_ogbl_biokg](https://graphcore-research.github.io/bess-kge/generated/besskge.dataset.KGDataset.html#besskge.dataset.KGDataset.build_ogbl_biokg)), in this notebook — for didactic purposes — we will show how to import the dataset from scratch." ] }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Environment setup\n", - "\n", - "While this notebook doesn't contain any code that needs to be run on IPUs or other accelerating hardware, the best way to run it is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - " [![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F0_custom_KG_dataset.ipynb)" - ] - }, { "attachments": {}, "cell_type": "markdown", @@ -828,7 +816,7 @@ "print(\n", " biokg.ht_types[\"train\"].shape,\n", " biokg.ht_types[\"valid\"].shape,\n", - " biokg.ht_types[\"test\"].shape\n", + " biokg.ht_types[\"test\"].shape,\n", ")" ] }, diff --git a/notebooks/1_biokg_training_inference.ipynb b/notebooks/1_biokg_training_inference.ipynb index 3e7a78f..cf63752 100644 --- a/notebooks/1_biokg_training_inference.ipynb +++ b/notebooks/1_biokg_training_inference.ipynb @@ -21,11 +21,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - " [![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F1_biokg_training_inference.ipynb)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/notebooks/2_yago_topk_prediction.ipynb b/notebooks/2_yago_topk_prediction.ipynb index ff9e0ef..a48db9d 100644 --- a/notebooks/2_yago_topk_prediction.ipynb +++ b/notebooks/2_yago_topk_prediction.ipynb @@ -31,11 +31,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F2_yago_topk_prediction.ipynb)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/notebooks/3_wikikg2_fp16.ipynb b/notebooks/3_wikikg2_fp16.ipynb index 26de2a0..3645f8b 100644 --- a/notebooks/3_wikikg2_fp16.ipynb +++ b/notebooks/3_wikikg2_fp16.ipynb @@ -25,11 +25,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://console.paperspace.com/github/graphcore-research/bess-kge?container=graphcore%2Fpytorch-paperspace%3A3.3.0-ubuntu-20.04-20230703&machine=Free-IPU-POD4&file=%2Fnotebooks%2F3_wikikg2_fp16.ipynb)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/setup.sh b/setup.sh deleted file mode 100755 index 3edcd03..0000000 --- a/setup.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash -# Copyright (c) 2022 Graphcore Ltd. All rights reserved. -# Script to be sourced on launch of the Gradient Notebook - -DETECTED_NUMBER_OF_IPUS=$(python .gradient/available_ipus.py) -if [[ "$1" == "test" ]]; then - IPU_ARG="${DETECTED_NUMBER_OF_IPUS}" -else - IPU_ARG=${1:-"${DETECTED_NUMBER_OF_IPUS}"} -fi - -export NUM_AVAILABLE_IPU=${IPU_ARG} -export GRAPHCORE_POD_TYPE="pod${IPU_ARG}" - -export POPLAR_EXECUTABLE_CACHE_DIR="/tmp/exe_cache" -export DATASET_DIR="/tmp/dataset_cache" -export CHECKPOINT_DIR="/tmp/checkpoints" - -# mounted public dataset directory (path in the container) -# in the Paperspace environment this would be ="/datasets" -export PUBLIC_DATASET_DIR="/datasets" - -export POPTORCH_CACHE_DIR="${POPLAR_EXECUTABLE_CACHE_DIR}" -export POPTORCH_LOG_LEVEL=ERR -export RDMAV_FORK_SAFE=1 - -export PIP_DISABLE_PIP_VERSION_CHECK=1 CACHE_DIR=/tmp -jupyter lab --allow-root --ip=0.0.0.0 --no-browser --ServerApp.trust_xheaders=True \ - --ServerApp.disable_check_xsrf=False --ServerApp.allow_remote_access=True \ - --ServerApp.allow_origin='*' --ServerApp.allow_credentials=True \ No newline at end of file