Skip to content

Latest commit

 

History

History
30 lines (21 loc) · 1022 Bytes

spark-rapids-installation.md

File metadata and controls

30 lines (21 loc) · 1022 Bytes
Raw

How to install Spark RAPIDS

  1. Download the RAPIDS jars:
$ wget https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.5.0/rapids-4-spark_2.12-0.5.0.jar
$ wget https://repo1.maven.org/maven2/ai/rapids/cudf/0.19.2/cudf-0.19.2-cuda11.jar
  • cudf-0.19.2-cuda11.jar
  • rapids-4-spark_2.12-0.5.0.jar
  1. Export the location to these jars:
export SPARK_RAPIDS_DIR=/home/mijin/sparkRapidsPlugin
export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-0.19.2-cuda11.jar
export SPARK_RAPIDS_PLUGIN_JAR=${SPARK_RAPIDS_DIR}/rapids-4-spark_2.12-0.5.0.jar
export MASTER_HOST=server01
  1. Install the GPU discovery script:
$ wget https://github.com/apache/spark/blob/master/examples/src/main/scripts/getGpusResources.sh

Reference