How to correctly install Spark NLP on Windows 8 and 10

[maziyarpanahi]

The steps:

• Download OpenJDK from here: https://adoptopenjdk.net/?variant=openjdk8&jvmVariant=hotspot
  • Make sure it is 64-bit
  • Make sure you install it in the root C:\java Windows doesn't like space in the path.
  • During installation after changing the path, select setting Path
• Download winutils and put it in C:\hadoop\bin https://github.com/cdarlint/winutils/blob/master/hadoop-2.7.3/bin/winutils.exe
• Download Anaconda 3.6 from Archive, I didn't like the new 3.8 : https://repo.anaconda.com/archive/Anaconda3-2020.02-Windows-x86_64.exe
• Download Apache Spark 2.4.6 and extract it in C:\spark\
• Set the env for HADOOP_HOME to C:\hadoop and SPARK_HOME to C:\spark
• Set Paths for %HADOOP_HOME%\bin and %SPARK_HOME%\bin
• Install Microsoft Visual C++ 2010 Redistributed Package (x64) https://www.microsoft.com/en-us/download/details.aspx?id=26999
• Create C:\tmp and C:\tmp\hive
• Fix permissions:
  • C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/hive
  • C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/

Either create a conda env for python 3.6, install pyspark==2.4.6 spark-nlp numpy and use Jupyter/python console, or in the same conda env you can go to spark bin for pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.5.

[cpoptic]

Thank you for this guide. Does sparknlp only work with Java version 8? Does it work with Java 15.0?

Also is there a Docker container alternative to install sparknlp environment on a Windows machine?

[maziyarpanahi]

Hi @cpoptic

Apache Spark 2.3 and 2.4 require only Java 8 so that is one of the requirements. We don't officially provide a Docker container but it's pretty simple to make a Docker container based on Ubuntu or other Linux images as long as you set it up correctly. I'll make another post for Docker setup, not officially supported by us but it could be a good start.

Update: Hope this could be a good start for setting up Docker container: #1714

[resul-ai]

Fix permissions:
C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/hive
C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/
According to above explanation those ../tmp/.. folders should be temp.

[maziyarpanahi]

Thanks @ResulCaliskan
I made a mistake here: Create C:\temp and C:\temp\hive it should have been Create C:\tmp and C:\tmp\hive which I fixed. The rest is correct now:

- Create C:\tmp and C:\tmp\hive
- Fix permissions:
  - C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/hive
  - C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/

[ESteanes]

Something that I also had to add in order to successfully install Pyspark was to add the following environment variable

PYSPARK_PYTHON=python

Otherwise, when I tried launching PySpark from the command line, it would always say that it is unable to find the python executable

[ESteanes]

I also had a couple of suggestions to possibly add to the main guide:

• Perhaps adjust the wording of what hadoop version to download to get the user to check what is appropriate for the version of Spark?
  • I installed Spark 3.1.2 and it took me a while to finally deduce that one of the issues was that I needed Hadoop 3.2+ from https://github.com/cdarlint/winutils, not hadoop 2.7
• I also found that using a venv environment worked fine on Python 3.9.7
• So far, I didn't have to do anything special about changing the permissions for the /tmp directories

But overall thank you very much for the guide!

[nobubblegums]

Thanks for the hadoop heads-up, my experience was pretty much the same as yours, except I ran python 3.8. Another thing to note if anyone's having problems, be sure to set environment variables for pyspark. You can either do this in your virtual/conda environment, I guess it would also work on Windows level. For conda, after activating your environment:
conda env config vars set PYSPARK_PYTHON=python (re-activate to apply this, do not leave spaces before&after "=")
Some people also needed to add PYSPARK_DRIVER_PYTHON=jupyter , worked fine for me without it though.