Skip to content

Commit

Permalink
Add python documentation (#988)
Browse files Browse the repository at this point in the history
  • Loading branch information
robzor92 authored Jan 3, 2022
1 parent e61b122 commit 7dcdb2f
Show file tree
Hide file tree
Showing 41 changed files with 202 additions and 52 deletions.
File renamed without changes.
File renamed without changes.
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added docs/assets/images/python/conflicts.png
Binary file added docs/assets/images/python/create_env.png
Binary file added docs/assets/images/python/export_env.png
Binary file added docs/assets/images/python/install_dep.gif
Binary file added docs/assets/images/python/install_git.gif
Binary file added docs/assets/images/python/install_search.gif
Binary file added docs/assets/images/python/list.png
Binary file added docs/assets/images/python/navigate.gif
Binary file added docs/assets/images/python/remove_env.png
Binary file added docs/assets/images/python/uninstall.png
40 changes: 20 additions & 20 deletions docs/compute/jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ sections below.

<p align="center">
<figure>
<a href="../../assets/images/newjob1.png">
<img src="../../assets/images/newjob1.png" alt="New job quickstart">
<a href="../../assets/images/jobs/newjob1.png">
<img src="../../assets/images/jobs/newjob1.png" alt="New job quickstart">
</a>
<figcaption>New job quickstart</figcaption>
</figure>
Expand All @@ -45,8 +45,8 @@ For example, in the image below the default arguments "a b c" will be passed as

<p align="center">
<figure>
<a href="../../assets/images/newjob2.png">
<img src="../../assets/images/newjob2.png" alt="Job default arguments">
<a href="../../assets/images/jobs/newjob2.png">
<img src="../../assets/images/jobs/newjob2.png" alt="Job default arguments">
</a>
<figcaption>Job default arguments</figcaption>
</figure>
Expand All @@ -59,8 +59,8 @@ the job's specification which you can then import from the `advanced options` pa

<p align="center">
<figure>
<a href="../../assets/images/newjob3.png">
<img src="../../assets/images/newjob3.png" alt="Job export">
<a href="../../assets/images/jobs/newjob3.png">
<img src="../../assets/images/jobs/newjob3.png" alt="Job export">
</a>
<figcaption>Job export</figcaption>
</figure>
Expand Down Expand Up @@ -105,8 +105,8 @@ You can set the following properties for Spark/PySpark jobs:

<p align="center">
<figure>
<a href="../../assets/images/newjob4.png">
<img src="../../assets/images/newjob4.png" alt="Spark job configuration properties">
<a href="../../assets/images/jobs/newjob4.png">
<img src="../../assets/images/jobs/newjob4.png" alt="Spark job configuration properties">
</a>
<figcaption>Spark job configuration properties</figcaption>
</figure>
Expand All @@ -125,8 +125,8 @@ You can set the following properties for Python jobs:

<p align="center">
<figure>
<a href="../../assets/images/newjob5.png">
<img src="../../assets/images/newjob5.png" alt="Python job configuration properties">
<a href="../../assets/images/jobs/newjob5.png">
<img src="../../assets/images/jobs/newjob5.png" alt="Python job configuration properties">
</a>
<figcaption>Python job configuration properties</figcaption>
</figure>
Expand Down Expand Up @@ -171,8 +171,8 @@ The potential severities these alerts should report are:

<p align="center">
<figure>
<a href="../../assets/images/newjob6.gif">
<img src="../../assets/images/newjob6.gif" alt="Job alerts">
<a href="../../assets/images/jobs/newjob6.gif">
<img src="../../assets/images/jobs/newjob6.gif" alt="Job alerts">
</a>
<figcaption>Job alerts</figcaption>
</figure>
Expand All @@ -189,8 +189,8 @@ and it is up to the user to clean them up if needed.

<p align="center">
<figure>
<a href="../../assets/images/newjob7.png">
<img src="../../assets/images/newjob7.png" alt="Job deletion">
<a href="../../assets/images/jobs/newjob7.png">
<img src="../../assets/images/jobs/newjob7.png" alt="Job deletion">
</a>
<figcaption>Job deletion</figcaption>
</figure>
Expand All @@ -208,17 +208,17 @@ executions page by clicking the `view all executions` link.

<p align="center">
<figure>
<a href="../../assets/images/newjob8.png">
<img src="../../assets/images/newjob8.png" alt="Job preview">
<a href="../../assets/images/jobs/newjob8.png">
<img src="../../assets/images/jobs/newjob8.png" alt="Job preview">
</a>
<figcaption>Job preview</figcaption>
</figure>
</p>

<p align="center">
<figure>
<a href="../../assets/images/newjob9.png">
<img src="../../assets/images/newjob9.png" alt="Executions overview">
<a href="../../assets/images/jobs/newjob9.png">
<img src="../../assets/images/jobs/newjob9.png" alt="Executions overview">
</a>
<figcaption>Executions overview</figcaption>
</figure>
Expand Down Expand Up @@ -256,8 +256,8 @@ It is also possible to work on jobs while developing in your IntelliJ/PyCharm ID

<p align="center">
<figure>
<a href="../../assets/images/newjob10.gif">
<img src="../../assets/images/newjob10.gif" alt=" Working with jobs from Hopsworks IntelliJ/PyCharm plugin">
<a href="../../assets/images/jobs/newjob10.gif">
<img src="../../assets/images/jobs/newjob10.gif" alt=" Working with jobs from Hopsworks IntelliJ/PyCharm plugin">
</a>
<figcaption>Working with jobs from Hopsworks IntelliJ/PyCharm plugin</figcaption>
</figure>
Expand Down
62 changes: 31 additions & 31 deletions docs/compute/jupyter.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ can access from the main project menu. The image below shows the Jupyter service

<p align="center">
<figure>
<a href="../../assets/images/jupyter1.png">
<img src="../../assets/images/jupyter1.png" alt="Jupyter service in Hopsworks">
<a href="../../assets/images/jupyter/jupyter1.png">
<img src="../../assets/images/jupyter/jupyter1.png" alt="Jupyter service in Hopsworks">
</a>
<figcaption>Jupyter dashboard in Hopsworks</figcaption>
</figure>
Expand All @@ -23,8 +23,8 @@ open Jupyter in a new tab if your browser is blocking this operation by default.

<p align="center">
<figure>
<a href="../../assets/images/jupyter2.png">
<img src="../../assets/images/jupyter2.png" alt="Classic Jupyter">
<a href="../../assets/images/jupyter/jupyter2.png">
<img src="../../assets/images/jupyter/jupyter2.png" alt="Classic Jupyter">
</a>
<figcaption>Classic Jupyter</figcaption>
</figure>
Expand All @@ -45,8 +45,8 @@ resources will be allocated based on the configuration options Jupyter was start

<p align="center">
<figure>
<a href="../../assets/images/jupyter3.png">
<img src="../../assets/images/jupyter3.png" alt="Jupyter dashboard page showing currently running applications">
<a href="../../assets/images/jupyter/jupyter3.png">
<img src="../../assets/images/jupyter/jupyter3.png" alt="Jupyter dashboard page showing currently running applications">
</a>
<figcaption>Jupyter dashboard page showing currently running applications</figcaption>
</figure>
Expand All @@ -72,8 +72,8 @@ figure below.

<p align="center">
<figure>
<a href="../../assets/images/jupyter4.png">
<img src="../../assets/images/jupyter4.png" alt="Jupyter logs">
<a href="../../assets/images/jupyter/jupyter4.png">
<img src="../../assets/images/jupyter/jupyter4.png" alt="Jupyter logs">
</a>
<figcaption>Jupyter logs</figcaption>
</figure>
Expand All @@ -88,7 +88,7 @@ also shows you how much time is currently left before the Jupyter server shuts d
## Debug Jupyter installation
Jupyter is installed in the Python environment of your project. This means that if a dependency of Jupyter is removed,
or an incorrect version is installed it may not work properly. If the Python environment ends up in a state with
conflicting libraries installed then an alert will be shown in the Python management page under the project settings menu.
conflicting libraries installed then an alert will be shown in the `Python libraries` page under the `Project settings` menu.

## Jupyter + Spark on Hopsworks

Expand Down Expand Up @@ -129,8 +129,8 @@ After you have started the Jupyter notebook server, you can create a PySpark not

<p align="center">
<figure>
<a href="../../assets/images/jupyter5.png">
<img src="../../assets/images/jupyter5.png" alt="Create a pyspark notebook">
<a href="../../assets/images/jupyter/jupyter5.png">
<img src="../../assets/images/jupyter/jupyter5.png" alt="Create a pyspark notebook">
</a>
<figcaption>Create a pyspark notebook</figcaption>
</figure>
Expand All @@ -140,8 +140,8 @@ When you execute the first cell in a PySpark notebook, the Spark session is auto

<p align="center">
<figure>
<a href="../../assets/images/jupyter6.png">
<img src="../../assets/images/jupyter6.png" alt="SparkSession creation with pyspark kernel">
<a href="../../assets/images/jupyter/jupyter6.png">
<img src="../../assets/images/jupyter/jupyter6.png" alt="SparkSession creation with pyspark kernel">
</a>
<figcaption>SparkSession creation with pyspark kernel</figcaption>
</figure>
Expand All @@ -152,8 +152,8 @@ on a Spark driver in the cluster. You can execute regular Python code:

<p align="center">
<figure>
<a href="../../assets/images/jupyter7.png">
<img src="../../assets/images/jupyter7.png" alt="Executing python code on the spark driver in the cluster">
<a href="../../assets/images/jupyter/jupyter7.png">
<img src="../../assets/images/jupyter/jupyter7.png" alt="Executing python code on the spark driver in the cluster">
</a>
<figcaption>Executing python code on the spark driver in the cluster</figcaption>
</figure>
Expand All @@ -163,8 +163,8 @@ Since you are executing on the Spark driver, you can also run applications on sp

<p align="center">
<figure>
<a href="../../assets/images/jupyter8.png">
<img src="../../assets/images/jupyter8.png" alt="Starting a spark application from Jupyter">
<a href="../../assets/images/jupyter/jupyter8.png">
<img src="../../assets/images/jupyter/jupyter8.png" alt="Starting a spark application from Jupyter">
</a>
<figcaption>Starting a spark application from Jupyter</figcaption>
</figure>
Expand All @@ -178,8 +178,8 @@ magic commands provided by sparkmagic. You can view a list of all commands by ex

<p align="center">
<figure>
<a href="../../assets/images/jupyter9.png">
<img src="../../assets/images/jupyter9.png" alt="Printing a list of all sparkmagic commands">
<a href="../../assets/images/jupyter/jupyter9.png">
<img src="../../assets/images/jupyter/jupyter9.png" alt="Printing a list of all sparkmagic commands">
</a>
<figcaption>Printing a list of all sparkmagic commands</figcaption>
</figure>
Expand All @@ -202,8 +202,8 @@ are illustrated below. Using this approach, you can have large scale cluster com

<p align="center">
<figure>
<a href="../../assets/images/jupyter10.png">
<img src="../../assets/images/jupyter10.png" alt="Creating a spark dataframe">
<a href="../../assets/images/jupyter/jupyter10.png">
<img src="../../assets/images/jupyter/jupyter10.png" alt="Creating a spark dataframe">
</a>
<figcaption>Creating a spark dataframe</figcaption>
</figure>
Expand All @@ -217,8 +217,8 @@ local Jupyter server for plotting.
Using **%%sql**:
<p align="center">
<figure>
<a href="../../assets/images/jupyter11.png">
<img src="../../assets/images/jupyter11.png" alt="Downloading the spark dataframe to a pandas dataframe using %%sql">
<a href="../../assets/images/jupyter/jupyter11.png">
<img src="../../assets/images/jupyter/jupyter11.png" alt="Downloading the spark dataframe to a pandas dataframe using %%sql">
</a>
<figcaption>Downloading the spark dataframe to a pandas dataframe using %%sql</figcaption>
</figure>
Expand All @@ -227,8 +227,8 @@ Using **%%sql**:
Using **%%spark**:
<p align="center">
<figure>
<a href="../../assets/images/jupyter12.png">
<img src="../../assets/images/jupyter12.png" alt="Printing a list of all sparkmagic commands">
<a href="../../assets/images/jupyter/jupyter12.png">
<img src="../../assets/images/jupyter/jupyter12.png" alt="Printing a list of all sparkmagic commands">
</a>
<figcaption>Downloading the spark dataframe to a pandas dataframe using %%spark</figcaption>
</figure>
Expand All @@ -243,24 +243,24 @@ To do this, use the sparkmagic `%%local` to access the local pandas dataframe an

<p align="center">
<figure>
<a href="../../assets/images/jupyter13.png">
<img src="../../assets/images/jupyter13.png" alt="Import plotting libraries locally on the Jupyter server">
<a href="../../assets/images/jupyter/jupyter13.png">
<img src="../../assets/images/jupyter/jupyter13.png" alt="Import plotting libraries locally on the Jupyter server">
</a>
<figcaption>Import plotting libraries locally on the Jupyter server</figcaption>
</figure>
</p>
<p align="center">
<figure>
<a href="../../assets/images/jupyter14.png">
<img src="../../assets/images/jupyter14.png" alt="Printing a list of all sparkmagic commands">
<a href="../../assets/images/jupyter/jupyter14.png">
<img src="../../assets/images/jupyter/jupyter14.png" alt="Printing a list of all sparkmagic commands">
</a>
<figcaption>Plot a local pandas dataframe using seaborn and the magic %%local</figcaption>
</figure>
</p>
<p align="center">
<figure>
<a href="../../assets/images/jupyter15.png">
<img src="../../assets/images/jupyter15.png" alt="Plot a local pandas dataframe using matplotlib and the magic %%local">
<a href="../../assets/images/jupyter/jupyter15.png">
<img src="../../assets/images/jupyter/jupyter15.png" alt="Plot a local pandas dataframe using matplotlib and the magic %%local">
</a>
<figcaption>Plot a local pandas dataframe using matplotlib and the magic %%local</figcaption>
</figure>
Expand Down
Loading

0 comments on commit 7dcdb2f

Please sign in to comment.