Skip to content
This repository has been archived by the owner on Jul 10, 2024. It is now read-only.

Commit

Permalink
SUBMARINE-861. Trim the size of pysubmarine
Browse files Browse the repository at this point in the history
### What is this PR for?
<!-- A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://submarine.apache.org/contribution/contributions.html
-->
Trim the size of pysubmarine, in other words,we let the user install the package such as pytorch and Tensorflow by themself instead of pre-installation from pysubmarine
### What type of PR is it?
 Improvement

### Todos
 [ ] - Deal with Jupiter Lab

### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-861
### How should this be tested?
<!--
* First time? Setup Travis CI as described on https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
-->
### Screenshots (if appropriate)

### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No

Author: featherchen <[email protected]>

Signed-off-by: byronhsu <[email protected]>

Closes #619 from featherchen/SUBMARINE-861 and squashes the following commits:

70807f5 [featherchen] add package in IT environment
8245e4a [featherchen] add tf-latest
42d1009 [featherchen] SUBMARINE-861. Add extras_reqire
4ed0f42 [featherchen] SUBMARINE-861. Add extras and modifile Jupiter dockerfile
eb07371 [featherchen] SUBMARINE-861. Delete installation of pytorch and Tensorflow
  • Loading branch information
featherchen authored and ByronHsu committed Jul 4, 2021
1 parent 5ef3a42 commit c0fe073
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 10 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: '1.8'
java-version: "1.8"
- name: Set up Maven 3.6.3
uses: stCarolas/setup-maven@v4
with:
Expand Down Expand Up @@ -130,7 +130,7 @@ jobs:
- name: Install dependencies
run: |
pip install --upgrade pip
pip install --no-cache-dir ./submarine-sdk/pysubmarine/.
pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf,pytorch]
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
- name: Run integration test
run: |
Expand All @@ -141,4 +141,4 @@ jobs:
kubectl get pods
kubectl -n default get events --sort-by='{.lastTimestamp}'
kubectl describe nodes
if: ${{ failure() }}
if: ${{ failure() }}
2 changes: 1 addition & 1 deletion dev-support/docker-images/jupyter-gpu/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ RUN wget --quiet https://repo.continuum.io/miniconda/Miniconda3-py37_${MINICONDA

# Install latest sumbarine python sdk and notebook
RUN git clone https://github.com/apache/submarine && \
pip install submarine/submarine-sdk/pysubmarine && \
pip install -e submarine/submarine-sdk/pysubmarine[tf,pytorch] && \
conda install nodejs && \
conda install -c conda-forge jupyterlab jupyterlab-git && \
jupyter lab build
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker-images/jupyter/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ RUN wget --quiet https://repo.continuum.io/miniconda/Miniconda3-py37_${MINICONDA

# Install latest sumbarine python sdk and notebook
RUN git clone https://github.com/apache/submarine && \
pip install submarine/submarine-sdk/pysubmarine && \
pip install -e submarine/submarine-sdk/pysubmarine[tf,pytorch] && \
conda install nodejs && \
conda install -c conda-forge jupyterlab jupyterlab-git && \
jupyter lab build
Expand Down
2 changes: 1 addition & 1 deletion dev-support/pysubmarine/install-conda.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,6 @@ pip install --upgrade pip
# Install pysubmarine
git clone https://github.com/apache/submarine.git
cd submarine/submarine-sdk/pysubmarine
pip install .
pip install -e .[tf,pytorch]
pip install -r github-actions/lint-requirements.txt
pip install -r github-actions/test-requirements.txt
10 changes: 6 additions & 4 deletions submarine-sdk/pysubmarine/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,19 @@
'sqlalchemy',
'sqlparse',
'pymysql',
'tensorflow>=1.14.0,<2.0.0',
'requests',
'urllib3 >= 1.15.1',
'certifi >= 14.05.14',
'python-dateutil >= 2.5.3',
'pyarrow==0.17.0',
'torch>=1.5.0',
'torchvision>=0.6.0',
'mlflow>=1.15.0',
'boto3>=1.17.58 '
'boto3>=1.17.58',
],
extras_require={
'tf':['tensorflow>=1.14.0,<2.0.0'],
'tf-latest':['tensorflow'],
'pytorch':['torch>=1.5.0','torchvision>=0.6.0'],
},
classifiers=[
'Intended Audience :: Developers',
'Programming Language :: Python :: 3.5',
Expand Down

0 comments on commit c0fe073

Please sign in to comment.