Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Colab Env Check & Install #9

Open
wants to merge 45 commits into
base: master
Choose a base branch
from

Commits on Oct 17, 2019

  1. fix spacing

    gumdropsteve committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    d1c1a00 View commit details
    Browse the repository at this point in the history
  2. new script to check GPU type of colab and provide next step guidance …

    …(if any) specific to allocation
    gumdropsteve committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    9d27e40 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    83853c3 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2019

  1. Configuration menu
    Copy the full SHA
    6d0dc43 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    28385a9 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #1 from gumdropsteve/feature_utils-colab-env-check

    Create colab-env-check.sh
    gumdropsteve authored Oct 21, 2019
    Configuration menu
    Copy the full SHA
    4a5c430 View commit details
    Browse the repository at this point in the history
  4. Switch to pynvml from !nvidia-smi

    Thinking this is more runable, having issues executing !nvidia-smi
    gumdropsteve authored Oct 21, 2019
    Configuration menu
    Copy the full SHA
    59c4732 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #2 from gumdropsteve/feature_utils-colab-env-check

    Switch to pynvml from !nvidia-smi
    gumdropsteve authored Oct 21, 2019
    Configuration menu
    Copy the full SHA
    299a416 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2019

  1. Configuration menu
    Copy the full SHA
    72e37e4 View commit details
    Browse the repository at this point in the history
  2. installing cuml issues

    gumdropsteve committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    7b48188 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5498066 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2019

  1. add taxi colab button

    gumdropsteve committed Oct 27, 2019
    Configuration menu
    Copy the full SHA
    8a75074 View commit details
    Browse the repository at this point in the history
  2. too much data (RuntimeError: Exception occured! file=/conda/conda-bld…

    …/libcuml_1571339163826/work/cpp/src_prims/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory)
    gumdropsteve committed Oct 27, 2019
    Configuration menu
    Copy the full SHA
    45fd58b View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2019

  1. Configuration menu
    Copy the full SHA
    7a45cd3 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2019

  1. Create colab_env.py

    reformatted; use pynvml (preinstalled in base Colab) - remove other scripts or format like so if kept
    gumdropsteve authored Nov 11, 2019
    Configuration menu
    Copy the full SHA
    1f5f0af View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    90f9980 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #5 from gumdropsteve/feature/utils

    Colab Environment Check (function & script)
    gumdropsteve authored Nov 11, 2019
    Configuration menu
    Copy the full SHA
    7989b43 View commit details
    Browse the repository at this point in the history
  4. Merge pull request #6 from gumdropsteve/feature_utils

    Feature utils branch
    gumdropsteve authored Nov 11, 2019
    Configuration menu
    Copy the full SHA
    a5ba9e5 View commit details
    Browse the repository at this point in the history
  5. colab env check

    gumdropsteve committed Nov 11, 2019
    Configuration menu
    Copy the full SHA
    463fe6f View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2019

  1. Update colab_env.py

    make inot script
    gumdropsteve authored Nov 12, 2019
    Configuration menu
    Copy the full SHA
    1f95713 View commit details
    Browse the repository at this point in the history
  2. Update colab_env.py

    make into script, formatting
    gumdropsteve authored Nov 12, 2019
    Configuration menu
    Copy the full SHA
    69b0e4f View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2019

  1. via Adam + colab_env.py

    Thank you, Adam.
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    133c20d View commit details
    Browse the repository at this point in the history
  2. Update bsql-colab.sh

    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    34cb23a View commit details
    Browse the repository at this point in the history
  3. Update colab_env.py

    testing elimination of temp fix colab install
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    6694e86 View commit details
    Browse the repository at this point in the history
  4. Update README.md

    sync with master
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    346a176 View commit details
    Browse the repository at this point in the history
  5. Update colab_env.py

    Checking for GPU type:
    Woo! You got the right kind of GPU!
    Traceback (most recent call last):
      File "colab_env.py", line 46, in <module>
        subprocess.Popen(['blazingsql-orchestrator', '9100', '8889', '127.0.0.1', '8890'],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
      File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
        restore_signals, start_new_session)
      File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'blazingsql-orchestrator': 'blazingsql-orchestrator'
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    e34a4d4 View commit details
    Browse the repository at this point in the history
  6. Create set_paths.py

    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    e5e7653 View commit details
    Browse the repository at this point in the history
  7. Update bsql-colab.sh

    Trying to complete the setting paths part of temp p-100 colab install within bsql-colab.sh
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    b4ddc23 View commit details
    Browse the repository at this point in the history
  8. Update bsql-colab.sh

    takes too long
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    e3a6118 View commit details
    Browse the repository at this point in the history
  9. Update and rename set_paths.py to set_paths.sh

    does not work as python script, forever pends connection 8889
    
    ```
    *********************************************
    Your Google Colab instance is BlazingSQL ready!
    *********************************************
    --2019-11-13 02:19:45--  https://github.com/gumdropsteve/bsql-demos/raw/feature/utils/utils/set_paths.py
    Resolving github.com (github.com)... 140.82.118.4
    Connecting to github.com (github.com)|140.82.118.4|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://raw.githubusercontent.com/gumdropsteve/bsql-demos/feature/utils/utils/set_paths.py [following]
    --2019-11-13 02:19:46--  https://raw.githubusercontent.com/gumdropsteve/bsql-demos/feature/utils/utils/set_paths.py
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 571 [text/plain]
    Saving to: ‘set_paths.py’
    
    set_paths.py        100%[===================>]     571  --.-KB/s    in 0s      
    
    2019-11-13 02:19:46 (116 MB/s) - ‘set_paths.py’ saved [571/571]
    
    Setting RAPIDS paths
    SLF4J: The requested version 1.6 by your slf4j binding is not compatible with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10]
    SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details.
    /usr/local/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: 
    Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so.
    
    For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup')
      warnings.warn(errors.NumbaWarning(msg))
    /usr/local/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: 
    Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice/.
    
    For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup')
      warnings.warn(errors.NumbaWarning(msg))
    Usage: blazingsql-engine <RAL_ID> <GPU_ID> <ORCHESTRATOR_HTTP_COMMUNICATION_[IP|HOSTNAME]> <ORCHESTRATOR_HTTP_COMMUNICATION_PORT> <RAL_HTTP_COMMUNICATION_[IP|HOSTNAME]> <RAL_HTTP_COMMUNICATION_PORT> <RAL_TCP_PROTOCOL_PORT> <OPTIONAL_NETWORK_INTERFACE_NAME_ETH0_DEFAULT>
    RAL Engine starting
    Using the network interface: eth0
    RAL ID: 1
    GPU ID: 0
    Orchestrator HTTP communication host: 127.0.0.1
    Orchestrator HTTP communication port: 9100
    RAL HTTP communication host: 172.28.0.2
    RAL HTTP communication port: 9001
    RAL protocol port: 8891
    Socket Name: 172.28.0.2
    Log Name: RAL.1.log
    RAL TCP protocol port: 8891
    java.net.SocketException: Connection reset
    	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
    	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
    	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:126)
    	at com.blazingdb.calcite.application.TCPClient.send(TCPClient.java:38)
    	at com.blazingdb.calcite.application.TCPClient.send(TCPClient.java:55)
    	at com.blazingdb.calcite.application.TCPService.synchronizeSchema(TCPService.java:63)
    	at com.blazingdb.calcite.application.TCPService.<init>(TCPService.java:53)
    	at com.blazingdb.calcite.application.CalciteApplication.main(CalciteApplication.java:204)
    schema synchronization with Orchestrator failed
    Waiting for messages in TCP port: 8890
    ```
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    f16046c View commit details
    Browse the repository at this point in the history
  10. Delete set_paths.sh

    Does not work
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    35e2991 View commit details
    Browse the repository at this point in the history
  11. Delete taxi_fare_prediction.ipynb

    Shouldn't be here, bad fork
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    27ccfea View commit details
    Browse the repository at this point in the history
  12. Delete taxi_fare_prediction.ipynb

    Merge pull request #8 from gumdropsteve/gumdropsteve-patch-1
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    0fe2c63 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    b3cb7e8 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    92d49e9 View commit details
    Browse the repository at this point in the history
  15. Remove env check

    Doing this in a separate cell works more logically in notebook.
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    b3a600e View commit details
    Browse the repository at this point in the history
  16. Update colab_env.py

    More clear success output.
    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    6efa6f5 View commit details
    Browse the repository at this point in the history
  17. sp

    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    ee1848d View commit details
    Browse the repository at this point in the history
  18. clear success output

    gumdropsteve authored Nov 13, 2019
    Configuration menu
    Copy the full SHA
    77315eb View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2019

  1. Configuration menu
    Copy the full SHA
    f4b7156 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2019

  1. convert to include latest notebooks-contrib

    plus working bsql colab solution
    gumdropsteve authored Nov 15, 2019
    Configuration menu
    Copy the full SHA
    d81571e View commit details
    Browse the repository at this point in the history
  2. Update bsql-colab.sh

    remove extra
    gumdropsteve authored Nov 15, 2019
    Configuration menu
    Copy the full SHA
    b0e6347 View commit details
    Browse the repository at this point in the history
  3. fi

    gumdropsteve authored Nov 15, 2019
    Configuration menu
    Copy the full SHA
    62cffe9 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2019

  1. Create blazingsql.sh (#11)

    * Create blazingsql.sh
    
    * Update blazingsql.sh
    gumdropsteve authored Nov 18, 2019
    Configuration menu
    Copy the full SHA
    62a5837 View commit details
    Browse the repository at this point in the history
  2. Update bsql-colab.sh (#12)

    * Update bsql-colab.sh
    
    * Update bsql-colab.sh
    
    * Delete bsql-colab.sh
    gumdropsteve authored Nov 18, 2019
    Configuration menu
    Copy the full SHA
    079340e View commit details
    Browse the repository at this point in the history