Skip to content

Latest commit

 

History

History
147 lines (113 loc) · 5.74 KB

README.md

File metadata and controls

147 lines (113 loc) · 5.74 KB
Raw

TRIMS (MXNet with Persistent GPU Memory)

Reference

Dakkak, A., Li, C., De Gonzalo, S. G., Xiong, J., & Hwu, W-M. W. (2019). TrIMS: Transparent and isolated model sharing for low latency deep learning inference in function-as-a-service. In E. Bertino, C. K. Chang, P. Chen, E. Damiani, M. Goul, & K. Oyama (Eds.), Proceedings - 2019 IEEE International Conference on Cloud Computing, CLOUD 2019 - Part of the 2019 IEEE World Congress on Services (pp. 372-382). [8814494] (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2019-July). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2019.00067

Installation

Requirements

Setup your target location (make sure that LD_LIBRARY_PATH and PATH has been updated to include this)

export UPR_INSTALL_PREFIX=$HOME/.usr
export UPR_BASE_DIR=$HOME/carml/data/mxnet

make sure that both directories exist

mkdir -p $UPR_INSTALL_PREFIX
mkdir -p $UPR_BASE_DIR

Remember to update your PKG_CONFIG_PATH

export PKG_CONFIG_PATH=$UPR_INSTALL_PREFIX/lib/pkgconfig/:$PKG_CONFIG_PATH

C-ARES

wget https://c-ares.haxx.se/download/c-ares-1.13.0.tar.gz
tar -xf c-ares-1.13.0.tar.gz
cd c-ares-1.13.0
./buildconf
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug
make install

Protobuf

wget https://github.com/google/protobuf/archive/v3.5.1.tar.gz
tar -xf v3.5.1.tar.gz
cd protobuf-3.5.1
./autogen.sh
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug --with-zlib
make
make install

Protobuf-c

wget https://github.com/protobuf-c/protobuf-c/releases/download/v1.3.0/protobuf-c-1.3.0.tar.gz
tar -xf protobuf-c-1.3.0.tar.gz
cd protobuf-c-1.3.0
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug
make install

GRPC

wget https://github.com/grpc/grpc/archive/v1.9.1.tar.gz
tar -xf v1.9.1
cd v1.9.1
make install prefix=$UPR_INSTALL_PREFIX
make install-plugins prefix=$UPR_INSTALL_PREFIX

OpenCV (Optional)

wget https://github.com/opencv/opencv/archive/3.4.1.tar.gz
tar -xf 3.4.1.tar.gz
cd opencv-3.4.1/
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$UPR_INSTALL_PREFIX -DWITH_CUDA=OFF -DWITH_OPENCL=OFF ..
make
make install

From APT-GET

Other requirements can be installed using APT . The base requirements are listed in the (MXNet installation guide)[https://mxnet.apache.org/install/index.html].

Regenerate GRPC Code

It is not recommended to regenerate the protofile, since it's dependent on the protobuf version. The only time you need to regenerate it is after updating it.

cd src/c_api
make

Build Server

The server is part of the MXNet build process.

Downloading Models

Running

Server

Client

Profiling

Environment Variables

Name Description Default Value
UPR_ENABLED true
UPR_PROFILE_IO only makes sense if UPR_ENABLED=false true
UPR_RUN_ID [undefined]
git build_git_sha
UPR_GIT_BRANCH build_git_branch
UPR_GIT_TIME build_git_time
UPR_CLIENT
UPR_BASE_DIR
UPR_MODEL_NAME
UPR_PROFILE_TARGET profile.json
UPR_INITIALIZE_EAGER false
UPR_INITIALIZE_EAGER_ASYNC false
UPR_INPUT_CHANNELS 3
UPR_INPUT_WIDTH 224
UPR_INPUT_HEIGHT 224
UPR_INPUT_MEAN_R 0
UPR_INPUT_MEAN_G 0
UPR_INPUT_MEAN_B 0
UPR_ENABLE_MEMORY_PROFILE false
UPR_ENABLE_CUDA_FREE false
UPR_SHARING_GRANULARITY model
-------------------------- ----------- -------------
UPRD_EVICTION_POLICY LRU
UPRD_ESTIMATION_RATE 1.0
UPRD_MEMORY_PERCENTAGE 0.8
UPRD_PERSIST_CPU true
UPRD_PERSIST_ONLY_CPU only persist on cpu memory false
UPRD_WRITE_PROFILE write server profile file false
UPRD_ESTIMATE_WITH_INTERNAL_MEMORY use internal memory info for estimate true