TensorMSA : Tensorflow Micro Service Architecture

Install

History

17.2.3 Initial
17.2.4 Firefox Fix, Chrome Installed, Korean Font Installed
17.3.1 Tensorflow 1.0, RabbitMQ, Vnc Server, Xfce4 Installed
17.4.11 Tensorflow 1.1(Complie), Neo4j, flower, mecab Installed

Summeries

python 3.5
conda
Tensorflow r0.12
Django
postgres 9.6
Pycharm Comunity
Chrome
python packages for hoyai
pgadmin3
rabbit mq
hdfview
vnc xfce4 setup
Neo4j
mecab

1.Prerequisite

Docker 1.13 on ubuntu 16.04
How to install : https://docs.docker.com/engine/installation/linux/ubuntu/
- Add 127.0.0.1 ip-xxx-xx-x-xx to /etc/hosts (ubuntu@ip-XX-XX-X-XXX in console)

- unset DISPLAY
 ```

<b>2.User add ubuntu </b> </br>
```bash
  sudo groupadd docker
  sudo gpasswd -a ubuntu docker

Test command : docker ps(By ubuntu id) - restart AWS

3.Download Docker Images

  docker pull hoyai/hoyai_dev_docker:squashed

4.Run Docker Container

Changes Resolution for vnc = VNC_RESOLUTION="1920x1080"

  docker run -itd --env="VNC_RESOLUTION=1920x1080" --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --name hoyai_dev -p 5672:5672 -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 -p 5901:5901 hoyai/hoyai_dev_docker:squashed2
  
  docker run -itd --env="VNC_RESOLUTION=1920x1080" --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --volume="/hoya_src_root:/hoya_src_root" --volume="/hoya_model_root:/hoya_model_root" --volume="/hoya_str_root:/hoya_str_root" --volume="/hoya_data_root:/hoya_data_root" --name hoyai_dev -p 5672:5672 -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 -p 5901:5901 hoyai/hoyai_dev_docker:squashed2
  
  gpu
 nvidia-docker run -itd --env="VNC_RESOLUTION=1920x1080" --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --name hoyai_dev -p 5672:5672 -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 -p 5901:5901 hoyai/hoyai_gpu_dev_docker:v1.0
 
 ai 137
docker run -itd --env="VNC_RESOLUTION=1920x1080" --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --volume="/data1/hoyai_playground:/home/dev/hoyai_playground"  --volume="/data1/hoya_src_root:/hoya_src_root" --volume="/data1/hoya_model_root:/hoya_model_root" --volume="/data1/hoya_str_root:/hoya_str_root" --name hoyai_dev_2 -p 5672:5672 -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 -p 5901:5901 -p 5555:5555 hoyai/hoyai_dev_docker:flower

5. Register Docker Service

  cd /etc/systemd/system/
  vi docker_hoyai.service

docker_hoyai.service

  [Unit]
  Description=hoyai container
  Requires=docker.service
  After=docker.service

  [Service]
  Restart=always
  ExecStart=/usr/bin/docker start -a hoyai_dev
  ExecStop=/usr/bin/docker stop -t 2 hoyai_dev

  [Install]
  WantedBy=default.target

Service enable & start

   sudo systemctl enable docker_hoyai.service
   
   sudo systemctl start docker_hoyai.service

Service disable & stop

   sudo systemctl disable docker_hoyai.service
   
   sudo systemctl stop docker_hoyai.service

6. Restart aws

Check for hoyai_dev_docker started after aws reboot

   
   docker ps

7. Install VNC

Go to chrome app store
Find VNC Viewer for Google Chrome

7. Connect VNC

Make New tab in chrome
Find apps icon on top of screen
Click vnc and input your ip(port : XXXX)
- Passwd : *********

8. ETC

pycharm.sh &
firefox
pgadmin3

9. Merge Another Repository to My Repository

Make git repository for docker auto build
Add remode repository

git remote add hoyai_docker https://github.com/TensorMSA/hoyai_docker.git

cd hoyai_dev_docker/

git pull hoyai_docker

git branch -d dd
git checkout -b dd hoyai_docker/master

git checkout master
git merge dd
git push

10. Squash Docker images

docker build --rm -t hoyai/hoyai_dev_docker:v2.1 .

pip install docker-squash

docker history hoyai/hoyai_dev_docker:v2.1
select image id for starting point
docker-squash -f f49eec89601e -t hoyai/hoyai_dev_docker:squashed hoyai/hoyai_dev_docker:v2.0


docker-squash -f f49eec89601e -t hoyai/hoyai_dev_docker:squashed hoyai/hoyai_dev_docker

11. Docker Container size up(1.7.1 RHEL)

service docker stop
vi /etc/sysconfig/docker
other_args="--storage-driver=devicemapper --storage-opt dm.basesize=20G"
rm -Rf /var/lib/docker (다 지워짐 조심)
service docker start or reboot
이미지를 새로 받아야함

b>12. Docker rebuild setup list

1. jenkins git pull
2. jenkins jupyter git pull
3. inside docker : passwd root
4. inside docker : /home/dev 에서 ln -s로 hoya_src_root연결
5. inside docker : /home/dev에서 jupyter notebook 실행
6. inside docker : start_hoyai.sh 복사
7. inside docker : ./start_hoyai.sh 6 8 10 으로 서버 스타트
8. jupyter rule setup
9. tensorboard --logdir=/hoya_model_root

b>13. Docker rebuild setup list

su postgres


pg_dump -Fc tensormsa > /hoya_src_root/postgres_backup/postgres_backup_07_10_2.dump

cp /var/nfs/hoya_src_root/postgres_backup/postgres_backup_07_10_2.dump ./


pg_restore -c -d tensormsa tensormsa.dump

GRANT ALL ON schema public TO tfmsauser;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO tfmsauser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tfmsauser;

---------------------------before--------------------------------------------------------
1.Install Xming

download Xming : https://sourceforge.net/projects/xming/
install

2.Install Putty

install putty : http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html?

3.Connect AWS EC2 Instance

.....

4.Git clone

5.Docker build

  docker build --rm -t hoyai/client:v0.1 .

You can change name and version [hoyai/client:v0.1]

6.Docker run

  docker run -itd --name hoyai_dev -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 --volume /root/data/:/root/lib/ hoyai/client:v0.1
  
  docker run -itd  --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --name hoyai_dev -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 --volume /root/data/:/root/lib/ hoyai/client:v0.1
  
docker run -itd  --env="VNC_RESOLUTION=1920x1080" --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --name hoyai_dev -p 5672:5672 -p 2266:2266 -p 5432:5432 -p 8000:8000 -p 6006:6006 -p 8888:8888 -p 5901:5901 --volume /root/data/:/root/lib/ hoyai/client:v0.2

7.Connect hoyai_dev directly and Change root password

   docker exec -it hoyai_dev bash

    passwd root

8.Connect hoyai_dev by SSH

putty -> connection -> ssh -> X11 -> Enable X11 forwarding check
Connect AWS, Port : 2266

9.Run Pycharm

    /home/dev/pyc/bin/pycharm.sh &

10.Run Chrome

    /usr/bin/google-chrome --no-sandbox

11.Docker VNC
Browser it doesn't crash

docker run -itd -p 6080:80 -p 5902:5900 -e VNC_PASSWORD=mypassword hoyai/ubuntu_vnc --name hoyai_documents

http://xxx.xx.xx.xxx:6080

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README_bk.md

README_bk.md

TensorMSA : Tensorflow Micro Service Architecture

Install

Files

README_bk.md

Latest commit

History

README_bk.md

File metadata and controls

TensorMSA : Tensorflow Micro Service Architecture

Install