Skip to content

Latest commit

 

History

History
293 lines (234 loc) · 7.58 KB

README.md

File metadata and controls

293 lines (234 loc) · 7.58 KB
Raw

💁 Week #1

Environment setup

  1. Install pyenv (kivabe korben niche dewa ase)
  2. Install python 3.8.6 using pyenv (Pyevn cheat sheet added below)
  3. Install vscode
  4. Install following extentions in vscode: 
    chrmarti.regex
donjayamanne.githistory
dzhavat.bracket-pair-toggler
eamodio.gitlens
GrapeCity.gc-excelviewer
humao.rest-client
ionutvmi.path-autocomplete
iterative.dvc
mechatroner.rainbow-csv
ms-azuretools.vscode-docker
ms-python.autopep8
ms-python.flake8
ms-python.isort
ms-python.pylint
ms-python.python
ms-python.vscode-pylance
ms-toolsai.jupyter
ms-toolsai.jupyter-keymap
ms-toolsai.jupyter-renderers
ms-toolsai.vscode-jupyter-cell-tags
ms-toolsai.vscode-jupyter-slideshow
ms-vscode-remote.remote-containers
ms-vscode-remote.remote-ssh
ms-vscode-remote.remote-ssh-edit
ms-vscode.remote-explorer
njpwerner.autodocstring
PKief.material-icon-theme
Shan.code-settings-sync
shardulm94.trailing-spaces
shd101wyy.markdown-preview-enhanced
VisualStudioExptTeam.intellicode-api-usage-examples
VisualStudioExptTeam.vscodeintellicode
wayou.vscode-todo-highlight
yzhang.markdown-all-in-one
  5. Create virtual environment (kivabe korben niche dewa ase)
  6. Install all python required libs- 
     pip install -r requriements.txt

🐍 pyenv cheat-sheet

Installation of pyenv

Windows

  • Open windows powershell and run this command:

    Invoke-WebRequest -UseBasicParsing -Uri "https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1" -OutFile "./install-pyenv-win.ps1"; &"./install-pyenv-win.ps1"

  • If you are getting any UnauthorizedAccess error as below then start Windows PowerShell with the Run as administrator option and run -

    Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine

  • Then re-run the previous powershell link code.

  • For more details visit this link.

Linux
  • If you wish to install a specific release of Pyenv rather than the latest head, set the PYENV_GIT_TAG environment variable (e.g. export PYENV_GIT_TAG=v2.2.5). 
    curl https://pyenv.run | bash
  • For more details visit this link.

some common command

Here's a cheat sheet of some commonly used commands with pyenv:

  • To list all the available Python versions that can be installed with pyenv:

    pyenv install --list

  • pyenv install: Install a specific Python version.

    pyenv install <version>

  • pyenv versions: List all installed Python versions.

    pyenv versions

  • pyenv global: Set the global Python version to be used.

    pyenv global <version>

  • pyenv local: Set a Python version for the current directory.

    pyenv local <version>

  • pyenv shell: Set a Python version for the current shell session.

    pyenv shell <version>

  • pyenv uninstall: Uninstall a specific Python version.

    pyenv uninstall <version>

  • pyenv rehash: Rehash the installed executables.

    pyenv rehash

  • pyenv which: Display the full path to the executable of a Python version.

    pyenv which <version>

  • pyenv exec: Run a command using a specified Python version.

    pyenv exec <version> <command>

🌱 Creation of virtual environments

  1. Install and create virtual environment:

    pip install virtualenv
python -m venv mlops_env

  2. Activate it: Linux:

    source mlops_env/bin/activate

    Windows:

    C:\Users\path\mlops_env\Scripts\activate

💁 Week 2-3

All about EDA. Check notebook folder.

💁 Week 4

Install & Run Apache Airflow with Docker 👌

  1. Install Docker.
  2. Download apache airflow YAML file Link
  3. Update yaml file and add dockerfile (provided updated files in repo)
  4. Init Airflow: 
    docker-compose up airflow-init
  5. Start Airflow services 
    docker-compose up
  6. Stop Airflow 
    docker-compose down

💁 Week 5

Important links 🔗

  1. For Cornjob stuff : https://crontab.guru/
  2. Scheduler Doc : https://airflow.apache.org/docs/apache-airflow/1.10.1/scheduler.html

Python code format fixing using Black

  1. install black 
    pip install black
  2. fix code format. black then code directory. 
    black .\dags\airline_price.py

Issues:

  1. airflow docker-compose up issue. Showing this error message:
File "/usr/local/lib/python3.9/logging/config.py", line 571, in configure mlops-airflow-webserver-1 | raise ValueError('Unable to configure handler ' mlops-airflow-webserver-1 | ValueError: Unable to configure handler 'processor'

Solution : create logs, dags, data, config, plugins folder manually.

💁 Week 6

To get your access key ID and secret access key

  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. On the navigation menu, choose Users.

  3. Choose your IAM user name (not the check box).

  4. Open the Security credentials tab, and then choose Create access key.

  5. To see the new access key, choose Show. Your credentials resemble the following:

    Access key ID: AKIAIOSFODNN7EXAMPLE
Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

  6. To download the key pair, choose Download .csv file. Store the .csv file with keys in a secure location.

for more details : https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html

MLFlow

We will use mlflow for experiment tracking and model deploying. MLFlow official doc: https://mlflow.org/docs/latest/index.html

💁 Week 7

  1. Updated dag
  2. docker-compose YAML file updated
  3. AWS credential added inside YAML file
  4. RUN docker compose

Runing MLFlow using docker (without Airflow)

  1. To build the Docker image, navigate to the directory containing the Dockerfile and run the following command: 
    docker build -t mlflow-server -f Docker-mlflow .
  2. Once the image is built, you can run the MLflow server using the following command: 
    docker run -p 5000:5000 -v mlflow:/mlflow mlflow-server

Issue

MLflow stopped working after docker-compose up without any error. But work properly without Airflow. 😟

Solution: Reason was there was no space in my C drive for running it. I reomved few big files. and restarted the docker. I worked with the issue.

💁 Week 8

  • We solved MLFlow docker issue
  • Then we discuss about DVC

Commonly used DVC commands:

  • DVC install 
    pip install dvc
  • Init DVC 
    dvc init
  • data versioning 
    dvc add data/
git add data.dvc
  • model versioning 
    dvc add models/

Week #9

Week #10

  • Model deploy with Flask & Docker
  • Prepare Docker-app to deploy flask app
  • Added flask app code and requirement
  • Test notebook added

Runing Flask API using Docker

  1. Build command: 
    docker image build -t flaskapp -f .\Docker-app .
  2. Run command: 
    docker run -p 80:80 -t flaskapp