Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docker] add image-building workflow and update docker file #18

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/workflows/docker_image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Docker Image CI

on:
workflow_dispatch:

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push-image:
runs-on: [self-hosted, Linux, X64, Docker]

permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Log in to the Container registry
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
flavor: |
latest=auto
tags: |
type=sha,format=long
type=raw,value=latest,enable={{is_default_branch}}

- name: Build and push Docker image
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: MAX_JOBS=64
19 changes: 14 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,36 +1,45 @@
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04

ARG MAX_JOBS=32

WORKDIR /app

COPY ./patches /app

ENV TZ=Etc/UTC \
DEBIAN_FRONTEND=noninteractive \
NINJA_MAX_JOBS=32 \
MAX_JOBS=32 \
TORCH_CUDA_ARCH_LIST="8.0 8.6 8.9 9.0"
NINJA_MAX_JOBS=$MAX_JOBS \
MAX_JOBS=$MAX_JOBS \
TORCH_CUDA_ARCH_LIST="8.0 8.6 8.9 9.0"\
TORCH_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1"\
_GLIBCXX_USE_CXX11_ABI=1\
PYTORCH_VERSION="2.2.0a0+git6c8c5ad"


RUN apt-get update \
&& apt-get install -y software-properties-common \
&& apt-get install -y python3.10 python3-pip git git-lfs cmake ninja-build \
&& apt-get install -y python3.10 python3-pip git git-lfs cmake ninja-build build-essential gcc g++ \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& ln -sf /usr/bin/python3.10 /usr/bin/python3 \
&& ln -sf /usr/bin/pip3.10 /usr/bin/pip3 \
&& pip3 install --upgrade pip setuptools wheel \
&& apt update \
&& apt install -y python-is-python3
&& apt install -y python-is-python3 openssh-server \
&& apt clean

RUN bash build_pytorch_w_patch.sh

RUN bash build_torchvision.sh

RUN bash build_torchdistX_w_patch.sh


RUN pip3 install --no-cache-dir packaging \
&& pip3 install --no-cache-dir mpmath==1.3.0 \
&& pip3 install --no-cache-dir "setuptools>=69.0.0" \
&& pip3 install --no-cache-dir regex \
&& pip3 install --no-cache-dir pillow \
&& pip3 install --no-cache-dir pybind11 \
&& pip3 install --no-cache-dir einops \
&& pip3 install --no-cache-dir expecttest \
Expand Down
4 changes: 2 additions & 2 deletions patches/build_pytorch_w_patch.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#! usr/bin/bash
#!/usr/bin/bash
set -e

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
Expand All @@ -13,7 +13,7 @@ fi

git clone --branch v2.2.1-rc3 --depth 1 https://github.com/pytorch/pytorch.git
pushd pytorch
git apply $PATCH_PATH
git apply "$PATCH_PATH"
git submodule sync
git submodule update --init --recursive --depth 1
pip3 install -r requirements.txt
Expand Down
4 changes: 2 additions & 2 deletions patches/build_torchdistX_w_patch.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#! usr/bin/bash
#!/usr/bin/bash
set -e

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
Expand All @@ -15,7 +15,7 @@ git clone --depth 1 https://github.com/pytorch/torchdistx.git
pushd torchdistx
git pull
git checkout 9c1b9f5cb2fa36bfb8b70ec07c40ed42a33cc87a
git apply $PATCH_PATH
git apply "$PATCH_PATH"
git submodule sync
git submodule update --init --recursive --depth 1
cmake -DTORCHDIST_INSTALL_STANDALONE=ON -GNinja -DCAKE_CXX_COMPILER_LAUNCHER=ccache -B build
Expand Down
11 changes: 11 additions & 0 deletions patches/build_torchvision.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/usr/bin/bash
set -e

# build torchvision
git clone -b v0.17.0 https://github.com/pytorch/vision.git
pushd vision
python3 setup.py install
popd
rm -rf vision
python3 -c "import torchvision"
pip3 list | grep torch