deploy.sh

An open source CI/CD system

中文 README_zh.md

Introduction

deploy.sh is a versatile CI/CD program that can be executed manually or automated. It supports various deployment scenarios and integrates with popular CI/CD tools like GitLab-Runner and Jenkins.

Features

• Code style: phpcs, phpcbf, java code style, jslint, shfmt, hadolint...
• Code quality: sonarqube scan, OWASP, ZAP, vulmap...
• Unit test: phpunit, junit...
• Build: npm build, composer install, maven build, gradle build, docker build, pip install ...
• Deploy method: rsync+ssh, rsync, rsync + container image, rsync jar/war, ftp, sftp, kubectl, helm...
• Function test: Jmeter, pytest...
• Performance test: stress test, jmeter, loadrunner
• Notify deploy result: work-weixin, Telegram, Element(Matrix), dingding...
• Renew cert: acme.sh renew cert for https
• Cloud vendors: AWS, Aliyun, Qcloud, Huaweicloud...

Installation

Prerequisites:

• Git
• Bash shell environment

git clone --depth 1 https://github.com/xiagw/deploy.sh.git $HOME/runner

How to automatically detect the programming language

Language	Detection Method
node	Exists package.json or include project_lang=node in README.md
php	Exists composer.json or include project_lang=php in README.md
java	Exists pom.xml or include project_lang=java in README.md
python	Exists requirements.txt or include project_lang=python in README.md
other	Include project_lang=[other] in README.md

project_lang=shell

Quickstart

option [1]. deploy.sh manually

## If your project repository already exists
cd /path/to/<your_project.git>
$HOME/runner/deploy.sh

## If your project repository dose not exist. (deploy.sh will clone it)
$HOME/runner/deploy.sh --git-clone https://github.com/<some_name>/<some_project>.git

option [2]. deploy.sh automated

## crontab
*/5 * * * * for d in /path/to/src/*/; do (cd $d && git pull && $HOME/runner/deploy.sh --cron); done

## run in screen or tmux
while true; do for d in /path/to/src/*/; do (cd $d && git pull && $HOME/runner/deploy.sh --loop); done; sleep 300; done

option [3]. deploy.sh with GitLab-Runner

1. Prepare a gitlab-server and gitlab-runner-server
2. Install gitlab-runner, register to gitlab-server, and start gitlab-runner
3. cd $HOME/runner
4. cp conf/example-deploy.json data/deploy.json ## !!!change to yours!!!
5. cp conf/example-deploy.env data/deploy.env ## !!!change to yours!!!
6. Refer to conf/.gitlab-ci.yaml of this project, setup <your_project.git>/.gitlab-ci.yaml

option [4]. deploy.sh with Jenkins

1. Create job,
2. setup job, run custom shell, bash $HOME/runner/deploy.sh

Examples (with GitLab Server and GitLab-Runner)

Step 1: Prepair Gitlab server

There is already a gitlab server (if not, you can refer to xiagw/docker-gitlab to start one with docker-compose)

Step 2: Prepair Gitlab Runner server

There is already a server that has installed gitlab-runner and register to Gitlab server, (executer is shell)

and make sure gitlab-runner is running properly. sudo gitlab-runner status

Step 3: Prepair Application server (Linux/k8s/microk8s/k3s)

if use rsync+ssh to deploy: The ssh key file had been prepared, and you can login to the Application server without a password from the gitlab-runner server (the key file can be in $HOME/.ssh/, or in the $HOME/runner/data/.ssh/)

if use k8s to deploy: on the gitlab-runner server, prepare ~/.kube/config

Step 4: install, git clone deploy.sh

SSH login to the gitlab-runner server

git clone https://github.com/xiagw/deploy.sh.git $HOME/runner

Step 5: Update $HOME/runner/data/deploy.json, $HOME/runner/data/deploy.env

Refer to the conf/example-deploy.json, conf/example-deploy.env, change to yours configure

cd $HOME/runner
cp conf/example-deploy.json data/deploy.json      ## change to yours
cp conf/example-deploy.env data/deploy.env        ## change to yours

Step 6: Create Gitlab project on gitlab server

Example: created project-A under the root account on gitlab-server (root/project-A)

Step 7: Create root/project-A/.gitlab-ci.yml on gitlab server

Create and submit .gitlab-ci.yml on Gitlab project-A

Step 8: Enjoy CI/CD

FAQ

How to create Helm files for applications project

If you use helm to deploy to k8s, change helm-new.sh for yours (default open port 8080 and 8081)

bash $HOME/runner/bin/helm-new.sh
## change to yours [$HOME/runner/data/helm/<your_project>]

How to resolve gitlab-runner fail

If you use Ubuntu, just rm -f $HOME/.bash_logout

Diagram

graph TB;

Dev -- pull/push --> Java;
Dev -- pull/push --> PHP;
Dev -- pull/push --> VUE;
Dev -- pull/push --> Python;
Dev -- pull/push --> more[More...];
Java -- pull/push --> GIT;
PHP -- pull/push --> GIT;
VUE -- pull/push --> GIT;
Python -- pull/push --> GIT;
more -- pull/push --> GIT;
GIT --> CICD[deploy.sh];
OPS -- shell --> GIT;
OPS -- shell --> CICD;
UI_UE -- sketch --> PD;
PD -- issues --> GIT[GitLab Server];
QA -- issues--> GIT;
testm[Manuel tests] -- test--> QA;
testa[Auto tests] -- test--> QA;
CICD -- deploy --> K8S[k8s/helm];
CICD -- build --> Build;
CICD -- database --> db1[Database manage];
CICD -- cert --> cert[Cert manage];
CICD -- notify --> notify[Notify manage];
CICD -- check --> rev[Code Check];
CICD -- test --> test[Test Center];
notify -- notify --> weixin/dingding/telegram/element;
db1 -- sql --> flyway;
cert -- shell --> acme[acme.sh];
acme -- dns api --> dnsapi;
acme -- web --> www;
dnsapi -- dns api --> dns1[dns api CF];
dnsapi -- dns api --> dns2[dns api ali];
test -- test --> testu[Unit tests];
testu -- test --> testf[Function tests];
testf -- test --> testl[Load tests];
rev -- style --> format[Style];
format --> Lint;
rev -- quality--> quality[Quality];
quality --> Sonarqube;
Build -- push --> Repo[Docker Registry];
K8S -- pri --> ENV_D[ENV develop];
K8S -- pri --> ENV_T[ENV testing];
K8S -- pri --> ENV_M[ENV master];
ENV_D -- pri --> app_d[app 1,2,3...];
app_d -- pri --> cache_d[redis cluster];
cache_d -- pri --> db_d[mysql cluster];
ENV_T -- pri --> app_t[app 1,2,3...];
app_t -- pri --> cache_t[redis cluster];
cache_t -- pri --> db_t[mysql cluster];
ENV_M -- pri --> app_m[app 1,2,3...];
app_m -- pri --> cache_m[redis cluster];
cache_m -- pri --> db_m[mysql cluster];

Loading

Contributing

We welcome contributions to deploy.sh!

Please make sure to update tests as appropriate and adhere to the project's coding standards.

deploy.sh Issue

deploy.sh PR

Donation

If you find this project helpful, consider making a small donation to support its development:

Alipay	WeChat Pay

Digital Currency:

BitCoin

BTC native segwit Address: bc1qaphg63gygfelzq5ptssv3rq6eayhwclghucf8r

BTC segwit Address: 3LzwrtqD6av77XVN68UXWLKaHEtAPEQiPt

ETH/ERC20

ETH/ERC20 Address 0x007779971b2Df368E75F1a660c1308A51f45A02e

BSC/ERC20

BSC/ERC20 Address 0x007779971b2Df368E75F1a660c1308A51f45A02e

TRX/TRC20

TRX/TRC20 Address TAnZ537r98Jo63aKDTfbWmBeooz29ASd73