Merge pull request #650 from BC-SECURITY/release/5.0.3

v5.0.3 into main

.github/CONTRIBUTING.md

@@ -31,14 +31,17 @@ The more information you provide in a Github issue the easier it will be for us
 * For additional guidelines for your PowerShell code itself, check out the [PowerSploit style guide](https://github.com/PowerShellMafia/PowerSploit/blob/master/README.md).
 * For more in-depth docs on developing modules, see the [Module Development docs](https://bc-security.gitbook.io/empire-wiki/module-development)
 
-## Code Formatting
+## Code Formatting and Linting
 * As of Empire 4.4, we are using [psf/black](https://github.com/psf/black) for code formatting.
     * Black is a Python code formatter that helps to keep the codebase uniform and easy to read
 * As of Empire 4.4, we are using [PyCQA/isort](https://github.com/PyCQA/isort)
     * Isort is a Python utility that sorts and formats imports.
+* As of Empire 5.0.1, we are using [charliermarsh/ruff](https://github.com/charliermarsh/ruff) for linting.
+    * Ruff is a python linter that helps identify common bugs and style issues.
 * After implementing your changes:
     1. run `black .` (or `poetry run black .`).
     2. run `isort .` (or `poetry run isort .`).
+    3. run `ruff . --fix` (or `poetry run ruff . --fix`). 
 * The repo is also configured to use [pre-commit](https://pre-commit.com/) to automatically format code.
   * Once you have pre-commit installed, you can run `pre-commit install` to install the pre-commit hooks.
-  * Then pre-commit will execute black and isort automatically before committing.
+  * Then pre-commit will execute black, isort, and ruff automatically before committing.

.github/actions/update-starkiller/action.yml

@@ -34,9 +34,19 @@ runs:
       # but would lose the comments.
       run: |
         sed -i "s/ref: .*/ref: ${{ inputs.starkiller-version }}/" empire/server/config.yaml
+    # If use_temp_dir is true, Starkiller is cloned into a temp directory, the CI will fail,
+    # and the submodule will not be updated. So set it to false, make the changes, then set it back.
+    - name: Update config.yaml use_temp_dir
+      shell: bash
+      run: |
+        sed -i'.bak' "s/use_temp_dir: .*/use_temp_dir: false/" empire/server/config.yaml
     - name: Run starkiller update script
       shell: bash
       run: python empire.py sync-starkiller
+    - name: Reset use_tmp_dir
+      shell: bash
+      run: |
+        mv empire/server/config.yaml.bak empire/server/config.yaml
     - name: Update changelog
       shell: bash
       run: |

.github/ci-and-release.md

@@ -1,12 +1,14 @@
 # CI Processes
 
-## Build and Test
+## Pull Requests - Build and Test
 All pull requests will run the `Lint and Test` workflow.
 
 * The workflow will run `black` and `isort` checks and then run `pytest` on Python 3.8, 3.9, and 3.10.
 * If the pull request is coming from a `release/*` branch, it will build the docker image and run `pytest` on it
 * If the pull request changes the `install.sh` script, it will run the install script on the supported OS and check for errors
 
+When submitting a pull request to `private-main`, the label `auto-merge-downstream` can be added. If the label is present, then merging a branch to `private-main` will automatically trigger the prerelease step of merging `private-main` into `sponsors-main` and `kali-main`.
+
 ## BC-SECURITY/Empire-Sponsors Sponsors & Kali Release Process
 *Note: Starting in 2023, the Kali team will be pulling from the public repo.
 I am keeping the Kali workflows running for now with the exception of the tagging.
@@ -189,7 +191,7 @@ Tagged releases will push to the corresponding tag in DockerHub.
 Requires secrets in the repo `DOCKER_USERNAME` and `DOCKER_PASSWORD` as well as `RELEASE_TOKEN` that has `repo` and `workflow` access.
 
 ## More Information
-TODO: Link to CI/CD blog post once it is written.
+https://www.bc-security.org/using-github-actions-to-manage-ci-cd-for-empire/
 
 ## Contributing
 To update the workflows if you don't have access to the `Empire-Sponsors` repo:

.github/cst-config-base.yaml

@@ -23,7 +23,7 @@ commandTests:
   - name: "ps-empire version"
     command: "./ps-empire"
     args: ["server", "--version"]
-    expectedOutput: ["4.* BC Security Fork"]
+    expectedOutput: ["5.* BC Security Fork"]
 fileExistenceTests:
   - name: 'profiles'
     path: '/empire/empire/server/data/profiles/'

.github/cst-config-docker.yaml

@@ -7,4 +7,4 @@ commandTests:
   - name: "python3 version"
     command: "python3"
     args: ["--version"]
-    expectedOutput: ["Python 3.9.*"]
+    expectedOutput: ["Python 3.11.*"]

.github/docker-compose.yml

@@ -4,7 +4,26 @@ version: '3'
 
 services:
   test:
+    depends_on:
+      - db
+    links:
+      - 'db:db'
     build: ../
     image: bcsecurity/empire-test
-    entrypoint: poetry
-    command: run python -m pytest .
+    entrypoint: /bin/bash
+    platform: linux/amd64
+    command: >
+      -c "DATABASE_USE=sqlite poetry run python -m pytest .
+      && sed -i 's/localhost:3306/db:3306/g' empire/test/test_server_config.yaml
+      && DATABASE_USE=mysql poetry run python -m pytest ."
+  db:
+    image: mysql:8.0
+    restart: always
+    environment:
+      MYSQL_ROOT_PASSWORD: 'root'
+      MYSQL_DATABASE: test_empire
+    volumes:
+     - db:/var/lib/mysql
+volumes:
+  db:
+    driver: local

.github/install_tests/Debian10.Dockerfile

@@ -1,6 +1,7 @@
 FROM debian:buster
 WORKDIR /empire
 COPY . /empire
+RUN sed -i 's/use: mysql/use: sqlite/g' empire/server/config.yaml
 # No to all extras except yes to "Python 3.8"
 RUN echo 'n\nn\nn\ny\n' | /empire/setup/install.sh
 RUN rm -rf /empire/empire/server/data/empire*

.github/install_tests/Debian11.Dockerfile

@@ -1,6 +1,7 @@
 FROM debian:bullseye
 WORKDIR /empire
 COPY . /empire
+RUN sed -i 's/use: mysql/use: sqlite/g' empire/server/config.yaml
 RUN yes n | /empire/setup/install.sh
 RUN rm -rf /empire/empire/server/data/empire*
 RUN yes | ./ps-empire server --reset

.github/install_tests/KaliRolling.Dockerfile

@@ -1,6 +1,7 @@
 FROM kalilinux/kali-rolling:latest
 WORKDIR /empire
 COPY . /empire
+RUN sed -i 's/use: mysql/use: sqlite/g' empire/server/config.yaml
 RUN yes n | /empire/setup/install.sh
 RUN rm -rf /empire/empire/server/data/empire*
 RUN yes | ./ps-empire server --reset

.github/install_tests/ParrotRolling.Dockerfile

@@ -1,6 +1,7 @@
 FROM parrotsec/core:latest
 WORKDIR /empire
 COPY . /empire
+RUN sed -i 's/use: mysql/use: sqlite/g' empire/server/config.yaml
 RUN yes n | /empire/setup/install.sh
 RUN rm -rf /empire/empire/server/data/empire*
 RUN yes | ./ps-empire server --reset

.github/install_tests/Ubuntu2004.Dockerfile

@@ -1,6 +1,7 @@
 FROM ubuntu:20.04
 WORKDIR /empire
 COPY . /empire
+RUN sed -i 's/use: mysql/use: sqlite/g' empire/server/config.yaml
 RUN yes n | /empire/setup/install.sh
 RUN rm -rf /empire/empire/server/data/empire*
 RUN yes | ./ps-empire server --reset

.github/install_tests/Ubuntu2204.Dockerfile

@@ -1,6 +1,7 @@
 FROM ubuntu:22.04
 WORKDIR /empire
 COPY . /empire
+RUN sed -i 's/use: mysql/use: sqlite/g' empire/server/config.yaml
 RUN yes n | /empire/setup/install.sh
 RUN rm -rf /empire/empire/server/data/empire*
 RUN yes | ./ps-empire server --reset

.github/install_tests/cst-config-debian10.yaml

@@ -8,3 +8,11 @@ commandTests:
     command: "python3.8"
     args: ["--version"]
     expectedOutput: ["Python 3.8.*"]
+  - name: "mysql which"
+    command: "which"
+    args: ["mysql"]
+    expectedOutput: ["/usr/bin/mysql"]
+  - name: "mysql version"
+    command: "mysql"
+    args: ["--version"]
+    expectedOutput: ["mysql  Ver 8.0.*"]

.github/install_tests/cst-config-debian11.yaml

@@ -8,3 +8,11 @@ commandTests:
     command: "python3"
     args: ["--version"]
     expectedOutput: ["Python 3.9.*"]
+  - name: "mysql which"
+    command: "which"
+    args: ["mysql"]
+    expectedOutput: ["/usr/bin/mysql"]
+  - name: "mysql version"
+    command: "mysql"
+    args: ["--version"]
+    expectedOutput: ["mysql  Ver 8.0.*"]

.github/install_tests/cst-config-kalirolling.yaml

@@ -7,4 +7,12 @@ commandTests:
   - name: "python3 version"
     command: "python3"
     args: ["--version"]
-    expectedOutput: ["Python 3.10.*"]
+    expectedOutput: ["Python 3.11.*"]
+  - name: "mysql which"
+    command: "which"
+    args: ["mysql"]
+    expectedOutput: ["/usr/bin/mysql"]
+  - name: "mysql version"
+    command: "mysql"
+    args: ["--version"]
+    expectedOutput: ["mysql  Ver 15.*10.*-MariaDB"]

.github/install_tests/cst-config-parrotrolling.yaml

@@ -8,3 +8,11 @@ commandTests:
     command: "python3"
     args: ["--version"]
     expectedOutput: ["Python 3.9.*"]
+  - name: "mysql which"
+    command: "which"
+    args: ["mysql"]
+    expectedOutput: ["/usr/bin/mysql"]
+  - name: "mysql version"
+    command: "mysql"
+    args: ["--version"]
+    expectedOutput: ["mysql  Ver 15.*10.*-MariaDB"]

.github/install_tests/cst-config-ubuntu2004.yaml

@@ -8,3 +8,11 @@ commandTests:
     command: "python3"
     args: ["--version"]
     expectedOutput: ["Python 3.8.*"]
+  - name: "mysql which"
+    command: "which"
+    args: ["mysql"]
+    expectedOutput: ["/usr/bin/mysql"]
+  - name: "mysql version"
+    command: "mysql"
+    args: ["--version"]
+    expectedOutput: ["mysql  Ver 8.0.*"]

.github/install_tests/cst-config-ubuntu2204.yaml

@@ -8,3 +8,11 @@ commandTests:
     command: "python3"
     args: ["--version"]
     expectedOutput: ["Python 3.10.*"]
+  - name: "mysql which"
+    command: "which"
+    args: ["mysql"]
+    expectedOutput: ["/usr/bin/mysql"]
+  - name: "mysql version"
+    command: "mysql"
+    args: ["--version"]
+    expectedOutput: ["mysql  Ver 8.0.*"]

.github/workflows/dockerimage.yml

@@ -12,7 +12,7 @@ jobs:
     if: ${{ github.repository == 'BC-SECURITY/Empire' }}
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: 'recursive'
       - name: Publish Docker

.github/workflows/lint-and-test.yml

@@ -13,63 +13,65 @@ jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-      - uses: psf/black@stable
+      - uses: actions/checkout@v3
+      - uses: psf/black@23.1.0
       - uses: isort/isort-action@master
+      - name: Run ruff
+        run: |
+          pip install ruff
+          ruff .
   test:
     needs: lint
     timeout-minutes: 15
     runs-on: ubuntu-latest
     name: Test Python ${{ matrix.python-version }}
     strategy:
       matrix:
-        python-version: [ '3.8', '3.9', '3.10' ]
+        python-version: [ '3.8', '3.9', '3.10', '3.11' ]
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         if: ${{ github.repository == 'BC-SECURITY/Empire' }}
         with:
           submodules: 'recursive'
       # token is only needed in sponsors repo because of private submodules
       # don't use token in public repo because prs from forks cannot access secrets
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         if: ${{ github.repository == 'BC-SECURITY/Empire-Sponsors' }}
         with:
           submodules: 'recursive'
           token: ${{ secrets.RELEASE_TOKEN }}
+      - name: Install Poetry
+        run: |
+          curl -sL https://install.python-poetry.org | python - -y
       # Poetry cache depends on OS, Python version and Poetry version.
       # https://gist.github.com/gh640/233a6daf68e9e937115371c0ecd39c61
-      - name: Cache Poetry cache
-        uses: actions/cache@v2
-        with:
-          path: ~/.cache/pypoetry
-          key: poetry-cache-${{ runner.os }}-${{ matrix.python-version }}
-      # virtualenv cache should depends on OS, Python version and `poetry.lock` (and optionally workflow files).
-      - name: Cache Packages
-        uses: actions/cache@v2
-        with:
-          path: ~/.local
-          key: poetry-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}-${{ hashFiles('.github/workflows/*.yml') }}
       - name: Setup Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
-      - name: Install Poetry
+          cache: 'poetry'
+      - name: Set up MySQL
         run: |
-          curl -sL https://install.python-poetry.org | python - -y
+          sudo systemctl start mysql
       - name: Install dependencies
         run: |
+          poetry env use ${{ matrix.python-version }}
           poetry install
-      - name: Run test suite
+      - name: Run test suite - mysql
+        run: |
+          DATABASE_USE=mysql poetry run pytest . -v
+      - name: Run test suite - sqlite
         run: |
-          poetry run pytest
+          DATABASE_USE=sqlite poetry run pytest . -v
+
   test_image:
     # To save CI time, only run these tests on the release PRs
     if: ${{ startsWith(github.head_ref, 'release/') }}
     timeout-minutes: 20
     runs-on: ubuntu-latest
     name: Test Docker Image
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: 'recursive'
           token: ${{ secrets.RELEASE_TOKEN }}
@@ -93,10 +95,11 @@ jobs:
     runs-on: ubuntu-latest
     name: Test Install Script
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: 'recursive'
           depth: 0
+          token: ${{ secrets.RELEASE_TOKEN }}
       # To save CI time, only run these tests when the install script is changed
       - name: Get changed files using defaults
         id: changed-files