Skip to content
This repository has been archived by the owner on Nov 18, 2024. It is now read-only.

Add org-wide maintenance workflow #12

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add org-wide maintenance workflows
ferferga committed Nov 1, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit debd3c4a3db134fbc6e15c322770b127246c3204
113 changes: 113 additions & 0 deletions .github/workflows/maintenance.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
name: Org-wide maintenance 🧰

on:
schedule:
- cron: "0 0 * * *"
workflow_dispatch:

jobs:
workflow_removal:
runs-on: ubuntu-latest
name: Workflow cleanup 🧹👷‍♂️
# Errors will probably be caused by excesses in API quota, so we can safely continue.
# Remaining workflows will be removed in the next scheduled run.
continue-on-error: true
strategy:
fail-fast: false
matrix:
repo:
- 'jellyfin/jellyfin'
- 'jellyfin/jellyfin-web'
- 'jellyfin/jellyfin-androidtv'
- 'jellyfin/jellyfin-android'
- 'jellyfin/jellyfin-chromecast'
- 'jellyfin/jellyfin-sdk-csharp'
- 'jellyfin/jellyfin-sdk-kotlin'
- 'jellyfin/jellyfin-sdk-swift'
- 'jellyfin/jellyfin-client-axios'
- 'jellyfin/jellyfin-roku'
- 'jellyfin/jellyfin-vue'
- 'jellyfin/jellyfin-webos'
- 'jellyfin/jellyfin-plugin-anilist'
- 'jellyfin/jellyfin-plugin-anisearch'
- 'jellyfin/jellyfin-plugin-artwork'
- 'jellyfin/jellyfin-plugin-anidb'
- 'jellyfin/jellyfin-plugin-webhook'
- 'jellyfin/jellyfin-plugin-tvmaze'
- 'jellyfin/jellyfin-plugin-ldpauth'
- 'jellyfin/jellyfin-plugin-tvdb'
- 'jellyfin/jellyfin-plugin-fanart'
- 'jellyfin/jellyfin-plugin-tvheadend'
- 'jellyfin/jellyfin-plugin-bookshelf'
- 'jellyfin/jellyfin-plugin-opensubtitles'
- 'jellyfin/jellyfin-plugin-covertartarchive'
- 'jellyfin/jellyfin-plugin-kodisyncqueue'
- 'jellyfin/jellyfin-plugin-nextpvr'
- 'jellyfin/jellyfin-plugin-playbackreporting'
- 'jellyfin/jellyfin-plugin-musicbrainz'
- 'jellyfin/jellyfin-plugin-trakt'
- 'jellyfin/jellyfin-plugin-reports'
- 'jellyfin/jellyfin-plugin-autoorganize'
- 'jellyfin/jellyfin-plugin-tmdb'
- 'jellyfin/jellyfin-plugin-omdb'
- 'jellyfin/jellyfin-plugin-kitsu'
- 'jellyfin/jellyfin-plugin-tmdbboxsets'
- 'jellyfin/jellyfin-plugin-template'
- 'jellyfin/jellyfin-meta-plugins'
- 'jellyfin/jellyfin-media-player'
- 'jellyfin/jellyfin-mpv-shim'
- 'jellyfin/jellyfin-expo'
- 'jellyfin/jellyfin-meta'
- 'jellyfin/jellyfin-artwork'
- 'jellyfin/jellyfin.org'
- 'jellyfin/jellyfin-docs'
- 'jellyfin/jellyfin-kodi'
- 'jellyfin/jellyfin-metapackages'
- 'jellyfin/jellyfin-ffmpeg'
- 'jellyfin/Swiftfin'
- 'jellyfin/jellyfin-apiclient-javascript'
- 'jellyfin/jellyfin-exoplayer-ffmpeg-extension'
- 'jellyfin/Jellyfin.XmlTv'
- 'jellyfin/JavascriptSubtitlesOctopus'

steps:
- name: Clone repository
uses: actions/[email protected]

- name: Set correct permissions
run: chmod +x delete_workflows.sh
working-directory: ./maintenance

- name: Run deletion script 🗑
run: ./delete_workflows.sh ${{ matrix.repo }}
working-directory: ./maintenance
env:
GITHUB_TOKEN: ${{ secrets.JF_BOT_TOKEN }}

dangling_images_removal:
runs-on: ubuntu-latest
name: Cleanup dangling images from GHCR 🧹📦
# Errors will probably be caused by excesses in API quota, so we can safely continue.
# Remaining workflows will be removed in the next scheduled run.
continue-on-error: true
# Pay attention that the org name is not necessary here, as gh will automatically take the one from the logged in user.
strategy:
fail-fast: false
matrix:
container:
- 'jellyfin'
- 'jellyfin-vue'

steps:
- name: Clone repository
uses: actions/[email protected]

- name: Set correct permissions
run: chmod +x delete_ghcr_dangling_images.sh
working-directory: ./maintenance

- name: Run deletion script 🗑
run: ./delete_ghcr_dangling_images.sh ${{ matrix.container }}
working-directory: ./maintenance
env:
GITHUB_TOKEN: ${{ secrets.JF_BOT_TOKEN }}
33 changes: 33 additions & 0 deletions maintenance/delete_ghcr_dangling_images.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash
set -e

# Simple script to remove dangling images from GHCR
# You need to be logged to 'gh' first.

# package_name syntax. i.e: jellyfin-vue
container="$1"
temp_file="ghcr_prune.ids"
rm -rf $temp_file

echo "Fetching dangling images from GHCR..."
gh api /user/packages/container/${container}/versions --paginate > $temp_file

ids_to_delete=$(cat "$temp_file" | jq -r '.[] | select(.metadata.container.tags==[]) | .id')

if [ "${ids_to_delete}" = "" ]
then
echo "There are no dangling images to remove for this package"
exit 0
fi

echo -e "\nDeleting dangling images..."
while read -r line; do
id="$line"
## Workaround for https://github.com/cli/cli/issues/4286 and https://github.com/cli/cli/issues/3937
echo -n | gh api --method DELETE /user/packages/container/${container}/versions/${id} --input -
echo Dangling image with ID $id deleted successfully
done <<< $ids_to_delete

rm -rf $temp_file
echo -e "\nAll the dangling images have been removed successfully"
exit 0
54 changes: 54 additions & 0 deletions maintenance/delete_workflows.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/bin/bash
set -e

# Simple script to remove workflows that are not successful or failed (with completion status like skipped, cancelled, etc.)
# It also removes successful or failed workflows that doesn't have artifacts or logs.
# You need to be logged to 'gh' first

# owner/repo syntax
repo="$1"

temp_file="workflow_runs.payload"
rm -rf $temp_file

echo "Fetching workflows..."
echo $(gh api /repos/${repo}/actions/runs --paginate | jq '{count: .total_count, runs: [.workflow_runs[] | select(.status=="completed") | {id, conclusion}]}') | jq '.' > $temp_file

ids_to_delete=$(cat "$temp_file" | jq -r '.runs | .[] | select((.conclusion!="success") and (.conclusion!="failure")) | .id')

if [ "${ids_to_delete}" = "" ]
then
echo "All workflows are exited successfully or failed. Nothing to remove"
else
echo "Removing all the workflows that are not successful or failed..."
while read -r line; do
id="$line"
## Workaround for https://github.com/cli/cli/issues/4286 and https://github.com/cli/cli/issues/3937
echo -n | gh api --method DELETE /repos/${repo}/actions/runs/${id} --input -

echo "Stale workflow run with ID $id deleted successfully!"
done <<< $ids_to_delete
fi

ids_to_delete=$(cat "$temp_file" | jq -r '.runs | .[] | select((.conclusion=="success") or (.conclusion=="failure")) | .id')

if [ "${ids_to_delete}" = "" ]
then
echo "No workflows to check for logs or artifacts. Exiting..."
else
echo -e "\nDeleting workflows without logs and artifacts..."
while read -r line; do
id="$line"
artifact_count=$(gh api /repos/${repo}/actions/runs/${id}/artifacts | jq -r '.total_count')
if [ "${artifact_count}" = "0" ]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the artifacts count is > 0 if there are logs but 0 once they where cleaned up? (wasn't able to confirm that with the docs)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, artifacts != logs. Artifacts are the binaries produced by workflows and uploaded using this action. More info here.

Right now we don't use artifacts anywhere but in JMP. See this workflow as an example. Artifacts are located at the bottom. Logs are the output of the job

Artifacts also expire, but might have a different expiration date than logs. That's why we need to set a preference order on what's more important, logs or artifacts.

For my script's logic, I choose that artifacts > logs. I assume that maintainers/teams that have configured the artifact retention period to be larger than the retention for logs have their reasons to do so, so I'm not allowed to remove workflows runs that might have valuable data.

How the script works:

As I mentioned, for each workflow, the script checks if there are artifacts:

  • IF TRUE (total_count != 0): We continue the loop to the next one and ignore if the workflow run has logs or not because artifacts are more relevant.
  • IF FALSE (total_count == 0): We call the API endpoint to download the logs (there is no API endpoint for querying for its existence, so I must perform the download straightaway). If the logs doesn't exist, GitHub will reply with HTTP 401 Gone, so we can remove the workflow run without further ado.

gh exits with code 1 when it receives the 401, so we can use the standard || in bash to run the deletion command. This happens in this line:

gh api --silent /repos/${repo}/actions/runs/${id}/logs || \ 
				echo -n | gh api --method DELETE /repos/${repo}/actions/runs/${id} --input - && \
				echo "Workflow run without logs and artifacts with ID $id deleted successfully!"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow I must have been blind not seeing that gh api --silent /repos/${repo}/actions/runs/${id}/logs, all good then xD

then
gh api --silent /repos/${repo}/actions/runs/${id}/logs || \
echo -n | gh api --method DELETE /repos/${repo}/actions/runs/${id} --input - && \
echo "Workflow run without logs and artifacts with ID $id deleted successfully!"
fi
done <<< $ids_to_delete
fi

rm -rf $temp_file
echo -e "\n\nFinished the workflow run cleaning process"
exit 0