Skip to content

Commit d743754

Browse files
pietroalbinipietroalbini
authored
Scrape the details page when determining which XML to upload to Trunk (#9275)
Before this PR, to retrieve the XML file to upload from Buildomat to Trunk we relied on hardcoding the supported job names and using the artifacts published with Buildomat's `[[publish]]`. While investigating why Trunk wasn't marking tests as flaky (even after retrying the same job multiple times), Rain discovered that the published artifact remained the same even after retrying CI on the same commit. Turns out this was intended behavior, as [Buildomat doesn't allow overriding already published artifacts](https://github.com/oxidecomputer/buildomat/blob/cc723d4029d1acb687779994a43c52288e6e34c1/server/src/db/mod.rs#L1778-L1788). To temporarily work around the problem, this PR switches the Trunk integration to scrape the HTML details page looking for `junit.xml` files. Hopefully once Buildomat provides a better way to retrieve the file we will be able to remove the hacky scraping code. More details on the rationale and the options considered is available in code comments.
1 parent 163acd1 commit d743754

File tree

1 file changed

+77
-35
lines changed

1 file changed

+77
-35
lines changed

‎tools/upload_buildomat_junit_to_trunk.sh‎

Lines changed: 77 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -94,28 +94,71 @@ for check_run_id in $(jq -r .check_runs[].id api/check_suite_runs); do
9494

9595
gh api "repos/${GITHUB_REPOSITORY}/check-runs/${check_run_id}" > api/check_run
9696
job_url="$(jq -r .details_url api/check_run)"
97+
job_name="$(jq -r .name api/check_run)"
9798

98-
# Figure out where the JUnit XML report lives and the Trunk variant based on the job name. If new
99-
# jobs are added or renamed, you will need to change this too.
99+
# We need to somehow download the JUnit XML from the job artifacts, and there are no good
100+
# solutions available in Buildomat to do so:
100101
#
101-
# The Trunk variant allows to differentiate flaky tests by platform, and should be the OS name.
102-
job_name="$(jq -r .name api/check_run)"
103-
case "${github_app} / ${job_name}" in
104-
"buildomat / build-and-test (helios)")
105-
artifact_series="junit-helios"
106-
artifact_name="junit.xml"
107-
variant="helios"
108-
;;
109-
"buildomat / build-and-test (ubuntu-22.04)")
110-
artifact_series="junit-linux"
111-
artifact_name="junit.xml"
112-
variant="ubuntu"
113-
;;
114-
*)
115-
echo "unsupported job name, skipping upload: ${job_name}"
116-
continue
117-
;;
118-
esac
102+
# - Buildomat supports publishing artifacts at pre-defined URLs with its [[publish]] config
103+
# block, and that's what this script originally used. The problem is, if jobs run multiple
104+
# times for the same commit (for example in case of a retry) Buildomat will only publish the
105+
# artifact of the first job, not for any followup jobs.
106+
#
107+
# - Buildomat has a JSON API that does expose the job outputs, but it's behind authentication
108+
# and this workflow cannot access it. It would also require everyone tweaking the script to
109+
# obtain credentials before running the script locally.
110+
#
111+
# - Buildomat provides an unauthenticated HTML page with all the job details, including download
112+
# links for the job outpouts. It requires scraping the HTML to figure out the correct link.
113+
#
114+
# Out of the three options, the second would be ideal, but cannot implemented now. The first
115+
# option was implemented previously, but to detect flaky tests Trunk needs the correct JUnit
116+
# files when a job is retried. This only leaves us with the third option, which while not ideal
117+
# seems to work fine for now. Sorry.
118+
log_step "detecting the presence of a JUnit report in job ${job_name}"
119+
found_link=""
120+
# curl flags:
121+
# -f: exit 1 if the response is a 4xx or a 5xx
122+
# -L: follow redirects
123+
#
124+
# grep flags:
125+
# -h: don't output file names
126+
# -o: only print what matches, not the whole line
127+
# -P: use perl regexes (needed for \K)
128+
#
129+
# Regex details:
130+
# - We first match `href="`, to only get links as part of an <a> tag.
131+
# - The \K then instructs grep to only print matching text *after* the \K.
132+
# - We finally include everything that looks like a link until a `"` is found.
133+
for link in $(curl -fL --retry 3 "${job_url}" | grep -hoP 'href="\Khttps://[^"]+'); do
134+
case "${link}" in
135+
# Look for Buildomat links for an artefact called "junit.xml".
136+
https://buildomat.eng.oxide.computer/wg/0/artefact/*/junit.xml)
137+
if [[ "${found_link}" == "${link}" ]]; then
138+
echo "note: found the same JUnit link multiple times, only using the first" >&2
139+
elif [[ "${found_link}" == "" ]]; then
140+
echo "success: found ${link}" >&2
141+
rm -f git/junit.xml
142+
curl -fL -o git/junit.xml --retry 3 "${link}"
143+
found_link="${link}"
144+
else
145+
# It could be possible to support multiple JUnit files, but we don't need that
146+
# so far. If you encounter this error feel free to add support for it.
147+
echo "error: found more than one JUnit reports:" >&2
148+
echo "- ${found_link}" >&2
149+
echo "- ${link}" >&2
150+
exit 1
151+
fi
152+
;;
153+
*)
154+
echo "note: ignoring non-JUnit URL ${link}" >&2
155+
;;
156+
esac
157+
done
158+
if [[ "${found_link}" == "" ]]; then
159+
echo "skipping job ${job_name}, as it doesn't contain any JUnit report"
160+
continue
161+
fi
119162

120163
# Configure the environment to override Trunk's CI detection (with CUSTOM=true) and to provide all
121164
# the relevant information about the CI run. Otherwise Trunk will either pick up nothing (when
@@ -133,25 +176,24 @@ for check_run_id in $(jq -r .check_runs[].id api/check_suite_runs); do
133176
vars+=("PR_TITLE=${pr_title}")
134177
fi
135178

179+
echo >&2
136180
echo "these environmnent variables will be set:" >&2
137181
for var in "${vars[@]}"; do
138182
echo "${var}" >&2
139183
done
140184

141-
# The URL is configured through a [[publish]] block in the Buildomat job configuration.
142-
log_step "downloading the JUnit XML report for check run ${check_run_id}..."
143-
junit_url="https://buildomat.eng.oxide.computer/public/file/${GITHUB_REPOSITORY}/${artifact_series}/${commit}/${artifact_name}"
144-
rm -f git/junit.xml
145-
curl -fL -o git/junit.xml --retry 3 "${junit_url}"
146-
147185
# Uploading to Trunk has to happen inside of the git repository at the current commit.
148-
log_step "uploading the JUnit XML report of check run ${check_run_id} to Trunk..."
149-
cd git/
150-
env "${vars[@]}" \
151-
../trunk flakytests upload \
152-
--junit-paths junit.xml \
153-
--variant "${variant}" \
154-
--org-url-slug "${TRUNK_ORG_SLUG}" \
155-
--token "${TRUNK_TOKEN}"
156-
cd ..
186+
if [[ -z "${DRY_RUN+x}" ]]; then
187+
log_step "uploading the JUnit XML report of check run ${check_run_id} to Trunk..."
188+
cd git/
189+
env "${vars[@]}" \
190+
../trunk flakytests upload \
191+
--junit-paths junit.xml \
192+
--variant "${variant}" \
193+
--org-url-slug "${TRUNK_ORG_SLUG}" \
194+
--token "${TRUNK_TOKEN}"
195+
cd ..
196+
else
197+
log_step "skipped upload to Trunk, we are in a dry run"
198+
fi
157199
done

0 commit comments

Comments
 (0)