rsync to stagingyum per distro

[ehelms]

The rsync script on web01 needs the version to exist on the rsync target in order to properly know what release to work with. In order to include the release we need to rsync per OS rather than one command for the whole version.

[ekohl]

The idea behind sync a whole version is that it's atomic. For Debian (where we have multiple directories) we join the paths:

jenkins-jobs/theforeman.org/pipelines/lib/packaging.groovy

Line 462 in d78f0f3

/usr/bin/rsync -avPx ${deb_paths.join(' ')} ${target_path}

That way it's a single atomic action and everything gets promoted as one. You don't have the risk (*) that EL9 is newer than EL8 because the network failed after EL9 was done.

I still think that we should solve it similar to theforeman/foreman-infra#1994 instead. I'll leave some comments there.

• Technically, everything can crash, but you minimize the risk

[evgeni]

The idea behind sync a whole version is that it's atomic. For Debian (where we have multiple directories) we join the paths:

jenkins-jobs/theforeman.org/pipelines/lib/packaging.groovy

Line 462 in d78f0f3

/usr/bin/rsync -avPx ${deb_paths.join(' ')} ${target_path}

While the above excerpt looks like it, we don't do that. deb_paths are the paths of the actual debs.

The place where we actually call rsync methods from is here:

jenkins-jobs/theforeman.org/pipelines/lib/packaging.groovy

Lines 186 to 197 in d78f0f3

	echo "Repos to rsync: ${repos}"
	repos.each { key, value ->
	if (value.type == 'stage') {
	steps.rsync_packages[key] = {
	rsync_to_debian_stage(value.suite, value.component, value.deb_paths)
	}
	} else {
	steps.rsync_packages[key] = {
	rsync_to_debian_release(value.suite, value.component, value.deb_paths)
	}
	}
	}

And repos has keys of the form ${suite}-${component}, which is bullseye-3.9, focal-plugins etc.

You can also see it in Jenkins where we have separate "push-{version}-{os}" steps for stage→prod:
https://ci.theforeman.org/blue/organizations/jenkins/foreman-nightly-deb-pipeline/detail/foreman-nightly-deb-pipeline/1353/pipeline
https://ci.theforeman.org/blue/organizations/jenkins/foreman-3.8-release-pipeline/detail/foreman-3.8-release-pipeline/6/pipeline/166

And "{os}-{version}" for packaging → stage:
https://ci.theforeman.org/blue/organizations/jenkins/foreman-packaging-deb-3.8-release/detail/foreman-packaging-deb-3.8-release/12/pipeline/131

[ehelms]

So what's the verdict on this PR? Pipelines continue to be broken without a resolution.

…

On Wed, Nov 22, 2023, 3:02 AM Evgeni Golov ***@***.***> wrote: The idea behind sync a whole version is that it's atomic. For Debian (where we have multiple directories) we join the paths: https://github.com/theforeman/jenkins-jobs/blob/d78f0f3b828444b109105fa5d27927be3f158c31/theforeman.org/pipelines/lib/packaging.groovy#L462 While the above excerpt looks like it, we don't do that. deb_paths are the paths of the actual debs. The place where we actually call rsync methods from is here: https://github.com/theforeman/jenkins-jobs/blob/d78f0f3b828444b109105fa5d27927be3f158c31/theforeman.org/pipelines/lib/packaging.groovy#L186-L197 And repos has keys of the form ${suite}-${component}, which is bullseye-3.9, focal-plugins etc. You can also see it in Jenkins where we have separate "push-{version}-{os}" steps for stage→prod: https://ci.theforeman.org/blue/organizations/jenkins/foreman-nightly-deb-pipeline/detail/foreman-nightly-deb-pipeline/1353/pipeline https://ci.theforeman.org/blue/organizations/jenkins/foreman-3.8-release-pipeline/detail/foreman-3.8-release-pipeline/6/pipeline/166 And "{os}-{version}" for packaging → stage: https://ci.theforeman.org/blue/organizations/jenkins/foreman-packaging-deb-3.8-release/detail/foreman-packaging-deb-3.8-release/12/pipeline/131 — Reply to this email directly, view it on GitHub <#381 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHT47QOZVMV3C3Q642QIDYFWWPZAVCNFSM6AAAAAA7TFM7AOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRSGI3TSMRYGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ekohl]

I was playing a bit with it and didn't finish it yesterday, but in theforeman/foreman-infra#1994 (comment) I wrote down my findings.

The short version is to add trailing slashes, so my recommendation is to rsync ${collection}/${version}/ ${user}@${host}:rsync_cache/${target}/${version}/

[ekohl]

I didn't test this out end-to-end, but since it's what I recommended based on local testing I approve :)