Skip excluded architectures #2769

FrostyX · 2023-06-11T22:11:53Z

I am updating the existing tools to still run only for RPM builds, so this commit changes nothing, only allows for creating automation that can be hooked after SRPM builds.

FrostyX · 2023-06-11T22:17:34Z

rpmbuild/copr_rpmbuild/automation/srpm_results.py

+        srpm = locate_srpm(self.resultdir)
+        hdr = get_rpm_header(srpm)
+        keys = ["name", "exclusivearch", "excludearch"]
+        return {key: hdr[key] for key in keys}


I know we agreed on using python3-specfile but I tried it and couldn't easily get the exclusivearch from it so I used python3-rpm instead. We already use it on some places, so it was trivial to use here.

IMHO there is no advantage in using one over the other, so I would just pick whatever is easiest to use. But if we really want specfile, I can try to figure out how to use it.

Seems like OK for the first step at least. specfile could resolve some some issues like

%if 0%{?fedora} > 39 ExcludeEarch: x86_64 %endif

Now thinking again, are such spec files actually resolvable on the single builder that did source build?

@nforro can you provide a guidance to @FrostyX please?

IMHO there is no advantage in using one over the other, so I would just pick whatever is easiest to use.

With specfile you can get ExclusiveArch like this:

spec = Specfile(...) with spec.tags() as tags: try: exclusivearch = tags.exclusivearch.value except AttributeError: exclusivearch = None

or (to get all of them):

spec = Specfile(...) with spec.tags() as tags: exclusivearches = [t.value for t in tags if t.normalized_name == "Exclusivearch"]

IMHO there is no advantage in using one over the other, so I would just pick whatever is easiest to use.

There is a difference, with specfile you can access all tags, even those that are enclosed in conditions that would resolve to false on the system where the code is running, while rpm gives you only resolved tags.

BTW, you can use specfile and still access the underlying rpm.spec object with Specfile.rpm_spec property.

Can I affect the conditions, to get the answer to question like "What is the ExclusiveArch value for Fedora 38+"?

Yes, expanded_value is fine, but that doesn't answer the question; we want expanded_value on specific distribution (e.g. RHEL 9). Can we use python-specfile for this?

Yes. You can specify macros you want to be defined before every expansion (you can't delete macros loaded from macro files, but you can redefine them):

spec = Specfile("python-specfile.spec", macros=[("fedora", "39"), ("rhel", "")]) # explicitly specify parsed preamble section here, so that macros and conditions are resolved # be aware though that if you make any changes the original section will be overwritten with spec.tags(spec.parsed_sections.package) as tags: try: exclusivearch = tags.exclusivearch.expanded_value except AttributeError: exclusivearch = None # for read-only access it's probably better to avoid the context manager tags = spec.tags(spec.parsed_sections.package).content try: exclusivearch = tags.exclusivearch.expanded_value except AttributeError: exclusivearch = None

You can do the same thing with rpm, but it would be rather cumbersome.

This way of working with spec files is quite revolutionary, thanks.

Thank you for the code example @nforro, worked for me perfectly.
I updated the PR.

I submitted a documentation request packit/specfile#243

FrostyX · 2023-06-11T22:21:01Z

The #1701 added the python-specfile into the copr-backend.spec, and I don't quite feel this is ideal.
Would you @FrostyX mind moving that (spec / SRPM analysis from backend to builder, and create the
results.json or alike for SRPM builds? (similar to what we have for BuildChroot handling)

I agree, and I have no problem to add it to this PR or to create a follow-up PR for it. But first, let's figure out the exclude-arch part.

This is needed because otherwise backend can set some chroots as skipped after the SRPM build is completed but then DistGit imports the package and sets all (even the skipped) chroots as pending.

FrostyX · 2023-06-12T18:53:16Z

@tstellar, if the results for your package looked like this, would that be okay?

tstellar · 2023-06-12T19:06:04Z

The UI looks good, what will the json returned by the monitor API call look like?

FrostyX · 2023-06-12T19:16:08Z

Like this:

[
    {
        "build_id": 2914549,
        "chroot": "fedora-37-ppc64le",
        "name": "biosdevname",
        "pkg_version": "0.7.3-11",
        "state": "skipped"
    }
    ,
    {
        "build_id": 2914549,
        "chroot": "fedora-37-x86_64",
        "name": "biosdevname",
        "pkg_version": "0.7.3-11",
        "state": "succeeded"
    }
    ,
    {
        "build_id": 2914549,
        "chroot": "fedora-37-aarch64",
        "name": "biosdevname",
        "pkg_version": "0.7.3-11",
        "state": "skipped"
    }
    ,
    {
        "build_id": 2914549,
        "chroot": "fedora-37-s390x",
        "name": "biosdevname",
        "pkg_version": "0.7.3-11",
        "state": "skipped"
    }
    ,
    {
        "build_id": 2914549,
        "chroot": "fedora-37-armhfp",
        "name": "biosdevname",
        "pkg_version": "0.7.3-11",
        "state": "skipped"
    }
]

Please ignore the build ID difference from the screenshot.

tstellar · 2023-06-12T21:19:58Z

@FrostyX That looks great. Thank You!

xsuchy · 2023-06-12T22:47:54Z

beaker-tests/Sanity/copr-cli-basic-operations/runtest-exclusivearch.sh

+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see http://www.gnu.org/licenses/.


Copyright year does not match. And the header is too long. Can you either omit it or use the two-liner from https://github.com/david-a-wheeler/spdx-tutorial#spdx-license-expressions-in-source-code-files ?

praiskup · 2023-06-19T07:00:49Z

beaker-tests/Sanity/copr-cli-basic-operations/runtest-exclusivearch.sh

+        chroots+=" --chroot fedora-$FEDORA_VERSION-s390x"
+
+        rlRun "copr-cli create ${NAME_PREFIX}ExclusiveArch $chroots"
+        rlRun "copr-cli build-distgit ${NAME_PREFIX}ExclusiveArch --name biosdevname --commit $BRANCH"


Is this testing both excludearch and exclusivearch? It would be nice to have both tested

This test was only for ExclusivAarch. I added another one for ExcludeArch

praiskup · 2023-06-19T07:09:47Z

rpmbuild/copr_rpmbuild/automation/srpm_results.py

+        spec_path = locate_spec(self.resultdir)
+        spec = Spec(spec_path)
+        keys = ["name", "exclusivearch", "excludearch"]
+        return {key: getattr(spec, key) for key in keys}


The exclusive arch is a bit different, I think it can be specified per subpackage.

Do you have a tip for any package that does either of those per subpackage, please?

binutils has ExcludeArch for sub-packages, but it's hidden behind the --with crossbuilds flag.

lammps package also has this.

procyon package has ExclusiveArch for its sub-packages, so that should give you test cases for both ExcludeArch and ExclusiveArch.

Nice, procyon is even more specific - that's the variant of package with BuildArch: noarch with ExclusiveArch for sub-packages. https://bugzilla.redhat.com/show_bug.cgi?id=1394853

Thank you @tstellar, and @praiskup.
I added one more test for procyon just to make sure the parsing doesn't fail on it.
The PR can IMHO be merged now.

FrostyX · 2023-06-19T10:44:11Z

rpmbuild/copr_rpmbuild/helpers.py

+
+    def __init__(self, path):
+        try:
+            macros = [("fedora", "39"), ("rhel", "")]


We should loop over all chroots in the future. For now, we should have macros = [] and put a TODO into the code.

Fix fedora-copr#2137

…tory

FrostyX · 2023-07-17T22:34:04Z

I finally created an issue for rpm.org, to have the macros documented for sub-packages
rpm-software-management/rpm-web#43

rpmbuild: run automation classes for both SRPM and RPM builds

d2d9853

I am updating the existing tools to still run only for RPM builds, so this commit changes nothing, only allows for creating automation that can be hooked after SRPM builds.

FrostyX commented Jun 11, 2023

View reviewed changes

FrostyX added the needs-tests label Jun 11, 2023

frontend: don't touch skipped chroots when import is completed

e196fd7

This is needed because otherwise backend can set some chroots as skipped after the SRPM build is completed but then DistGit imports the package and sets all (even the skipped) chroots as pending.

FrostyX force-pushed the exclusivearch branch from 1c554c6 to 8fe246b Compare June 12, 2023 18:50

FrostyX removed the needs-tests label Jun 12, 2023

xsuchy reviewed Jun 12, 2023

View reviewed changes

FrostyX force-pushed the exclusivearch branch from 8fe246b to 63d8869 Compare June 13, 2023 19:21

praiskup reviewed Jun 19, 2023

View reviewed changes

FrostyX commented Jun 19, 2023

View reviewed changes

FrostyX force-pushed the exclusivearch branch from 8f17261 to 4645e5d Compare June 20, 2023 23:28

praiskup added the needs-tests label Jun 26, 2023

FrostyX added 3 commits June 26, 2023 22:42

frontend, backend, rpmbuild: skip excluded chroots

6cf9499

Fix fedora-copr#2137

rpmbuild: query exclusivearch and excludearch from spec instead of SRPM

dbf9ced

rpmbuild: extract spec into resultdir regardless of the current direc…

b2530cc

…tory

FrostyX force-pushed the exclusivearch branch from 4645e5d to b2530cc Compare June 26, 2023 20:57

FrostyX removed the needs-tests label Jun 27, 2023

praiskup approved these changes Jun 27, 2023

View reviewed changes

praiskup merged commit 48c4d99 into fedora-copr:main Jun 27, 2023
9 checks passed

praiskup mentioned this pull request Jun 28, 2023

Document "how to overwrite/undefine macros" packit/specfile#243

Closed

2 tasks

FrostyX mentioned this pull request Jul 17, 2023

What does ExclusiveArch and ExcludeArch do for sub-packages? rpm-software-management/rpm-web#43

Open

FrostyX mentioned this pull request Aug 10, 2023

Unstoppable "running" job #2841

Closed

FrostyX mentioned this pull request Dec 13, 2023

Copr build system should respect ExcludeArch and ExclusiveArch #2630

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip excluded architectures #2769

Skip excluded architectures #2769

FrostyX commented Jun 11, 2023

FrostyX Jun 11, 2023

praiskup Jun 12, 2023

xsuchy Jun 12, 2023

nforro Jun 13, 2023

praiskup Jun 13, 2023

praiskup Jun 13, 2023

nforro Jun 13, 2023

praiskup Jun 13, 2023

FrostyX Jun 13, 2023 •

edited

Loading

praiskup Jun 28, 2023

FrostyX commented Jun 11, 2023

FrostyX commented Jun 12, 2023

tstellar commented Jun 12, 2023

FrostyX commented Jun 12, 2023

tstellar commented Jun 12, 2023

xsuchy Jun 12, 2023

FrostyX Jun 13, 2023

praiskup Jun 19, 2023

FrostyX Jun 20, 2023

praiskup Jun 19, 2023

FrostyX Jun 20, 2023

tstellar Jun 21, 2023

tstellar Jun 21, 2023

tstellar Jun 21, 2023

praiskup Jun 21, 2023

FrostyX Jun 26, 2023

FrostyX Jun 19, 2023

FrostyX Jun 20, 2023

FrostyX commented Jul 17, 2023

Skip excluded architectures #2769

Skip excluded architectures #2769

Conversation

FrostyX commented Jun 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrostyX Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrostyX commented Jun 11, 2023

FrostyX commented Jun 12, 2023

tstellar commented Jun 12, 2023

FrostyX commented Jun 12, 2023

tstellar commented Jun 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrostyX commented Jul 17, 2023

FrostyX Jun 13, 2023 •

edited

Loading