Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TRACKER] collection test failures #1

Open
30 of 49 tasks
gotmax23 opened this issue Dec 3, 2023 · 20 comments
Open
30 of 49 tasks

[TRACKER] collection test failures #1

gotmax23 opened this issue Dec 3, 2023 · 20 comments

Comments

@gotmax23
Copy link
Collaborator

gotmax23 commented Dec 3, 2023 •
edited
Loading

(All the collections that have been retired from Ansible 10.3.0 or otherwise no longer have sanity test failures in that version should be checked off.)

./main.py -i data/9/ansible-9.1.0-sanity.yaml -t build/ansible-build-data/9/ansible-9.1.0-tags.yaml -u data/9/ansible-9.1.0-missing-upstreams.yaml failures --bullet="- [ ]" --reverse | wl-copy
@gotmax23 gotmax23 changed the title [TRACKER] Collection test failures [TRACKER] 9.1.0 collection test failures Jan 23, 2024
@Andersson007 Andersson007 mentioned this issue Jan 24, 2024
Merged
@patchback patchback bot mentioned this issue Jan 24, 2024
Merged
@mariolenz
Copy link

mariolenz commented Jan 27, 2024 •
edited
Loading

I think you can ignore some of those collections since they'll be removed from Ansible 10 anyway:

@mariolenz
Copy link

BTW I see purestorage.fusion on your list. It looks like this collection will be deprecated. So maybe you can ignore this, too.

@mariolenz
Copy link

mariolenz commented Feb 17, 2024 •
edited
Loading

BTW I see purestorage.fusion on your list. It looks like this collection will be deprecated. So maybe you can ignore this, too.

@gotmax23 This collection has been deprecated by it's maintainers and will be removed from Ansible 10: ansible-community/ansible-build-data#374

@gotmax23

This comment was marked as outdated.

Sign in to view
@gotmax23

This comment was marked as resolved.

Sign in to view
@gotmax23

This comment was marked as resolved.

Sign in to view
@gotmax23
Copy link
Collaborator Author

gotmax23 commented Aug 22, 2024 •
edited
Loading

are also a problem. The maintainer says that our results are broken and differ from the results reported by Automation Hub. Perhaps there are sanity ignore entries that are not included in the git repository. Otherwise, I am at a loss.

@gotmax23
Copy link
Collaborator Author

@gundalow, any idea what's going on with the Cisco collections? Looking at the output in https://console.redhat.com/ansible/automation-hub/repo/published/cisco/intersight/import-log/, it seems that AH doesn't run the validate-modules test which is one that we require to pass.

@gotmax23
Copy link
Collaborator Author

A discussion about community.network has been started in https://forum.ansible.com/t/should-we-remove-community-network-from-the-ansible-community-package/8030.

@felixfontein
Copy link

are also a problem. The maintainer says that our results are broken and differ from the results reported by Automation Hub. Perhaps there are sanity ignore entries that are not included in the git repository. Otherwise, I am at a loss.

It turned out that this is actually due to a bug in ansible-test (ansible/ansible#83842). On the other hand, if Automation Hub's sanity tests accepted it, it means that AH doesn't run the validate-modules sanity test. This basically means that successful import in AH doesn't mean that all sanity tests pass. (But it also doesn't mean that the ones not run fail...)

@gotmax23

This comment was marked as outdated.

Sign in to view
@mariolenz
Copy link

@gotmax23 You've started and named this issue [TRACKER] 9.1.0 collection test failures. But in your last comment you mention 10.4.0. I'm a little bit confused...

I suggest to open an new [TRACKER] 10.x collection test failures (and a [TRACKER] 11.x collection test failures once it's been released) issue. And close this one once Ansible 9.x is EOL. I think this would make it easier for us to keep track.

What do you think?

@gotmax23 gotmax23 changed the title [TRACKER] 9.1.0 collection test failures [TRACKER] collection test failures Oct 4, 2024
@gotmax23
Copy link
Collaborator Author

gotmax23 commented Oct 4, 2024

I removed the version number from the issue name. The idea is — for now — to use this issue on a rolling basis to track changes for whatever the latest release is.

@mariolenz
Copy link

@gotmax23 Could you update this now that 11.0.0 is out? 9 and 10 will be dead soon, so the test failures there aren't really interesting anymore.

@gotmax23
Copy link
Collaborator Author

gotmax23 commented Nov 25, 2024 •
edited by mariolenz
Loading

(Duplicates that are already listed in the issue description and that have had issues filled already should be checked off here.)

11.0.0

  • fortinet.fortimanager
    14261 validate-modules
  • community.network (Remove community.network from Ansible 12 ansible-build-data#475)
    654 BANNED_SANITY
  • google.cloud (Remove google.cloud from Ansible 12 ansible-build-data#476)
    203 validate-modules
  • ovirt.ovirt
    136 WRONG_HASH
    1 WRONG_GALAXY_YML_NAME
  • theforeman.foreman
    122 yamllint
  • dellemc.unity
    18 validate-modules
  • awx.awx
    14 validate-modules
    1 WRONG_HASH
  • community.sap_libs
    9 validate-modules
  • sensu.sensu_go (Remove sensu.sensu_go from Ansible 12 ansible-build-data#463)
    7 validate-modules
    1 WRONG_HASH
  • wti.remote
    8 WRONG_HASH
  • community.okd
    6 validate-modules
  • cisco.dnac
    5 validate-modules
  • cisco.mso
    5 validate-modules
  • cyberark.conjur
    3 yamllint
    1 validate-modules
  • netapp.ontap
    4 validate-modules
  • cisco.aci
    3 validate-modules
  • community.mongodb
    2 yamllint
    1 validate-modules
  • cyberark.pas
    2 WRONG_HASH
    1 validate-modules
  • ibm.spectrum_virtualize
    3 WRONG_HASH
  • kubernetes.core
    3 BANNED_SANITY
  • check_point.mgmt
    2 WRONG_HASH
  • inspur.ispim
    2 WRONG_HASH
  • kubevirt.core
    2 WRONG_HASH
  • openstack.cloud
    2 yamllint
  • azure.azcollection
    1 validate-modules
  • cisco.meraki
    1 BANNED_SANITY

Total: 26

@gotmax23
Copy link
Collaborator Author

gotmax23 commented Nov 25, 2024 •
edited
Loading

Follow ups:

@mariolenz
Copy link

community.network, google.cloud and sensu.sensu_go will be removed from Ansible 12: ansible-community/ansible-build-data#504

Should we mention this in your list somehow? Maybe something like:

* [x] community.network (ansible-community/ansible-build-data#475)
  654 BANNED_SANITY

* [x] google.cloud (ansible-community/ansible-build-data#476)
  203 validate-modules

* [x] sensu.sensu_go (ansible-community/ansible-build-data#463)
  7 validate-modules
  1 WRONG_HASH

@gotmax23
Copy link
Collaborator Author

My goal for this issue is more to keep track of whether or not the issues have been resolved as opposed to how (either it's been fixed or the collection has been scheduled for removal), but if you feel that information is helpful, feel free to add it. At some point, I'll also update the script to not list collections that are already scheduled for removal, as we now have structured data about removed collections in the collection-meta file.

@mariolenz
Copy link

Since you don't object, I've added the information about collection removal. At the end of the day, I think this information is helpful. If we will remove a collection because it's officially / we consider it unmaintained, we know that we don't have to have a closer look at it.

BTW Do you think that duplicates that are already listed in the issue description and that have had issues filled already should be checked off? For example, are the ovirt.ovirt issues in Ansible 9 really the same ones as in Ansible 11?

@gotmax23
Copy link
Collaborator Author

BTW Do you think that duplicates that are already listed in the issue description and that have had issues filled already should be checked off? For example, are the ovirt.ovirt issues in Ansible 9 really the same ones as in Ansible 11?

The issue with ovirt.ovirt is that upstream runs the tests in CI but has their own script to preprocess their sources which this repo's tooling does not use. I've just been ignoring it for now. In general, I'm using the issue description to keep track of all open issues. Some collections in the description have multiple issues listed under them, because the original issues were closed, and then I opened new issues after new failures were found in later versions of the collection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mariolenz @felixfontein @gotmax23