CentOS 6 deprecation

[bastelfreak]

This issue is a result of #228

CentOS 6 main support is end of life. In the past we removed operating systems when their main support cycle is reached. We never cared about LTS/extended support (like Debian does for oldoldstable or RedHat offers for money). I want to continue this for CentOS 6 and remove it from metadata.json. This will automatically disable unit/acceptance tests for those versions as well. Such a PR should be marked backwards-incompatible. This ensures a major release happens.

[bastelfreak]

@voxpupuli/collaborators please provide feedback :)

[ekohl]

👍 since not a lot of modules still support it. While doing the last modulesync we also saw that a few that did claim support were actually broken.

IMHO we should also drop any compatibility code. Typically this is sysvinit support for the Red Hat OS family since EL7 moved to systemd but there may be more.

[crazymind1337]

Sounds good for me. 👍

[ekohl]

FYI: with VP msync 4.0.0 we automatically enable acceptance tests if they're there for all supported operating systems. In doing so I found a few broken EL6 versions so some modules have dropped it already. To me this is just more proof that with 4.0.0 we're making a big step forward in maintainability of modules.

[b4ldr]

no issue from me dropping EOL OS's

[jhoblitt]

👍 "He's dead, Jim"

[ekohl]

They also removed the vagrant image and dropped it from the mirror networks. That means we can't test it anymore. I've submitted more PRs to drop it.

[bastelfreak]

I'm also +1 for dropping it. In many modules it was already broken, we just didn't test it. All others are now untestable. I don't see us investing a huge amount of time to reimplement acceptance tests + fix existing issues for CentOS 6. Normally I would say a week is enough for such an issue, but given the holidays in the US we might want to keep it open a few more days.

[seanmil]

While CentOS 6 may be EOL, RHEL 6 has years left (in extended support). While much of my codebase is internal I do use a number of Voxpupuli modules as I can and when looking at overhauling modules I do try to look at the community for replacements. Not having support for supported OSes my customers use, even if they are at normal EOL, is a deal-breaker. I'd love to be running only the latest and greatest, but not everyone can dictate the platforms they have to support.

Might I suggest disabling CentOS 6 support for those modules known to be broken, but for those with working tests and capability leave it enabled and continue supporting it until RHEL 6 EOL?

I don't, however, have a great suggestion on what to do given the removal of the vagrant image.

My 2 cents worth. Thanks!

[ekohl]

Without the ability to test it, it's really hard. Remember that we can't use RHEL in our CI. CentOS mirrors are offline for EL6. Quoting https://lists.centos.org/pipermail/centos-devel/2020-December/056208.html:

Just to let you all know that starting from today, mirrorlist.centos.org nodes answer "Invalid release/repo/arch combination" and also that content was removed from mirrors. Johnny pushed the last updates yesterday, that went out to external mirrors .

Today most modules are struggling even to get by. Maintainers like myself don't even know the projects and have to rely on tests, preferably acceptance tests.

I'd love to hear a better solution.

I have been thinking about how to better deal with that. I often get back to the concept of maintainers for sets of modules. Let's say I depend on extlib, nginx and unattended_upgrades. It should be possibly for me to be recognized there. This means some modules can have a bit different policy (support EL6 for longer or similarly, Puppet 5) but those maintainers would be responsible for it. To me that's open source: you need it, you make sure it happens. If you don't, there's no guarantee.

If you're formally recognized (for example, via CODEOWNERS), then it also comes with some responsibility. For example, if a modulesync fails or a build breaks, you should be ready to help fix it.

[attachmentgenie]

i believe with the upstream infrastructure being removed the discussion is now moot. We pride ourselves in being a test driven organisation. So not being able to test is a road block we wont be able to overcome easily. Bar for some sponsor stepping up and keeping extensive mirror infrastructure in place for this to work. RHEL support IMHO has always only been by proxy due it being compatible with centos.

[btravouillon]

The upstream repos are still available at https://vault.centos.org/centos/6/.

[roidelapluie]

I 👍 the removal too

[seanmil]

i believe with the upstream infrastructure being removed the discussion is now moot. We pride ourselves in being a test driven organisation. So not being able to test is a road block we wont be able to overcome easily. Bar for some sponsor stepping up and keeping extensive mirror infrastructure in place for this to work. RHEL support IMHO has always only been by proxy due it being compatible with centos.

How extensive of a mirror system are you looking for? Would two Internet accessible hosts in different datacenters with a mirror of the CentOS 6 vault be sufficient? No promises, but I could look into providing them.

[bastelfreak]

I don't think it's a good idea that our whole CI infrastructure depends on mirrors/work from a single person. the whole idea of vox pupuli is that we work on module as a collective to avoid issues that happen when a single person maintains them.

while I highly appreciate the offer, it adds a huge burden on you. We run CI jobs the whole day, the whole week. the stuff needs to be available and maintained. And adding a miror only solves half of the issues. we would still need people to implement the new mirror infra into the test environment we run. (and we would still need a majority of collaborators saying 👍 to keep maintaining CentOS 6).

[binford2k]

For what it's worth, we're having the same conversations internally. We also explored the idea of hosting our own CentOS 6 mirror, but it's likely that we'll remove CentOS 6 testing & support across the board and continue to support RHEL 6 until its extended EOL in 2024.

[jhoblitt]

I think the only real argument for continuing to support el6 was that it was that it was even less effort to leave existing module support in place rather than actively remove it. However, acceptance testing for an open project is now all but impossible unless effort is invested to keep something alive which "upstream" is actively trying to kill. I think that's a pretty terrible value proposition unless a sponsor steps forward to pay for "extended voxpupuli support" and even then it would require ongoing centos builds of upstream package updates.

[ekohl]

Exactly what @jhoblitt said: unless someone steps up to actually do the work, I'd like to actively remove CentOS 6 support from our modules. Offering a mirror is a nice gesture, but the hard part is actually making sure our CI can use it. I would also like to see a list of modules that people actually want to use on EL6.

Note that it doesn't have to be CentOS 6. If someone can (legally) offer RHEL 6 to use, that would work as well.

IMHO VP is severely understaffed for the amount of modules that we have. Even with CI it's hard to keep up. Without CI, it's impossible to maintain.

[alexjfisher]

However, acceptance testing for an open project is now all but impossible unless effort is invested

I had a little of a mess around this afternoon. I don't actually think it's that much effort.

All I think is actually needed is for me to push a very slightly customised centos6 docker image to our voxpupuli dockerhub namespace.

Dockerfile

FROM centos:6

RUN /bin/rm /etc/yum.repos.d/*
COPY CentOS.repo /etc/yum.repos.d/CentOS.repo
RUN yum update -y \
  && yum clean all

CentOS.repo

[base]
name=CentOS-$releasever - Base
baseurl=https://vault.centos.org/centos/6/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

#released updates
[updates]
name=CentOS-$releasever - Updates
baseurl=https://vault.centos.org/centos/6/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=https://vault.centos.org/centos/6/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

Then BEAKER_PUPPET_COLLECTION=puppet6 BEAKER_setfile=centos6-64m{image=voxpupuli/centos6:latest} bundle exec rake beaker worked for me when I tested on puppet-redis.

@seanmil If I set this up, have you got a few modules you would particularly like to help out keeping CentOS 6 support going for a bit longer?

[alexjfisher]

@Flipez Could you add me (alexjfisher) to our org on dockerhub? I think you and the bot are the only 2 members, and the bot didn't have admin permissions, so I couldn't add myself! ;)

[roidelapluie]

Let's be clear that all modules releases are not going away. You can keep the current version and it will still work for you. Centos 6. I do think that people should migrate to a newer version of CentOS before thinking about new features in Puppet modules.

[roidelapluie]

I'd also note that for puppet-corosync, when we dropped old puppet releases, we did a minor release on an old puppet version supported module. So that is also something, but it would be up to the maintainers of that module to do such releases.

[alexjfisher]

Centos 6. I do think that people should migrate to a newer version of CentOS before thinking about new features in Puppet modules.

Migrating to a newer OS is a lot easier said than done. There are plenty of times where maybe 90% of servers are on the latest OS, but the remaining 10% are stuck until some much bigger project is completed. I think wanting to upgrade a puppet module for bug fixes and new features is reasonable.

[jhoblitt]

@alexjfisher If someone wants to keep using an unsupported OS release then all they need to do is put the unsupported platform in its own environment with pinned version of modules that support it.

[roidelapluie]

Yes, in général people whivh stay behind should have mirrors and also be able to patch old puppet modules if needed.

[igalic]

@jhoblitt wrote:

@alexjfisher If someone wants to keep using an unsupported OS release then all they need to do is put the unsupported platform in its own environment with pinned version of modules that support it.

i was quite surprised how little use people make of environments in these or similar situations where it would be so appropriate and such an easy solution to so many problems

[binford2k]

FWIW, only 1.9% of nodes managed by our Enterprise customers are still on CentOS 6. I do not have similar stats for open source users because we haven't gotten enough telemetry data back yet to project.

[bastelfreak]

Also puppet started to drop CentOS 6 testing: puppetlabs/puppetlabs-apache#2107

@alexjfisher do you have a lost of modules where you would like to see centos6 support / would maintain it?

[attachmentgenie]

if puppet has also started dropping centos6 support. than who are we to keep discussing our support for it. We could propose to those that want to keep supporting it to do this from a branch (similar to the puppet5 proposal) but drop it from main/master

[roidelapluie]

Yes I guess we wouldn't win much being the last ones

[seanmil]

alexjfisher:

@seanmil If I set this up, have you got a few modules you would particularly like to help out keeping CentOS 6 support going for a bit longer?

@alexjfisher so, I looked over my 228 modules I currently have deployed and I was a little surprised that only 10 of them are Voxpupuli. And then, looking at those, the ones that are widely used tend to have little to no particularly specific OS support (just based on a quick glance), the ones that do we use in limited capacity on a small number of systems. Giving it some thought, I believe we didn't ever use more because I think SLES support tended to be lacking (probably because SLES is another difficult to test in free CI environments distribution) and we needed SLES too. So I guess this won't impact me as much as I thought. But I do wish the commercial distros provided some better options for public CI pipelines and testing.

jhoblitt:

@alexjfisher If someone wants to keep using an unsupported OS release then all they need to do is put the unsupported platform in its own environment with pinned version of modules that support it.

Which is a great solution, unless you are already heavily using environments to achieve other means. In our environment we have about 24 environments, and have LTS-release distros in all of them. While I'm working on consolidating / refactoring environment layout we have 8+ years of entrenched stuff doing it this way and it has been taking time. I couldn't practically pin old module versions in their own environments at the moment.

binford2k:

FWIW, only 1.9% of nodes managed by our Enterprise customers are still on CentOS 6. I do not have similar stats for open source users because we haven't gotten enough telemetry data back yet to project.

@binford2k I wouldn't expect that number to be super high, but think you might find that orgs stuck supporting older distros may also be more averse to enabling analytics to be reported, so that number is probably skewed low.

So, for my part, I wish RH provided a reasonable way to run open source CI jobs against LTS releases, but they don't and I concur that it doesn't seem to make sense to continue support in Voxpupuli.

[seanmil]

Oh, not specifically germane to the CentOS 6 discussion, but more around testing commercial distros, this is possibly another wrinkle upcoming? https://blog.centos.org/2020/12/future-is-centos-stream/ It is not entirely clear to me what type of impact this would have from a Puppet module CI testing standpoint. Hopefully little.

[roidelapluie]

I'd indeed create another issue for the centos 8 thing, not taking over this one :)

[jhoblitt]

Related to centos 6, COPR is going to purge el6 packages in ~6months: https://docs.pagure.org/copr.copr/copr_outdated_chroots_removal_policy.html

[towo]

For what it's worth, some Docker builds are already failing because the repositories have either been moved to some archive location or unpublished entirely.

[towo]

Do we have a consensus in the text on #232?