split and move web to EL9

[evgeni]

• puppet should be able to deploy a webserver with EL8 (according to tests/vagrant)
• need to check if software used actually accepts our config (e.g. freight, awstats)

plan:

• install web02 with el8
• copy data
• make publishing publish to both
• verify
• switch DNS

side-plan:

• split into three: yum / deb / rest?
• "probably not"

[ekohl]

When replacing this, keep #1692 in mind and migrate it to postfix.

[ekohl]

Next year CentOS Stream 8 will go EOL, so perhaps we should move straight to EL9.

[evgeni]

AI for @evgeni: write up how to split that box into nice chunks

[evgeni]

web01 currently hosts 5 "services":

• deb (archivedeb: 45G, deb: 45G, stagingdeb: 6G)
• yum (yum: 57G, stagingyum: 7G)
• web (180M)
• downloads (12G)
• debugs (3M)

All of these are available via HTTP, all but web is also available via rsync (and debugs allows writing via rsync).

I would like to

• split this box up into 3 distinct machines, each doing one service only:
  • deb
  • yum
  • downloads
• move web to GitHub pages.
  • One of the reasons we kept this on "own" infra is to analyze RSS logs, which we didn't do in a while. So let's drop this and then get back to that tiny detail once we need it again.
  • Ewoud mentioned that in the past GH had issues to render our page
  • I configred https://gh.theforeman.org/ to serve via GH Pages and it seems to work just fine.
• drop debugs, it's seldomly used and if people want to share files with us they usually find other ways.
• stop doing rsync completely (sorry @alexjfisher)

yum and downloads can trivially be deployed on EL9 (downloads just needs apache and ssh/rsync, yum also needs createrepo stack, but that's easy).
deb is a bit harder, as it's using freight and there is no freight in EPEL9 -- nothing we can't fix, but it's additional work.

[ekohl]

Reminder we talked about starting rpm.theforeman.org with a new layout: #1937 (comment)

[ehelms]

Does stopping rsync affect how we get RPMs from Jenkins / Local into staging? and from staging to production?

[evgeni]

Pushing no, that happens via rsync-via-ssh, which would still work
Pulling maybe, will research

[evgeni]

So the only place we use rsync:// today is https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/web/files/deploy-yumrepo.sh.
This is getting executed by Jenkins on web01 by calling ssh yumrepo@web01 and the above being set as a force command.
https://github.com/theforeman/jenkins-jobs/blob/39d06222770cd02a83b98cf7490cb93f8bbddc7b/theforeman.org/pipelines/lib/release.groovy#L10-L14

The idea is that it copies things from stagingyum to yum. There is no need for it to use rsync:// at all -- all files are available locally.

Untested, but I think #2040 should do it.

[evgeni]

Should update privacy policy when moving to Github, see https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages#data-collection and https://docs.github.com/en/site-policy/privacy-policies/github-general-privacy-statement

[evgeni]

rsync drop

• discourse announce: https://community.theforeman.org/t/sunsetting-rsync-theforeman-org/37884/
• rss announce: Sunsetting rsync.theforeman.org theforeman.org#2142
• use a local path when copying things from stagingyum to yum #2040
• local rsync fixes #2057
• don't advertise rsync:// now that we want to drop it #2061
• drop stagingyum from rsync config #2059
• drop rsync from web #2066
• drop DNS record

[evgeni]

I realized moving our website to GHP would break the /latest/ and "next" redirects we have:

foreman-infra/puppet/modules/web/manifests/vhost/web.pp

Lines 33 to 36 in e3a6a09

	$docs_rewrites = [
	{ 'rewrite_rule' => ["^/manuals/latest(.*) /manuals/${stable}\$1 [R,L]"] },
	{ 'rewrite_rule' => ["^/manuals/${next}(.*) /manuals/nightly\$1 [R,L]"] },
	]

Not sure it's worth the hassle, but wanted to mention it.

Alternatively, host it on the same box as new-downloads and front it by fastly?

[ekohl]

I realized moving our website to GHP would break the /latest/ and "next" redirects we have:

For next I propose to avoid it with theforeman/foreman#10155.

As for latest: it's often mentioned in our theforeman.org repo. Most common offenders appear to be plugins.

[evgeni]

Question: What should happen with https://debugs.theforeman.org/awstats/? The stats for all vhosts (but web) are useless, because they are already behind the CDN. When we either move www to GH, or front it by Fastly, it will become useless too.

I vote on removing that early (as in: now)?

[ekohl]

I don't recall looking at it for quite some time, so I wouldn't miss it.

[evgeni]

AI:

• evgeni create VMs for DEB and RPM
• evgeni do DEB (freight needs to be built based on fedora for epel9)
• eric do RPM (createrepo version matches :yay:)
• rest later

[evgeni]

o goth, naming bikeshedding. I suggested debrepo01 and rpmrepo01 but @ekohl wants repo-<type>01, so it sorts.

@ehelms please play tie breaker

[ehelms]

repo-<type>01 works for me

[evgeni]

repo-deb01.osuosl.theforeman.org and repo-rpm01.osuosl.theforeman.org are up.
they have a second (yet unused) disk for lvm/data.

[evgeni]

Evgeni did freight: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-87076f97c3

[evgeni]

[root@repo-deb01 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.
  Creating devices file /etc/lvm/devices/system.devices
[root@repo-deb01 ~]# vgcreate repo_deb01_data /dev/sdb
  Volume group "repo_deb01_data" successfully created
[root@repo-deb01 ~]# pvs
  PV         VG              Fmt  Attr PSize    PFree   
  /dev/sdb   repo_deb01_data lvm2 a--  <150.00g <150.00g
[root@repo-deb01 ~]# lvcreate -L100G -n www repo_deb01_data
  Logical volume "www" created.
[root@repo-deb01 ~]# mkfs.xfs /dev/mapper/repo_deb01_data-www 
meta-data=/dev/mapper/repo_deb01_data-www isize=512    agcount=4, agsize=6553600 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=26214400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.
[root@repo-deb01 ~]# echo "/dev/mapper/repo_deb01_data-www /var/www xfs defaults 0 0" >> /etc/fstab 
[root@repo-deb01 ~]# systemctl daemon-reload
[root@repo-deb01 ~]# mount -a
[root@repo-deb01 ~]# ls /var/www/
[root@repo-deb01 ~]# 

[evgeni]

repo-deb01 has the signing key and the repo content from yesterday imported, things seem to work fine so far.

[evgeni]

needs #2129 as the next thing to happen

[evgeni]

• debian stuff is now handled by repo-deb01
• web/downloads by website01 (and behind fastly)
• yum/rpm outstanding

[evgeni]

plan for yum/rpm:

• add web::vhost::yum and web::vhost::stagingyum to profile::repo::rpm
• mount the existing (unused) disk via LVM as /var/www
• rsync things from web01
• point fastly at it
• point publishing configs at it (jenkins, rel-eng)
• [some time in the future] actually do RPM.theforeman.org with new layout

[evgeni]

[root@web01 ~]# shutdown -h now
Connection to web01 closed by remote host.