Plan B for parallel join ( try to get a certificate before joining ) #10

GinoHarlos · 2023-05-01T12:26:39Z

Plan B for parallel join ( try to get a certificate before joining )

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#54311

Description of the changes

Please describe the changes with a few sentences.

Motivation

Ensuring the ability to deploy the "Univention Corporate Server - Container Mode" in future and minimize default live patching. ( minimizes the risk of missing a certificate )

Description

In case of starting many directory nodes or managed nodes at the same time, there are coming a lot of certificates in a short period of time too and maybe you will get a join failed. To prevent this fact, the container try to get a certificate before joining to a primary directory node. This functionality is particularly useful with the container option dcwait and using docker-compose too.

…option

GinoHarlos · 2023-05-01T12:33:59Z

Preparation for UCS 5.2-0, fix some issues and add systemd OnFailure option

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation ( preparation for UCS 5.2-0 with backwards compatibility )

fix latest version stability and switch to ucs-releases.json with fallback to GitHub default branch
add systemd unit option OnFailure for firstboot and recreate ( super useful when starting from very old containers with active latest option )
new development/test version ( bootstrap.sh.json )
fix missing GPG keys for APT ( bootstrap.sh and inside the latest loop "00-aA-APT-SOURCES-LIST-Aa-00" too )
fix restore/recreate ( slapadd will have an empty /etc/ldap/slapd.d directory )
fix ntpsec service unit ( ntpsec would like to run as root inside the container )
fix nslcd service unit ( needs to be restarted )
fix man-db service unit ( systemd PrivateTemp )
fix docker in docker ( add systemd reset-failed )
fix minimal postfix support ( regenarate postfix lookup tables )
fix dns/forwarder and ignore loopback addresses ( docker's internal DNS server 127.0.0.11 is not reliable enough as a forwarder )
fix dbus/avahi and bee nice to systemd-logind.service for UCS >= 5.0-0 ( failed to start dbus-org.freedesktop.login1.service )
fix named with samba as backend ( docker vs. podman "named: the working directory is not writable" set systemd WorkingDirectory to /var/cache/bind )
( docker has permission to write files into the "/" dirrectory with the systemd Group as bind )
add switching appcenter repository url for testing environment
( Is appcenter-test.software-univention.de really active and usable for the test environment? )

Documentation changes

fix missing command in basic examples by adding a fallback to watch -n 1 docker exec ${FQDN} systemctl status univention-container-mode-firstboot.service ...
fix latest version description
add description for systemd unit option OnFailure for firtboot and recreate ( univention-container-mode-firstboot-on-failure.service and univention-container-mode-recreate-on-failure.service )

Additional notes

With the "fix latest version" patch it is posible to start a container with UCS 4.4-0 and it will update/upgrade itself to version 5.2-0 ( univention-corporate-server:4.4-0-test ) or 5.0-3 ( univention-corporate-server:4.4-0 ) with default mirror url.

docker \
  run \
    --detach \
    --env DEBUG=TRUE \
    --cap-add SYS_ADMIN \
    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
    --restart unless-stopped \
    --tmpfs /tmp:exec \
    --tmpfs /run/lock \
    --tmpfs /run \
    --hostname dc.ucs.example \
    --name dc.ucs.example \
      docker.io/ginoharlos/univention-corporate-server:4.4-0

watch -n 1 docker exec dc.ucs.example systemctl status univention-container-mode-firstboot.service univention-container-mode-firstboot-on-failure.service
...
● univention-container-mode-firstboot.service - Univention container mode firstboot
   Loaded: loaded (/lib/systemd/system/univention-container-mode-firstboot.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/univention-container-mode-firstboot.service.d
           └─override.conf
...
● univention-container-mode-firstboot-on-failure.service - Univention container mode firstboot
   Loaded: loaded (/lib/systemd/system/univention-container-mode-firstboot-on-failure.service; static; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/univention-container-mode-firstboot-on-failure.service.d
           └─override.conf

docker \
  exec dc.ucs.example \
    systemd-analyze blame | egrep -- univention-container-mode
...
   13min 20.335s univention-container-mode-firstboot.service

docker \
  exec dc.ucs.example /bin/bash -c 'systemd-detect-virt --container && univention-check-join-status && \
    ucr search --brief ^repository/.*/server ^version ^appcenter/apps ^appcenter/docker$'
...
docker
Joined successfully
appcenter/docker: false
repository/app_center/server: appcenter.software-univention.de
repository/mirror/server: <empty>
repository/online/component/.*/server: <empty>
repository/online/server: https://updates.software-univention.de/
version/erratalevel: 648
version/patchlevel: 3
version/version: 5.0

CLAassistant · 2023-05-19T18:54:57Z

All committers have signed the CLA.

…ainer apps from univention app center by replacing the clean environment documentation including Dockerfile

GinoHarlos · 2023-06-06T16:00:23Z

Add feature of pre installed role container images including non-container apps from univention app center by replacing the clean environment documentation including Dockerfile

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation

Adding the feature of new container images with pre installed role as primary, replica, backup directory node or managed node.

Documentation changes

del clean environment documentation
add pre installed role container images ( build with docker or podman, including non-container apps )
add new LATEST option value for pre installed role container images

Dockerfile changes

del clean environment Dockerfile ( clean.environment.Dockerfile )
add pre installed role Dockerfile ( pre.installed.role.Dockerfile )

Description

With amount of minimal changes to the firstboot deploy, get a "fork" but with pre installed system role to save time. The default firstboot needs 15 to 20 minutes for a first start/boot. If you run the same with a pre installed container image, it takes less the 10 minutes.

fix 00-aA-APT-USER-AGENT-Aa-00, 00-aA-DEPENDENCIES-Aa-00, 41-setup-pre-certificate, 52-setup-join, 70-docker-in-docker, 70-ssh-server, 99-system-cleanup and utils ( the APT User Agent get's a new flag )
fix recreate/00-aA-APT-SOURCES-LIST-Aa-00, recreate/00-aA-RESTORE-Aa-00 and recreate/52-setup-restore ( allow default to upgrade the system before restoring, only for UCS > 5.0-0 )
fix univention container mode service units ( switch to ConditionPathExists=/etc/machine.secret for backup, joined and init )
add {join,recreate}/{30,70}-ssh-server-keys ( be sure to have ssh private and public keys, also very usefull for every normal UCS container mode image )
add {join,recreate}/99-zZ-UPGRADE-LATEST-Zz-99 ( LATEST=SKIPP is an alternative option to LATEST=TRUE and will trigger the univention upgrade process for UCS > 5.0-0 )

for dc in pdc sdc bdc mdc; do docker exec ${dc}.ucs.example systemd-analyze blame | egrep -- univention-container-mode-pre-installed-role.service; done
...
    6min 12.302s univention-container-mode-pre-installed-role.service ( primary directory node )
    5min 17.301s univention-container-mode-pre-installed-role.service ( replica directory node )
      6min 529ms univention-container-mode-pre-installed-role.service ( backup directory node )
    4min 27.356s univention-container-mode-pre-installed-role.service ( managed node )
...

fix utils - UniventionAddApp - for container build process to add non-container apps ( directly support for Active Directory-compatible Domain Controller )

for dc in pdc sdc bdc mdc; do docker exec ${dc}.ucs.example systemd-analyze blame | egrep -- univention-container-mode-pre-installed-role.service; done
...
   11min 38.519s univention-container-mode-pre-installed-role.service ( primary directory node --ad-dc-- included )
    8min 19.857s univention-container-mode-pre-installed-role.service ( replica directory node --ad-dc-- included )
     9min 9.014s univention-container-mode-pre-installed-role.service ( backup directory node  --ad-dc-- included )
    4min 49.985s univention-container-mode-pre-installed-role.service ( managed node )
...

Additional notes

For an slimify container image it will take 300 to 400 MB of space and as normal 50 to 100 MB on top ( gziped tar archive on the container registry ).

BugFixes

fix systemd OnFailure for restore/recreate, firstboot and pre installed role(s)
fix systemd firstboot and recreate service units for requested RESTART option ( skipp univention-check-join-status )
fix setup pre certificate for pre installed roles ( generate host certificate only for a vaild or empty server role )
fix the stability of join process ( allow to fail once max twice )
fix the stability of automated setup ( counter overflow in utils )
fix slimify and prevent the reinstalling of all packages ( save time and avoid unnecessary reinstallations )
fix mimimal postfix mail server support ( missing lookup tables )
fix nameserver and forwarder ( be sure to overwrite any old nameservers and forwarders )
fix the stability of namespace error detection ( be nice to systemd-logind, this will benefit cifs/dbus and avahi )
fix app center test repository mirror ( ignore gpg for all.tar ... ucr set appcenter/index/verify=false )
fix restore/recreate stability and add system upgrade process ( one step further to automatic system restore/recreate )

CleanUp

del {join,recreate}/{40,60}-setup-*-certificate-patch ( no longer needed, Plan B for parallel join works very well )

GinoHarlos · 2023-06-11T16:42:35Z

ToDo item systemd -- fix private tmp and restrict namespace

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation

One point less on the ToDo list and a clean start of systemd service units.

Documentation changes

add a comment for hugepages volume ( docker run ... --volume /dev/hugepages:/dev/hugepages ... )
update container build and run with podman ( podman run ... --systemd true ... since version 3.1.0 )

Description

Depending on the container start and its privileges, there are more and more errors in individual systemd service units. This is now recognized for every container start and repaired in a nice or hard way.

journalctl --full --all --no-pager --no-hostname --boot | awk '/^.*systemd\[1\]\:.*(NAMESPACE|BPF.*cgroup.*failed.*)$/{ print $0 }'
... ${unit} Attaching egress BPF program to cgroup ... ${unit} failed ...
... ${unit} Main process exited, code=exited, status=226/NAMESPACE

…ey files

…-upgrade

GinoHarlos · 2024-04-15T18:15:43Z

Preparation for UCS 5.2-0 and fix some issues

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation

update to new version ( bootstrap.sh.json and docs )
fix systemd detect new machine, be sure to remove log, identity and key files ( Dockerfile )
fix slimify mode and allow a new path ( /usr/share/doc/univention*/changelog.gz )
fix pre installed role and be sure to skipp the first and second dist-upgrade
fix systemd private tmp and restrict namespace detection
fix systemd service unit firstboot ( univention-container-mode-firstboot )
fix restore/recreate ( cups )
fix and prepare sso web service for UCS >= 5.1-0 ( umc/web/sso/enabled=false )
fix and prepare sso certificate
fix and prepare systemd sssd service unit
fix and prepare systemd wait online service unit ( we don't need this service unit in the container )
fix and prepare systemd default services and timers ( add inetd, slapd and ntpsec )
prepare restore/recreate for containerbases apps
fix running under Docker >= 25.0.0 by changing control groups mount

Documentation changes

update to new version 5.0-7
add a comment for appcenter volume
add a comment for Docker >= 25.0.0
add a comment for keycloak app

Known issues

you can't realy restore/recreate with container based apps from univention appcenter

…erged )

…ntax )

…ersions don't accept empty 'WantedBy' )

… config )

…end )

…d one )

GinoHarlos · 2024-05-06T18:39:46Z

Preparation for UCS 5.2-0, fix some issues and improve stability

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation

fix debootstrap usr-is-merged ( E: Couldn't find these debs: usr-is-merged )
fix Dockerfiles ( missing packages in pre installed roles and code syntax )
fix systemd on failure and set a nice start condition ( old systemd versions don't accept empty 'WantedBy' )
fix restore/recreate LDAP ( loop databases from static and/or dynamic config )
fix restore/recreate dns ( restart any name server, frontend and backend )
fix restore/recreate ssh ( cleanup known host public key cache )
fix restore/recreate ( force cleanup to get a fresh backup )
fix restore/recreate ( disable certificates json file )
fix restore/recreate ( restore local secrets and try to backup the old one )
add restore/recreate for PostgreSQL ( dump databases )
add restore/recreate for containerbases apps like keycloak

Documentation changes

add a comment for sso restore/recreate ( saml or keycloak )

Dockerfile changes

fix missing packages in pre installed roles

…nup for better readability in exported json file )

…nivention-corporate-server-debootstrap is no longer needed )

GinoHarlos · 2024-08-12T05:43:12Z

Preparation for UCS 5.2-0, move the debootstrap to the docker build process and fix some issues

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation

fix restore/recreate ssl ( add keycloak/server/sso/fqdn )
fix slimify mode ( save another megabyte )
del bootstrap.sh and move debootstrap to docker build ( Dockerfile )

Documentation changes

update to new version 5.0-8
update docker and podman build ( the image univention-corporate-server-debootstrap is no longer needed )
update and complete the container name with univention namespace ( univention-corporate-server is not an official image )

Dockerfile changes

add debootstrap
fix slimify

GinoHarlos · 2024-10-02T14:04:50Z

Update documentation to next patchlevel release

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Documentation changes

update to new version 5.0-9

…s the major and minor in the name. )

…e.conf files are not consistent. )

GinoHarlos · 2024-11-25T20:07:06Z

Preparation for UCS 5.2-0 and fix some issues

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Motivation

fix repository public key, add new public key type, this now contains the major and minor in the name
fix timezone data, different entries in /etc/timezone and /etc/locale.conf files are not consistent

GinoHarlos · 2025-02-07T06:45:30Z

Update documentation to next minor release

Please make sure you considered the following things

I read the contribution guidelines.
I read the code of conduct.
I updated a bug report in the Univention Bugzilla.
I've added a bugzilla comment about this pull request.

Link to the issue in Bugzilla

Bug#

Description of the changes

Please describe the changes with a few sentences.

Documentation changes

update to new version 5.2-0

GinoHarlos added 2 commits May 1, 2023 14:24

Plan B for parallel join ( try to get a certificate before joining )

78684a5

Preparation for UCS 5.2-0, fix some issues and add systemd OnFailure …

104615b

…option

Add feature of pre installed role container images including non-cont…

3ec389d

…ainer apps from univention app center by replacing the clean environment documentation including Dockerfile

ToDo item systemd -- fix private tmp and restrict namespace

a05d61d

GinoHarlos added 15 commits April 14, 2024 21:19

Update README(S) and bootstrap config to UCS version 5.0-7

6148621

Fix systemd detect new machine, be sure to remove log, identity and k…

61d47a4

…ey files

Fix slimify mode and allow a new path

1878a69

fix pre installed role and be sure to skipp the first and second dist…

00600bb

…-upgrade

fix systemd private tmp and restrict namespace detection

43f82eb

fix systemd service unit firstboot

57ec9e5

fix restore/recreate ( cups )

14f93b9

fix sso web service for UCS >= 5.1-0 ( umc/web/sso/enabled=false )

c39562e

fix and prepare sso certificate

c27c771

fix and prepare systemd sssd service unit

7936857

fix and prepare systemd wait online service unit

9f1624e

fix and prepare systemd default services and timers

94c429a

prepare restore/recreate for containerbases apps

2792a07

fix running under Docker >= 25.0.0 by changing control groups mount

38a907f

add a comment for keycloak app

e364654

GinoHarlos added 6 commits May 6, 2024 20:04

fix debootstrap usr-is-merged ( E: Couldn't find these debs: usr-is-m…

eae6de2

…erged )

fix Dockerfiles ( missing packages in pre installed roles and code sy…

fefeb77

…ntax )

fix systemd on failure and set a nice start condition ( old systemd v…

ed2bb6d

…ersions don't accept empty 'WantedBy' )

fix restore/recreate LDAP ( loop databases from static and/or dynamic…

e6f9208

… config )

fix restore/recreate dns ( restart any name server, frontend and back…

cfb64ac

…end )

fix restore/recreate ssh ( cleanup known host public key cache )

f80e465

GinoHarlos added 6 commits May 6, 2024 20:23

fix restore/recreate ( force cleanup to get a fresh backup )

e273881

fix restore/recreate ( disable certificates json file )

1a06447

fix restore/recreate ( restore local secrets and try to backup the ol…

ef5bfae

…d one )

add restore/recreate for PostgreSQL ( dump databases )

1c01c32

add restore/recreate for containerbases apps like keycloak

a6990fa

add restore/recreate for containerbases apps like keycloak

adea1a4

GinoHarlos added 4 commits August 11, 2024 16:49

fix restore/recreate ssl ( add keycloak/server/sso/fqdn and some clea…

3edaf54

…nup for better readability in exported json file )

fix slimify ( save another megabyte )

29e6520

del bootstrap.sh and move debootstrap into docker build ( the image u…

8c94f0c

…nivention-corporate-server-debootstrap is no longer needed )

fix docs ( complete the container name with univention namespace )

1ec7d94

update to next patchlevel release

1d4ed56

GinoHarlos added 2 commits November 25, 2024 19:28

fix repository public key ( Add new public key type, this now contain…

bbb07b0

…s the major and minor in the name. )

fix timezone data ( Different entries in /etc/timezone and /etc/local…

c96d6aa

…e.conf files are not consistent. )

update to new version 5.2-0

52fbc50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plan B for parallel join ( try to get a certificate before joining ) #10

Plan B for parallel join ( try to get a certificate before joining ) #10

GinoHarlos commented May 1, 2023

GinoHarlos commented May 1, 2023

CLAassistant commented May 19, 2023 •

edited

Loading

GinoHarlos commented Jun 6, 2023

GinoHarlos commented Jun 11, 2023

GinoHarlos commented Apr 15, 2024

GinoHarlos commented May 6, 2024

GinoHarlos commented Aug 12, 2024

GinoHarlos commented Oct 2, 2024

GinoHarlos commented Nov 25, 2024

GinoHarlos commented Feb 7, 2025

Plan B for parallel join ( try to get a certificate before joining ) #10

Are you sure you want to change the base?

Plan B for parallel join ( try to get a certificate before joining ) #10

Conversation

GinoHarlos commented May 1, 2023

Plan B for parallel join ( try to get a certificate before joining )

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

Description

GinoHarlos commented May 1, 2023

Preparation for UCS 5.2-0, fix some issues and add systemd OnFailure option

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation ( preparation for UCS 5.2-0 with backwards compatibility )

Documentation changes

Additional notes

CLAassistant commented May 19, 2023 • edited Loading

GinoHarlos commented Jun 6, 2023

Add feature of pre installed role container images including non-container apps from univention app center by replacing the clean environment documentation including Dockerfile

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

Documentation changes

Dockerfile changes

Description

Additional notes

BugFixes

CleanUp

GinoHarlos commented Jun 11, 2023

ToDo item systemd -- fix private tmp and restrict namespace

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

Documentation changes

Description

GinoHarlos commented Apr 15, 2024

Preparation for UCS 5.2-0 and fix some issues

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

Documentation changes

Known issues

GinoHarlos commented May 6, 2024

Preparation for UCS 5.2-0, fix some issues and improve stability

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

Documentation changes

Dockerfile changes

GinoHarlos commented Aug 12, 2024

Preparation for UCS 5.2-0, move the debootstrap to the docker build process and fix some issues

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

Documentation changes

Dockerfile changes

GinoHarlos commented Oct 2, 2024

Update documentation to next patchlevel release

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Documentation changes

GinoHarlos commented Nov 25, 2024

Preparation for UCS 5.2-0 and fix some issues

Please make sure you considered the following things

Link to the issue in Bugzilla

Description of the changes

Motivation

GinoHarlos commented Feb 7, 2025

Update documentation to next minor release

Please make sure you considered the following things

Link to the issue in Bugzilla

CLAassistant commented May 19, 2023 •

edited

Loading