DLPX-89763 DLPX-86523 delphix-platform changes #477

justsanjeev · 2024-04-18T14:34:45Z

Problem

CIS is looking or a single home directory filesystem mounted at the /home location, currently we have the home dataset is mounted on /export/home

Due to that we see the below issues in the CIS Report

(1.45) 7402 Status of the '/home' partition in the '/etc/fstab' file
(1.46) 13248 Status of Mount Partition '/home' using mount command
(1.47) 7403 Status of the 'nodev' mount option setting for the '/home partition' defined in the '/etc/ fstab' file
(1.48) 14601 Status of the 'nodev' option for '/home' partition using 'mount' command

Solution

Mounting the home dataset to /home.
Post upgrade creating a softlink /export/home to /home so that the exiting tests and dependencies have minimal impact.

We have now implemented the delphix-legacy-link.service for the /export/home -> /home creation, we earlier tried to do the sae in the delphix-platform.services and tested to implement the change with ExecStart and ExecStartPost. It dod not work and then this change was implemented to achieve the same after the delphix-platform.services are up completely.

Testing Done

-------Start of Testing of Build 11635 13 July 2025 ----------

Git ab-pre-push : 11635 - 🟠

Pre-checkin: Initial pre-checking job with above was failed and later passed with https://github.com/delphix/qa-infra/pull/2315

Successful Pre-chekin: https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/blackbox-self-service/183042/consoleFull - 🟢

Additional Automated Upgrade 2025.20.0 --> develop : https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/blackbox-chained/7870/consoleFull - 🟢

Manual Upgrade upgrade from dlpx-2025.3.0.0 --> develop : 🟢 | Engine : dlpx2025300.dlpxdc.co | Alive : 🟢
In manual upgrade we created an appdata dsource and created a vdb on the target engine, after upgrade our dsource was Active & the vdb was founf in Running state.

Scan Results
Created a Engine from the AMI published from the appliance job above

dc clone  -s ami-03c02c00e258efd64 dlpx-sanjeev.rohila-develop dlpx-dev-sr-03c02c00e258efd64

As we are now upgraded to Ubuntu 24.04 , there are changes in the CIS test, we have Ran the CIS Scans with the now with a new policy. we have some changes 7402 , 7403, 14601 and 13248 are not present in the new policy instead below 3 new controls closer to

29044 : 🟢
29045: 🟢
29046: 🟢

CIS Scan Report

lsof output, no dependency on /export/home -

delphix@ip-10-110-199-125:~$ sudo lsof +f -- /export/home
COMMAND     PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
delphix-s  1814    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1819    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1820    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1827    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1828    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1829    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1831    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1832    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1834    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1836    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1838    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1840    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1841    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1842    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1844    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1845    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1846    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1847    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1849    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1851    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1853    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1854    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1859    root  cwd    DIR   0,36       13    3 /home/delphix
nfs_delet  1860    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1892    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1895    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1897    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1898    root  cwd    DIR   0,36       13    3 /home/delphix
vmstat     1909    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1920    root  cwd    DIR   0,36       13    3 /home/delphix
arcstat    1924    root  cwd    DIR   0,36       13    3 /home/delphix
mpstat     1967    root  cwd    DIR   0,36       13    3 /home/delphix
iostat     1970    root  cwd    DIR   0,36       13    3 /home/delphix
arcstat    1978    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1999    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    2002    root  cwd    DIR   0,36       13    3 /home/delphix
zpool      2020    root  cwd    DIR   0,36       13    3 /home/delphix
zpool      2031    root  cwd    DIR   0,36       13    3 /home/delphix
sleep      2041    root  cwd    DIR   0,36       13    3 /home/delphix
sleep      2143    root  cwd    DIR   0,36       13    3 /home/delphix
bpftrace   2153    root  cwd    DIR   0,36       13    3 /home/delphix
nfs_delet  2154    root  cwd    DIR   0,36       13    3 /home/delphix
sleep      3339    root  cwd    DIR   0,36       13    3 /home/delphix
java       7179    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9361    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9364    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9368    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9372    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14453    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14523    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14524    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14652    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15600    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15604    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15608    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15612    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15616    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15620    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15624    root  cwd    DIR   0,36       13    3 /home/delphix
script    15723 delphix  cwd    DIR   0,36       13    3 /home/delphix
bash      15734 delphix  cwd    DIR   0,36       13    3 /home/delphix
sleep     15764    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15768    root  cwd    DIR   0,36       13    3 /home/delphix
sudo      15769    root  cwd    DIR   0,36       13    3 /home/delphix
sudo      15770    root  cwd    DIR   0,36       13    3 /home/delphix
lsof      15771    root  cwd    DIR   0,36       13    3 /home/delphix
lsof      15772    root  cwd    DIR   0,36       13    3 /home/delphix
delphix@ip-10-110-199-125:~$

Softlink /export/home -> /home - 🟢

delphix@ip-10-110-199-125:~$  ls -ltr /export
total 1
lrwxrwxrwx 1 root root 5 Jul 14 11:34 home -> /home
delphix@ip-10-110-199-125:~$

-------End of Testing of Build 11635 13 July 2025----------

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml

debian/preinst

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml

justsanjeev · 2025-04-16T15:53:23Z

We have made the latest changes here and added the below logic to delphix-platform.service to take care of the /export/home mount point and soft link /export/home -> /home

# Create /export/home symlink to /home if it doesn't already exist
ExecStartPost=/bin/sh -c '\
  echo "Ensuring /export/home is a symlink to /home..."; \
  if mountpoint -q /export/home; then \
    echo "/export/home is a mountpoint, unmounting..."; \
    umount /export/home || { echo "Failed to unmount /export/home"; exit 1; }; \
  fi; \
  if [ -e /export/home ] && [ ! -L /export/home ]; then \
    echo "Removing existing /export/home directory..."; \
    rm -rf /export/home; \
  fi; \
  if [ ! -d /export ]; then \
    mkdir /export; \
  fi; \
  if [ -d /export ] && [ ! -L /export/home ]; then \
    echo "Creating symlink: /export/home -> /home"; \
    ln -s /home /export/home; \
  fi;'

We this change we got a build : appliance-build-orchestrator-pre-push/10842.

Here we have the successful upgrade tests.
But to verify @sebroy 's comment, I again the same testing

-- Created an app data unstructured vdb on a target
-- Upgraded the engine
-- delphix-mgmt did not come up due to delphix-platform

Also my /var/log/syslog show

Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1218 [main]initcall_blacklisted =p "initcall %s blacklisted\012"
Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Ensuring /export/home is a symlink to /home...
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: /export/home is a mountpoint, unmounting...
Apr 15 12:04:11 ip-10-110-212-88 sh[156704]: umount: /export/home: target is busy.
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Failed to unmount /export/home
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Control process exited, code=exited, status=1/FAILURE
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Failed with result 'exit-code'.
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: Failed to start Delphix Appliance Platform Service.

Upon executing below

delphix@ip-10-110-212-88:~$ sudo systemctl enable delphix-platform.service
delphix@ip-10-110-212-88:~$ sudo systemctl restart delphix-mgmt

And my log shows that the services has started successfully

Apr 15 12:37:42 ip-10-110-212-88 dynamic-debug[165503]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Ensuring /export/home is a symlink to /home...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: /export/home is a mountpoint, unmounting...
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 systemd[164396]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Removing existing /export/home directory...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Creating symlink: /export/home -> /home
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Finished Delphix Appliance Platform Service.
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Starting Delphix management service...

Setup Present for the same setup: dlpx-qa-2025200-qar-170885-766f97e2.dlpxdc.co
Upgraded from: 2025.2.0.0
Upgrade Tar: s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4235/upgrade-artifacts/internal-qa.upgrade.tar
An Engine retained in the Error State: dlpx-qa-2025200-qar-170888-766f97e2.dlpxdc.co

justsanjeev · 2025-04-16T15:57:56Z

Now here, we have created an alternative PR where we have added the logic to cleanup the /export/home mount point and creating the soft link /export/home -> /home.

We have added another service delphix-legacy-link.service

This Build is Validated : https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/10846/
ManualUpgrade with an Unstructured database mounted: ✅
Manual Upgraded Engine, with deffered and Full Upgrade : dlpx-qa-2025200-qar-170959-766f97e2.dlpxdc.co
Failed Pre Checkin Test Rebuild: pre_checkin

justsanjeev · 2025-04-18T10:08:00Z

Post changing my code as per above comments, I have Build appliance-build-orchestrator-pre-push/10897 running.

sebroy

I think we're almost there with this. The only remaining detail is the mechanism used to run one-time boot-time logic.

files/common/lib/systemd/system/delphix-legacy-link.service

files/common/lib/systemd/system/delphix-platform.service

sebroy · 2025-07-16T02:14:39Z

debian/postinst

+  if [[ -f "/var/lib/delphix-platform/export-home" ]]; then
+    chmod 755 "/var/lib/delphix-platform/export-home"
+  fi


This shouldn't be necessary. Note that there is no equivalent code needed for the other executables delivered by this package. You should be able to set the permissions on the file directly in the repo.

sebroy · 2025-07-16T02:18:15Z

files/common/var/lib/delphix-platform/export-home

+    echo 'Ensuring /export/home is a symlink to /home...'
+    if [ -e /export/home ]; then
+      echo 'Removing existing /export/home directory...'
+      rm -rf /export/home


This seems risky. If there's some bug in our code and we fail to unmount home directories for whatever reason, this will delete all home directories. At this point, if everything works as designed, /export/home should be an empty directory (it's the mountpoint for a filesystem that should have been unmounted already), and so rmdir should succeed.

…rt/home Adding the mount point logic back to delphix-platform.sh script Changing as per Seb-s comment to alter the softlink logic. Changing the code to previous version to validate the comments. Changing the delphix-legacy-link.service to perform the same steps next reboot like deferred upgrade. Creating a different service for taking care of soft /export/home -> /home DLPX-89763 DLPX-86523 delphix-platform changes Moved the code to create symlink to different script Removing the autofs handling as per the concerns raised by Seb and our validation Removing the softlink loginc from main.yml PR URL: https://www.github.com/delphix/delphix-platform/pull/477

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from ec9dada to 317e58e Compare April 18, 2024 14:34

prakashsurya reviewed May 8, 2024

View reviewed changes

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml Show resolved Hide resolved

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 317e58e to 68118d0 Compare June 24, 2024 09:31

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 68118d0 to e56b7de Compare July 2, 2024 11:54

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 4 times, most recently from a4f876c to d356ef1 Compare July 11, 2024 10:29

justsanjeev mentioned this pull request Jul 12, 2024

DLPX-89763 DLPX-86523 delphix-platform changes #487

Closed

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 5 times, most recently from 38fba97 to 28cab6b Compare July 16, 2024 07:08

justsanjeev mentioned this pull request Jul 16, 2024

DLPX-89763 DLPX-86523 delphix-platform changes #473

Closed

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from b1d61e7 to 0c718f3 Compare September 25, 2024 17:25

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from b984a71 to f389cbe Compare October 7, 2024 10:17

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from 64091f7 to 5022542 Compare October 22, 2024 11:48

justsanjeev mentioned this pull request Oct 24, 2024

DLPX-86523 CIS: /home filesystem and mount options delphix/appliance-build#756

Open

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 5022542 to 744dc46 Compare November 4, 2024 07:54

justsanjeev marked this pull request as ready for review November 5, 2024 10:42

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 3 times, most recently from d39c466 to 52eee14 Compare November 25, 2024 07:26

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 3 times, most recently from b5743b5 to 3a9629e Compare December 3, 2024 08:11

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 4e5c764 to d0ba9f9 Compare December 3, 2024 11:33

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from d0ba9f9 to 43d194d Compare January 16, 2025 06:30

sebroy requested changes Mar 26, 2025

View reviewed changes

debian/preinst Outdated Show resolved Hide resolved

debian/preinst Outdated Show resolved Hide resolved

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml Outdated Show resolved Hide resolved

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 5 times, most recently from 24f6877 to 5a17a79 Compare April 15, 2025 09:26

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 27a3f5e to 8f0067f Compare April 22, 2025 06:00

sebroy reviewed May 12, 2025

View reviewed changes

files/common/lib/systemd/system/delphix-legacy-link.service Outdated Show resolved Hide resolved

files/common/lib/systemd/system/delphix-legacy-link.service Outdated Show resolved Hide resolved

files/common/lib/systemd/system/delphix-legacy-link.service Outdated Show resolved Hide resolved

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from f2356e1 to e003914 Compare May 19, 2025 15:54

sebroy requested changes May 27, 2025

View reviewed changes

files/common/lib/systemd/system/delphix-platform.service Outdated Show resolved Hide resolved

files/common/lib/systemd/system/delphix-platform.service Outdated Show resolved Hide resolved

files/common/lib/systemd/system/delphix-platform.service Outdated Show resolved Hide resolved

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 7 times, most recently from 1c3baff to e22f588 Compare June 4, 2025 17:01

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from e22f588 to 26210f6 Compare June 30, 2025 16:20

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from 8b5f0fb to a5ef067 Compare July 13, 2025 07:15

sebroy requested changes Jul 16, 2025

View reviewed changes

justsanjeev added 2 commits July 21, 2025 17:15

Taking care the comments from Seb on 16 July 2025.

b36fe93

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from eba0198 to b36fe93 Compare July 21, 2025 11:47

Since /export/home has empty delphix directory, taking care

562800b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

DLPX-89763 DLPX-86523 delphix-platform changes #477

DLPX-89763 DLPX-86523 delphix-platform changes #477

Uh oh!

justsanjeev commented Apr 18, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justsanjeev commented Apr 16, 2025 •

edited

Loading

Uh oh!

justsanjeev commented Apr 16, 2025 •

edited

Loading

Uh oh!

justsanjeev commented Apr 18, 2025

Uh oh!

sebroy left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sebroy Jul 16, 2025

Uh oh!

sebroy Jul 16, 2025

Uh oh!

Uh oh!

DLPX-89763 DLPX-86523 delphix-platform changes #477

Are you sure you want to change the base?

DLPX-89763 DLPX-86523 delphix-platform changes #477

Uh oh!

Conversation

justsanjeev commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Testing Done

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justsanjeev commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justsanjeev commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justsanjeev commented Apr 18, 2025

Uh oh!

sebroy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sebroy Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

sebroy Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

justsanjeev commented Apr 18, 2024 •

edited

Loading

justsanjeev commented Apr 16, 2025 •

edited

Loading

justsanjeev commented Apr 16, 2025 •

edited

Loading