Skip to content

sap_ha_pacemaker_cluster: Add support for SAP Web Dispatcher #974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

rob0d
Copy link
Contributor

@rob0d rob0d commented Feb 16, 2025

@ja9fuchs @marcelmamula Hopefully this one is correct.
Replacement for PR #929

@marcelmamula
Copy link
Contributor

@rob0d @ja9fuchs It looks much cleaner now. We will review it and let you know.

@marcelmamula
Copy link
Contributor

marcelmamula commented Feb 24, 2025

@rob0d Can you share installation details that you used for testing?

Installing WD on same hosts with ASCS/ERS would require using same SID, so installing them together.
Otherwise you could not specify standalone WD SID in same sap_ha role execution as it takes only on SID.

  • If that is one under same SID, what would be Filesystem RA NFS mount? Same as ASCS/ERS ?

@rob0d
Copy link
Contributor Author

rob0d commented Feb 24, 2025

Hi @marcelmamula,

WD must not have the same SID as ASCS/ERS. Hence a different parameter as per the example in PR #929 (sap_ha_pacemaker_cluster_wdp_sid). WD is a separate netweaver-like instance and is installed as a separate set of resources (in the same way HANA would be installed as a separate set of resources when installed on the same ASCS/ERS cluster).

The installation of WD on both cluster nodes is slightly more painful as there is only one WD instance unlike with ASCS + ERS where the installation of one on each node takes care of all the Linux changes. With WD, the installation has to be done on both nodes (as it would be the case if you had a 3-node ASCS/ERS cluster).

After many tests I've decided that the most stable and least complicated option is:

  • Mount WD NFS on node 1 + add WD VIP to node 1
  • Install WD on node 1
  • Remove WD NFS and WD VIP from node 1
  • Add WD VIP to node 2 (DO NOT mount WD NFS on node 2!)
  • Install WD on node 2
  • Rename /usr/sap/WDSID to /usr/sap/WDSID.old on node 2 (possibly remove it later)
  • Now all the setup on both nodes is the same with all the files (from installation on node 1) are on NFS
  • Run sap_ha_pacemaker

Code looks like this (not fully functional as it's an extract):

    # TODO add code to handle when these are predefined in inventory in sap_vips
    - name: Get wdp_host IP from DNS
      ansible.builtin.shell: "dig {{ wdp_host }} +short"
      changed_when: false
      register: _wdp_host_ip
      failed_when: _wdp_host_ip.rc != 0 or _wdp_host_ip.stdout | length == 0
      when: wdp_is_virtual | d(false)
 
    - name: Adding WebDisp VIP
      ansible.builtin.shell:  ip address add "{{ _wdp_host_ip.stdout }}"/26 dev ens192
      ignore_errors: true
      when: wdp_is_virtual | d(false)
 
    - name: Import variables for sap_swpm Ansible Role (Default Mode)
      ansible.builtin.include_vars:
         file: ./vars/variables-sap-swpm-default-mode-webdisp-standalone-install.yml
 
    # Prepare temporary NFS mounted storage only on primary node
    # we will ignore and scrap /usr/sap/WDP/Wxx on secondary node
    - name: Prepare temporary NFS mounted storage
      when:
        - wdp_is_virtual | d(false)
        - "'webdisp_primary' in group_names"
      ansible.builtin.include_tasks:
        file: swpm-distributed-cs-prep.yml   # This deals with /usr/sap/WDSID folders and NFS
      vars:
        __sap_sid: "{{ wdp_system_sid }}"
        __instance_folder: "W{{ sap_swpm_wd_instance_nr }}"
 
    - name: Execute Ansible Role sap_swpm
      ansible.builtin.include_role:
        name: { role: ../roles/sap_swpm }
 
    - name: Block to handle HA WebDisp setup
      when: wdp_is_virtual | d(false)
      block:
 
        # For WebDisp these tasks are handled at the end of install rather than at the beginning of HA setup
        # because WebDisp is symmetrical (unlike ASCS/ERS)
        - name: Execute Cluster preparation tasks - WebDisp
          ansible.builtin.include_tasks:
            file: cs-cluster-prep.yml   # This stops WD and unmounts the temp NFS
          vars:
            __sap_sid: "{{ wdp_system_sid }}"
            __instance_no: "{{ sap_swpm_wd_instance_nr }}"
            __instance_folder: "W{{ sap_swpm_wd_instance_nr }}"
 
        - name: Removing WEBDISP VIP after installation
          ansible.builtin.shell:  ip address del "{{ _wdp_host_ip.stdout }}"/26 dev ens192
          ignore_errors: true
 
        - name: Rename instance directory on secondary host
          ansible.builtin.command:
            cmd: mv "/usr/sap/{{ wdp_system_sid }}/W{{ sap_swpm_wd_instance_nr }}" "/usr/sap/{{ wdp_system_sid }}/W{{ sap_swpm_wd_instance_nr }}.old"
          when: "'webdisp_secondary' in group_names"
 
        - name: Create instance directory mount point on secondary host
          ansible.builtin.file:
            path: "/usr/sap/{{ wdp_system_sid }}/W{{ sap_swpm_wd_instance_nr }}"
            state: directory
            mode: '0755'
            owner: "{{ wdp_system_sid | lower }}adm"
            group: sapsys
          when: "'webdisp_secondary' in group_names"

@marcelmamula
Copy link
Contributor

@rob0d No wonder that this is not supported scenario since it has so many peculiarities.
You originally said standalone or combined, that is why I asked if you used same SID. But your explanation points towards standalone in both cases, where one is dedicated standalone cluster and other is cost optimized standalone cluster along with ASCS/ERS.

I supposed you did not use sap_storage_setup role to setup NFS mounts for WD since it does not accept WD as input?

  • Filesystem RA needs to be tied to specific NFS. You are using NFS only for /usr/sap/<WD_SID/ and leave rest on / ?
  • I was thinking about updating sap_storage_setup to allow for WD, but I need to clarify this with you.

@marcelmamula
Copy link
Contributor

@rob0d Please look at #972 while we are preparing tests, as it directly affects this PR.

#972 changed how we handle some SAPinstance related steps and we moved them to run before ha_cluster.

@rob0d
Copy link
Contributor Author

rob0d commented Mar 5, 2025

@rob0d No wonder that this is not supported scenario since it has so many peculiarities. You originally said standalone or combined, that is why I asked if you used same SID. But your explanation points towards standalone in both cases, where one is dedicated standalone cluster and other is cost optimized standalone cluster along with ASCS/ERS.

I supposed you did not use sap_storage_setup role to setup NFS mounts for WD since it does not accept WD as input?

* Filesystem RA needs to be tied to specific NFS. You are using NFS only for `/usr/sap/<WD_SID/` and leave rest on `/` ?

* I was thinking about updating `sap_storage_setup` to allow for WD, but I need to clarify this with you.

I'm not sure if I understood you correctly, but no sap_storage_setup is needed. Everything that WD needs is on NFS in the same way ASCS/ERS. All it needs is /usr/sap/ which is a cluster resource and /sapmnt which (in my case) is not managed by the cluster and mounted from /etc/fstab.

However, generally speaking sap_storage_setup will not care if it's webdisp or not. As far as everybody is concerned it's just another SAP system.

Standalone/combined is in the context of coexistence on the same cluster as ASCS/ERS/HANA. Both options work as from cluster point of view it's just another resource group with resources in the same way if you installed 5 x ASCS/ERS instances for five different SAP systems on the same cluster. They are all independent.

I am guessing the confusion comes from standalone/embedded (note 3115889) webdisp? Embedded webdisp would be pretty much invisible to the cluster (with the exception of the restart parameter in instance profile), but it's not something that's widely used.

@rob0d
Copy link
Contributor Author

rob0d commented Mar 5, 2025

@rob0d Please look at #972 while we are preparing tests, as it directly affects this PR.

#972 changed how we handle some SAPinstance related steps and we moved them to run before ha_cluster.

I had a look. I can see possibly two changes that I need to make, but one would require changing @ja9fuchs code which is very specific to ascs/ers pre_steps_nwas_cs_ers.yml in roles/sap_ha_pacemaker_cluster/tasks/main.yml and I'm not sure if to change it.

There have been quite a few commits into since I've created this second PR it's looking quite difficult for me to rebase/remerge and add extra changes without messing it up.

@marcelmamula
Copy link
Contributor

@rob0d Please look at #972 while we are preparing tests, as it directly affects this PR.
#972 changed how we handle some SAPinstance related steps and we moved them to run before ha_cluster.

I had a look. I can see possibly two changes that I need to make, but one would require changing @ja9fuchs code which is very specific to ascs/ers pre_steps_nwas_cs_ers.yml in roles/sap_ha_pacemaker_cluster/tasks/main.yml and I'm not sure if to change it.

There have been quite a few commits into since I've created this second PR it's looking quite difficult for me to rebase/remerge and add extra changes without messing it up.

I mentioned that PR, because we had to rethink why our clusters were unstable, which was caused by incorrect order of steps.
This was our prompt to split it into pre and post tasks. If you are fine after testing WD, then you don't need to do any changes in relation to this PR.

@marcelmamula
Copy link
Contributor

I'm not sure if I understood you correctly, but no sap_storage_setup is needed. Everything that WD needs is on NFS in the same way ASCS/ERS. All it needs is /usr/sap/ which is a cluster resource and /sapmnt which (in my case) is not managed by the cluster and mounted from /etc/fstab.

However, generally speaking sap_storage_setup will not care if it's webdisp or not. As far as everybody is concerned it's just another SAP system.

Standalone/combined is in the context of coexistence on the same cluster as ASCS/ERS/HANA. Both options work as from cluster point of view it's just another resource group with resources in the same way if you installed 5 x ASCS/ERS instances for five different SAP systems on the same cluster. They are all independent.

I am guessing the confusion comes from standalone/embedded (note 3115889) webdisp? Embedded webdisp would be pretty much invisible to the cluster (with the exception of the restart parameter in instance profile), but it's not something that's widely used.

I was following your train of thought. If Standalone WD is different SID, then its NFS mounts (shared across cluster) are under different mount than ASCS/ERS SID mounts.

This is usual simple setup when ASCS/ERS is mounted up, which would mean that sap_storage_setup role would have to be enhanced and executed for WD, to create new mount/folder for WD SID.

fs-x.amazonaws.com:/AE1/usr/sap/AE1/SYS     8.0E  3.0G  8.0E   1% /usr/sap/AE1/SYS
fs-x.amazonaws.com:/AE1/usr/sap/AE1/ASCS00  8.0E  3.0G  8.0E   1% /usr/sap/AE1/ASCS00
fs-x.amazonaws.com:/AE1/usr/sap/AE1/ERS01   8.0E  3.0G  8.0E   1% /usr/sap/AE1/ERS01
fs-x.amazonaws.com:/AE1/usr/sap/trans       8.0E  3.0G  8.0E   1% /usr/sap/trans
fs-x.amazonaws.com:/AE1/sapmnt              8.0E  3.0G  8.0E   1% /sapmnt

@rob0d
Copy link
Contributor Author

rob0d commented Mar 6, 2025

@rob0d Please look at #972 while we are preparing tests, as it directly affects this PR.
#972 changed how we handle some SAPinstance related steps and we moved them to run before ha_cluster.

I had a look. I can see possibly two changes that I need to make, but one would require changing @ja9fuchs code which is very specific to ascs/ers pre_steps_nwas_cs_ers.yml in roles/sap_ha_pacemaker_cluster/tasks/main.yml and I'm not sure if to change it.
There have been quite a few commits into since I've created this second PR it's looking quite difficult for me to rebase/remerge and add extra changes without messing it up.

I mentioned that PR, because we had to rethink why our clusters were unstable, which was caused by incorrect order of steps. This was our prompt to split it into pre and post tasks. If you are fine after testing WD, then you don't need to do any changes in relation to this PR.

Yes, I'd like to apply similar changes to this branch (at least in relation to the monitoring interval) but I now have three different codebases and can't test any complicated changes in this branch. If we can merge this to dev, I can then take dev and merge everything back to the original private repo where this is developed, add the changes based on PR #972 and create a new PR here. I hope that makes sense :).

@rob0d
Copy link
Contributor Author

rob0d commented Mar 6, 2025

I was following your train of thought. If Standalone WD is different SID, then its NFS mounts (shared across cluster) are under different mount than ASCS/ERS SID mounts.

This is usual simple setup when ASCS/ERS is mounted up, which would mean that sap_storage_setup role would have to be enhanced and executed for WD, to create new mount/folder for WD SID.

fs-x.amazonaws.com:/AE1/usr/sap/AE1/SYS     8.0E  3.0G  8.0E   1% /usr/sap/AE1/SYS
fs-x.amazonaws.com:/AE1/usr/sap/AE1/ASCS00  8.0E  3.0G  8.0E   1% /usr/sap/AE1/ASCS00
fs-x.amazonaws.com:/AE1/usr/sap/AE1/ERS01   8.0E  3.0G  8.0E   1% /usr/sap/AE1/ERS01
fs-x.amazonaws.com:/AE1/usr/sap/trans       8.0E  3.0G  8.0E   1% /usr/sap/trans
fs-x.amazonaws.com:/AE1/sapmnt              8.0E  3.0G  8.0E   1% /sapmnt

I am still confused :), but maybe I'm starting to understand.
Are you referring to sap_storage_setup_host_type which utilises sap_storage_setup_nfs_dirs_usr_sap?
Strangely enough I had no need for that in relation to WebDisp and I am wondering what I'm missing.
My (partial) config for ASCS/ERS cluster:

  storage_definition:
 
    - name: usr_sap
      mountpoint: /usr/sap
      disk_size: "{{ usr_sap_fs_size }}"
      filesystem_type: xfs
      lvm_lv_name: lv_usrsap
      lvm_vg_name: vg_usrsap
      lvm_pv_device: "{{ lvm_pv_device_usrsap }}"
 
    - name: sapmnt
      mountpoint: /sapmnt
      nfs_path: /sapmnt
      nfs_server: "{{ sap_storage_setup_nfs_local }}"
      nfs_mount_options: "{{ sap_storage_setup_nfs_mount_options }}"
 
    - name: nfs_install
      mountpoint: /install
      nfs_path: /install
      nfs_server: "{{ sap_storage_setup_nfs_central }}"
      nfs_mount_options: "{{ sap_storage_setup_nfs_mount_options }}"
 
 
    - name: nfs_trans
      mountpoint: /usr/sap/trans
      nfs_path: /usr/sap/trans
      nfs_server: "{{ sap_storage_setup_nfs_central }}"
      nfs_mount_options: "{{ sap_storage_setup_nfs_mount_options }}"
 
    - name: swap
      disk_size: "{{ swap_fs_size }}"
      filesystem_type: swap
      lvm_lv_name: lv_swap
      lvm_vg_name: vg_root
      lvm_pv_device: "{{ lvm_pv_device_swap }}"

/usr/sap/ is local, /install,/usr/sap/trans and /install are always mounted. I don't have any cluster filesystems in the storage_definiton as I didn't want them mounted permanently in /etc/fstab by the role.
I create the mountpoints and mount from NFS just before I install ASCS/ERS/WD and then unmount them and then kick off sap_ha_pacemaker_cluster.

Is this not how it's supposed to work?

@marcelmamula
Copy link
Contributor

@rob0d This is example how filesystems would be created by sap_storage_setup role if we add support for WD:

fs-x.amazonaws.com:/AE1/usr/sap/AE1/SYS     8.0E  3.0G  8.0E   1% /usr/sap/AE1/SYS
fs-x.amazonaws.com:/AE1/usr/sap/AE1/ASCS00  8.0E  3.0G  8.0E   1% /usr/sap/AE1/ASCS00
fs-x.amazonaws.com:/AE1/usr/sap/AE1/ERS01   8.0E  3.0G  8.0E   1% /usr/sap/AE1/ERS01
fs-x.amazonaws.com:/AE1/usr/sap/trans       8.0E  3.0G  8.0E   1% /usr/sap/trans
fs-x.amazonaws.com:/AE1/sapmnt              8.0E  3.0G  8.0E   1% /sapmnt
fs-x.amazonaws.com:/WD1/usr/sap/WD1/SYS     8.0E  3.0G  8.0E   1% /usr/sap/WD1/SYS
fs-x.amazonaws.com:/WD1/usr/sap/WD1/W20     8.0E  3.0G  8.0E   1% /usr/sap/WD1/W20

@rob0d
Copy link
Contributor Author

rob0d commented Mar 10, 2025

fs-x.amazonaws.com:/WD1/usr/sap/WD1/SYS     8.0E  3.0G  8.0E   1% /usr/sap/WD1/SYS
fs-x.amazonaws.com:/WD1/usr/sap/WD1/W20     8.0E  3.0G  8.0E   1% /usr/sap/WD1/W20

Ah ok. I get it. I see no reason why we shouldn't enhance the storage role to support this WD deployment pattern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants