Red Hat Enterprise Linux (RHEL) provides high availability (HA) for SAP business-critical workloads at the application and database level in order to maximize uptime and minimize unplanned disruptions of service.
This blog describes an example deployment of a scale-up SAP HANA installation configured with SAP HANA System Replication (HSR) in a RHEL HA Pacemaker cluster. Background on this configuration is available in the Red Hat documentation for SAP. The scope includes:
Target audience: Linux and SAP administrators evaluating SAP HANA on RHEL HA with plans to deploy in non-production and production.
Red Hat’s Ansible Automation Platform provides out-of-the box code (Ansible system roles) written in YAML that will automate the installation and configuration of SAP HANA with Red Hat HA. The RHEL System Roles for SAP were used to automate the following steps, enabling a faster installation:
The lab setup is shown below:
Note: Production implementations may vary depending on the best practices of the server, network, and/or cloud vendors as well as the networking guidelines of your organization.
The output of the command “nmcli con show” shows the three different networks:
[root@hana1 ~]# nmcli con show
NAME UUID TYPE DEVICE
Wired connection 1 c6838ef4-751f-35f8-a8df-8200246827d0 ethernet eth0
Wired connection 2 2d23d504-7789-3a20-940e-024c73818847 ethernet eth1
Wired connection 3 aa2980e0-9e69-4119-94ec-1bb0d92f1431 ethernet eth2
[root@hana1 ~]#
Separate networks were used to segregate the network traffic corresponding to the different SAP HANA network zones:
Network segregation can help with performance in situations where there is a high workload on the servers; for example, batch processing, which can generate significant network traffic in the client and internal zones.
The setup of a separate network for HSR is defined in the SAP HANA Administration Guide. Extra entries were added to the “/hana/shared/RHE/global/hdb/custom/config/global.ini'' file on hana1 and hana2, as shown below:
rheadm@hana1:/hana/shared/RHE/global/hdb/custom/config> more global.ini
[system_replication_communication]
listeninterface = .global
[system_replication_hostname_resolution]
192.168.2.23 = hana2
< not all output shown>
rheadm@hana2:/hana/shared/RHE/global/hdb/custom/config> more global.ini
[system_replication_communication]
listeninterface = .global
[system_replication_hostname_resolution]
192.168.2.22 = hana1
< not all output shown>
A cluster can be configured to use multiple independent network interfaces for communication and network redundancy. In this example, the cluster was configured to use the private and SAP HANA internal networks. The cluster stores the networking configuration in the /etc/corosync/corosync.conf file:
[root@hana1 ~]#more /etc/corosync/corosync.conf
….
….
nodelist {
node {
ring0_addr: hana1.private.example.com
name: hana1.private.example.com
nodeid: 1
ring1_addr: 192.168.2.22
}
node {
ring0_addr: hana2.private.example.com
name: hana2.private.example.com
nodeid: 2
ring1_addr: 192.168.2.23
}
}
,,,
The above shows the cluster is using two network links identified by ring0 and ring1.
If cluster communications need to be separated completely from the SAP HANA networks and cluster heartbeat redundancy is required, then a fourth network would need to be provisioned.
The status of the cluster is obtained using the command “pcs status --full” (not all the output is shown below):
root@hana1 ~]# pcs status --full
…
Node List:
* Online: [ hana1.private.example.com hana2.private.example.com ]
Full List of Resources:
* fence_hana1 (stonith:fence_ipmilan): Started hana1.private.example.com
* fence_hana2 (stonith:fence_ipmilan): Started hana2.private.example.com
* vip_RHE_00_primary (ocf::heartbeat:IPaddr2): Started hana1.private.example.com
* Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
* Started: [ hana1.private.example.com hana2.private.example.com ]
* Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
* Promoted: [ hana1.private.example.com ]
* Unpromoted: [ hana2.private.example.com ]
Node Attributes:
* Node: hana1.private.example.com (1):
* hana_rhe_clone_state : PROMOTED
* hana_rhe_op_mode : logreplay
* hana_rhe_remoteHost : hana2
* hana_rhe_roles : 4:P:master1:master:worker:master
* hana_rhe_site : DC01
* hana_rhe_srmode : sync
* hana_rhe_sync_state : PRIM
* hana_rhe_version : 2.00.067.04.1700654562
* hana_rhe_vhost : hana1
* lpa_rhe_lpt : 1716157304
* master-SAPHana_RHE_00 : 150
* Node: hana2.private.example.com (2):
* hana_rhe_clone_state : DEMOTED
* hana_rhe_op_mode : logreplay
* hana_rhe_remoteHost : hana1
* hana_rhe_roles : 4:S:master1:master:worker:master
* hana_rhe_site : DC02
* hana_rhe_srmode : sync
* hana_rhe_sync_state : SOK
* hana_rhe_version : 2.00.067.04.1700654562
* hana_rhe_vhost : hana2
* lpa_rhe_lpt : 30
* master-SAPHana_RHE_00 : 100
The output above shows the following:
The SAP HANA resource agents gather attributes that are regularly updated and stored locally on each cluster node. These attributes, visible under "Node Attributes" (output of the command “pcs status --full”), are used to assess cluster health and make failover decisions. One such attribute is “hana_rhe_sync_state,” which is set to “SOK” for the secondary/demoted SAP HANA instance, indicating a healthy status for HSR.
The application server connection to the primary SAP HANA instance is via the SAP HANA client and the SAP HANA user store “hdbuserstore” (reference: SAP Note 3158257 - FAQ: SAP HANA User Store).
The command “hdbuserstore list” executed on the application server host, appnode1, provides the SAP HANA database connection details for the application server:
appnode1:rheadm 17> hdbuserstore list
DATA FILE
: /home/rheadm/.hdb/appnode1/SSFS_HDB.DAT
KEY FILE
: /home/rheadm/.hdb/appnode1/SSFS_HDB.KEY
KEY DEFAULT
ENV : virtIP:30013
USER: SAPHANADB
DATABASE: RHE
ACTIVE RECORDS : 5
DELETED RECORDS : 4
NUMBER OF COMPLETE KEY: 1
Operation succeed.
appnode1:rheadm 18
As shown above, the SAP HANA user store uses the virtual IP hostname “virtIP” to connect to the primary SAP HANA instance. The virtual IP is managed by the cluster and is always started by the cluster on the host running the primary SAP HANA instance.
The command “R3trans -d” can be used to check a successful connection to the primary SAP HANA instance (reference: SAP Note 2461656 - R3trans shows DB connection error even DB is up and running).
appnode1:rheadm 16> R3trans -d
This is R3trans version 6.26 (release 785 - 03.08.22 - 08:30:00).
unicode enabled version
R3trans finished (0000).
appnode1:rheadm 17>
During a cluster failover, the cluster virtual IP resource is stopped, the application server will disconnect, and the above command when run, will return a non-zero error code.
Here we validate some Pacemaker functionality via tests that simulate network errors:
Note: Cluster functionality should be validated by running a number of different failure scenarios. Not all of the scenarios are covered here; for a list of such tests and expected results, see here.
The primary SAP HANA instance is on hana1, and the secondary on hana2.
The SAP HANA internal network interface on hana1 is deactivated:
[root@hana1 ~]# nmcli con down ‘Wired connection 3’
Connection 'Wired connection 3' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)
This caused HSR to fail between hana1 and hana2. This can be verified using a SAP supplied Python script to check HSR status:
rheadm@hana1:/usr/sap/RHE/HDB00/exe/python_support> python systemReplicationStatus.py
|Database |Host |Port |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary |Replication |Replication |Replication |Secondary |
| | | | | | | |Host |Port |Site ID |Site Name |Active Status |Mode |Status |Status Details |Fully Synced |
|-------- |----- |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------------ |----------- |----------- |----------------------------- |------------ |
|SYSTEMDB |hana1 |30001 |nameserver | 1 | 1 |DC01 |hana2 | 30001 | 2 |DC02 |CONNECTION TIMEOUT |SYNC |ACTIVE | | True |
|RHE |hana1 |30007 |xsengine | 2 | 1 |DC01 |hana2 | 30007 | 2 |DC02 |CONNECTION TIMEOUT |SYNC |ACTIVE | | True |
|RHE |hana1 |30003 |indexserver | 3 | 1 |DC01 |hana2 | 30003 | 2 |DC02 |CONNECTION TIMEOUT |SYNC |ERROR |Log shipping timeout occurred | False |
….
The cluster status at this point shows the following:
[root@hana1 ~]# pcs status --full
…
…
Node List:
* Online: [ hana1.private.example.com (1) hana2.private.example.com (2) ]
Full List of Resources:
* fence_hana1 (stonith:fence_ipmilan): Started hana2.private.example.com
* fence_hana2 (stonith:fence_ipmilan): Started hana1.private.example.com
* vip_RHE_00_primary (ocf::heartbeat:IPaddr2): Started hana1.private.example.com
* Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
* SAPHanaTopology_RHE_00 (ocf::heartbeat:SAPHanaTopology): Started hana1.private.example.com
* SAPHanaTopology_RHE_00 (ocf::heartbeat:SAPHanaTopology): Started hana2.private.example.com
* Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
* SAPHana_RHE_00 (ocf::heartbeat:SAPHana): Promoted hana1.private.example.com
* SAPHana_RHE_00 (ocf::heartbeat:SAPHana): Unpromoted hana2.private.example.com
Node Attributes:
* Node: hana1.private.example.com (1):
* hana_rhe_clone_state : PROMOTED
* hana_rhe_op_mode : logreplay
* hana_rhe_remoteHost : hana2
* hana_rhe_roles : 4:P:master1:master:worker:master
* hana_rhe_site : DC01
* hana_rhe_srmode : sync
* hana_rhe_sync_state : PRIM
* hana_rhe_version : 2.00.067.04.1700654562
* hana_rhe_vhost : hana1
* lpa_rhe_lpt : 1716486893
* master-SAPHana_RHE_00 : 150
* Node: hana2.private.example.com (2):
* hana_rhe_clone_state : DEMOTED
* hana_rhe_op_mode : logreplay
* hana_rhe_remoteHost : hana1
* hana_rhe_roles : 4:S:master1:master:worker:master
* hana_rhe_site : DC02
* hana_rhe_srmode : sync
* hana_rhe_sync_state : SFAIL
* hana_rhe_version : 2.00.067.04.1700654562
* hana_rhe_vhost : hana2
* lpa_rhe_lpt : 10
* master-SAPHana_RHE_00 : -INFINITY
…
The above output shows the following:
At this point, the network error would need to be fixed, which would restart HSR.
Before fixing the HSR network, the next step kills the primary SAP HANA instance using the “HDB kill -9” command:
rheadm@hana1:/usr/sap/RHE/HDB00/exe/python_support> HDB kill -9
…
killing HDB processes:
kill -9 108402 /usr/sap/RHE/HDB00/hana1/trace/hdb.sapRHE_HDB00 -d -nw -f /usr/sap/RHE/HDB00/hana1/daemon.ini pf=/usr/sap/RHE/SYS/profile/RHE_HDB00_hana1
kill -9 108428 hdbnameserver
…..
The cluster first shows a failure of the primary SAP HANA instance on hana1:
[root@hana1 ansible-files]# pcs status
…
…
Node List:
* Online: [ hana1.private.example.com hana2.private.example.com ]
Full List of Resources:
* fence_hana1 (stonith:fence_ipmilan): Started hana1.private.example.com
* fence_hana2 (stonith:fence_ipmilan): Started hana2.private.example.com
* vip_RHE_00_primary (ocf::heartbeat:IPaddr2): Stopped
* Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
* Started: [ hana1.private.example.com hana2.private.example.com ]
* Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
* SAPHana_RHE_00 (ocf::heartbeat:SAPHana): FAILED hana1.private.example.com
* Unpromoted: [ hana2.private.example.com ]
Failed Resource Actions:
* SAPHana_RHE_00_monitor_119000 on hana1.private.example.com 'promoted (failed)' (9): call=40, status='complete', exitreason='', last-rc-change='2024-05-19 17:53:19 -04:00', queued=0ms, exec=0ms
…..
The cluster restarts the primary SAP HANA instance on hana1, i.e., no failover to hana2:
[root@hana1 ansible-files]# pcs status
..
..
Node List:
* Online: [ hana1.private.example.com hana2.private.example.com ]
Full List of Resources:
* fence_hana1 (stonith:fence_ipmilan): Started hana1.private.example.com
* fence_hana2 (stonith:fence_ipmilan): Started hana2.private.example.com
* vip_RHE_00_primary (ocf::heartbeat:IPaddr2): Started hana1.private.example.com
* Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
* Started: [ hana1.private.example.com hana2.private.example.com ]
* Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
* Promoted: [ hana1.private.example.com ]
* Unpromoted: [ hana2.private.example.com ]
…..
The above is expected behavior as HSR has failed, which the cluster knows about; the HSR error status is stored in the node attributes as shown above. This means that data replication to the secondary instance has failed, and the data in the secondary SAP HANA system may not be up-to-date. Any failover in this situation may result in data loss, and the cluster has prevented this.
The primary SAP HANA instance is on hana1, and the secondary on hana2.
The private internal network interface on hana2 is deactivated:
[root@hana2 ~]# nmcli con down 'Wired connection 2'
Connection 'Wired connection 2' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
The cluster communication continues as normal with the remaining link 1 (the SAP HANA internal network). This was observed in the cluster log file /var/log/cluster/corosync.log:
[root@hana2 ~]# more corosync.log
….
….
Jun 05 14:34:11 [66151] hana2 corosync info [KNET ] link: host: 1 link: 0 is down
Jun 05 14:34:11 [66151] hana2 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)
….
….
Meanwhile, there was no impact on the cluster, and the primary SAP HANA instance continued to run on hana1 and the secondary on hana2.
The article describes an example deployment of SAP HANA and SAP HANA System Replication (HSR) with a RHEL HA Pacemaker cluster using multiple networks corresponding to the network zones defined by SAP and communications required by the cluster.
Some tests were conducted to simulate network errors and demonstrate cluster functionality. Note that not all the required failover testing was shown here - for a full list of tests and expected results, see here.
Thank you Vas Mitra and Frank Danapfel for the content :)!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
4 | |
4 | |
3 | |
3 | |
3 | |
2 | |
2 | |
2 | |
2 | |
2 |