Technology Blogs by Members
Explore a vibrant mix of technical expertise, industry insights, and tech buzz in member blogs covering SAP products, technology, and events. Get in the mix!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

SAP HANA on RHEL HA Pacemaker Deployment Example

Prerna_Mohta
Explorer
‎07-09-2024 7:33 AM
692

Introduction

Red Hat Enterprise Linux (RHEL) provides high availability (HA) for SAP business-critical workloads at the application and database level in order to maximize uptime and minimize unplanned disruptions of service.

This blog describes an example deployment of a scale-up SAP HANA installation configured with SAP HANA System Replication (HSR) in a RHEL HA Pacemaker cluster. Background on this configuration is available in the Red Hat documentation for SAP. The scope includes:

  • Description of a lab setup using multiple networks used by Pacemaker and the different SAP HANA network zones.
  • Validation of some cluster functionality via tests simulating network failure.

Target audience: Linux and SAP administrators evaluating SAP HANA on RHEL HA with plans to deploy in non-production and production. 

Lab Setup

Red Hat’s Ansible Automation Platform  provides out-of-the box code (Ansible system roles) written in YAML that will automate the installation and configuration of SAP HANA with Red Hat HA. The RHEL System Roles for SAP were used to automate the following steps, enabling a faster installation: 

  • Install SAP HANA on hosts hana1 and hana2; SID=RHE.
  • Configure SAP HANA System Replication (HSR).
  • Install the SAP NetWeaver ABAP Central Services (ASCS) and Primary Application Server (PAS) on host appnode1.
  • Configure RHEL HA for SAP HANA. The Ansible system role installs the RHEL HA software and performs multiple configuration steps to set up the SAP cluster resource agents as per the required guidelines.

The lab setup is shown below:

unnamed (4).png

Note: Production implementations may vary depending on the best practices of the server, network, and/or cloud vendors  as well as the networking guidelines of your organization.

The output of the command “nmcli con show” shows the three different networks:

 

 

[root@hana1 ~]# nmcli con show
NAME            	UUID                              	TYPE  	DEVICE
Wired connection 1  c6838ef4-751f-35f8-a8df-8200246827d0  ethernet  eth0   
Wired connection 2  2d23d504-7789-3a20-940e-024c73818847  ethernet  eth1
Wired connection 3  aa2980e0-9e69-4119-94ec-1bb0d92f1431  ethernet  eth2  
[root@hana1 ~]#

 

 

SAP HANA Networks

Separate networks were used to segregate the network traffic corresponding to the different SAP HANA network zones:

  • Client zone used by SAP application servers.
  • Internal zone for HSR.

Network segregation can help with performance in situations where there is a high workload on the servers; for example, batch processing, which can generate significant network traffic in the client and internal zones. 

The setup of a separate network for HSR is defined in the SAP HANA Administration Guide.  Extra entries were added to the “/hana/shared/RHE/global/hdb/custom/config/global.ini'' file on hana1 and hana2, as shown below:

 

 

rheadm@hana1:/hana/shared/RHE/global/hdb/custom/config> more global.ini

[system_replication_communication]
listeninterface = .global

[system_replication_hostname_resolution]
192.168.2.23 = hana2

< not all output shown>
rheadm@hana2:/hana/shared/RHE/global/hdb/custom/config> more global.ini

[system_replication_communication]
listeninterface = .global

[system_replication_hostname_resolution]
192.168.2.22 = hana1

< not all output shown>

 

 

Cluster Networking

A cluster can be configured to use multiple independent network interfaces for communication and network redundancy. In this example, the cluster was configured to use the private and SAP HANA internal networks. The cluster stores the networking configuration in the /etc/corosync/corosync.conf file:

 

 

[root@hana1 ~]#more /etc/corosync/corosync.conf
….
….
nodelist {
    node {
        ring0_addr: hana1.private.example.com
        name: hana1.private.example.com
        nodeid: 1
        ring1_addr: 192.168.2.22
    }

    node {
        ring0_addr: hana2.private.example.com
        name: hana2.private.example.com
        nodeid: 2
        ring1_addr: 192.168.2.23
    }
}
,,,

 

 

The above shows the cluster is using two network links identified by ring0 and ring1.

If cluster communications need to be separated completely from the SAP HANA networks and cluster heartbeat redundancy is required, then a fourth network would need to be provisioned. 

Cluster Status

The status of the cluster is obtained using the command “pcs status --full” (not all the output is shown below):

 

 

root@hana1 ~]# pcs status --full
…
Node List:
  * Online: [ hana1.private.example.com hana2.private.example.com ]

Full List of Resources:
  * fence_hana1    (stonith:fence_ipmilan):     Started hana1.private.example.com
  * fence_hana2    (stonith:fence_ipmilan):     Started hana2.private.example.com
  * vip_RHE_00_primary    (ocf::heartbeat:IPaddr2):     Started hana1.private.example.com
  * Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
	* Started: [ hana1.private.example.com hana2.private.example.com ]
  * Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
	* Promoted: [ hana1.private.example.com ]
	* Unpromoted: [ hana2.private.example.com ]


Node Attributes:
  * Node: hana1.private.example.com (1):
	* hana_rhe_clone_state       		 : PROMOTED  
	* hana_rhe_op_mode           		 : logreplay
	* hana_rhe_remoteHost        		 : hana2	 
	* hana_rhe_roles             		 : 4:P:master1:master:worker:master
	* hana_rhe_site              		 : DC01 	 
	* hana_rhe_srmode            		 : sync 	 
	* hana_rhe_sync_state        		 : PRIM 	 
	* hana_rhe_version           		 : 2.00.067.04.1700654562
	* hana_rhe_vhost             		 : hana1	 
	* lpa_rhe_lpt                		 : 1716157304
	* master-SAPHana_RHE_00      		 : 150  	 
  * Node: hana2.private.example.com (2):
	* hana_rhe_clone_state       		 : DEMOTED   
	* hana_rhe_op_mode           		 : logreplay
	* hana_rhe_remoteHost        		 : hana1	 
	* hana_rhe_roles             		 : 4:S:master1:master:worker:master
	* hana_rhe_site              		 : DC02 	 
	* hana_rhe_srmode            		 : sync 	 
	* hana_rhe_sync_state        		 : SOK  	 
	* hana_rhe_version           		 : 2.00.067.04.1700654562
	* hana_rhe_vhost             		 : hana2	 
	* lpa_rhe_lpt                		 : 30   	 
	* master-SAPHana_RHE_00      		 : 100

 

 

The output above shows the following:

  • Virtual IP address cluster resource “vip_RHE_00_primary” - this is based on the IP address of hostname “virtIP” defined in /etc/hosts on all hosts.
  • SAP HANA cluster resources are based on agents “SAPHanaTopology” and “SAPHana”. These resource agents interact with SAP HANA. For more details, see SAP Note 3288971 - FAQ (SUSE HAE/RedHat HAA Pacemaker Cluster Resource Manager in SAP HANA System Replication Environments). 
  • The primary (promoted) SAP HANA instance is on hana1, and the secondary (demoted) SAP HANA instance is on hana2.
  • Node attributes collected by the SAP resource agents. 

The SAP HANA resource agents gather attributes that are regularly updated and stored locally on each cluster node. These attributes, visible under "Node Attributes" (output of the command “pcs status --full”), are used to assess cluster health and make failover decisions. One such attribute is “hana_rhe_sync_state,” which is set to “SOK” for the secondary/demoted SAP HANA instance, indicating a healthy status for HSR. 

Application Server Connectivity

The application server connection to the primary SAP HANA instance is via the SAP HANA client and the SAP HANA user store “hdbuserstore” (reference: SAP Note  3158257 - FAQ: SAP HANA User Store).

The command “hdbuserstore list” executed on the application server host, appnode1, provides the SAP HANA database connection details for the application server:

 

 

appnode1:rheadm 17> hdbuserstore list
DATA FILE
: /home/rheadm/.hdb/appnode1/SSFS_HDB.DAT
KEY FILE
: /home/rheadm/.hdb/appnode1/SSFS_HDB.KEY
KEY DEFAULT
ENV : virtIP:30013
USER: SAPHANADB
DATABASE: RHE
ACTIVE RECORDS : 5
DELETED RECORDS : 4
NUMBER OF COMPLETE KEY: 1
Operation succeed.
appnode1:rheadm 18

 

 

As shown above, the SAP HANA user store uses the virtual IP hostname “virtIP” to connect to the primary SAP HANA instance. The virtual IP is managed by the cluster and is always started by the cluster on the host running the primary SAP HANA instance.

The command “R3trans -d” can be used to check a successful connection to the primary SAP HANA instance (reference: SAP Note 2461656 - R3trans shows DB connection error even DB is up and running).

 

 

appnode1:rheadm 16> R3trans -d
This is R3trans version 6.26 (release 785 - 03.08.22 - 08:30:00).
unicode enabled version
R3trans finished (0000).
appnode1:rheadm 17>

 

 

During a cluster failover, the cluster virtual IP resource is stopped, the application server will disconnect, and the above command when run, will return a non-zero error code.

Validate Setup

Here we validate some Pacemaker functionality via tests that simulate network errors:

  • Drop the SAP HANA internal network interface. 
  • Drop the private network interface.

Note: Cluster functionality should be validated by running a number of different failure scenarios. Not all of the scenarios are covered here; for a list of such tests and expected results, see here.

SAP HANA Internal Network Error Test

The primary SAP HANA instance is on hana1, and the secondary on hana2.

The SAP HANA internal network interface on hana1 is deactivated:

 

 

[root@hana1 ~]# nmcli con down ‘Wired connection 3’
Connection 'Wired connection 3' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)

 

 

This caused HSR to fail between hana1 and hana2. This can be verified using a SAP supplied Python script to check HSR status:

 

 

rheadm@hana1:/usr/sap/RHE/HDB00/exe/python_support> python systemReplicationStatus.py
|Database |Host  |Port  |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary          |Replication |Replication |Replication                   |Secondary    |
|         |      |      |             |          |        |          |Host      |Port      |Site ID   |Site Name |Active Status      |Mode        |Status      |Status Details                |Fully Synced |
|-------- |----- |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------------ |----------- |----------- |----------------------------- |------------ |
|SYSTEMDB |hana1 |30001 |nameserver   |        1 |      1 |DC01      |hana2     |    30001 |        2 |DC02      |CONNECTION TIMEOUT |SYNC        |ACTIVE      |                              |        True |
|RHE      |hana1 |30007 |xsengine     |        2 |      1 |DC01      |hana2     |    30007 |        2 |DC02      |CONNECTION TIMEOUT |SYNC        |ACTIVE      |                              |        True |
|RHE      |hana1 |30003 |indexserver  |        3 |      1 |DC01      |hana2     |    30003 |        2 |DC02      |CONNECTION TIMEOUT |SYNC        |ERROR       |Log shipping timeout occurred |       False |
….

 

 

The cluster status at this point shows the following:

 

 

[root@hana1 ~]# pcs status --full
…
…
Node List:
  * Online: [ hana1.private.example.com (1) hana2.private.example.com (2) ]

Full List of Resources:
  * fence_hana1    (stonith:fence_ipmilan):     Started hana2.private.example.com
  * fence_hana2    (stonith:fence_ipmilan):     Started hana1.private.example.com
  * vip_RHE_00_primary    (ocf::heartbeat:IPaddr2):     Started hana1.private.example.com
  * Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
    * SAPHanaTopology_RHE_00    (ocf::heartbeat:SAPHanaTopology):     Started hana1.private.example.com
    * SAPHanaTopology_RHE_00    (ocf::heartbeat:SAPHanaTopology):     Started hana2.private.example.com
  * Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
    * SAPHana_RHE_00    (ocf::heartbeat:SAPHana):     Promoted hana1.private.example.com
    * SAPHana_RHE_00    (ocf::heartbeat:SAPHana):     Unpromoted hana2.private.example.com

Node Attributes:
  * Node: hana1.private.example.com (1):
    * hana_rhe_clone_state                : PROMOTED  
    * hana_rhe_op_mode                    : logreplay
    * hana_rhe_remoteHost                 : hana2     
    * hana_rhe_roles                      : 4:P:master1:master:worker:master
    * hana_rhe_site                       : DC01      
    * hana_rhe_srmode                     : sync      
    * hana_rhe_sync_state                 : PRIM      
    * hana_rhe_version                    : 2.00.067.04.1700654562
    * hana_rhe_vhost                      : hana1     
    * lpa_rhe_lpt                         : 1716486893
    * master-SAPHana_RHE_00               : 150       
  * Node: hana2.private.example.com (2):
    * hana_rhe_clone_state                : DEMOTED   
    * hana_rhe_op_mode                    : logreplay
    * hana_rhe_remoteHost                 : hana1     
    * hana_rhe_roles                      : 4:S:master1:master:worker:master
    * hana_rhe_site                       : DC02      
    * hana_rhe_srmode                     : sync      
    * hana_rhe_sync_state                 : SFAIL     
    * hana_rhe_version                    : 2.00.067.04.1700654562
    * hana_rhe_vhost                      : hana2     
    * lpa_rhe_lpt                         : 10        
    * master-SAPHana_RHE_00               : -INFINITY
…

 

 

The above output shows the following:

  • The cluster does not initiate any failover, and the SAP HANA primary and secondary instances continue to run on their respective hosts.
  • The SAP HANA system is available and accessible by the application server, i.e., users can still access SAP.
  • The cluster is aware there is a problem with HSR; the attribute “hana_rhe_sync_state” for the secondary/demoted node on hana2 is “SFAIL”.

At this point, the network error would need to be fixed, which would restart HSR. 

Before fixing the HSR network, the next step kills the primary SAP HANA instance using the “HDB kill -9” command:

 

 

rheadm@hana1:/usr/sap/RHE/HDB00/exe/python_support> HDB kill -9
…
killing HDB processes:
kill -9 108402 /usr/sap/RHE/HDB00/hana1/trace/hdb.sapRHE_HDB00 -d -nw -f /usr/sap/RHE/HDB00/hana1/daemon.ini pf=/usr/sap/RHE/SYS/profile/RHE_HDB00_hana1
kill -9 108428 hdbnameserver
…..

 

 

The cluster first shows a failure of the primary SAP HANA instance on hana1:

 

 

[root@hana1 ansible-files]# pcs status
…
…

Node List:
  * Online: [ hana1.private.example.com hana2.private.example.com ]

Full List of Resources:
  * fence_hana1    (stonith:fence_ipmilan):     Started hana1.private.example.com
  * fence_hana2    (stonith:fence_ipmilan):     Started hana2.private.example.com
  * vip_RHE_00_primary    (ocf::heartbeat:IPaddr2):     Stopped
  * Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
	* Started: [ hana1.private.example.com hana2.private.example.com ]
  * Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
	* SAPHana_RHE_00    (ocf::heartbeat:SAPHana):     FAILED hana1.private.example.com
	* Unpromoted: [ hana2.private.example.com ]

Failed Resource Actions:
  * SAPHana_RHE_00_monitor_119000 on hana1.private.example.com 'promoted (failed)' (9): call=40, status='complete', exitreason='', last-rc-change='2024-05-19 17:53:19 -04:00', queued=0ms, exec=0ms
…..

 

 

The cluster restarts the primary SAP HANA instance on hana1, i.e., no failover to hana2:

 

 

[root@hana1 ansible-files]# pcs  status
..
..
Node List:
  * Online: [ hana1.private.example.com hana2.private.example.com ]

Full List of Resources:
  * fence_hana1    (stonith:fence_ipmilan):     Started hana1.private.example.com
  * fence_hana2    (stonith:fence_ipmilan):     Started hana2.private.example.com
  * vip_RHE_00_primary    (ocf::heartbeat:IPaddr2):     Started hana1.private.example.com
  * Clone Set: SAPHanaTopology_RHE_00-clone [SAPHanaTopology_RHE_00] (promotable):
	* Started: [ hana1.private.example.com hana2.private.example.com ]
  * Clone Set: SAPHana_RHE_00-clone [SAPHana_RHE_00] (promotable):
	* Promoted: [ hana1.private.example.com ]
	* Unpromoted: [ hana2.private.example.com ]


…..

 

 

The above is expected behavior as HSR has failed, which the cluster knows about; the HSR error status is stored in the node attributes as shown above. This means that data replication to the secondary instance has failed, and the data in the secondary SAP HANA system may not be up-to-date. Any failover in this situation may result in data loss, and the cluster has prevented this.

Cluster Network Error Test

The primary SAP HANA instance is on hana1, and the secondary on hana2. 

The private internal network interface on hana2 is deactivated:

 

 

[root@hana2 ~]# nmcli con down 'Wired connection 2'
Connection 'Wired connection 2' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)

 

 

The cluster communication continues as normal with the remaining link 1 (the SAP HANA internal network). This was observed in the cluster log file /var/log/cluster/corosync.log:

 

 

[root@hana2 ~]# more corosync.log
….
….
Jun 05 14:34:11 [66151] hana2 corosync info    [KNET  ] link: host: 1 link: 0 is down
Jun 05 14:34:11 [66151] hana2 corosync info    [KNET  ] host: host: 1 (passive) best link: 1 (pri: 1)
….
….

 

 

Meanwhile, there was no impact on the cluster, and the primary SAP HANA instance continued to run on hana1 and the secondary on hana2.

Summary

The article describes an example deployment of SAP HANA and SAP HANA System Replication (HSR) with a RHEL HA Pacemaker cluster using multiple networks corresponding to the network zones defined by SAP and communications required by the cluster.

Some tests were conducted to simulate network errors and demonstrate cluster functionality. Note that not all the required failover testing was shown here - for a full list of tests and expected results, see here.

Thank you Vas Mitra and Frank Danapfel for the content :)!

Labels:
Labels in this area
Popular Blog Posts