Hitachi Unified Compute Platform for the SAP
Hitachi Unified Compute Platform for the SAP HANA Platform with Logical Partitioning in a Scale-Up Configuration Using Hitachi Compute Blade 2500 and Hitachi Virtual Storage Platform
Reference Architecture Guide By Stephan Kreitz and David Pascuzzi
September 2015
Feedback Hitachi Data Systems welcomes your feedback. Please share your thoughts by sending an email message to [email protected] To assist the routing of this message, use the paper number in the subject and the title of this white paper in the text.
Contents Solution Overview................................................................................................................ 3 Key Solution Elements......................................................................................................... 4 Hardware Elements......................................................................................................................... 4 Software Elements .......................................................................................................................... 7
Solution Design................................................................................................................... 10 Hitachi Compute Blade 2500 Chassis Configuration ................................................................... Hitachi's Server Blade Architecture .............................................................................................. Fibre Channel SAN Architecture ................................................................................................... Network Architecture .................................................................................................................... Storage Architecture ..................................................................................................................... SAP HANA Configuration..............................................................................................................
10 10 11 12 13 15
1
Hitachi Unified Compute Platform for the SAP HANA Platform with Logical Partitioning in a Scale-Up Configuration Using Hitachi Compute Blade 2500 and Hitachi Virtual Storage Platform Reference Architecture Guide Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) in a scale-up configuration is a pre-configured virtual appliance ready to plug into a network to provide real-time access to operational data for use in analytic models. SAP HANA on Hitachi solutions with logical partitioning are based on a number of bare-metal appliance configurations with modifications that are required for running logical partitions (LPARs) in dedicated mode. The basis configurations are shown in Table 1, and possible LPAR sizes are shown in Table 2 on page 2. The modifications necessary to run LPARs for productive SAP HANA instances are described in this document. Table 1. Supported Scale-up Configurations for SAP HANA with Logical Partitioning Small (S)
Medium (M)
Large (L)
Number of CPU sockets
2
4
8
Number of Server Blades
1
2 (using a 2-blade SMP)
4 (using a 4-blade SMP)
Memory (in GB)
128, 256, 384, 512, 768, 1024, 1536
256, 384, 512, 768, 1024, 1536, 2048, 3072
512, 768, 1024, 1536, 2048, 3072, 4096, 6144
4 × 1-socket
8 × 1-socket
2 × 2-socket
4 × 2-socket
2 × 1- socket + 1 × 2socket
6 × 1-socket + 1 × 2-socket
Number of LPARs for 2 × 1-socket Productive Usage 1 × 2-socket
4 × 1-socket + 2 × 2-socket 2 × 1-socket + 3 × 2-socket
Max. Memory Size per LPAR
1-socket
50% of the memory
25% of the memory
12.5% of the memory
2-socket
100% of the memory
50% of the memory
25% of the memory
1
2 Based on these configurations, LPARs can be created following the memory and CPU socket restrictions shown in Table 2. Throughout this document the term CPU socket is used to differentiate between the LPAR size and the bare metal system size. CPU socket does not refer to an empty CPU socket, but includes the Intel E7 8880v3 Haswell CPU. Table 2. Supported LPAR Memory Configurations for SAP Application Scenarios LPAR Configuration
SAP Business Warehouse (BW) on HANA or Data Mart Scenarios
SAP Business Suite on HANA Scenarios
1-socket LPARs memory size (in GB)
64, 128, 192, 256
128, 256, 384, 512, 768
2-socket LPARs memory size (in GB)
128, 192, 256, 384, 512
128, 256, 384, 512, 768, 1024, 1536
A partition on the CPU core level instead of the CPU socket level is not supported for LPARs running productive SAP HANA instances. This reference architecture guide describes how to deploy Unified Compute Platform for SAP HANA on Hitachi solutions with LPAR in a scale-up configuration using the following:
Hitachi Compute Blade 2500 (with 520X B2 server blades)
Hitachi Virtual Storage Platform G600
SAP HANA
This technical paper assumes familiarity with the following:
Storage Area Network (SAN) based storage systems
General storage concepts
SAP HANA
Hitachi's logical partitioning concept
Common IT storage practices
2
3
Solution Overview Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) in a scale-up configuration reference solution is a blade server solution for SAP HANA using logical partitioning. This reference architecture uses the following:
Hitachi Compute Blade 2500 (CB 2500) — Enterprise computing power and performance with flexible I/O architecture and logical partitioning so multiple applications can easily and securely co-exist in the same chassis.
This solution uses a different number of 520X B2 server blades for a different number of CPU sockets and memory sizes. See Table 1, “Supported Scale-up Configurations for SAP HANA with Logical Partitioning,” on page 1 for a list of supported configurations.
Hitachi Virtual Storage Platform G600 (VSP G600) — Storage virtualization system designed to manage storage assets more efficiently. The persistent storage of the SAP HANA server resides on this storage device. SAP HANA — A multi-purpose, in-memory database to analyze transactional and analytical data. Logical Partitioning (LPAR) — A feature of Hitachi Compute Blade family, this firmware-based hypervisor creates logical partitions on one physical server.
3
4
Key Solution Elements These are the key hardware and software components used in this reference architecture.
Hardware Elements Based on Hitachi Unified Compute Platform for the SAP HANA Platform, it is possible to create scale-up configurations with logical partitioning.
Hitachi Compute Blade 2500 Hitachi Compute Blade 2500 delivers enterprise computing power and performance with unprecedented scalability and configuration flexibility. Lower your costs and protect your investment. Flexible I/O architecture and logical partitioning allow configurations to match application needs exactly with Hitachi Compute Blade 2500. Multiple applications easily and securely co-exist in the same chassis. Add server management and system monitoring at no cost with Hitachi Compute Systems Manager. Seamlessly integrate with Hitachi Command Suite in Hitachi storage environments. The configuration uses a different number (one, two, or four) of 520X B2 server blades in the Hitachi Compute Blade 2500 chassis for different size systems as outlined in Table 6, “Server Blade Configuration,” on page 11. Table 3 shows the specifications for the 520X B2 server blades used in the solution for different size configurations. Table 3. 520X B2 Server Blade Configuration Feature
Configuration Small (2-Socket) Medium (4-Socket)
Processors
2 Intel Xeon E7-8880 processors
Processor SKU
Intel Xeon E7-8880 v3
Processor frequency
2.3 GHz
Processor cores
18 cores
Number of blades
1
Large (8-Socket) 4 Intel Xeon E7-8880 processors
8 Intel Xeon E7-8880 processors
2
4
4
5 Table 3. 520X B2 Server Blade Configuration (Continued) Feature
Configuration Small (2-Socket) Medium (4-Socket)
Number of 128 GB DIMMs per blade 256 GB
Large (8-Socket)
8 × 16 GB DIMMs
8 × 8 GB DIMMs
N/A
16 × 16 GB DIMMs
8 × 16 GB DIMMs
8 x 8 GB DIMMs
384 GB
24 × 16 GB DIMMs
24 × 8 GB DIMMs
N/A
512 GB
32 × 16 GB DIMMs
16 × 16 GB DIMMs
16 × 8 GB DIMMs
768 GB
48 × 16 GB DIMMs
24 × 16 GB DIMMs
24 × 8 GB DIMMs
1024 GB
32 × 32 GB DIMMs
32 × 16 GB DIMMs
16 × 16 GB DIMMs
1536 GB
48 × 32 GB DIMMs
48 × 16 GB DIMMs
24 × 16 GB DIMMs
2048 GB
N/A
32 × 32 GB DIMMs
32 × 16 GB DIMMs
3072 GB
N/A
48 × 32 GB DIMMs
48 × 16 GB DIMMs
4096 GB
N/A
N/A
32 × 32 GB DIMMs
6144 GB
N/A
N/A
48 × 32 GB DIMMs
Network Ports
2 × 2-port 10GBASE-SR LAN PCI-E adapter on two I/O board modules for each blade:
IOBD 01B
IOBD 01B
IOBD 01B
IOBD 02B
IOBD 02B
IOBD 02B
IOBD 03B
IOBD 03B
IOBD 04B
IOBD 04B
IOBD 05B
IOBD 06B
IOBD 07B
IOBD 08B
5
6 Table 3. 520X B2 Server Blade Configuration (Continued) Feature
Configuration Small (2-Socket) Medium (4-Socket)
Fibre Channel Ports
Other Interfaces
Large (8-Socket)
2 × Hitachi 16 Gb/sec 2-port Fibre Channel PCI-E adapters on two I/O board modules for each blade:
IOBD 01A
IOBD 01A
IOBD 01A
IOBD 02A
IOBD 02A
IOBD 02A
IOBD 03A
IOBD 03A
IOBD 04A
IOBD 04A
IOBD 05A
IOBD 06A
IOBD 07A
IOBD 08A
1 USB 3.0 port KVM connector (VGA, COM, USB2.0 port)
Figure 1 shows the layout of the I/O board modules from the back of the Hitachi Compute Blade 2500 chassis. For each solution the maximum number of I/O board modules is used, ranging from four I/O board modules for a single blade solution up to 16 I/O board modules for a 4-blade solution.
Figure 1 6
7 Symmetric Multiprocessing Connector For multiple server blades, the solution uses symmetric multiprocessing (SMP) technology to combine multiple server blade resources into a single server. The 520X B2 server blades use SMP in one of two ways, depending on the size of the solution:
Combine two server blades with a 2-blade SMP connection board.
Combine four server blades with a 4-blade SMP connection board.
Hitachi Virtual Storage Platform Family The Hitachi Virtual Storage Platform family systems are based on industry-leading enterprise storage technology. With flash-optimized performance, these systems provide advanced capabilities previously available only in high-end storage arrays. With the Virtual Storage Platform family, you can build a high performance, software-defined infrastructure to transform data into valuable information. Hitachi Storage Virtualization Operating System provides storage virtualization, high availability, superior performance, and advanced data protection for all models in the Virtual Storage Platform family. This proven, mature software provides common features to consolidate assets, reclaim space, extend life, and reduce migration effort. New management software improves ease of use to save time and reduce complexity. The infrastructure of Storage Virtualization Operating System creates a management framework for improved IT response to business demands. This solution uses a single server with up to eight LPARs connected to one VSP G600. Further consolidation of multiple servers with up to eight LPARs each is possible using VSP G800 and VSP G1000. Both storage models have been certified for Hitachi bare metal scale-out appliances.
Software Elements Table 4 describes the software products used to deploy the one active node configuration. Table 4. Software Elements Software
Version
SUSE Linux Enterprise Server for SAP Applications or
11 SP3
Red Hat Enterprise Linux
6.6
SAP HANA
1.0 SPS09, Rev. 97 or later
Logical Partitioning Manager
02-28 or later
Hitachi Compute Blade 2500 management module firmware pack
A0135/00
Hitachi Compute Blade 2500 management module firmware
A0135-A-1200
SEL dictionary for Hitachi Compute Blade 2500
A0021
520X server blade integrated firmware
09-14
Hitachi Virtual Storage Platform
83-01-21-40/00
Hitachi Storage Navigator
Microcode dependent
Hitachi Command Suite
Version 11, release 8.1.2 or higher
7
8 SAP HANA SAP HANA converges database and application platform capabilities in-memory to transform transactions, analytics, text analysis, predictive and spatial processing so businesses can operate in real-time. This combines database, data processing, and application platform capabilities in a single in-memory platform. Also, the platform provides libraries for predictive, planning, text processing, spatial, and business analytics — all on the same architecture. This architecture comes from leading hardware partners of SAP, including Hitachi. By eliminating the divide between transactions and analytics, SAP HANA allows you to answer any business question anywhere in real time. Flexible views quickly expose analytic information. External data can be added to analytic models from across an entire organization. SAP customers can download more information on the SAP HANA Platform at the SAP Service Marketplace. See the installation and upgrade guides download section for SAP In-Memory Computing (SAP In-Memory Appliance — SAP HANA). The following are available:
SAP HANA Master Guide — this is the central starting point for the technical implementation of the SAP HANA platform. Use this for basic concepts and for planning the SAP HANA application system landscape. SAP HANA Installation and Initial Configuration Guides — use the various installation guides to install the required SAP In-Memory Database and the other software components for the different replication technologies. Refer to the SAP HANA Server Installation Guide for an overview on how to install SAP HANA. SAP HANA Technical Operations Manual — provides an end-to-end picture of the available SAP HANA appliance administration tools and the key tasks for a system administrator to perform. SAP HANA Master Update Guide — this explains how to update SAP HANA and its components. SAP Integration and Certification Center (SAP ICC) — this page provides information about SAP HANA appliances certified by SAP hardware partners.
The following is a link to all SAP-related documentation: http://help.sap.com/hana_platform/.
Information on productive SAP HANA systems on Hitachi systems with LPAR can be found in SAP Note 2063057.
For SAP Community Network documents on Hitachi systems with LPAR, use the following documents as a starting point: SAP on LPAR and Best Practices Guide for Hitachi LPAR.
SUSE Linux Enterprise Server (SLES) for SAP Applications and Red Hat Enterprise Linux (RHEL) The following options are available as guest operating system for the LPARs: 64-bit SUSE Linux Enterprise Server (SLES) for SAP Applications 11 SP3 64-bit Red Hat Enterprise Linux (RHEL) 6.6 Changing the configuration settings is only supported along the guidelines of SAP and the operating system distributor and may otherwise cause significant performance problems. A good starting point for information on this topic is provided in the following SAP Notes for SLES and RHEL: 1944799 - SAP HANA Guidelines for SLES Operating System Installation 2136965 - SAP HANA DB: Recommended OS settings for RHEL 6.6 For more details, see section 2.1.4.1, “Updating and Patching the Operating System,” in the SAP HANA Technical Operations Manual.
8
9 Logical Partitioning The logical partitioning feature from Hitachi partitions the physical resources of one single-blade or multi-blade server logically into independent server environments called LPARs (logical partitions). CPUs, memory, and I/O devices can be assigned in dedicated mode to LPARs to create fully isolated server environments for SAP HANA single node installations within each LPAR running on the same physical host without any noisy neighbor effects between the different server environments. 520X B2 server blades must be set to LP mode in order to enable logical partitioning capabilities. LPARs using one or two CPU sockets in dedicated mode, accessing the CPU sockets’ local memory bank, can be created for running productive SAP HANA instances. Combined with dedicated HBA cards for each LPAR, and separated controller ports and RAID groups in Hitachi Virtual Storage Platform, the LPAR manager enables the consolidation of multiple productive SAP HANA instances on the same physical server using a single Virtual Storage Platform. An overview of the supported LPAR configurations can be found in Table 1, “Supported Scale-up Configurations for SAP HANA with Logical Partitioning,” on page 1 and Table 2, “Supported LPAR Memory Configurations for SAP Application Scenarios,” on page 2. More information on the operations and administration of LPARs can be found in the Hitachi Compute Blade 2500 Series Logical Partition Manager User’s Guide (MK-99CB2500006-00). The licenses listed in Table 5 are required for running SAP HANA on Unified Compute Platform for the SAP HANA Platform with logical partitioning. Table 5. Licenses Required for UCP for SAP HANA with Logical Partitioning License Name
Mandatory
LPAR manager Enterprise license
1 per blade
9
10
Solution Design The detailed design for Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) in a scale-up configuration reference solution includes the following:
Hitachi Compute Blade 2500 Chassis Configuration
Hitachi's Server Blade Architecture
Fibre Channel SAN Architecture
Network Architecture
Storage Architecture
SAP HANA Configuration
Hitachi Compute Blade 2500 Chassis Configuration This solution uses one Hitachi Compute Blade 2500 chassis with the following components:
One Hitachi Compute Blade 2500 chassis can have a maximum of 8 full-width blades, but the SAP HANA scale-up configurations can have one, two, or four 520X B2 server blades. Refer to Table 6, “Server Blade Configuration,” on page 11. There are two management modules on the Hitachi Compute Blade 2500 chassis to connect to the management network. A maximum of 28 I/O board modules (IOBD) can be mounted on one Hitachi Compute Blade 2500 chassis. Depending on the size of the solution, between four and 16 I/O board modules are used.
Hitachi FIVE-FX 16 Gb/sec 2-port Fibre channel PCI-E adapters are installed on the A-slot of each I/O board module. 10GBase-SR 2-port network PCI-E adapters are installed on B-slot of each I/O board module.
Hitachi's Server Blade Architecture Each solution size has a different number of server blades, from one up to four full-width server blades. Multiple server blades use a two- or four-server blade SMP connection board to create a single four- or eight-socket SMP node with a total of 72 or 144 cores and different memory sizes.
10
11 Table 6 lists the server blade configuration details for small, medium, and large solution sizes. Table 6. Server Blade Configuration
Server Blades
Small (2-Socket)
Medium (4-Socket)
Large (8-Socket)
Total of 1 server blade:
Total of 2 server blades:
Total of 4 server blades (bottom-up):
Blade 3 (non-primary)
Blade 7 (non-primary)
Blade 1 (primary)
Blade 5 (non-primary)
Blade 3 (non-primary)
Blade 1 (primary)
Total Number of CPU Cores
Blade 1 (primary)
36
72
144
Fibre Channel SAN Architecture The Fibre Channel SAN architecture consists of the following components on 520X B2 blades:
Each blade uses two Hitachi 16 Gb/sec 2-port Fibre Channel Adapters installed on the PCI-E A-slots of the I/O board modules This solution uses between two and eight Hitachi 16 Gb/sec 2-port Fibre Channel Adapters in total
This solution uses between four and sixteen 16 Gb/sec Fibre Channel ports on the VSP G600 storage directly attached to the Hitachi Compute Blade 2500 server chassis via the Fibre Channel PCI-E adapters. Table 7 shows the Fibre Channel port mappings between the ports on the VSP G600 and the Fibre Channel adapters on the I/O board modules. Table 7. Fibre Channel Port Mapping Blade #
PCI-E Slot #
Port #
VSP G600 Port
Blade 1
IOBD 01A
0
1A
1
2A
0
3A
1
4A
0
1B
1
2B
0
3B
1
4B
0
1C
1
2C
0
3C
1
4C
IOBD 02A
Blade 3
IOBD 03A IOBD 04A
Blade 5
IOBD 05A IOBD 06A
11
12 Table 7. Fibre Channel Port Mapping (Continued) Blade #
PCI-E Slot #
Port #
VSP G600 Port
Blade 7
IOBD 07A
0
1D
1
2D
0
3D
1
4D
IOBD 08A
This configuration provides two dedicated paths from the LPAR within Hitachi Compute Blade 2500 to the ports on VSP G600 for each 1-socket LPAR, and four dedicated paths for 2-socket LPARs. Set the port properties for the point-to-point connection between Hitachi Compute Blade 2500 and VSP G600 storage, as shown in Table 8. Table 8. Port Properties Property
Value
Port Security
Disabled
Port Speed
Auto (16 Gb/sec)
Fabric
ON
Connection Type
P-to-P
Mixed configurations with both 1-socket and 2-socket LPARs are realized by first creating all 2-socket LPARs, and then creating the 1-socket LPARs.
Network Architecture This solution uses 10GBASE-SR 2-port LAN adapters installed on the PCI-E B-slots of each I/O board module. Between two and eight LAN adapters are used for connecting to external 10 GbE switches in the customer data center as shown in Table 9. Each network controller is dedicated to an LPAR providing two 10 GbE network ports for each 1-socket LPAR, and four 10GbE network ports for each 2-socket LPAR. Table 9. Available 10 GbE LAN Ports Number of 520X B2 Server Blades
Number of Available 10 GbE LAN Ports
1
2 NIC controllers with a total of 4 ports
2
4 NIC controllers with a total of 8 ports
4
8 NIC controllers with a total of 16 ports
12
13 1-socket LPARs, bond the corresponding two network interfaces as bond0 at the operating system level using activeFor active network bonding mode with the options shown in Table 10, which acts as the client network for the SAP HANA node. The additional two network interfaces available to 2-socket LPARs can be used freely for a bonding device bond1, i.e. to separate the client and backup network or the SAP HANA system replication network, or they can also be added to bond0. Table 10. Network Bonding Parameters Parameter
Value
mode
802.3ad
miimon
100
xmit_hash_policy
layer3+4
updelay
5000
lacp_rate
fast
Management Network Logical partitioning manager requires two additional network ports for the LPAR management network path. These two ports are provided by two 1 GbE 4-port LAN mezzanine cards located in slots 1 and 3 of blade 1. The LAN mezzanine cards are connected to the management network through two 1/10Gb LAN switch modules located in the back of the Hitachi Compute Blade 2500 chassis. All 1 GbE management network connections can be consolidated using these 1G/10Gb LAN switch modules, especially the LAN connection to the Compute Blade 2500 chassis management modules. Logical partitioning manager management network links must be aggregated using active-backup mode. The IP address of LPAR manager and the chassis management modules (SVP) have to be in the same subnet. For more information on VLAN configuration for the LPAR management path, see the Hitachi Compute Blade 2500 Series Logical Partitioning Manager User's Guide (MK-99CB2500006-00). The Hitachi Compute Blade 2500 chassis has two management modules to secure fault tolerance with the following:
Manage the power supply of each module and monitor the status of the system unit.
Support the management functionality of the network within the system unit for server blades and various modules.
Connect between a management module and an external network through a management LAN module
Storage Architecture There are many factors impacting the drive size and storage configuration of Hitachi Unified Compute Platform for the SAP HANA Platform solutions including I/O and capacity requirements. With fully dedicated LPARs running on these systems, the SAS drive RAID groups that are used for the HANA data and log volumes are dedicated to each LPAR and must not be shared between the LPARs. This increases the storage requirements for Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) solution compared to the standard scale-up solution that is used as a basis for this solution.
13
14 RAID and LUN Configuration This reference architecture uses slightly different RAID configurations on Hitachi Virtual Storage Platform G600 depending on the size of the server.
One RAID-6 (6D+2P) group using 8 × 600 GB SAS 10k RPM drives for hosting operating system and HANA shared volumes. On systems with a memory size of 3 TB or more, one RAID-6 (14D+2P) group using 16 × 600 GB SAS 10k RPM drives is used to meet the combined size requirements for the operating system and /hana/shared directory of all LPARs. For each LPAR, two or three dedicated RAID groups for HANA data and log volumes are created depending on the LPAR memory size requirements. 1 × 600 GB SAS 10k RPM spare drive per 32 SAS drives.
For each LPAR on this solution the following LUNs are created:
One 100 GB OS LUN to host the LPAR guest operating system.
One LUN to host the HANA shared volume that matches the LPAR memory size.
Four 150 GB LUNs to host the HANA log volume located on the same RAID group. A striped volume over the four LUNs is used. Four or eight LUNs to host the HANA data volume. A striped volume over these LUNs is used. See Table 11 for RAID group and LUN details.
Table 11. RAID Groups for HANA Data Volume and HANA Log Volume per LPAR
HANA data volume
LPAR memory size
Reference Architecture Guide By Stephan Kreitz and David Pascuzzi
September 2015
Feedback Hitachi Data Systems welcomes your feedback. Please share your thoughts by sending an email message to [email protected] To assist the routing of this message, use the paper number in the subject and the title of this white paper in the text.
Contents Solution Overview................................................................................................................ 3 Key Solution Elements......................................................................................................... 4 Hardware Elements......................................................................................................................... 4 Software Elements .......................................................................................................................... 7
Solution Design................................................................................................................... 10 Hitachi Compute Blade 2500 Chassis Configuration ................................................................... Hitachi's Server Blade Architecture .............................................................................................. Fibre Channel SAN Architecture ................................................................................................... Network Architecture .................................................................................................................... Storage Architecture ..................................................................................................................... SAP HANA Configuration..............................................................................................................
10 10 11 12 13 15
1
Hitachi Unified Compute Platform for the SAP HANA Platform with Logical Partitioning in a Scale-Up Configuration Using Hitachi Compute Blade 2500 and Hitachi Virtual Storage Platform Reference Architecture Guide Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) in a scale-up configuration is a pre-configured virtual appliance ready to plug into a network to provide real-time access to operational data for use in analytic models. SAP HANA on Hitachi solutions with logical partitioning are based on a number of bare-metal appliance configurations with modifications that are required for running logical partitions (LPARs) in dedicated mode. The basis configurations are shown in Table 1, and possible LPAR sizes are shown in Table 2 on page 2. The modifications necessary to run LPARs for productive SAP HANA instances are described in this document. Table 1. Supported Scale-up Configurations for SAP HANA with Logical Partitioning Small (S)
Medium (M)
Large (L)
Number of CPU sockets
2
4
8
Number of Server Blades
1
2 (using a 2-blade SMP)
4 (using a 4-blade SMP)
Memory (in GB)
128, 256, 384, 512, 768, 1024, 1536
256, 384, 512, 768, 1024, 1536, 2048, 3072
512, 768, 1024, 1536, 2048, 3072, 4096, 6144
4 × 1-socket
8 × 1-socket
2 × 2-socket
4 × 2-socket
2 × 1- socket + 1 × 2socket
6 × 1-socket + 1 × 2-socket
Number of LPARs for 2 × 1-socket Productive Usage 1 × 2-socket
4 × 1-socket + 2 × 2-socket 2 × 1-socket + 3 × 2-socket
Max. Memory Size per LPAR
1-socket
50% of the memory
25% of the memory
12.5% of the memory
2-socket
100% of the memory
50% of the memory
25% of the memory
1
2 Based on these configurations, LPARs can be created following the memory and CPU socket restrictions shown in Table 2. Throughout this document the term CPU socket is used to differentiate between the LPAR size and the bare metal system size. CPU socket does not refer to an empty CPU socket, but includes the Intel E7 8880v3 Haswell CPU. Table 2. Supported LPAR Memory Configurations for SAP Application Scenarios LPAR Configuration
SAP Business Warehouse (BW) on HANA or Data Mart Scenarios
SAP Business Suite on HANA Scenarios
1-socket LPARs memory size (in GB)
64, 128, 192, 256
128, 256, 384, 512, 768
2-socket LPARs memory size (in GB)
128, 192, 256, 384, 512
128, 256, 384, 512, 768, 1024, 1536
A partition on the CPU core level instead of the CPU socket level is not supported for LPARs running productive SAP HANA instances. This reference architecture guide describes how to deploy Unified Compute Platform for SAP HANA on Hitachi solutions with LPAR in a scale-up configuration using the following:
Hitachi Compute Blade 2500 (with 520X B2 server blades)
Hitachi Virtual Storage Platform G600
SAP HANA
This technical paper assumes familiarity with the following:
Storage Area Network (SAN) based storage systems
General storage concepts
SAP HANA
Hitachi's logical partitioning concept
Common IT storage practices
2
3
Solution Overview Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) in a scale-up configuration reference solution is a blade server solution for SAP HANA using logical partitioning. This reference architecture uses the following:
Hitachi Compute Blade 2500 (CB 2500) — Enterprise computing power and performance with flexible I/O architecture and logical partitioning so multiple applications can easily and securely co-exist in the same chassis.
This solution uses a different number of 520X B2 server blades for a different number of CPU sockets and memory sizes. See Table 1, “Supported Scale-up Configurations for SAP HANA with Logical Partitioning,” on page 1 for a list of supported configurations.
Hitachi Virtual Storage Platform G600 (VSP G600) — Storage virtualization system designed to manage storage assets more efficiently. The persistent storage of the SAP HANA server resides on this storage device. SAP HANA — A multi-purpose, in-memory database to analyze transactional and analytical data. Logical Partitioning (LPAR) — A feature of Hitachi Compute Blade family, this firmware-based hypervisor creates logical partitions on one physical server.
3
4
Key Solution Elements These are the key hardware and software components used in this reference architecture.
Hardware Elements Based on Hitachi Unified Compute Platform for the SAP HANA Platform, it is possible to create scale-up configurations with logical partitioning.
Hitachi Compute Blade 2500 Hitachi Compute Blade 2500 delivers enterprise computing power and performance with unprecedented scalability and configuration flexibility. Lower your costs and protect your investment. Flexible I/O architecture and logical partitioning allow configurations to match application needs exactly with Hitachi Compute Blade 2500. Multiple applications easily and securely co-exist in the same chassis. Add server management and system monitoring at no cost with Hitachi Compute Systems Manager. Seamlessly integrate with Hitachi Command Suite in Hitachi storage environments. The configuration uses a different number (one, two, or four) of 520X B2 server blades in the Hitachi Compute Blade 2500 chassis for different size systems as outlined in Table 6, “Server Blade Configuration,” on page 11. Table 3 shows the specifications for the 520X B2 server blades used in the solution for different size configurations. Table 3. 520X B2 Server Blade Configuration Feature
Configuration Small (2-Socket) Medium (4-Socket)
Processors
2 Intel Xeon E7-8880 processors
Processor SKU
Intel Xeon E7-8880 v3
Processor frequency
2.3 GHz
Processor cores
18 cores
Number of blades
1
Large (8-Socket) 4 Intel Xeon E7-8880 processors
8 Intel Xeon E7-8880 processors
2
4
4
5 Table 3. 520X B2 Server Blade Configuration (Continued) Feature
Configuration Small (2-Socket) Medium (4-Socket)
Number of 128 GB DIMMs per blade 256 GB
Large (8-Socket)
8 × 16 GB DIMMs
8 × 8 GB DIMMs
N/A
16 × 16 GB DIMMs
8 × 16 GB DIMMs
8 x 8 GB DIMMs
384 GB
24 × 16 GB DIMMs
24 × 8 GB DIMMs
N/A
512 GB
32 × 16 GB DIMMs
16 × 16 GB DIMMs
16 × 8 GB DIMMs
768 GB
48 × 16 GB DIMMs
24 × 16 GB DIMMs
24 × 8 GB DIMMs
1024 GB
32 × 32 GB DIMMs
32 × 16 GB DIMMs
16 × 16 GB DIMMs
1536 GB
48 × 32 GB DIMMs
48 × 16 GB DIMMs
24 × 16 GB DIMMs
2048 GB
N/A
32 × 32 GB DIMMs
32 × 16 GB DIMMs
3072 GB
N/A
48 × 32 GB DIMMs
48 × 16 GB DIMMs
4096 GB
N/A
N/A
32 × 32 GB DIMMs
6144 GB
N/A
N/A
48 × 32 GB DIMMs
Network Ports
2 × 2-port 10GBASE-SR LAN PCI-E adapter on two I/O board modules for each blade:
IOBD 01B
IOBD 01B
IOBD 01B
IOBD 02B
IOBD 02B
IOBD 02B
IOBD 03B
IOBD 03B
IOBD 04B
IOBD 04B
IOBD 05B
IOBD 06B
IOBD 07B
IOBD 08B
5
6 Table 3. 520X B2 Server Blade Configuration (Continued) Feature
Configuration Small (2-Socket) Medium (4-Socket)
Fibre Channel Ports
Other Interfaces
Large (8-Socket)
2 × Hitachi 16 Gb/sec 2-port Fibre Channel PCI-E adapters on two I/O board modules for each blade:
IOBD 01A
IOBD 01A
IOBD 01A
IOBD 02A
IOBD 02A
IOBD 02A
IOBD 03A
IOBD 03A
IOBD 04A
IOBD 04A
IOBD 05A
IOBD 06A
IOBD 07A
IOBD 08A
1 USB 3.0 port KVM connector (VGA, COM, USB2.0 port)
Figure 1 shows the layout of the I/O board modules from the back of the Hitachi Compute Blade 2500 chassis. For each solution the maximum number of I/O board modules is used, ranging from four I/O board modules for a single blade solution up to 16 I/O board modules for a 4-blade solution.
Figure 1 6
7 Symmetric Multiprocessing Connector For multiple server blades, the solution uses symmetric multiprocessing (SMP) technology to combine multiple server blade resources into a single server. The 520X B2 server blades use SMP in one of two ways, depending on the size of the solution:
Combine two server blades with a 2-blade SMP connection board.
Combine four server blades with a 4-blade SMP connection board.
Hitachi Virtual Storage Platform Family The Hitachi Virtual Storage Platform family systems are based on industry-leading enterprise storage technology. With flash-optimized performance, these systems provide advanced capabilities previously available only in high-end storage arrays. With the Virtual Storage Platform family, you can build a high performance, software-defined infrastructure to transform data into valuable information. Hitachi Storage Virtualization Operating System provides storage virtualization, high availability, superior performance, and advanced data protection for all models in the Virtual Storage Platform family. This proven, mature software provides common features to consolidate assets, reclaim space, extend life, and reduce migration effort. New management software improves ease of use to save time and reduce complexity. The infrastructure of Storage Virtualization Operating System creates a management framework for improved IT response to business demands. This solution uses a single server with up to eight LPARs connected to one VSP G600. Further consolidation of multiple servers with up to eight LPARs each is possible using VSP G800 and VSP G1000. Both storage models have been certified for Hitachi bare metal scale-out appliances.
Software Elements Table 4 describes the software products used to deploy the one active node configuration. Table 4. Software Elements Software
Version
SUSE Linux Enterprise Server for SAP Applications or
11 SP3
Red Hat Enterprise Linux
6.6
SAP HANA
1.0 SPS09, Rev. 97 or later
Logical Partitioning Manager
02-28 or later
Hitachi Compute Blade 2500 management module firmware pack
A0135/00
Hitachi Compute Blade 2500 management module firmware
A0135-A-1200
SEL dictionary for Hitachi Compute Blade 2500
A0021
520X server blade integrated firmware
09-14
Hitachi Virtual Storage Platform
83-01-21-40/00
Hitachi Storage Navigator
Microcode dependent
Hitachi Command Suite
Version 11, release 8.1.2 or higher
7
8 SAP HANA SAP HANA converges database and application platform capabilities in-memory to transform transactions, analytics, text analysis, predictive and spatial processing so businesses can operate in real-time. This combines database, data processing, and application platform capabilities in a single in-memory platform. Also, the platform provides libraries for predictive, planning, text processing, spatial, and business analytics — all on the same architecture. This architecture comes from leading hardware partners of SAP, including Hitachi. By eliminating the divide between transactions and analytics, SAP HANA allows you to answer any business question anywhere in real time. Flexible views quickly expose analytic information. External data can be added to analytic models from across an entire organization. SAP customers can download more information on the SAP HANA Platform at the SAP Service Marketplace. See the installation and upgrade guides download section for SAP In-Memory Computing (SAP In-Memory Appliance — SAP HANA). The following are available:
SAP HANA Master Guide — this is the central starting point for the technical implementation of the SAP HANA platform. Use this for basic concepts and for planning the SAP HANA application system landscape. SAP HANA Installation and Initial Configuration Guides — use the various installation guides to install the required SAP In-Memory Database and the other software components for the different replication technologies. Refer to the SAP HANA Server Installation Guide for an overview on how to install SAP HANA. SAP HANA Technical Operations Manual — provides an end-to-end picture of the available SAP HANA appliance administration tools and the key tasks for a system administrator to perform. SAP HANA Master Update Guide — this explains how to update SAP HANA and its components. SAP Integration and Certification Center (SAP ICC) — this page provides information about SAP HANA appliances certified by SAP hardware partners.
The following is a link to all SAP-related documentation: http://help.sap.com/hana_platform/.
Information on productive SAP HANA systems on Hitachi systems with LPAR can be found in SAP Note 2063057.
For SAP Community Network documents on Hitachi systems with LPAR, use the following documents as a starting point: SAP on LPAR and Best Practices Guide for Hitachi LPAR.
SUSE Linux Enterprise Server (SLES) for SAP Applications and Red Hat Enterprise Linux (RHEL) The following options are available as guest operating system for the LPARs: 64-bit SUSE Linux Enterprise Server (SLES) for SAP Applications 11 SP3 64-bit Red Hat Enterprise Linux (RHEL) 6.6 Changing the configuration settings is only supported along the guidelines of SAP and the operating system distributor and may otherwise cause significant performance problems. A good starting point for information on this topic is provided in the following SAP Notes for SLES and RHEL: 1944799 - SAP HANA Guidelines for SLES Operating System Installation 2136965 - SAP HANA DB: Recommended OS settings for RHEL 6.6 For more details, see section 2.1.4.1, “Updating and Patching the Operating System,” in the SAP HANA Technical Operations Manual.
8
9 Logical Partitioning The logical partitioning feature from Hitachi partitions the physical resources of one single-blade or multi-blade server logically into independent server environments called LPARs (logical partitions). CPUs, memory, and I/O devices can be assigned in dedicated mode to LPARs to create fully isolated server environments for SAP HANA single node installations within each LPAR running on the same physical host without any noisy neighbor effects between the different server environments. 520X B2 server blades must be set to LP mode in order to enable logical partitioning capabilities. LPARs using one or two CPU sockets in dedicated mode, accessing the CPU sockets’ local memory bank, can be created for running productive SAP HANA instances. Combined with dedicated HBA cards for each LPAR, and separated controller ports and RAID groups in Hitachi Virtual Storage Platform, the LPAR manager enables the consolidation of multiple productive SAP HANA instances on the same physical server using a single Virtual Storage Platform. An overview of the supported LPAR configurations can be found in Table 1, “Supported Scale-up Configurations for SAP HANA with Logical Partitioning,” on page 1 and Table 2, “Supported LPAR Memory Configurations for SAP Application Scenarios,” on page 2. More information on the operations and administration of LPARs can be found in the Hitachi Compute Blade 2500 Series Logical Partition Manager User’s Guide (MK-99CB2500006-00). The licenses listed in Table 5 are required for running SAP HANA on Unified Compute Platform for the SAP HANA Platform with logical partitioning. Table 5. Licenses Required for UCP for SAP HANA with Logical Partitioning License Name
Mandatory
LPAR manager Enterprise license
1 per blade
9
10
Solution Design The detailed design for Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) in a scale-up configuration reference solution includes the following:
Hitachi Compute Blade 2500 Chassis Configuration
Hitachi's Server Blade Architecture
Fibre Channel SAN Architecture
Network Architecture
Storage Architecture
SAP HANA Configuration
Hitachi Compute Blade 2500 Chassis Configuration This solution uses one Hitachi Compute Blade 2500 chassis with the following components:
One Hitachi Compute Blade 2500 chassis can have a maximum of 8 full-width blades, but the SAP HANA scale-up configurations can have one, two, or four 520X B2 server blades. Refer to Table 6, “Server Blade Configuration,” on page 11. There are two management modules on the Hitachi Compute Blade 2500 chassis to connect to the management network. A maximum of 28 I/O board modules (IOBD) can be mounted on one Hitachi Compute Blade 2500 chassis. Depending on the size of the solution, between four and 16 I/O board modules are used.
Hitachi FIVE-FX 16 Gb/sec 2-port Fibre channel PCI-E adapters are installed on the A-slot of each I/O board module. 10GBase-SR 2-port network PCI-E adapters are installed on B-slot of each I/O board module.
Hitachi's Server Blade Architecture Each solution size has a different number of server blades, from one up to four full-width server blades. Multiple server blades use a two- or four-server blade SMP connection board to create a single four- or eight-socket SMP node with a total of 72 or 144 cores and different memory sizes.
10
11 Table 6 lists the server blade configuration details for small, medium, and large solution sizes. Table 6. Server Blade Configuration
Server Blades
Small (2-Socket)
Medium (4-Socket)
Large (8-Socket)
Total of 1 server blade:
Total of 2 server blades:
Total of 4 server blades (bottom-up):
Blade 3 (non-primary)
Blade 7 (non-primary)
Blade 1 (primary)
Blade 5 (non-primary)
Blade 3 (non-primary)
Blade 1 (primary)
Total Number of CPU Cores
Blade 1 (primary)
36
72
144
Fibre Channel SAN Architecture The Fibre Channel SAN architecture consists of the following components on 520X B2 blades:
Each blade uses two Hitachi 16 Gb/sec 2-port Fibre Channel Adapters installed on the PCI-E A-slots of the I/O board modules This solution uses between two and eight Hitachi 16 Gb/sec 2-port Fibre Channel Adapters in total
This solution uses between four and sixteen 16 Gb/sec Fibre Channel ports on the VSP G600 storage directly attached to the Hitachi Compute Blade 2500 server chassis via the Fibre Channel PCI-E adapters. Table 7 shows the Fibre Channel port mappings between the ports on the VSP G600 and the Fibre Channel adapters on the I/O board modules. Table 7. Fibre Channel Port Mapping Blade #
PCI-E Slot #
Port #
VSP G600 Port
Blade 1
IOBD 01A
0
1A
1
2A
0
3A
1
4A
0
1B
1
2B
0
3B
1
4B
0
1C
1
2C
0
3C
1
4C
IOBD 02A
Blade 3
IOBD 03A IOBD 04A
Blade 5
IOBD 05A IOBD 06A
11
12 Table 7. Fibre Channel Port Mapping (Continued) Blade #
PCI-E Slot #
Port #
VSP G600 Port
Blade 7
IOBD 07A
0
1D
1
2D
0
3D
1
4D
IOBD 08A
This configuration provides two dedicated paths from the LPAR within Hitachi Compute Blade 2500 to the ports on VSP G600 for each 1-socket LPAR, and four dedicated paths for 2-socket LPARs. Set the port properties for the point-to-point connection between Hitachi Compute Blade 2500 and VSP G600 storage, as shown in Table 8. Table 8. Port Properties Property
Value
Port Security
Disabled
Port Speed
Auto (16 Gb/sec)
Fabric
ON
Connection Type
P-to-P
Mixed configurations with both 1-socket and 2-socket LPARs are realized by first creating all 2-socket LPARs, and then creating the 1-socket LPARs.
Network Architecture This solution uses 10GBASE-SR 2-port LAN adapters installed on the PCI-E B-slots of each I/O board module. Between two and eight LAN adapters are used for connecting to external 10 GbE switches in the customer data center as shown in Table 9. Each network controller is dedicated to an LPAR providing two 10 GbE network ports for each 1-socket LPAR, and four 10GbE network ports for each 2-socket LPAR. Table 9. Available 10 GbE LAN Ports Number of 520X B2 Server Blades
Number of Available 10 GbE LAN Ports
1
2 NIC controllers with a total of 4 ports
2
4 NIC controllers with a total of 8 ports
4
8 NIC controllers with a total of 16 ports
12
13 1-socket LPARs, bond the corresponding two network interfaces as bond0 at the operating system level using activeFor active network bonding mode with the options shown in Table 10, which acts as the client network for the SAP HANA node. The additional two network interfaces available to 2-socket LPARs can be used freely for a bonding device bond1, i.e. to separate the client and backup network or the SAP HANA system replication network, or they can also be added to bond0. Table 10. Network Bonding Parameters Parameter
Value
mode
802.3ad
miimon
100
xmit_hash_policy
layer3+4
updelay
5000
lacp_rate
fast
Management Network Logical partitioning manager requires two additional network ports for the LPAR management network path. These two ports are provided by two 1 GbE 4-port LAN mezzanine cards located in slots 1 and 3 of blade 1. The LAN mezzanine cards are connected to the management network through two 1/10Gb LAN switch modules located in the back of the Hitachi Compute Blade 2500 chassis. All 1 GbE management network connections can be consolidated using these 1G/10Gb LAN switch modules, especially the LAN connection to the Compute Blade 2500 chassis management modules. Logical partitioning manager management network links must be aggregated using active-backup mode. The IP address of LPAR manager and the chassis management modules (SVP) have to be in the same subnet. For more information on VLAN configuration for the LPAR management path, see the Hitachi Compute Blade 2500 Series Logical Partitioning Manager User's Guide (MK-99CB2500006-00). The Hitachi Compute Blade 2500 chassis has two management modules to secure fault tolerance with the following:
Manage the power supply of each module and monitor the status of the system unit.
Support the management functionality of the network within the system unit for server blades and various modules.
Connect between a management module and an external network through a management LAN module
Storage Architecture There are many factors impacting the drive size and storage configuration of Hitachi Unified Compute Platform for the SAP HANA Platform solutions including I/O and capacity requirements. With fully dedicated LPARs running on these systems, the SAS drive RAID groups that are used for the HANA data and log volumes are dedicated to each LPAR and must not be shared between the LPARs. This increases the storage requirements for Hitachi Unified Compute Platform for the SAP HANA Platform with logical partitioning (LPAR) solution compared to the standard scale-up solution that is used as a basis for this solution.
13
14 RAID and LUN Configuration This reference architecture uses slightly different RAID configurations on Hitachi Virtual Storage Platform G600 depending on the size of the server.
One RAID-6 (6D+2P) group using 8 × 600 GB SAS 10k RPM drives for hosting operating system and HANA shared volumes. On systems with a memory size of 3 TB or more, one RAID-6 (14D+2P) group using 16 × 600 GB SAS 10k RPM drives is used to meet the combined size requirements for the operating system and /hana/shared directory of all LPARs. For each LPAR, two or three dedicated RAID groups for HANA data and log volumes are created depending on the LPAR memory size requirements. 1 × 600 GB SAS 10k RPM spare drive per 32 SAS drives.
For each LPAR on this solution the following LUNs are created:
One 100 GB OS LUN to host the LPAR guest operating system.
One LUN to host the HANA shared volume that matches the LPAR memory size.
Four 150 GB LUNs to host the HANA log volume located on the same RAID group. A striped volume over the four LUNs is used. Four or eight LUNs to host the HANA data volume. A striped volume over these LUNs is used. See Table 11 for RAID group and LUN details.
Table 11. RAID Groups for HANA Data Volume and HANA Log Volume per LPAR
HANA data volume
LPAR memory size
Recommend Documents
No documents