-
Notifications
You must be signed in to change notification settings - Fork 200
ARM Tech Preview
DRAFT IN PROGRESS
This area houses the latest info/tips regarding the ARM tech preview included in the OpenHPC 1.2 release from November 2016. The packages are targeted at 64-bit server platforms, it is being released as a Tech Preview initially as there are some known issues around provisioning and a subset of development platforms on the target distributions versions and hardware platforms.
The information on this page is intended to suppliment the AARCH64 OpenHPC Installation Guides for SLES-12-SP1 and CentOS-7.2, in particular the provisioning steps outlined with Warewulf (most of the steps in sections 4.3 thru 4.9), which are not directly usable without additional modification to the PXE boot configuration.
- GSL: a small subset of tests performed with the GSL library failed precision releated tests. This is currently attributed to the fact that the tests included in GSL are tuned for x86 which does 80-bit extended precision.
- PAPI: hardware counter availability may not be available depending on the underlying ARM platform
- MPI: available hardware for this Tech Preview release was ethernet only. The available MPI stacks reflect this test environment.
- mpiP: appears to have trouble collecting certain information in certain scenarios causing it to fail integration tests
- Hypre and SuperLU-dist: the libraries build, but when linking test applications unresolved symbols remain
- Nagios and Ganglia: don't work on SLES-12-SP1 due to missing PHP5 dependencies
- Warewulf: the ARM Standard Base Boot Requirements and Standard Base System Architecture requires specific UEFI support duirng the boot process which doesn't seem to be compatible with the way warewulf currently autoprovisions worker nodes. There is a work-around, but it requires some manual intervention during installation and deployment of the nodes.
SLES 12 SP1 AARCH64 support was a Beta release which has now been supersceded by the commercial release of 64-bit ARM support in SLES 12 SP2. However, in order to match the x86_64 versions of the packages we built and tested against this beta for the 1.2 release of OpenHPC. If you were not already part of the SLES 12 SP1 beta, you may have trouble getting access to the necessary base OS installation. We are in the process of validating that these packages may be used on SLES 12 SP2 and hope to move to the commercially avaialble release in the next version of OpenHPC.
64-bit ARM support has been made available as an altarch variation of the CentOS 7.2 release. There is some information here and ISO images available here. You will also need to acquire access to certain EL7 packages in order to complete a full installation of OpenHPC.
On the three tested system configurations we found disparity with respect to the availability of performance counters. Since all ARM 64-bit hardware has performance counters and PAPI has support for ARM64 performance counter hardware, this is likely a problem with either the kernel or the device tree passed to the kernel from the firmware. You can determine whether or not you have access on your platform's configuration by running papi_avail(1):
# module load papi
# papi_avail
...
Number Hardware Counters : 0 (Xgene Mustang)
-- or --
Number Hardware Counters : 6 (Softiron Seattle)
-- or --
Number Hardware Counters : 0 (Cavium ThunderX)
The other thing to note is that while the ARM Architecture specifies a core set of performance counters, many more may be available depending on the microarchitecture. We are in the process of working with the various silicon partners to make sure support for these is available in PAPI. As we discover work arounds enabling additional counter support for various platforms we will include them here.
Lustre client support has been available for a while on both the 32-bit and 64-bit ARM platforms. However, since different ARM platforms require different kernels than the standard ones found in the SLES-12-SP1 and CentOS-7.2 distributions we couldn't easily build a lustre that would work for specific platform configurations. As better ARM support is added to commercial distributions (like the support now in SLES-12-SP2), this will become easier. For now, you'll have to build your own kernel if you want lustre support.
While both of these libraries build, we discovered anomolies during testing that we have not yet been able to resolve. Once we have a workaround we will include it here.
mvapich packages compiled but we did not have infiniband hardware support in our testbeds at the time of these release to validate the packages and/or any instructions relating to them. We are working with platform vendors to acquire sufficient hardware to test this in the future. If you have working infiniband support on your ARM platform you may be able to get existing libraries to work on your own.
SLES 12 SP1 did not contain PHP5 packages which were required for Nagios and Ganglia, CentOS-7.2 works fine. You may be able to work around this by building PHP5 yourself, or finding a compatible PHP5 package that can be installed on SLES 12 SP1.
Network booting is a bit different on ARM platforms
Cavium 2K CDK CDK
- Cavium ThunderX uArchitecture, armv8
- ThunderX (version a1)
- 2 socket, 48-core, 128GB of Memory
- Linux Version 4.4.21-64-default
- Tested against SLES-12-SP1 install
- EFI v2.40 by Cavium Thunder cn88xx EFI ThunderX-Firmware-Release-1.22.9-15-gcc66a09 Aug 4 2016 16:55:45
APM X-C1 Server Development Platform (Mustang)
- APM X-Gene uArchitecture, armv8
- APM X-Gene-1
- 1 socket, 8 core, 16GB of memory
- Linux Version 4.4.11-reference.135.aarch64
- Tested against CentOS-7.2 install
- EFI v2.40 by X-Gene Mustang Board EFI Nov 24 2015 13:22:41
- ARM Cortex-A57 uArchitecture, armv8
- AMD Seattle Processor (Rev.B0)
- 1 socket, 8 core, 16GB of Memory
- Linux version 4.4.21-64-default
- Tested against SLES-12-SP1 install
- EFI v2.40 by American Megatrends
Please email any questions on AARCH64 provisioning or configuration to the OpenHPC ARM Architecture Maintainer (Currently Eric Van Hensbergen [email protected]) and I will endeavor to answer them and include the response for others to benefit from.