Open Fabrics Enterprise Distribution (OFED) Version 3.5-2 Release Notes December 2013 =============================================================================== Table of Contents =============================================================================== 1. Overview, which includes: - OFED Distribution Rev 3.5-2 Contents - Supported Platforms and Operating Systems - Supported HCA and RNIC Adapter Cards and Firmware Versions - Tested Switch Platforms - Third party Test Packages - OFED sources 2. Change log 3. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of OpenFabrics Enterprise Distribution (OFED) release 3.5-2. The OFED software package is composed of several software modules, and is intended for use on a computer cluster constructed as an InfiniBand Fabric, an iWARP Network or a RoCE Fabric. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. 1.1 OFED 3.5-2 Contents ----------------------- The OFED package contains the following components: - OpenFabrics core and ULPs: - IB HCA drivers (mthca, mlx4, qib, ehca) - iWARP RNIC driver (cxgb3, cxgb4, nes) - RoCE drivers (mlx4) - ib core - Upper Layer Protocols: IPoIB, SRP Initiator and target, iSER Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA*. - OpenFabrics utilities: - OpenSM (OSM): InfiniBand Subnet Manager - Diagnostic tools - Performance tests - Extra packages: - infinipath-psm: Performance-Scaled Messaging API, an accelerated interface to Intel(R) HCAs - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) - Documentation 1.2 Supported Platforms and Operating Systems --------------------------------------------- o CPU architectures: - x86_64 - x86 - ppc64 o Linux Operating Systems: - RedHat EL6.2 2.6.32-220.el6 - RedHat EL6.3 2.6.32-279.el6 - RedHat EL6.4 2.6.32-358.el6 - SLES11 SP2 3.0.13-0.27-default - SLES11 SP3 3.0.76-0.9.1 - kernel.org 3.5 * * Minimal QA for these versions. 1.3 HCAs and RNICs Supported ---------------------------- This release supports IB HCAs by IBM, Intel and Mellanox Technologies, iWARP RNICs by Chelsio Communications and Intel and RoCE adapters by IBM and Mellanox. InfiniBand Adapters o IBM HCAs: - GX Dual-port SDR 4x IB HCA - GX Dual-port SDR 12x IB HCA - GX Dual-port DDR 4x IB HCA - GX Dual-port DDR 12x IB HCA o Intel (formerly QLogic) HCAs: - Intel(R) True Scale DDR PCIe x8 and x16 HCAs - Intel(R) True Scale QDR PCIe x8 Gen2 HCAs o Mellanox Technologies HCAs (SDR, DDR and QDR Modes are Supported): - ConnectX(R) and ConnectX EN (fw-25408 Rev 2.9.1000) - ConnectX-2 (fw-ConnectX2 Rev 2.9.1000) o Mellanox Technologies HCAs (FDR and FDR10 Modes are Supported): - ConnectX-3 (fw-ConnectX3 Rev 2.30.3000) For official firmware versions please see: http://www.mellanox.com/content/pages.php?pg=firmware_download iWARP Adapters o Chelsio RNICs: - S310/S320 10GbE Storage Accelerators - R310/R320 10GbE iWARP Adapters - T4 Based Adapters with the exception of the T420-SO-CR o Intel RNICs: - NE020 10Gb iWARP Adapter RoCE Adapters o IBM - IBM Flex System EN4132 2-port 10 GbE RoCE - IBM EL27 PCIe LP 2-Port 10GbE RoCE SFP+ adapter - IBM EC28 PCIe 2-Port 10GbE RoCE SFP+ adapter o Mellanox - ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1200) - ConnectX-3 EN (fw-ConnectX3 Rev 2.11.0500) 1.4 Switches Supported ---------------------- This release was tested with switches and gateways provided by the following companies: InfiniBand Switches o Flextronics - F-X430044 o Intel (formerly QLogic) - 12200 o Mellanox - IS-5030 - SX6025 - SX6036 iWARP Switches o Fujitsu - XG2000C 10Gb Ethernet Switch RoCE Switches o Arista o BLADE Network Technologies (BNT) o Mellanox - SX1036 - SX1024 - SX1016 1.5 Third Party Packages ------------------------ The following third party packages have been tested with OFED 3.5-2: - Open MPI - 1.6.5 - Intel MPI 4.1 1.6 OFED Sources ---------------- All sources are located under git://git.openfabrics.org/ Linux: ------ URL: git://git.openfabrics.org/compat-rdma/linux-3.5.git Branch: for-ofed-3.5 - Linux kernel sub-tree that includes files relevant for the OFED project only. Based on v3.5. Used to shorten git clone time. Note: the regular Linux git tree can be used as well. compat: ------- URL: git://git.openfabrics.org/compat-rdma/compat.git Branch: ofed - Based on compat project (https://github.com/mcgrof/compat). The compat module provides functionality introduced in newer kernels to older kernels through a set of header files and exported symbols. See https://github.com/mcgrof/compat/wiki for details. - Used to replace kernel_addons in the previous OFED kernel tree. compat-rdma: ------------ URL: git://git.openfabrics.org/compat-rdma/compat-rdma.git Branch: master User level Sources are downloaded from http://www.openfabrics.org/downloads/ as written in the BUILD_ID The kernel sources are based on Linux 3.5 mainline kernel. Its patches are included in the OFED sources directory. For details see HOWTO.build_ofed. The list of maintainers is available under: http://www.openfabrics.org/downloads/MAINTAINERS =============================================================================== 2. Change log =============================================================================== OFED-3.5-2 RC3 Main Changes from OFED 3.5-2 RC2 ------------------------------------------------------------------------------- 1. Updated packages: - librdmacm-1.0.17.1 - perftest-2.0-0.63.g5bb5fe6.tar.gz OFED-3.5-2 Main Changes from OFED 3.5-1 and OFED 3.5 ------------------------------------------------------------------------------- 1. Add support for the OSes - RedHat EL6.4 - SLES11 SP3 2. Removed packages - compat-dapl 3. Updated packages: - dapl-2.0.39 - ibacm-1.0.8 - infiniband-diags-1.6.2 - infinipath-psm-3.1-4 - libcxgb4-1.3.1 - libibmad-1.3.10 - libibverbs-1.1.7 - libmlx4-1.0.5 - libmthca-1.0.6 - librdmacm-1.0.17.1 - mstflint-3.0-0.6 - opensm-3.3.16 - perftest-2.0-0.58 New Patches 1. 0001-mlx4_core-Roll-back-round-robin-bitmap-allocation-co.patch See Bug 2452 =============================================================================== 3. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various components of the OFED 3.5-2 release. 01. When upgrading from an earlier OFED version, the installation script does not stop the earlier OFED version prior to uninstalling it. Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to OFED 3.5 or reboot the server after OFED installation. 02. Memory registration by the user is limited according to administrator setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for system configuration. 03. Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. 04. The qib driver is supported only on 64-bit platforms. 05. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that is that these utilities support devices of type Ethernet only. 06. In case uninstall is failing, check the error log and remove the remaining RPMs manually using 'rpm -e '. 07. On SLES11.x, set allow_unsupported_modules parameter to 1 in file: /etc/modprobe.d/unsupported-modules. Without this the modules will not load. 08. RDS is supported only on kernel.org 3.5 and not on any distro OS. 09. RHEL 6.5 is not supported in 3.5-2 but will be supported in the next OFED version which is OFED 3.12 10. Mellanox ConnectIB HCAs are not supported in OFED 3.5-2. See OFED 3.12 11. There are known issues with iWARP using Chelsio adapters which will be resolved in OFED 3.12. See Bugs 2402, 2441 and 2445 12. There is an issue with dapl which will be resolved in the next release dapl 2.0.40 which will be included in OFED 3.12 - see Bug 2448 13. There is an issue with PSM when compiling with gcc >= 4.6. This will be resolved in OFED 3.12 - see Bug 2430 14. There is an issue with NFSoRDMA when running on Power PC. This will be resolved in OFED 3.12 - see Bug 2449 15. Newer SLES or RHEL patch kernels that are updated to the original kernel that is included in the original SLES or RHEL distributions may not work with OFED-3.5-2. The list of kernels that are supported are listed in section 1.2. 16. The OFA OFED uninstall program only uninstalls packages installed by previous versions of OFED. OFED will not silently uninstall RPMs that are not a part of it. The user should be aware of the packages that should be uninstalled and do this by himself. For example: OFED will not uninstall RDMA components installed by the user when installing an OS from a Distro. 17. The existing ibutils package (released 7/28/11) does not support FDR, FDR 10 or EDR speeds when using ibdiagnet. Note: See the release notes of each component for additional issues.