The Windows OpenFabrics (WinOF) release package contains the following:
OpenFabrics InfiniBand core drivers and Upper Level Protocols (ULPs):
-
HCA
Drivers -
Mellanox
InfiniHost & ConnectX low level drivers.
-
Infiniband Core components: IBAL,
Winverbs, Winmad, OFED verbs and rdma_cm.
-
Upper Layer Protocols:
IPoIB, WSD, ND, VNIC, SRP Initiator and DAT/DAPL
OpenFabrics utilities:
Documentation
-
User's manual
-
Release Notes
<Return-to-Top>
CPU architectures
-
x64 (x86_64, amd64)
-
x86
-
IA64
Operating Systems
Supported HCAs (Host Channel Adapters)
Mellanox - all InfiniBand HCA products are supported.
Mellanox Firmware (FW) versions and update tools
Current HCA firmware version can be viewed from the 'vstat' command.
Supported Switches
-
QLogic
-
SilverStorm -
update your firmware for correct embedded Subnet Manager operation.
-
Voltaire
-
Flextronics
-
Due to lack of hardware
accessibility,
other vendor switches have not been tested.
<Return-to-Top>
User mode tools and diagnostics are installed in '%SystemDrive%\Program
Files\WinOF'. Although device driver modules initially reside in
'%SystemDrive%\Program Files\WinOF\Drivers', when installed the Windows device
installer copies driver files to %SystemRoot%\system32 & %SystemRoot%\SysWOW64'.
DAT & DAPL runtime libraries are installed into %SystemRoot% in order to be
accessible to user mode applications.
WinOF user mode executables are designed to be run from
a Command Prompt window; see 'Start->Program Files-> Windows OpenFabrics-> Command Prompt'.
'%SystemDrive%\Program Files\WinOF' is appended to the system wide search
path environment variable 'PATH'; new command windows created after the installation will have the updated
%PATH% environment variable, hence will be able to access WinOF executables.
The ‘default’ installation installs ‘released/free’ (not
checked/debug) versions of drivers and executables. Debug/Checked versions are
available as an install feature.
A single instance of a subnet manager, opensm.exe, must be
running on a fabric connected node in order for the Infiniband fabric to be
configured and useful; either Windows or Linux OFED opensm work well.
It is recommended that OpenSM be run as a Windows service. A typical
WinOF install will install OpenSM as a Windows service which
is disabled; after an installation, choose your openSM node and start the openSM
service.
See the WinOF Manual for details on opensm as a service.
By selecting the 'OpenSM_Service_Enabled' install feature, a local OpenSM subnet
management service will be automatically started.
Note, the opensm.exe process must continue to run in order
to maintain Infiniband fabric configuration. Should the opensm.exe process die,
restart the service if not automatic or re-run opensm.exe in order to continue
correct fabric operation.
For more subnet management information, consult the Windows
OpenFabrics manual.
<Return-to-Top>
Summary
- Double-click the .msi installer file, Except for
Vista.
- For Vista installs, from an privileged administrator command window, execute 'msiexec /I
WinOF_2-1_wlh_xxx.msi'.
Requirements
Install HCA hardware prior to installing the Windows OpenFabrics Release
package. Upon reboot, cancel any attempts to install new device hardware from
the "Found New Hardware Wizard".
If you have previously installed openib-windows, WinOF (Windows OpenFabrics)
packages or vendor supplied Infiniband packages, uninstall and reboot prior to
installing this version of Windows OpenFabrics (WinOF).
For an existing Windows OpenFabrics (WinOF) uninstall
Start-> Programs-> Windows OpenFabrics-> Uninstall WinOF
-or-
Control Panel -> Add or Remove Programs-> Windows
OpenFabrics->Remove.
openib-windows (IB stack before WinOF 1.0 release) uninstall: (not the normal
uninstall case)
- Disable any Local Area Connections which are bound to the
IPoIB network adapter.
- Uninstall the IPoIB driver - My
computer->Manage->Devices->Network Adapters->IPoIB*
- Stop the openSM subnet manager if it is running on the
local node.
- Uninstall the Infiniband HCA device; don't forget the
System Device-> InfiniBand Fabric
REBOOT
Upon system restart, cancel the 'Found New Hardware Wizard' attempts to install
drivers for newly discovered PCI device (HCA).
'My Computer->Manage->Device Manager' should display
'Other Devices->?PCI device' which is your Infiniband HCA device.
Install
Vista installation only; open a privileged Administrator command
window and type
start/wait msiexec /I WinOF_2-1_wlh_xxx.msi
For all other Windows variants, double-clicking the installer file (.msi)
works correctly to start a WinOF installation.
From a login session with administrator privileges, File Explorer view, double-click the WinOF_M-m{-p}_OSE_arch.msi file to begin the Windows OpenFabrics
installation.
Where 'M-n{-p}' indicates Major release number, Minor release number, and
optional point-release number.
OSE - Operating System Environment:
- wlh - (Windows LongHorn) Windows Server 2008, HPC or Vista
- wnet - Windows Server 2003 or Windows XP 64-bit version
- wxp - Windows XP x86/32-bit version
'_arch_' will be one of
- _x64_ for Intel EMT64 systems or amd64; see
x86-64
- _x86_ for x86
compatible systems
- _ia64_ for Intel IA64 systems.
Follow the on-screen instructions.
Answer 'yes' or 'Continue' to any notifier windows which reference non-WHQL'ed
driver installation.
These non-WHQL'ed driver install questions can not be disabled
via 'My Computer->Properties->Hardware->Driver Signing->Ignore'.
Bottom-line - The digitally signed WinOF drivers in this package are not WHQL'ed
per se, although they are
built from driver source which has been or is in process of being WHQL'ed by
hardware vendors who participate in the open source WinOF development process.
**** WARNING ****
Should any 'Welcome to the Found New Hardware Wizard' windows popup after the
WinOF install has started, just ignore (window will disappear in a few
seconds) or 'Cancel' the errant popup window.
For Windows Server 2003 (SP1) you can safely ignore the errant FNHW popup
window; for Windows XP (SP2) the errant FNHW popups need to be 'cancelled'
in order for the WinOF install to proceed. The report back from Microsoft on these 'Found New Hardware Wizard' windows
popping up, is they are due to the fact the WinOF drivers are not WHQL'ed
and hence are not trusted by Windows; MS claims this is a 'feature'.
An artifact of cancelling an errant 'Found New Hardware' popup window may
induce an errant taskbar Information notifier/bubble indicating there may
have been problems installing your hardware drivers;
IGNORE/cancel this information bubble, likely the device installation
proceeded without error. Check the Device Manager views for the
Infiniband HCA and IPoIB Network Adapters for initial verification of
correct installation.
You do need to answer 'Yes' or 'Continue' to those notifier windows which
reference non-WHQL driver installation. If you have previously
set My Computer->Properties->Hardware->Driver Signing->Ignore, the FNHW
windows likely will occur anyway.
Should the install appear to hang, look around for occluded notifier windows
waiting for a response which
are covered by existing windows.
Install type: Custom only
- InfiniBand Core:
- HCA driver - Mellanox: InfiniHost or ConnectX
Windows PNP will select the correct HCA driver.
- IB core stack and libraries - includes winverbs,
winmad and OFED verbs library.
- Release Notes, Manual
- Subnet Management service [OpenSM] (installed & disabled), Performance &
OFED Diagnostic tools
- Optional:
IPoIB - Internet Protocols over
InfiniBand
Winsock Direct (Win2008/3 only)
Network Direct (MS supported only for Server 2008 HPC).
VNIC - Virtual Ethernet device over Infiniband
SRP - SCSI over RDMA Protocol
DAT/DAPL
- RDMA device independent, user-mode Direct Access Transport & Direct Access
Provider
- DAT v1.1 runtime libraries.
- DAT v1.1 application build environment
(header files & link files).
- DAT/DAPL v2.0 runtime libraries
- DAT v2.0 application build environment
OpenSM_service_started - InfiniBand Subnet Management
enabled and started as a Windows Service.
By default OpenSM is installed as a disabled Windows Service.
Checked versions of driver files.
- The 'default' installation includes IB core, IPoIB, WSD (Win2K3 only),
DAT 1.1 runtime libraries, OpenSM service (disabled).
<Return-to-Top>
WinOF driver installation uses the Device Installation
Frameworks for Applications (DIFxApp) and the Windows Driver Store.
The Microsoft Plug-n-Play (PNP) subsystem
controls which Infiniband HCA (Host Channel Adapter) device driver is loaded
from the Windows Driver Store. HCA driver selection is based on PNP device
ID probing, thus InfiniHost, ConnectX or both drivers can be loaded
by the PNP subsystem.
Using DIFxApp and PNP device probing results in the WinOF installation not
requiring a specific HCA model [ConnectX or InfiniHost] feature selection; HCA
vendor is the feature selection criteria.
An unattended WinOF install will install the following 'default' options
- Infiniband Core modules - Mellanox HCA driver
(see HCA driver selection note), IB core stack, libraries, utilities and
documentation.
- IPoIB - Internet Protocols over InfiniBand
- WSD - (Win2K8/3 only, not installed on XP) Win Sock Direct
- ND - NetworkDirect [started automatically (via ndinstall.exe) only for Server
2008/HPC and Vista].
- DAT & DAPL (v1.1 & v2.0) runtime libraries + dapltest.exe
(dt-svr & dt-cli).
HCA selection for all Windows variants
(Server 2008, HPC & Vista)
Default HCA driver selection is driven by the
Windows PNP subsystem; no user selection required or supported.
Should WSD not be a desirable unattended install option (Win2008/3 only),
currently you would install unattended then execute the
command 'installsp -r'
on each node to remove WSD.
To perform a silent unattended installation for Server
2003/XP, invoke the following command
from a command window.
start/wait msiexec.exe /i WinOF_2-1_wnet_x64.msi /qn /quiet /log
%TEMP%\WOF-install.log
'/log dev:path\logFilename.log' is optional.
msiexec.exe /? for all options.
Examples:
Windows XP - Mellanox HCA Hardware - same as above with .msi
name change (wnet --> wxp).
- ConnectX HCA
start/wait msiexec.exe /i WinOF_2-1_wxp_x86.msi /qb /quiet HCA=cx
Server 2008 / Vista - any Mellanox HCA Hardware
- No need to specify HCA type as PNP (Plug-n-Play)
figures out the correct HCA to install.
start/wait msiexec.exe /i WinOF_2-1_wlh_x64.msi /qb /quiet
- If you desire an unattended WinOF install with a
progress bar, from a console window
start/wait msiexec.exe /i WinOF_2-1_wlh_x64.msi /passive
Non-HCA Install Features added to above msiexec command
line examples:
- Start a Subnet Manager running on the local system:
add 'OSMS=1'
Default install + OpenSM Started on the local
system
- Server 2003, Mellanox HCA - start/wait msiexec.exe /i WinOF_2-1_wnet_x64.msi
/passive OSMS=1
<Return-to-Top>
Device drivers in WinOF 2.0 and later releases are Digitally
signed by the 3rd party Software Publisher 'OpenFabrics Alliance', although they are not WHQL'ed (Windows Quality Hardware Labs) certified.
The WinOF source code is used by hardware vendors to achieve WHQL
certification.
The lack of WHQL driver certification places WinOF drivers
in the 'unknown 3rd party SW publishers' category.
When installing 'Unknown 3rd party SW publishers' drivers, Windows prompts on
the install console for instructions as what to do (Install, Trust or not)?
This prompting results in a failed remote node install when performing a
first-time unattended WinOF install.
By preloading the remote node's Trusted 3rd party Software Publisher Certificate
Store, the unattended install will proceed without prompting, thus allowing the WinOF install to complete
successfully.
Be aware: if a node is reimaged (same or different template applied) the
established trust of the OpenFabric Alliance is destroyed, thus the OFA certificate
needs to be reinstalled.
How to install WinOF drivers and components in the HPC
Server 2008 environment.
- Add the OpenFabrics Alliance Trusted SW Publisher certificate to the head-node's Trusted Publisher Certificate store by
- Installing WinOF on the head node - start Open Subnet
Manager if needed, otherwise depend on another OpenSM on the IB fabric.
- Our recommendation is to install WinOF on the
head-node; Set head-node IPoIB interface IPv4 address after WinOF install.
- Or - Installing the OFA certificate from the
installer .msi file by
- right-clicking the .msi file, select Properties
- Select the Digital Signatures tab
- Highlight the OpenFabrics Alliance
- Details
- View Certificate
- Install certificate
- Trusted Publisher certificate store
- Once the OFA certificate is installed, it will remain
until the node is wiped clean by reprovisioning.
- If remote nodes are operational (booted &
network accessible), then run the digital certificate install script 'cert-add.bat'
to 'add' the 'Open Fabrics Alliance' as a Trusted 3rd party Software Publisher
to the remote node 's certificate stores.
- CD your administrator command window to the WinOF
folder which contains the 'cert-add.bat' script file.
- If WinOF was installed on the head-node, then 'cd /d
%ProgramFiles%\WinOF
- Otherwise unpack, not install, the WinOF files to
gain access to cert-add.bat file by executing the following commands:
- Folder %TARGETDIR% folder must not exist, msiexec
/A will create it.
- msiexec /A WinOF_2-1_wlh.msi TARGETDIR=%TEMP%\WOF
# WOF folder does not exist.
- cd /d %TEMP%\WOF\PFiles\WinOF
- Select a filesystem share folder which is
accessible from the head-node and 'all' remote nodes specified in the
cert-add command; cert-add will write a batch script to this folder
which the remote node will execute to add the OFA SW Publishing
certificate to the local store.
- Cert add command format: cert-add
share-folder list-of-remote-nodes
- For the examples, assume
\\HN\WOF is the compute node visible share name;
head-node local name is \Program Files\Microsoft HPC Pack\Data\InstallShare\WOF
- example: add OFA Trusted Publisher
certificate to remote node's local certificate store
cert-add
\\HN\WOF cn01 cn02 cn03 cn04
cn05
- example: extract OFA TP certificate to remote
node visible share, creates
\\HN\WOF\OFA_TP.cer &
\\HN\WOF\rem-cert-ADD.bat
cert-add
\\HN\WOF
- For all compute nodes: clusrun /nodes:X msiexec /I \\HN\WOF\WinOF_xxx_yyy.msi
/quiet /qn /Lv msi.log
- Node Template Provisioning: if remote nodes are not
(not yet provisioned) then do the following:
- See %ProgramFiles%\WinOF\HPC\* for provisioning
example scripts; customization likely required.
- From the head node, extract the OFA certificate to a
file in a remote node visible share;
(example: \Program Files\Microsoft HPC Pack\Data\InstallShare\WOF); see
previous cert-add extract example.
- copy the following files to (example: \Program
Files\Microsoft HPC Pack\Data\InstallShare\WOF)
- WinOF installer .msi file
- %ProgramFiles%\WinOF\HPC\OFA-cert-install.bat
- %ProgramFiles%\WinOF\HPC\WinOF-install.bat
- Verify you have the required WinOF provisioning files
in the remote node visible share:
- WinOF installer .msi file, referened in
WinOF-install.bat
- OFA-cert-install.bat
- WinOF-install.bat
- rem-cert-ADD.bat
- OFA_TP.cer
- Run the cluster manager tool to create a compute node
template; include the following steps as part of the node provisioning
template steps after the HPC pack has been installed:
- copy entire contents of (example: \Program
Files\Microsoft HPC Pack\Data\InstallShare\WOF) to remote node as
%INSTALLDRIVE%\WOF
- execute the DOS command: %INSTALLDRIVE%\WOF\OFA-cert-install.bat
- execute the DOS command: %INSTALLDRIVE%\WOF\WinOF-install.bat
- Use 'cluster manager' to provision all compute nodes
using newly created compute node template.
- Set the IPoIB interface static IPv4 address on all compute
nodes; 1st IPoIB instance will be the next 'Local Area Connection X' after
the Ethernet port(s) on the compute node.
- clusrun /nodes:X netsh interface ip set address
"Local Area Connection 3" static 10.10.4.1 255.255.255.0
- clusrun /nodes:X netsh interface ip show address
"Local Area Connection 3"
- IPoIB supports DHCP IP address assignment for those
who choose to use it. Static IP address binding allows a direct mapping from
IP address to physical node number.
- IPoIB static IPv4 address assignment points of
Interest
When installing WinOF one must consider how the Interface name 'Local Area
Connection {x}' is assigned to the IPoIB ports in order to correctly target
IP address assignment.
Assume the target compute node has two Ethernet port and your HCA has two IB
ports. After the 1st WinOF installation, or node reprovisioing
you will see the following assignments:
- Ethernet port 0 is 'Local Area Connection'
- Ethernet port 1 is 'Local Area Connection 2'
- IB port 1 is 'Local Area Connection 3'
- IB port 2 is 'Local Area Connection 4'
A few minutes after the WinOF install completes, the
HPC manager will rename the active IPoIB 'Local Area Connection 3' network
interface to the 'Application' network interface along with the active
Ethernet interface being renamed to the 'Private' network; 'Local Area
Connection' is no longer in use.
If WinOF is uninstalled and then reinstalled without node reprovisioning,
the active IPoIB network interface is installed as 'Local Area Connection'.
Within a few minutes, the active IPoIB network interface will again be
renamed the 'Application' network interface.
Summary:
1st WinOF install, an IB cable is attached to HCA port 1,
IPoIB is named 'Local Area Connection 3'.
2nd WinOF install, IPoIB is named 'Local Area Connection' as
the Ethernet interface is renamed the 'Private' network.
- At this juncture, the WinOF drivers are installed.
Windows Deployment Manager usage for HPC compute node provisioning
Microsoft has chosen WDM to provision HPC
cluster compute nodes; see
Microsoft HPC
installs for details.
In support of the new Microsoft WDM HPC cluster
provisioning process, WinOF now supports an administrative install mode which is
a method to extract WinOF files to the local filesystem without actually installing them;
thus making WinOF files accessible to WDM provisioning.
Disclaimer - WDM 'driver' provisioning ONLY works for
WHQL'ed drivers; WinOF drivers are not WHQL'ed. For WinOF installs, see
previous node template provisioning discussion.
msiexec /A WinOF_2-1_wlh_x64.msi TARGETDIR=dev:\path-to-extracted-files
# last folder in path-to-extracted-files must NOT exist.
example: msiexec /A WinOF_2-1_wlh_x64.msi TARGETDIR=%TEMP%\WOF
(note: TARGETDIR is case sensitive, WOF must not exist)
The above command creates the folder structure:
%TEMP%\WOF\PFiles\WinOF\
WinOF device driver-less installs:
Normally WinOF device driver files are installed by processing the driver '.inf'
files. To install the WinOF package by selecting install features without
installing any device drivers [HCA, IPoIB, ND, WSD, SRP or VNIC] (assumed
reliance on WDM to install device drivers) can be accomplished with the following
interactive install command:
start/wait msiexec /I WinOF_2-1_wlh_x64.msi NODRV=1
To skip driver installation for an unattended install
start/wait msiexec /I WinOF_2-1_wlh_x64.msi /qn /quiet NODRV=1
Note: when uninstalling WinOF from an installation
which was done with NODRV=1, you MUST include NODRV=1 on the uninstall
command line; otherwise WinOF drivers will be uninstalled also.
start/wait msiexec /X WinOF_2-1_wlh_x64.msi /qn /quiet NODRV=1
See Unattended
Installations for further references.
<Return-to-Top>
The Windows command 'netsh' will assist in assigning an IP address to an
IPoIB interface.
By default, an IPoIB interface is configured to use DHCP. The following
discussion deals with setting a static IP address on an IPoIB interface.
If your system has two Ethernet ports, then IPoIB interfaces (HCA port 1, 2...)
will be assigned to 'Local Area Connection 3' and 'Local Area Connection 4'.
If you system has a single Ethernet port, then IPoIB interface is 'Local Area
Connection 2'.
For Server 2008/HPC systems, the WinOF installation follows the above
discussion. In defining HPC network interfaces, the IPoIB network will
eventually be
identified as the 'Application' network '. A few minutes after a
compute node is installed, the HPC network manager will rename the installed IPoIB 'Local Area Connection X' network interface to the 'Application' network.
Display all network interface configuration information
netsh interface ip show config
netsh interface ip show address
"Local Area Connection 3"
Config a static IP address for an IPoIB interface: [Local Area Connection 3
== 1st IPoIB interface], 10.10.4.23 is the assigned IP address
with a netmask of 255.255.255.0, the gateway is 10.10.4.1 metric is 2.
netsh interface ip set address
"Local Area Connection 3" static 10.10.4.23 255.255.255.0 10.10.4.1 2
See netsh for further
reference.
<Return-to-Top>
To uninstall a WinOF package from a single node
- Start--> All Programs-> Windows OpenFabrics-> Uninstall WinOF
- Control Panel-> Add Remove Programs-> Windows OpenFabrics-> Remove
Unattended
Uninstall
From a Command Window invoke the following command; a system reboot will
occur after the uninstall.
start/wait msiexec.exe /x {3A6A276E-A636-48E8-8624-9A0DE3F2A15E}
/quiet
/forcerestart
- or -
start/wait msiexec.exe /x WinOF_wlh_2-1_x64.msi /quiet /forcerestart
# WinOF_xxxx.msi is the file used to install WinOF.
msiexec.exe /help for details.
'/log dev:path\logfilename.log' if you wish to later view the install
results.
If for some reason the Product Code {GUID} changes, the latest code can
be located via
Programs-> Windows OpenFabrics-> Uninstall WinOF<right-click>Properties
Infrequently, the 'System Device->Infiniband Fabric' may not correctly uninstall. Make sure the Infiniband Fabric device is
completely uninstalled; use device manager for checking. The effect of a
partially uninstalled Infiniband Fabric device is IPoIB load failure
upon the next WinOF install.
Uninstall Notes
- If the SRP (SCSI RDMA Protocol) driver has been previously installed,
then in order to achieve a 'clean' uninstall, the SRP target drive(s) must
be released.
- Use the following diskpart.exe commands from an administrator
privileged command window:
- c:\windows\temp> diskpart.exe
SELECT VOLUME=<Drive Letter of the SRP target to be released>
# example: SELECT VOLUME=L
OFFLINE DISK
EXIT
- For additional SRP drives, repeat SELECT & OFFLINE DISK with new
Drive Letter.
The consequences of not releasing the SRP target drive(s) is that after the
uninstall reboot there are lingering InfiniBand driver files. These driver
files remain because while the SRP target is active they have references,
thus when the uninstall attempts to delete the files the operation fails.
<Return-to-Top>
From a command window type the following quick-start
commands:
- To start the Device Manager: devmgmt.msc
- To start the Services Manager: services.msc
A Vista WinOF installation can fail due to protection failures when
installing the HCA driver. Vista HCA driver installation failure can be caused
by double-clicking the .msi installer
file to start the installation.
WinOF installs for Vista (only) require the installation be started from a
privileged Administrator command window.
Start a privileged Administrator command window (Start->Command
Prompt->Right-click->Run as Administrator) and execute:
msiexec /I
WinOF_xxx_yyy.msi
To create a LOG file of your WinOF installation, execute this command:
msiexec /I WinOF_xxx_yyy.msi /Lv %TEMP%\WOF-install.log
View %TEMP%\WOF-install.log
msiexec /? for details.
Server 2008 or Vista can also fail to install due to driver install files
left over from previously failed install attempts; see %windir%\system32\DriverStore\{Temp|FileRepository}.
Look for folders with the following in the folder name: mthca, mlx4_, netipoib,
srp, vnic. Once you have located one of these folders you need to remove it as
win2k8/Vista will use these files instead of what your attempting to install.;
removing the folders is a pain.
Right-click the folder and select Properties->Security->Advanced, select owner
tab, select Edit button, select Administrator and check the 'Replace owner on
subcontainers and objects', now click 'OK', OK again, select Edit button from
the securities tab, check allow 'Full Control' box and click OK, OK again. At
this point you should be back to the file explorer. Now you can delete the
folder!
Setupapi.log has moved in Server 2008/Vista, see '%windir%\inf\setupapi-dev.log'.
Setupapi-*.log can at times contain useful install debug info.
Server 2008/Vista/HPC all use dpinst.exe to preinstall drivers into the
driver store. Windows PNP (Plug-n-Play) later will deamand drivers when the
hardware is recognized. dpinst.exe writes it's error information to '%windir%\dpinst.log'.
Down rev firmware will result in IPoIB not installing or other anonmolus
behavior. Consult the event
viewer, system error records, look for 'mthca' or 'mlx4*' entries generated when the HCA
driver loads. One of the mthca or mlx4 entries will display the current firmware
revision.
Firmware version information and updates
are available at
www.mellanox.com/support/firmware_download.php
•
–
•If
HCA FW is older than minimal FW
Error
reported to system event log
–HCA loads as memory controller to allow FW upgrade
•If HCA
FW is equal or newer than optimal version
•Else
HCA loads OK.
–Warning will be issued to system event log
–
My computer-> Manage-> Event Viewer-> System Events; search for
mthca, mlx4 or ipoib entries.<Return-to-Top>
Incomplete Previous Uninstall
Should the previous uninstall silently fail, the next installation
can fail in the Driver Install phase with the following error:
DriverInstall - HCA Ret 2 Error 0
At this juncture, check the Device Manager and uninstall the PCI
device from 'InfiniBand Host Channel Adapters', and make sure you uninstall the system device
'InfiniBand Fabric' if it is present.
REBOOT and retry the WinOF installation again.
See
ibscan.bat and ibcleanup.bat @
http://www.openfabrics.org/downloads/WinOF/etc/.
ibscan will display WinOF files on your system.
ibcleanup will attempt to remove all WinOF files from your syste.
WARNING - ibcleanup is not valid as a WinOF uninstall tool!
<Return-to-Top>
From the Device Manager you should find the following devices:
- From a command window type 'devmgmt.msc'
- InfiniBand Host Channel Adapters -> InfiniHost or Mellanox ConnectX
- The System Devices -> InfiniBand Fabric device is no longer present in
WinOF 2.0 and following releases.
- Network Adapters -> OpenIB IPoIB Adapter (an Instance per HCA port).
Validation Test
Open a WinOF command window
- Start->Program Files->Windows OpenFabrics -> Command
Window
- run vstat.exe to view HCA configuration.
- If IPoIB is installed (Device Manager->Network Adapters), ping another node on the InfiniBand fabric 'ping a.b.c.d'
<Return-to-Top>
Limit the number of Subnet Managers on your fabric; one SM per fabric is
sufficient, although redundant Subnet Managers are supported.
- After a 'typical' WinOF install, OpenSM is installed as a disabled
Windows service named "InfiniBand Subnet Management".
To start the OpenSM service, from a command window type 'services.msc' to
start the Services manager. From the Services manager, find and select
'InfiniBand Subnet Manager' view; Click the start button and set the service properties to 'auto' in order to restart on the next
system reboot.
- Install OpenSM as an automatic/running Windows service:
a) Select the 'OpenSM_service_Started' install feature. Once
the install has completed, check that the Infiniband Subnet Manager service
is running: From a command windows type 'services.msc' to start the Services
manager. Find and open the InfiniBand Subnet
Manager view.
- Consult the OpenSM log file @ %SystemRoot%\temp\osm.log to see what
OpenSM thinks is happening.
<Return-to-Top>
Local Network Adapter ordering, which results in the IP address advertised
for the system, can be managed from
'My Network Places-> Advanced Tab-> Advanced settings'
From the Advanced settings display, the ordering
of Local Network Adapters can be
reordered after the IPoIB Local Network Adapters have been installed. Please
check for your desired adapter ordering.
<Return-to-Top>
- Add/Remove Programs entry identifies the WinOF release version.
- Start->Programs->Windows OpenFabrics->Release Notes
(2nd Line)
- Driver confirmation from the Device Manager:
Query My Computer-> Manage-> Device Manager-> Network
Adapters-> OpenFabrics IPoIB Adapter-> Properties-> Driver-> Details
The subversion (svn) revision number will be listed under
'Driver Version'. The svn revision number will match what's listed in the
release notes.
- Hold the mouse point over \Program Files\WinOF\ib_read_bw.exe to display
the file version; [WinOF-Major.Minor.not-used.svn revision number].
- WinOF 1.0 is based on openib-windows svn revision 614 @
http://openib.tc.cornell.edu/downloads/binaries/ .
- WinOF 1.0.1 is based on openib-windows svn revision 864.
- WinOF 1.1 is based on openib-windows
(svn revision 1177 branches\WOF1-1).
- WinOF 2.0 is based on (svn.1763 branches\WOF2-0)
- WinOF 2.1 is based on (svn.2476 branches\WOF2-1)
<Return-to-Top>
The QLogic VNIC
(Virtual Network Interface Card) driver in conjunction with the QLogic Ethernet
Virtual I/O Controller (EVIC) provides virtual Ethernet interfaces and transport
for Ethernet packets over Infiniband.
Users can modify NIC parameters through User Interface icon in Network
Connections:
( Properties->"Configure..." button -> "Advanced" Tab).
Parameters
available:
Vlan Id (802.1Q)
values from 0 to
4094 ( default 0, disabled )
This specifies if VLAN ID-marked packet transmission is enabled and, if so,
specifies the ID.
Priority (802.1P)
values from 0 to 7
( default 0, feature disabled)
This specifies if priority-marked packet transmission is enabled.
Payload MTU size
values from 1500
to 9500 (default 1500)
This specifies the maximum transfer unit size in 100 bytes increments.
Recv ChkSum offload
(default enabled)
This specifies if IP protocols checksum calculations for receive is offloaded.
Send ChkSum offload
(default enabled)
This specifies if IP protocols checksum calculations for send is offloaded.
Secondary Path
(default
disabled)
Enabled - If more than one IB path to IOC exist then secondary IB instance of
virtual port will be created and configured with the same parameters as primary
one. Failover from Primary to Secondary IB path is transparent for user
application sending data through associated NIC.
Disabled – only one path at a time is allowed. If more than one path to IOC
exists then failed path will be destroyed and next available path will be used
for new connection. With this scenario there is a possibility new interface
instance will be assigned different MAC address when other hosts compete for
EVIC resources.
LBFO Bundle Id
(default disabled) Enabling support for OS provided Load Balancing and Fail
Over functionality on adapter level.
If enabled group ID can be selected from predefined names.
Heartbeat interval
configures
interval for VNIC protocol heartbeat messages in milliseconds.
0 – heartbeats disabled.
Note:
To take advantage of the features supported by these options, ensure that the
Ethernet gateway is also configured appropriately. For example, if the Payload
MTU for a VNIC interface is set to 4000,
the MTU at the EVIC module must also be set at least 4000
for the setting to take effect.
<Return-to-Top>
In order for DAT/DAPL programs to execute correctly, the 'dat.dll'
file must be present in the current directory,
%SystemRoot% or in the library search path.
WinOF installation places the dat.dll and dapl.dll
files in the '%SystemRoot%'
folder (a.k.a. \Windows).
The DAT/DAPL configuration file by convention is defined as
'%SystemDrive%\DAT\dat.conf'. If the file '%SystemDrive%\DAT\dat.conf' does not exist,
a template configuration file will be installed as '%SystemDrive%\DAT\dat.conf'.
In order to preserve existing installations, the dat.conf file is not
automatically installed if '%SystemDrive%\DAT\dat.conf' exists.
A sample dat.conf file is always installed as '\Program Files\WinOF\dat.conf
'.
The default DAT configuration file specification can be overridden by use
of the environment variable DAT_OVERRIDE. Such that 'DAT_OVERRIDE=D:\MyDAT\test.conf'
would be a valid override specification.
Within the dat.conf file, the DAPL library specification can be
located as the 5th whitespace separated line argument. By default the DAPL
library file is installed as %WINDIR%\dapl.dll'.
Should you choose to relocate the DAPL library file to a folder
where whitespace appears in the full library path specification, then the full
library file specification must be contained within double-quotes. A side effect
of the double-quotes is the library specification is treated as a Windows string
which implies the '\' (backslash character) is treated as an 'escape' character.
Hence all backslashes in the library path must be duplicated when enclosed in
double-quotes (e.g., "C:\\Programs Files\\WinOF\\dapl.dll").
DAT/DAPL version 2.0 runtime
libraries are identified as dat2.dll and dapl2.dll, both in %SystemRoot%; see
manual for further details.
<Return-to-Top>
The WinOF installer does not install the SRP driver as part of a default
installation.
If the SRP feature is selected in the custom installation window, an InfiniBand
SRP Miniport driver will be installed; see the device manager view under SCSI
and RAID controllers.
The system device 'InfiniBand I/O Unit' (IOU)
device is required for correct SRP operation. The WinOF installer will
install/load the IOU driver if the SRP feature is selected. See the device
manager view System Devices --> InfiniBand I/O Unit for conformation of correct
IOU driver loading.
In order for the SRP miniport driver installation to complete, an SRP target must be
detected by a Subnet Manager running somewhere on the InfiniBand fabric; either
a local or remote Subnet Manager works.
SRP Driver Uninstall
If the SRP (SCSI RDMA Protocol) driver has been previously
installed, then in order to achieve a 'clean' uninstall, the SRP target drive(s)
must be released. Unfortunately the 'offline disk' command is only
valid for diskpart (ver 6.0.6001) which is not distributed with Windows Server
2003 or XP.
- Use the following diskpart.exe commands from an administrator privileged
command window:
-
c:\windows\temp> diskpart.exe
SELECT VOLUME=<Drive Letter of the SRP target to be released>
# example: SELECT VOLUME=L
OFFLINE DISK
EXIT
- For additional SRP drives, repeat SELECT & OFFLINE DISK with new Drive
Letter.
The consequences of not releasing the SRP target drive(s) are that after the
WinOF uninstall reboot there are lingering InfiniBand driver files. These driver
files remain because while the SRP target is active they have references, thus
when the WinOF uninstall attempts to delete the files the operation fails.
<Return-to-Top>
WinVerbs is a userspace verbs and communication management interface
optimized
for the Windows operating system. Its lower interface is designed to support
any RDMA based device, including Infiniband and iWarp. Its upper interface is
capable of providing a low latency verbs interface, plus supports Microsoft's
NetworkDirect Interface, DAPL, and OFED libibverbs interfaces. It consists of
a userspace library and a kernel filter driver.
The WinVerbs and WinMAD drivers load as upper filter drivers for the Infiniband HCA
device.
(Open source iWarp drivers for Windows are not yet available.) A corresponding
WinVerbs.dll and libibmad.dll
userspace libraries install as part of the Winverbs driver installation package.
Additionally, a Windows port of the OFED libibverbs library, several test
programs and OFED InfiniBand diagnostic utilities are also included.
As of WinOF 2.1, Winverbs components are now integral
components of a default WinOF installation.
Although WinVerbs, WinMAD drivers, OFED libraries and utilities are install
selectable features, they are automatically included as part of the default
WinOF install.
<Return-to-Top>
-
DAPL2 providers Socket-CM
or DAPL2 rdma-CM
Configuring > 1 rank
per node can cause connection establishment problems; not an issue with the
IBAL provider.
-
IPoIB partition
multicast
support requires IGMP (Internet Gateway Management Protocol) v2 Windows default is v3.
Partition
multicast on ipoib will only work if the machine is configured to use
IGMP V2 (and not V3 which is the default).
To
configure your machine to use IGMP v2 please do the following:
Server
2003 or XP:
netsh routing ip igmp install
If the netsh command is not
available, then you will need to install via the command 'servermanagercmd.exe
-install NPAS-RRAS-Services'.
netsh
routing ip igmp
install
netsh
routing ip igmp install
add interface "interface name of IPoIB
adapter" igmpprototype=igmprtrv2
F
uture WinOF releases
will support IGMP V3.
-
Sometimes an
errant "Welcome to the Found New Hardware Wizard" window pops up during
InfiniBand device installation on XP or Server 2003.
Just 'cancel' the FNHW popup window and proceed with installation.
An artifact of the 'Cancel' operation may cause a taskbar Information bubble
to appear which incorrectly claims problems with the Infiniband device
installation; Check IB device status via the Device Manager.
