DAPL-ND - DAPL NetworkDirect Stand Alone installer v1.4.5 [06-02-2016] DAPL-ND installers for Windows DAPL over NetworkDirect v2 can be downloaded from http://www.openfabrics.org/downloads/Windows/DAPL-ND/latest 1.4.5 Increase dapls_ib_disconnect() WaitForSingle() timeout value to cover the case of 93 nodes (2322 MPI ranks). 1.4.4 remove developer object creation / destruction debug print statements. 1.4.3: no new feauters for 1.4.3, bugfix only. When Nd_Listen() fails with ND_INVALID_PARAMETER, return DAT_CONN_QUAL_IN_USE as a listen() for the specified port already exists. 1.4.2: new features: 1) A single installer now services: Windows Server 2008 (R2) Windows Server 2012 (R2) 2) %SystemDrive%\DAT\dat.conf is the new 'default' dat.conf path. 3) dat.conf: Environment variables are allowed in the DAPL provider path\specification - must be a quoted string! example: "%SystemRoot%\\system32\\dapl-ND.dll" Note: when the DAPL provider path is quoted, backslash '\' must be escaped, hence "%SystemRoot%\\system32\\dapl-ND.dll". 4) dat.conf: HCA name formats: "ND0 1" selects the first (0-th) IPv4 address in the NDlist.exe output. "ND0_v4 1" same as above. "ND0_v6 1" Future IPv6 address support - currently not supported. "ND1 1" Selects the 2nd IPv4 address in the NDlist.exe output. "ND0_10.10.4.0/24 1" Selects the first IPv4 address from NDlist.exe output which matches 10.10.4.0 after ANDing with the netmask(24 bits) 255.255.255.0. 5) dapl2test.exe default device is 'ND0'. right-click file to install (administrator), otherwise msiexec /I DAPL-ND_x64_v1.4.2.msi msiexec /X DAPL-ND_x64_v1.4.2.msi # to uninstall or Control Panel --> # Programs and Features to remove DAPL The installation of DAPL enables Intel MPI to utilize RDMA devices via the NetworkDirect v2 API; see %PROGRAMFILES%\DAPL\imb.bat discussion below. Intel MPI for Windows 5.0 has been validated using DAPL-ND over Mellanox WinOF/VPI. uDAT / uDAPL files (2.0.41) built using the Mellanox 4.95 release SDK. Install from account 'administrator' as it creates %windir%\system32\ Dat2.dll Dapl-ND.dll Dapl2test.exe NDList.exe %PROGRAMFILES%\DAPL\ %SystemDrive%\DAT\dat.conf Use Control Panel --> Programs and Features to remove DAPL Within %PROGRAMFILES%\DAPL\ you will find dt-svr.bat Dapltest server (server can support multiple clients) dt-cli.bat Dapltest client imb.bat Intel MPI Benchmarks driver script. After DAPL installation, verify ND devices are in sync w.r.t. ordering. For all hosts to be used in IMPI runs, execute NDlist.exe and verify ND0 or the NDx instance referenced in 'C:\DAT\dat.conf' is the IPoIB IPv4 address you are expecting. If not, disable the IPoIB instance which is not ACTIVE (aka associated port has no cable). If the above fails, then disable IPv6 on the IPoIB interfaces as IPv6 addressing can confuse DAPL2-ND.dll IPv4 address lookup; a known issue which is being addressed. Assuming Intel MPI and DAPL are installed. Running Dapltests – see manual.htm for further details. cd “%ProgramFiles%”\DAPL\ dt-svr.bat ND0 # ND0 == 1st NDv2 configured device, see NDlist, normally the 1st ACTIVE IPoIB instance. At the dapltest client host (can be the same although not very interesting performance wise). dt-cli.bat ND0 IPoIB-IPv4-addr-where-dt-svr-is-running dt-command # use cmd == ‘X’ valid cmds are displayed. Example: dt-cli ND0 10.10.4.101 X Use ‘dt-cli ND0 10.10.4.101 stop’ to shutdown the dapltest server. A good place to start is ‘dt-cli ND0 a.b.c.d conn’, then ‘dt-cli ND0 a.b.c.d transmete’ Running Intel MPI benchmark tests: (See start--> Intel SW-->MPI library reference for details cd “%ProgramFiles%”\DAPL imb.bat to display cmd-line format. Add MPI hosts to the file hostfile.txt , format is a single IPoIB hostname or IPv4 address per line. imb.bat num-nodes imb-test ranks-per-node MPI-device(rank-access-method:dat.conf-dev-name) imb 4 PingPong 2 rdma:ND0 # 4 nodes, PingPong test, 2 ranks-per-node, rdma access:ND0, # rdma uses IB for all rank communications. imb 4 all 2 rdssm:ND0 # 4 nodes ‘all’ IMB tests (takes about 15 minutes for 4 nodes), # rdssm:ND0, rdssm uses sysV shared memory intranode and IB RDMA internode.