uDAPL MCM Provider and MPXYD Daemon (CCL-proxy) dapl-2.1.x May 2015 MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support MIC to MIC, HOST to HOST, and MIC to HOST environments. The mcm client will NOT use MPXYD when running on the host. It requires a new MPXYD daemon service when clients are running on a MIC KNC adapter. This package installs all the host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion is currently included with MPSS and all libraries and services will be installed by default. Current release package: dapl-2.1.5.tar.gz * Sample host build from source package (ofed must installed) ./autogen.sh ./configure \ --enable-mcm \ --prefix=/usr \ --libdir=/usr/lib64 \ --sysconfdir=/etc make sudo make install * Sample /home/user1 MIC build from source package for MPSS 3.x (MPSS must be installed) source /opt/mpss/3.x/environment-setup-k1om-mpss-linux ./autogen.sh ./configure \ --enable-mcm --host=x86_64-k1om-linux \ --prefix=/home/user1/dapl-mic-install \ CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc \ CFLAGS="-I/opt/mpss/3.x/sysroots/k1om-mpss-linux/usr/include LDFLAGS="-L/opt/mpss/3.x/sysroots/k1om-mpss-linux/usr/lib64" make sudo make install * Sample /home/user1 MIC build from source package for MPSS 2.x (MPSS must be installed) export PATH=$PATH:/usr/linux-k1om-4.7/bin ./autogen.sh ./configure \ --enable-mcm \ --prefix=/home/user1/dapl-mic-install \ --libdir=/opt/intel/mic/ofed/card/usr/lib64 \ --sysconfdir=/opt/intel/mic/ofed/card/etc \ --host=x86_64-k1om-linux \ CFLAGS="-I/opt/intel/mic/ofed/card/usr/include" \ LDFLAGS="-L/opt/intel/mic/ofed/card/usr/lib64" make sudo make install * Cluster deployment (1) Build once on the head or on one of the nodes as described in the above steps. (2) Replicate these files on all the nodes: /etc/dat.conf /etc/mpxyd.conf /usr/sbin/mpxyd /usr/lib64/libdaplomcm.so.2 /opt/intel/mic/ofed/card/etc/dat.conf /opt/intel/mic/ofed/card/usr/lib64/libdaplomcm.so.2 /opt/intel/mic/ofed/card/ofed.filelist (3) Unload and then restart MPSS on all the nodes. * Start the proxy daemon on all the nodes (host only) sudo /usr/sbin/mpxyd * Use the MCM provider with Intel MPI 4.1.3 or greater for best out of box experiences. (1) Recommended settings: export I_MPI_MIC=1 export I_MPI_DEBUG=2 export I_MPI_FALLBACK=0 export I_MPI_MIC_DAPL_DIRECT_COPY_THRESHOLD=8192,262144 With these settings on MIC, messages less than 8192 bytes will be sent via pre-registered buffers; messages between 8192 and 262144 bytes will be sent via the Rendezvous protocol throught the first provider; and larger messages will be sent via the Rendezvous protocol through the second provider. Fine tune these two sizes for the best performance. * Setup for non-root CCL Proxy testing, MPXYD running as process with different service port from your /home directory: Using build instructions above, change prefix as follow and "make install": Build MIC: --prefix=/home/username/ccl-proxy-mic Build host: --prefix=/home/username/ccl-proxy-host edit /home/username/ccl-proxy-host/etc/mpxyd.conf and change the following entries: log_file /var/log/mpxyd.log to log_file /tmp/username/mpxyd.log lock_file /var/log/mpxyd.pid to lock_file /tmp/username/mpxyd.log scif_port_id 68 to scif_port_id 1068 start the mpxyd process on each node ssh node1-hostname /home/username/ccl-proxy-host/sbin/mpxyd -P -O /home/username/ccl-proxy-host/etc/mpxyd.conf& Note: override default port id using following environment variable: export DAPL_MCM_PORT_ID=1068 * Notes (1) Modify "/etc/mpxyd.conf" to change the settings for the proxy. Especially, try different values of "buffer_segment_size" for performance tuning. Use a smaller value for "buffer_pool_mb" to reduce the memory foorprint of mpxyd. Use a larger value for "scif_listen_qlen" to run more MPI ranks per card. Also modify mcm_affinity_base to the desired CPU_id to insure socket to adapter affinity. Best performance when HCA, MIC, and CPU are on same socket. Default settings are on CPU socket 0. (2) By default, only writes originated from MIC is proxied. However, it is also possible to proxy host-originated writes (e.g. for debugging purpose). To do this, set the environment variable "DAPL_MCM_ALWAYS_PROXY=1". This variable applies to the provider, not the proxy.