Direct Access Transport Libraries - (latest release, 12/14/16, includes MIC support)

User space libraries/utilities for Direct Access Transport (DAT). DAT is a transport-independent, platform-independent Application Programming Interface that supports RDMA (remote direct memory access) devices. Directory includes packages that include libdat, libdapl, and utilities for DAT API Spec Version 2. NOTE: v1 is no longer supported and will not be included with OFED releases. MIC support has been added in dapl-2.1.0, see README.mcm for build and install details.

MIC support is provided with the new MCM provider and MPXYD service. MCM requires the Intel(R) MPSS 3.x (YOCTO) release for Linux to be installed on your system. MPSS 3.x for Linux can be downloaded from: http://software.intel.com/mic-developer

Highlights for 2.1.10 - CCL Proxy optimization for post_send, dtest/dtest_suite improvements, bug fixes, UCM log for timers

Latest Release Notes: documentation/uDAPL_release_notes.txt

Latest Packages (see ChangeLog for recent changes, see README.mcm for MIC support):

For v2.1 package install RPM packages as follow:

Summary of changes:

  • Release 2.1.10-1
  • dtest_suite: add option to pause the test.
  • dtestcm: add client retry, give server time to queue up all listens
  • dtest: Add new man pages. (dtestx dtestcm dtestsrq)
  • cma: fix open_query mode, initialize attributes
  • ucm: up level CM timer logging, increase drep time at scale
  • dtest: fix return value check on do_rdma_write_with_msg
  • dtestx: check device capabilities and do atomic tests only if supported by HW
  • common: set atomic attributes based on provider/device capabilities
  • build: dtest_suite.sh was moved to test/scripts
  • mpxyd: let TX thread sleep if no open devices are referenced
  • MCM MIX: When mmap req from MIC return with fail stat print WARN.
  • dtest_suite: remove duplicate dtest_suite.sh
  • dtest: Enable -D option (data check) to work with scif provider
  • dtest_suite: fix typo in user_string var
  • mcm: remove logs from post send speed path
  • mcm proxy: push WR from MIC to host with scif mmap memory instead of scif_send.
  • dtest: the default size in pingpong test is set to 1 byte regardless to user input. Keep the user input if one provided.
  • dtest: Clean 4 printf from the middle of performance test. That may add time fot the test to be completed and reduce our performance.
  • Release 2.1.9-2
  • mcm: get GID from the tp struct which was initialized in open device
  • mcm: hca_handle in MFO mode is N/A and access it caused a seg fault
  • mpxyd: modify_qp uses wrong md->dev_attr with MTU changes
  • rpmbuild: fix specfile, don't overwrite build options
  • openib: new provider specific attribute - port GID
  • Release 2.1.9-1
  • dtestcm: exchange provider IA address info via sockets
  • ucm: increase default REQ/RTU timers on scaling threshold
  • mpxyd: m_req_event assert during large io streams, HST to MIC
  • mcm: HST->MXS IO streams can overrun MPXYD proxy-in WR queue
  • mpxyd: cleanup warnings in MIC proxy code
  • openib: cleanup warnings in openib providers
  • common: cleanup warnings in common code base
  • dapltest: cleanup warnings, unused variables, etc
  • scm: backward compatibility issue with MTU negotiation
  • mcm: fix mtu interop issues when MIC and HOST differ
  • dtest: clean up warnings, keep variables and functions local
  • Release 2.1.8
  • mpxyd: fix segfault in proxy_out debug logging
  • mpxyd: fix debug memory buffer log function
  • dtest: -D option is not valid with scif providers
  • dtest/dapltest: add new automated test suite for HOST to MIC testing
  • openib: update attributes correctly for iWARP transports
  • openib_common: set providers mtu to active_mtu instead of 2048
  • mpxyd: set affinity default to 2 for best performance
  • mcm: cleanup unused variable in dapls_ib_mr_register
  • dtest: enhancement to test, -D option for data check
  • mcm: add support for Intel Omni-Path driver (hfi) via mic MFO mode
  • mpxyd: fix ordering issues with the CCL Proxy receive side forwarding mechanism
  • dtest: report results only if one of the pingpong tests are run
  • mpxyd: with abnormal CM termination a CM object can be referenced after QP destroy
  • mpxyd: proxy out WR resources exhausted with MFO mode endpoints
  • release note update for CCL Proxy and Platform BIOS recommendations
  • dtestx: add dat_ib_open_query only option with -q
  • scm: CONN_PENDING: SOCKOPT ERR Connection refused ->
  • Release 2.1.7
  • dtest: add -a -i options, all data sizes, incremental size
  • dapl: Fix segfault while freeing qp
  • mpxyd: add P2P inline support for data size <= 96 bytes
  • dtest: change rdma_write_ping_pong so client is always last receiver
  • ucm: add DAPL_NETWORK_PROCESS_NUM option for total ranks
  • ucm: fca create group incorrectly using IB addr instead of socket address.
  • ucm: fca_comm_destroy called with NULL
  • dtest: add -W option for rdma write pinpong, similiar to ib_write_lat
  • docs: update release notes for collective build
  • mpxyd: reduce log level for rcv message flush
  • dapltest: dapltest with no argument not working in ppc64 arch
  • Release 2.1.6
  • ucm: add cluster size environments to adjust CM timers
  • mpxyd: proxy_in data transfers can improperly start before RTU received
  • mcm: forward open/query for MFO devices in query only mode
  • mpxyd: byte swap incorrect on WRC wr_len
  • dtest: remove ERR message from flush QP function
  • dapltest: Quit command with "-n port" number will core dump
  • config: update dat.conf for MFO qib devices, 2 adapters/ports
  • mpxyd: add MFO support on proxy side
  • mcm: add MFO proxy commands, device, and CM support
  • mcm: add MFO support to openib_common code base
  • mcm: add full offload (MFO) mode to provider to support qib on MIC
  • dtest: pre-allocated buffer too small for RMR, DTO ops timeout
  • mpxyd: fix buffer initialization when no-inline support is active
  • mpxyd: reduce log level on qp_flush to CM level
  • mcm: intra-node proxy missing LID setup on rejects
  • mcm: add intra-node support via ibscif device and mcm provider
  • mcm: provide MIC address info with proxy device open
  • mcm: add device info to non-debug log
  • common: add DAPL_DTO_TYPE_EXTENSION_IMM for rdma_write_imm DTO type checking
  • mpxyd: fix up some of the PI logging
  • dtest: modify rdma_write_with_msg to support uni-direction streaming
  • mcm,mpxyd: fix dreq processing to defer QP flush when proxy WRs still pending
  • mpxyd: update byte_len and comp_cnt for PO to remote HST communications
  • mcm: bug fixes for non-inline devices
  • mcm: return CM_rej with CM_req_in errors
  • mpxyd,mcm: RDMA write with immed data not signaled on request side
  • mcm: add WC opcode and wc_flags in debug log message
  • mpxyd: set options bug fix for mcm_ib_inline
  • Release 2.1.5
  • update release notes, readme files
  • dat.conf: update comments regarding versions
  • dtest: add logging of provider private data size with -v
  • scm: remove use of msg.resv field for process id logging
  • cma: report correct CM req private data size on query
  • mpxyd: memset ib_wr structure before post_send on WC and WR requests
  • mcm: add HST side provider support for device without inline data capability
  • ucm: CM changes for UD extended port space and indexer
  • ucm: add device support for new port space hash table
  • ucm: allocate/free AH hash table for UD endpoint types
  • ucm: check for AH caching when destroying via UD extension
  • ucm: optimizations for large scale UD communication management
  • mpxyd: use wr opcode instead of wc opcode to support logging on error cases
  • mcm: HST->MXS mode, using RDMA_WRITE_WITH_IMM, fails with dtest -w
  • dapl: aarch64 support for linux
  • dapltest: add scripts to dist, set default device to IPoIB
  • mpxyd: add wc_flags to proxy work completions
  • Release 2.1.4 (targeting OFED 3.18)
  • mpxyd: fix typo in configuration file
  • cma: RR attributes moved to common ib_cm struct
  • mpxyd: tx thread incorrectly sleeps with negative pi_rw_cnt value
  • dat.conf: add entries for True Scale qib device
  • mpxyd: add support for devices without inline data support
  • ucm: long disconnect times with many-to-one applications
  • openib: add inline data support check during device open
  • cleanup ib/cm attribute management across openib providers
  • dapltest: fix -Werror=format-security issue with printf
  • Release 2.1.3 (targeting OFED 3.18)
  • dapl: mpxyd service changes to support multi-thread single-core option
  • dapl: add rdma_write_imm and write only option to dtest
  • ucm: add time wait override capability for CM services
  • common: dapl_ep_free must serialize CM object destroy
  • dtestx: allow scale up to 1000 EP's
  • ucm: RTU not retransmitted in TIMEWAIT state
  • mpxyd: increase max open files for service
  • mpxyd: DTO completion ERR: status 12, op RDMA_WRITE running MPI alltoall test
  • mcm: HST->MXS mode incorrectly signals multiple fragments per WR
  • mcm: add segmentation to HST->MXS mode for improved performance
  • mpxyd: set global seg_sz to 128KB for proxy data service
  • openib: add port_num to provider named attributes
  • mcm: provide CPU family/model attribute on both host and mic sides
  • dtestx: update IB extension example test with new v2.0.9 features
  • dtest: add dtestsrq for SRQ example and provider testing
  • common: add srq support for openib verbs providers
  • openib: add IB UD cm_free/ah_free extension support in UCM provider
  • openib: add new TIMEWAIT state for CM
  • extension: add IB UD extensions to reduce provider CM and AH memory footprint
  • mpxyd/mcm: add provider specific attribute DAT_IB_PROXY_VERSION
  • mpxyd: log warning if running in COMPAT mode
  • add provider and proxy support for GUID across platform
  • common: return appropriate handles with affiliated EP and EVD async events
  • Release 2.1.2 (OFED 3.12-1)
  • mpxyd: add global routing support for proxy connections
  • mcm: only call mix_get_attr if running on MIC
  • openib: modify check for link_layer to handle unspecified
  • dapl: add support for the s390x platform
  • dtest server exchange connection info with client
  • mpxyd: 2 MICs in same numa_node will overlap CPU affinity, don't reset base
  • mcm: implement proxy mix_prov_attr function, add fields CPU model and family
  • mpxyd: tx thread may not be signaled on small segment writes
  • Release 2.1.1 (OFED 3.12-1 RC1)
  • common: add provider name to log messages
  • mpxyd: log warning message if numa_node invalid include debuginfo with build
  • build: include debuginfo with build
  • mpxyd: tx thread doesn't sleep during no pending IO state
  • mpxyd: change MIC cpu_mask to per numa node instead of adapter
  • mpxyd: set to MXS mode if device numa_node is invalid (-1)
  • mpxyd: MXS based alltoall benchmark hangs or returns post_send timeout
  • mpxyd: add IO profile capabilities to help debug alltoall stall cases
  • mpxyd: retry stalled inline post_send, init m_idx only when signaled
  • Release 2.1.0 (OFED 3.12-1, MIC support added)
  • build: add missing NEWS file
  • update autogen.sh
  • add MCM provider and MPXYD service to build
  • mpxyd: service startup script and configuration file
  • add readme for MCM provider and MPXYD service
  • update Copyright dates
  • add new MIC RDMA proxy service daemon (MPXYD)
  • add new dapl MIC provider (MCM) to support MIC RDMA proxy services
  • MCM: new MIC provider and proxy service definitions
  • cleanup build warnings
  • common: add CQ,QP,MR abstractions for new MIC provider and data proxy service
  • openib: cleanup, use inet_ntop for GIDs, remove some logs, destroy pipes on release
  • common: new dapls_evd_cqe_to_event call, cqe to event
  • common: init ring_buffer, assign hd/tl pos in range
  • allow log level changes during device open
  • ucm: fix cm rbuf setup, include grh pad on initialization
  • ucm: remove duplicate async_event code, use common async event call
  • new lightweight open_query/close_query IB extension for fast attribute query
  • dtestcm: add more detailed debug during disconnect phase
  • cma: long delays when opening cma provider with no IPoIB configured
  • common: new debug levels for low system memory, IA stats, and package info
  • build: remove library check for mverbs with --enable-fca
  • IB extension: segfault in create collective group with non-vector type IA handle"
  • build: change configure help to correctly state collective default=none

  • For more details see the following:

    uDAT/uDAPL source git tree is located at http://git.openfabrics.org/?p=~ardavis/dapl.git;a=summary