
The libfabric architecture is based on object-oriented design concepts.
At a high-level,
individual fabric services are associated with a set of interfaces.
For example, RMA services are accessible using a set of well-defined functions.
These interface sets are then associated with objects exposed by libfabric.
The relationship between an object and an interface set is roughly similar to
that between an object-oriented class and its member functions, although
the actual implementation differs for performance and scalability reasons.

An object is configured based on the results of the discovery services.
Providers dynamically associate objects with interface sets based on the
modes supported by the provider
and the capabilities requested by the application.
This enables optimized code paths between the application and fabric hardware
that is based on the expected usage model,
with the usage conveyed during initialization.

Figure~\ref{fig:object-model} shows a high-level view of the
parent-child relationships between libfabric objects,
with details described below.

\begin{figure}[ht]
\centering
%%\includegraphics[width=\columnwidth]{libfabric-objmod.png}
\includegraphics[scale=0.52]{libfabric-objmod.png}
\caption{Object Model of libfabric}
\label{fig:object-model}
\end{figure}

\emph{(i) Fabric}: A fabric represents a collection of hardware and software resources that access a single physical or virtual network. For example, a fabric may be a single network subnet. All network ports on a system that can communicate with each other through the fabric belong to the same fabric domain. A fabric can share network addresses with multiple providers.

A fabric not only includes local and remote NICs, but corresponding software, switches, routers, and any necessary fabric or subnet management components.  Fabrics are identified by a named string and encapsulate network based events and topology data.

\emph{(ii) Domain}: A domain represents a logical connection into a fabric. For example, a domain may map to a physical or virtual NIC. A domain defines the boundary within which fabric resources may be associated. Each domain belongs to a single fabric.

The properties of a domain describe how associated resources will be used.  Domain attributes include information about the application's threading model, and how fabric resources may be distributed among threads.  It also defines interactions that occur between endpoints, completion queues and counters, and address vectors.  The intent is for an application to convey enough data that the provider can select an optimized implementation tailored to its needs.

\emph{(iii) Passive Endpoint}: Passive endpoints are used by
connection-oriented protocols to listen for incoming connection requests.
Conceptually, they are equivalent to listening sockets.
Passive endpoints often map to software constructs,
and may span multiple domains.

\emph{(iv) Active Endpoint}: An active endpoint (or, simply, endpoint) represents a communication portal, and is conceptually similar to a socket.  All data transfer operations are initiated on endpoints.

Endpoints are usually associated with a transmit context and/or a receive context.  Transmit and receive contexts are often implemented using hardware queues that are mapped directly into the process's address space, which enables bypassing the operating system kernel for data transfers, though OFI does not require this implementation.  Data transfer requests are converted by the underlying provider into commands that are inserted into transmit and/or receive contexts.

A more advanced usage model of endpoints allows for resource sharing.  Because transmit and receive contexts may be associated with limited hardware resources, libfabric defines mechanisms for sharing contexts among multiple endpoints. Shared contexts allow an application or resource manager to prioritize where resources are allocated and how shared hardware resources should be used.

In contrast with shared contexts, the final endpoint model is known as a scalable endpoint.
Scalable endpoints allow a single endpoint to take advantage of multiple underlying hardware resources by having multiple transmit and/or receive contexts.
An application can direct data transfers to use a specific context,
or the provider can select which context to use.  Each context may be associated with its own completion queue.  Scalable contexts allow applications to separate resources to avoid thread synchronization or data ordering restrictions, without increasing the amount of memory needed for addressing.

\emph{(v) Event Queue}: An event queue (EQ) is used to collect and report the completion of asynchronous operations and
events. It handles control events that are not directly associated with data transfer operations, such as connection requests and asynchronous errors.

Applications directly or indirectly control the types of events that are received on an event queue.  An event queue supports an interface that is flexible enough to handle almost any type of event in an efficient manner.

\emph{(vi) Completion Queue}: A completion queue (CQ) is a high-performance queue used
to report the completion of data transfer operations.
An endpoint is associated with one or more completion queues.
An endpoint may direct completed transmit and receive operations
to separate completion queues, or the same queue.
The format of events read from a completion queue is determined by
an application.
This enables compact data structures with minimal writes to memory.
Additionally, the CQ interfaces are optimized around reporting completions
for operations that completed successfully,
with error completions handled ``out of band''.
This allows error events to report additional data
without incurring additional overhead
that would be unnecessary in the common case of a
successful transfer.

\emph{(vii) Completion Counter}: A completion counter is a lightweight alternative
to a completion queue,
in that its use simply increments a counter rather than placing
an entry into a queue.
Similar to CQs, an endpoint is associated with one or more counters.
However, counters provide finer granularity in the types of completions that they can track.  Completion counters and queues may be used together.

\emph{(viii) Wait Set}: A wait set provides a single underlying wait object to be signaled whenever a specified condition occurs on an event queue, completion queue, or counter belonging to the set. Wait sets enable optimized methods for applications to use in place of more generic native operating system constructs.  Applications can request that a specific type of wait object be used, such as a file descriptor, or allow the provider to select an optimal object.  The latter grants flexibility in current or future underlying implementations.

\emph{(ix) Poll Set}: Although libfabric is architected to support providers that offload data transfers directly into hardware, it supports providers that use the host CPU to progress data transfer operations.  In the latter case, a thread running on the host CPU may be needed to execute software needed to complete a data transfer. Libfabric defines a manual progress model where the application agrees to use its threads for this purpose, which avoids the need for additional threads being allocated by the underlying software libraries.  A poll set is defined to optimize for this situation.  A poll set allows an application to group together multiple objects such that progress can be driven across all associated data transfers.
 
\emph{(x) Memory Region}: A memory region describes an application's local memory buffers. In order for a fabric provider to access application memory during certain types of data transfer operations, such as RMA and atomic operations, the application must first grant the appropriate permissions to the fabric provider by constructing a memory region.

Libfabric defines multiple modes for creating memory regions.  It supports a method that aligns well with existing InfiniBand\textsuperscript{TM} and iWARP\textsuperscript{TM} hardware, but also allows for mechanisms needed to scale to millions of parallel peers.

\emph{(xi) Address Vector}: An address vector is used by connectionless endpoints to map higher-level addresses which may be more natural for an application to use, such as IP addresses, into fabric-specific addresses. This allows providers to reduce the amount of memory required to maintain large address look-up tables, and to eliminate expensive address resolution and look-up methods during
data transfer operations.



The libfabric object model that has been defined is extensible. Additional objects can easily be introduced, or new interfaces to an existing object can be added.  However, object definitions and interfaces are designed specifically to promote software scaling and low-latency, where needed.  Effort went into ensuring that objects provided the correct level of abstraction in order to avoid inefficiencies in either the application or the provider

