The CCL Offload (CCLO) kernel implements the ACCL primitives by orchestrating data movement between the network fabric, FPGA external memory, and FPGA compute kernels, with no host CPU involvement.
Data movement to and from the network is accomplished through custom interface blocks to the TCP/UDP network protocol offload engines,while FPGA external memory (DDR or HBM) is read and written through DataMover engines (DMA).
Orchestrating the required transfers and the interaction between various subsystems at high speed is a challenge. For this reason, the CCLO kernel consists of two subsystems: a software-programmable control plane which is extremely flexible, and a high-throughput data plane consisting of DMAs, configurable routing, and pipelined arithmetic, which is fast. Different parts of the data plane can be activated at the same time in parallel, e.g. to transmit and receive data simultaneously from the network.
Figures presents a high level overview of the CCLO data plane structure which consists of multiple functional units (FUs) - three AXI DataMover engines (
DMA0
, DMA1
,DMA2
), AXI Stream (AXIS
) interconnects (e.g. S0
, S1
), an internal arithmetic unit (A0
) and network interface logic (UD
,UP
,TP
,TD
). Data flows through all those components via 512bit wide AXIS interfaces [15], that are connected together via the central AXIS Switch.
Modules description:
-
Arithmetic
unit : CCLO uses either external or internal arithmetic units to performs sums in different way. This module would be used to optimize certain kind of operations like a gather that fuses all the data in an integer, which can be useful in certain cases (e.g. distributed training to perform stochastic gradient descent) -
UP
,TP
arePacketizer
s: collects data from the the rest of the system and broke them into smaller pieces that can be sent by the network stack in a single MTU. Those pieces are then sent via AXI Stream to the network stack. TheCTRL
module can orchestrate the start/end of procedure via ctrl AXIS interface. The module gets the total number of bytes to transfer, the destination and the tag fromCTRL
and prepends them to the outgoing stream. -
UD
,TD
areDepacketizer
s: they get data from the network stack and extracts application header from the payload, such as the source and the number of bytes in the packet. Those metadata are relayed to theCTRL
module viasts stream
. The payload are directly sent to theDMA0
orDMA2
depending on the protocol. -
The
DMA
X modules are AXI DataMovers, a Xilinx IP that enables high throughput transfer of data between the AXI4 memory-mapped and AXI4-Stream domains. The AXI DataMover provides the MM2S(read interface) and S2MM(write interface) AXI4-Stream channels that operate independently in a full-duplex like method. To be more specific: -
The AXI Switch is a Xilinx IP that allows to exchange AXI_stream packages between the modules and is controlled by the CTRL module.
The CCL_Offload relies extensively on AXI protocols (both MMAP and STREAM) and IP to exchange data. For more info on the protocol go to AXI MMAP spec, AXI STREAM spec, UG761 and UG1037.
The Figure illustrates the detailed structure of the control plane. Central to the control plane is a MicroBlaze single-core, FPGA-optimized microprocessor. The role of the Microblaze CPU in the CCLO design is to provide the re-quired flexibility to execute many different collectives utilizing a multitude of FUs in combination, which is difficult to achieve utilizing logic described in HLS or RTL.
Firmware executing on the Microblaze is able to generate commands to the various FUs - the DMAs, (de)packetizers, switch, etc. These commands can be assembled into data movement primitives, e.g. copy, transmit to network, which themselves are assembled into collectives. By implementing the control in software (compiled C code), the control logic can run at fairly high frequencies (up to 250 MHz) with reasonable resource utilization and can also be adapted and improved overtime with relative ease, without requiring re-synthesis of theCCLO kernel.
Compiling and debugging the firmware is possible utilizing Vitis and the Xilinx command-line tool XSCT, which enables all common software development features breakpoints, step-by-step execution, profiling and others.
However, the latency of firmware-directed control is higher when compared to RTL control logic. To counteract the higher latency of the MicroBlaze we employed a pipelined control plane architecture as illustrated in Figure 3 whereby most functional blocks are controlled through hardware First-In-First-Out (FIFO) memories, decoupling control issuing from the execution. Less performance-sensitive FUs (AXI Stream switches, arithmetic) are controlled through an AXI bus using register-mapped interfaces. These AXI interfaces to the FUs are utilized at most once per collective execution, to set long-duration configurations, such as the reduction function,network MTU, etc.
To be more specific, the control plane is divided in CTRL module and the exchange memory
The
CTRL
module takes the responsibility of orchestrating the data transfer between components inside the the CCLO kernel. As you can see from the Figure, a MicroBlaze
is currently employed to implement those functionalities.
The following Figure shows the main connections to and from the MicroBlaze
to other components in the design.
The CTRL module is divided in 7 sub-components:
AXI STREAM FIFOS
: that keep intermediate sts/data/cmd to be read/send to CCL_Offload modules. These are directly connected toAXIS (AXI Stream) M0-M10
andS0-12
. A thing that catches the eye is that the interrupt signal of the Microblaze is driven by a signal that originates from count of two FIFOs.- The interrupt for the MicroBlaze is raised either when:
- no data is present on
DMA0_s2mm(write)_cmd
queue or - when there is data on
DMA0_s2mm(write)_sts
queue - no data is present on
DMA2_s2mm(write)_cmd
queue or - when there is data on
DMA2_s2mm(write)_sts
queue. As we can see from the CCLO_kernel schematic the depacketizer is directly connected toDMA0
. Since we plan to use 100Gbit interface timing is critical to ensure that depacketizer output does not saturate theAXIS FIFO
s and eventually result in a packet drop. In that sense if we have no cmd written we should activate the MicroBlaze to identify the location where to put incoming data. In the same way, if theDMA0
completes an write operation we should reactivate the MicroBlaze to issue a new command to execute.
- no data is present on
- reset for all
FIFO
s is provided by aGPIO
, that strictly speaking is inside theexchange_memory
module. But the point is that this pin can be triggered by the MicroBlaze via a AXI LITE register write. - the
exchange_memory
submodule give access to theMicroBlaze
from the outside. There is a memory that can be read/write both from theMicroBlaze
and from the outside ofCTRL block
. An AXI MMAP port give physical access toexchange_memory
module. Among the other things, it contains a constant (accessible via anAXI GPIO IP
in Exchange memory module) that identifies the combination of CCLO and network stack in use.MicroBlaze S10/M10
are used to collect/send data from/toexchange_memory
. - The
AXI interconnect
empowers the communication fromM_AXI DP
to other AXI MMAP ìnterfaces (AXI switch ctrl
,depacketizer ctrl
,packetizer ctrl
andarithmetic ctrl
) - A memory for instruction and data that the MicroBlaze access via
ILMB/DLMB
which are Instruction/Data interface, Local Memory Bus from MicroBlaze reference guide. - The MicroBlaze. info about the ctrl sw in /sw/sw_apps/ccl_offload_control
For more information on how the MicroBlaze
works, take a look at sw/sw_apps/ccl_offload_control/ccl_offload_control.c
The exchange memory module is the user point of access to the CTRL module.
- hosts_sts comes from
MicroBlaze M10
; - host_ctrl receives configuration data from python driver (via
s_axi_control
,AXI crossbar0
) to request an operation to the CCLO kernel as specified earlier; - Moreover the user can write/read via
s_axi_control
->AXI crossbar0
theBRAM
in this module; - Furthermore MicroBlaze can write/read the
BRAM
in this module, but it has to useS00_AXI
interface passing throughcrossbar1
andcrossbar0
. As previous section showed, the Microblaze can access theexchange memory
module via an AXI MMAP interface.There are mainly 4 reasons for theMicroBlaze
to write/reading something fromS00_AXI
:- verifying that hw_id corresponds to what is exepected by ctrl_software
- to read some extra parameters in the
BRAM
- to start/end the
AXI_FIFO
reset procedure. i.e. setting a 0 onencore_aresetn
- to start/end the reset procedure by disabling/enabling the
AXI register
. This is done by enforcing theinit_done
signal to 1. As a consequence theAXI register
stops to reset (note:AXI register
reset is active low).
- The
AXI GPIO
is a Xilinx IP that offers a simple way to read values (e.g. thehw_id
constant) and write values (e.g. theencore_resetn
signal or theinit_done
signal).
The Makefile automates the CCLO kernel build, producing a Vitis-compatible object file (.xo). The Makefile takes the following variables:
PLATFORM
indicates the target Alveo shell.DEBUG
controls Chipscope insertion. There are three debugging levels:
You are ready to go.
- Source Vitis 2020.2
- run make to build
ccl_offload.xo
underccl_offload_ex/exports