DFX-MGR provides infrastructure to abstract configuration and hardware resource management for dynamic deployment of Xilinx based accelerator across different platforms.
DFX-MGR is merged in Petalinux 2021.1 onwards and the recipe is called
dfx-mgr. The recipe is not enabled by default and user is expected to enable
it. As of today, DFX-MGR can be used for dynamic loading/unloading of
accelerators to PL (Programmable Logic). The functionality is
tested for loading/unloading of Flat shell (i.e. shell which does not have any
reconfigurable partitions) and DFX shell (i.e. shell which contains static and
dynamic region). As you can see in the diagram below DFX-MGR can load a 3RP
design and a 2RP design with the corresponding accelerators dynamically without
having to reboot the board.
Once you compile Petalinux by enabling the dfx-mgr recipe, you should have
dfx-mgrd and dfx-mgr-client in /usr/bin of rootfs and libdfx-mgr.so in
/usr/lib. The config file daemon.conf can be found in /etc/dfx-mgrd/.
Config file is mandatory, refer the files section for details of it. A default
daemon.conf will be copied by the recipe and the user is expected to update as
required.
DFX-MGR recognizes the designs under /lib/firmware/xilinx on device
filesystem, location can be updated in daemon.conf. Designs could be
downloaded using dnf or any other package manager or manually copied to the
previously mentioned location. Each folder under /lib/firmware/xilinx will be
treated as a base shell design wherein the folder is expected to have a
shell.json file. Base design folder can then have sub-folder for each of the
accelerators. Each accelerator is expected to have a accel.json. Have a look at
below folder structure for more understanding and the details of json config
files.
The expected directory structure under /lib/firmware/xilinx which contains a
3x1 PL shell design, and a flat shell design. 3x1 base_design shell
has an accelerator called FFT which contains three different partial bitstreams
for three slots of base shell.
base_design needs to have base shell bitstream and shell.json.
DFX-MGR expects '_slot#' as subfolders for each of the accelerators for DFX designs.
Place the bitstream corresponding to that slot in the respective subfolder along
with accel.json file.
It is not mandatory to have all the partial bitstreams for each slot, but DFX-MGR will fail to load an accelerator to the slot if no partial design is found.
$ ls /lib/firmware/xilinx/base_design
base_Shell.pdi base.dtbo shell.json FFT
$ ls /lib/firmware/xilinx/base_design/FFT/
FFT_slot0 FFT_slot1 FFT_slot2
$ ls /lib/firmware/xilinx/base_design/FFT/FFT_slot0/
partial.pdi partial.dtbo accel.json
$ ls /lib/firmware/xilinx/base_design/FFT/FFT_slot1/
partial.pdi partial.dtbo accel.json
$ ls /lib/firmware/xilinx/base_design/FFT/FFT_slot2/
partial_slot2.pdi partial.dtbo accel.json
$ ls /lib/firmware/xilinx/flat_shell/
Flat_shell.bit.bin flat_shell.dtbo shell.json
As shown above, base_design has first three slots (0,1,2) corresponding to PL slots from the FFT subfolders. When DFX-MGR successfully loads FFT accel to one of the slots, -listPackage output would show active slot as 0,1 or 2.
DFX-MGR is started on Linux bootup and reads the config file
/etc/dfx-mgrd/daemon.conf from device for any config settings. Any change to
daemon.conf will need a restart of the /usr/bin/dfx-mgrd on target.
If you want to add another location for packages on the filesystem, append absolute path to the location in "firmware_location" and restart the daemon. The limitations around directory structure as explained above still apply as for "/lib/firmware/xilinx".
$ cat /etc/dfx-mgrd/daemon.conf
{
"default_accel":"/etc/dfx-mgrd/default_firmware", //Optional: echo the
//package-name to default_firmware for any
// accelerator to be loaded on start of daemon
"firmware_location": ["/lib/firmware/xilinx"] //Required:Package directories
// monitored by DFX-MGR
}
shell.json describes the base shell configuration information. Optional fields can be skipped if not desired.
One of the below type should be used for shell_type as per your design.
- XRT_FLAT: dfx-mgr will program the PL and update /etc/vart.conf on target with the path to the active xclbin on success.
- PL_FLAT: dfx-mgr will program the PL bitstream and treat the design as static.
- PL_DFX: dfx-mgr will treat the design as DFX with number of slots as mentioned in json.
$ cat shell.json
{
"shell_type" : "PL_DFX",// Required: valid values are XRT_FLAT/PL_FLAT/PL_DFX
"num_pl_slots": 3, //Required: Number of pl slots in your base shell design
"num_aie_slots":1, //Required: Number of aie slots in your base shell design
"load_base_design": "no" //Optional : If the shell was loaded during boot
//and you want to skip loading base shell when loading the
// accelerators to the slot.
"device_name" : "a0010000.SIHA_Manager", //optional: IP name
"reg_base" : "", //Optional: IP device base address
"reg_size" : "", //Optional
"clock_device_name" : "a0000000.clk_wiz", //optional
"clock_reg_base" : "",//optional
"clock_reg_size" : "", //optional
"isolation_slots": [
// optional, specify the register offsets and the corresponding values to be
// written for each slot to bring it out of isolation, expects the length
//of array equal to number of slots
{ "offset": ["0x4000","0x4004","0x4008","0x400c"],//Address offset for first slot
"values": ["0x01","0x01", "0x0b", "0x02"] }, //Value to be written to above regs
{ "offset": ["0x5000","0x5004","0x5008","0x500c"],
"values": ["0x01","0x01", "0x0b", "0x02"]},
{ "offset": ["0x6000","0x6004","0x6008","0x600c"],
"values": ["0x01","0x01","0x0b", "0x02"] }]
}
Note: The AIE support in dfx-mgr is preliminary and for a future production enablement capability. KRIA devices do not have AIE.
accel.json describes the accelerator configuration. Optional fields can be skipped if not desired. Flat shell designs are not required to have accel.json since they do not have reconfigurable partition.
- SIHA_PL_DFX: Use this option for PL accelerators build with SIHA manager, this enables some extra steps required to bring the slots out of isolation in addition to programming the bitstream.
- XRT_PL_DFX: Use this option for XRT based PL accelerator.
$ cat accel.json
{
"accel_type": "", // Required: supported types are SIHA_PL_DFX / XRT_PL_DFX
"accel_devices":[ // Optional: list of IP devices corresponding to this slot design
{ "dev_name": "20100000000.accel",
"reg_base":"",
"reg_size":"" }],
"sharedMemoryConfig": {
"sizeInKB": "",
"sharedMemType" : ""},
"dataMoverConfig": { // Optional: skip this if application handles its own dma
"dma_dev_name":"a4000000.dma",
"dma_driver":"vfio-platform",
"dma_reg_base":"",
"dma_reg_size":"",
"iommu_group":"0",
"Bus": "platform",
"HWType": "mcdma",
"max_buf_size":"8388608",
"dataMoverCacheCoherent": "Yes",
"dataMoverVirtualAddress": "Yes",
"dataMoverChnls":[
{"chnl_id": 0,
"chnl_dir":"ACAPD_DMA_DEV_W" },
{ "chnl_id": 0,
"chnl_dir": "ACAPD_DMA_DEV_R" }]
},
"AccelHandshakeType": "",
"fallbackBehaviour": "software" // Optional: If hw accelerator fails to load,
//DFX-MGR will try software fallback
}
DFX-MGR depends on external libraries/frameworks such as libdfx, XRT, inotify, etc. The recommended way to compile this repo is using yocto where the required dependency are taken care of in the recipe.
If not using yocto then dependent libraries will need to be provided to cmake using appropriate -DCMAKE_LIBRARY_PATH.
To compile using yocto in 2021.1 onwards, do bitbake dfx-mgr.
You would need to provide dependency libraries using -DCMAKE_LIBRARY_PATH for cmake.
There is generic cmake toolchian file for generic Linux which is in
cmake/platforms/cross-linux-gcc.cmake
Set the path where you have all the dependent libraries in build.sh.
$ cd dfx-mgr
$ mkdir build
$ cd build
$ ../build.sh
After build successfully completes, libdfx-mgr.so can be found
under build/usr/local/lib and binary under build/usr/local/bin.
The dfx-mgrd daemon should mostly be running on linux startup. Assuming it is running, you can use below commands from command line to load/unload accelerators.
- Command to list the packages present on target filesystem under
/lib/firmware/xilinx.
An alternative command line utility which some users might be familiar with can
also be used xmutil listapps.
$ dfx-mgr-client -listPackage
Accelerator Accel_type Base Pid Base_type #slots(PL+AIE) Active_slot
flat_shell PL_FLAT flat_shell id_ok PL_FLAT (0+0) -1
AIE_accel XRT_AIE_DFX base_design no_id PL_DFX (3+1) 3,
FFT SIHA_PL_DFX base_design id_err PL_DFX (3+1) 0,1
In the above output, AIE_accel is currently programmed to AIE and hence the slot shows as 3. FFT is programmed to first two slots out of three slots of PL. flat_shell is not programmed and show -1. Flat shell design do not have a dynamic reconfigurable partition and hence #slots shows as 0. Vivado's UID, PID information for tracking parent-to-child relationships is in the "Pid" column:
- "id_ok" When PID and the base UID are present and match as expected
- "id_err" When PID and the base UID are present but do not match.
- "no_id" When either PID or UID are not present.
Here is an example of 2-partition designs (see: kria-dfx-hw, kria-apps-firmware) from KR260 board with Ubuntu 22.04:
$ tree /lib/firmware/xilinx
/lib/firmware/xilinx
|-- k26-starter-kits
| |-- k26_starter_kits.bit.bin
| |-- k26_starter_kits.dtbo
| `-- shell.json
|-- k26_2rp_1409
| |-- AES128
| | |-- AES128_slot0
| | | |-- accel.json
| | | |-- opendfx_shell_i_RP_0_AES128_inst_0_partial.bit.bin
| | | |-- opendfx_shell_i_RP_0_AES128_inst_0_partial.bit.bin_i.dtbo
| | | `-- opendfx_shell_i_RP_0_AES128_inst_0_partial.bit.bin_i.dtsi
| | `-- AES128_slot1
| | |-- accel.json
| | |-- opendfx_shell_i_RP_1_AES128_inst_1_partial.bit.bin
| | |-- opendfx_shell_i_RP_1_AES128_inst_1_partial.bit.bin_i.dtbo
| | `-- opendfx_shell_i_RP_1_AES128_inst_1_partial.bit.bin_i.dtsi
| |-- AES192
| | |-- AES192_slot0
| | | |-- accel.json
| | | |-- opendfx_shell_i_RP_0_AES192_inst_0_partial.bit.bin
| | | |-- opendfx_shell_i_RP_0_AES192_inst_0_partial.bit.bin_i.dtbo
| | | `-- opendfx_shell_i_RP_0_AES192_inst_0_partial.bit.bin_i.dtsi
| | `-- AES192_slot1
| | |-- accel.json
| | |-- opendfx_shell_i_RP_1_AES192_inst_1_partial.bit.bin
| | |-- opendfx_shell_i_RP_1_AES192_inst_1_partial.bit.bin_i.dtbo
| | `-- opendfx_shell_i_RP_1_AES192_inst_1_partial.bit.bin_i.dtsi
| |-- DPU
| | |-- DPU_slot0
| | | |-- accel.json
| | | |-- opendfx_shell_i_RP_0_DPU_512_inst_0_partial.bit.bin
| | | |-- opendfx_shell_i_RP_0_DPU_512_inst_0_partial.bit.bin_i.dtbo
| | | `-- opendfx_shell_i_RP_0_DPU_512_inst_0_partial.bit.bin_i.dtsi
| | `-- DPU_slot1
| | |-- accel.json
| | |-- opendfx_shell_i_RP_1_DPU_512_inst_1_partial.bit.bin
| | |-- opendfx_shell_i_RP_1_DPU_512_inst_1_partial.bit.bin_i.dtbo
| | `-- opendfx_shell_i_RP_1_DPU_512_inst_1_partial.bit.bin_i.dtsi
| |-- FFT
| | |-- FFT_slot0
| | | |-- accel.json
| | | |-- opendfx_shell_i_RP_0_FFT_4channel_inst_0_partial.bit.bin
| | | |-- opendfx_shell_i_RP_0_FFT_4channel_inst_0_partial.bit.bin_i.dtbo
| | | `-- opendfx_shell_i_RP_0_FFT_4channel_inst_0_partial.bit.bin_i.dtsi
| | `-- FFT_slot1
| | |-- accel.json
| | |-- opendfx_shell_i_RP_1_FFT_4channel_inst_1_partial.bit.bin
| | |-- opendfx_shell_i_RP_1_FFT_4channel_inst_1_partial.bit.bin_i.dtbo
| | `-- opendfx_shell_i_RP_1_FFT_4channel_inst_1_partial.bit.bin_i.dtsi
| |-- FIR
| | |-- FIR_slot0
| | | |-- accel.json
| | | |-- opendfx_shell_i_RP_0_FIR_compiler_inst_0_partial.bit.bin
| | | |-- opendfx_shell_i_RP_0_FIR_compiler_inst_0_partial.bit.bin_i.dtbo
| | | `-- opendfx_shell_i_RP_0_FIR_compiler_inst_0_partial.bit.bin_i.dtsi
| | `-- FIR_slot1
| | |-- accel.json
| | |-- opendfx_shell_i_RP_1_FIR_compiler_inst_1_partial.bit.bin
| | |-- opendfx_shell_i_RP_1_FIR_compiler_inst_1_partial.bit.bin_i.dtbo
| | `-- opendfx_shell_i_RP_1_FIR_compiler_inst_1_partial.bit.bin_i.dtsi
| |-- PP_PIPELINE
| | |-- PP_PIPELINE_slot0
| | | |-- accel.json
| | | |-- opendfx_shell_i_RP_0_pp_pipeline_inst_0_partial.bit.bin
| | | |-- opendfx_shell_i_RP_0_pp_pipeline_inst_0_partial.bit.bin_i.dtbo
| | | `-- opendfx_shell_i_RP_0_pp_pipeline_inst_0_partial.bit.bin_i.dtsi
| | `-- PP_PIPELINE_slot1
| | |-- accel.json
| | |-- opendfx_shell_i_RP_1_pp_pipeline_inst_1_partial.bit.bin
| | |-- opendfx_shell_i_RP_1_pp_pipeline_inst_1_partial.bit.bin_i.dtbo
| | `-- opendfx_shell_i_RP_1_pp_pipeline_inst_1_partial.bit.bin_i.dtsi
| |-- opendfx_shell_wrapper.bit.bin
| |-- pl.dtbo
| |-- pl.dtsi
| `-- shell.json
`-- kr260-tsn-rs485pmod
|-- kr260-tsn-rs485pmod.bin
|-- kr260-tsn-rs485pmod.dtbo
`-- shell.json
$ sudo xmutil listapps
Accelerator Accel_type Base Pid Base_type #slots(PL+AIE) Active_slot
k26-starter-kits XRT_FLAT k26-starter-kits id_ok XRT_FLAT (0+0) 0,
kr260-tsn-rs485pmod XRT_FLAT kr260-tsn-rs485pmod id_ok XRT_FLAT (0+0) -1
FFT SIHA_PL_DFX k26_2rp_1409 no_id PL_DFX (2+0) -1
PP_PIPELINE SIHA_PL_DFX k26_2rp_1409 no_id PL_DFX (2+0) -1
FIR SIHA_PL_DFX k26_2rp_1409 no_id PL_DFX (2+0) -1
DPU SIHA_PL_DFX k26_2rp_1409 no_id PL_DFX (2+0) -1
AES128 SIHA_PL_DFX k26_2rp_1409 id_err PL_DFX (2+0) -1
AES192 SIHA_PL_DFX k26_2rp_1409 id_ok PL_DFX (2+0) -1
- Command to load one of the above listed accelerator. For dfx designs, base shell would be loaded first if not already loaded and then accelerator will be loaded to one of the free slots. If the shell is already loaded then only the accelerator will be loaded to one of free slots. If the device tree overlay (.dtbo) file contains external-fpga-config string the dfx-mgrd will use DFX_EXTERNAL_CONFIG_EN instead of the default DFX_NORMAL_EN flag when calling libdfx fetch function.
Equivalent xmutil command is xmutil loadapp FFT.
$ dfx-mgr-client -load FFT
- Command to remove accelerator from the slot. If there is not accel in the mentioned slot, this command will do nothing.
Equivalent xmutil command is xmutil unloadapp 1.
$ dfx-mgr-client -remove 1
Users can write applications to interact with daemon. Refer to example source
code in repo/example/sys/linux/load_accel.c for a simple example how to load
an accelerator. The applications, including dfx-mgr-client connect to the daemon
via /var/run/dfx-mgrd.socket file. This allows a client running in a docker
container to connect to the dfx_mgrd running outside the container.
-
DFX-MGR uses i-notify for firmware file updates and i-notify doesn't work with network filesystem. Hence it is recommended to NOT boot linux over NFS for correct functionality of DFX-MGR daemon.
-
DFX-MGR package names i.e. firmware folder names are limited to 64 character currently and absolute path lengths are limited to 512 char. Hence avoid creating long filenames.
-
I/O nodes don't support zero copy.