ResNet50 compiled success but the bitfile does not behave as expected

[laochonlam]

Hi FINN community,

I am using Vivado+Vitis 2020.1 with U250 and had success to compile the resnet50 model from finn-example to bitfile. However, I don't think it is a correct bitfile because when I run throughput_test, something doesn't make sense to me and some errors are generated after 2~3 times execution.

The nw_metrics.txt of ResNet50 running throughput_test

{'runtime[ms]': 0.28514862060546875, 'throughput[images/s]': 3506.943143812709, 'DRAM_in_bandwidth[Mb/s]': 527.8931375518395, 'DRAM_out_bandwidth[Mb/s]': 0.03506943143812709, 'DRAM_extw_idma0_bandwidth[Mb/s]': 7182.219558528428, 'fclk[mhz]': 162, 'batch_size': 1, 'fold_input[ms]': 0.009298324584960938, 'pack_input[ms]': 0.011205673217773438, 'copy_input_data_to_device[ms]': 0.23746490478515625, 'copy_output_data_from_device[ms]': 0.164031982421875, 'unpack_output[ms]': 0.4527568817138672, 'unfold_output[ms]': 0.004291534423828125}

The error message:

$ python driver.py --exec_mode throughput_test --bitfile ../bitfile/finn-accel.xclbin
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Execution failed: 6')>
RuntimeError: Execution failed: 6
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Execution failed: 6')>
RuntimeError: Execution failed: 6
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Execution failed: 6')>
RuntimeError: Execution failed: 6
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Execution failed: 6')>
RuntimeError: Execution failed: 6
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Execution failed: 6')>
RuntimeError: Execution failed: 6
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Execution failed: 6')>
RuntimeError: Execution failed: 6
Results written to nw_metrics.txt

No error in my build_dataflow.log. So I don't think it is a problem related to the compile flow. Maybe something wrong with the ResNet50 FINN code?

build_dataflow.log:[2021-12-22 04:03:12,955] Running step: step_resnet50_tidy [1/13]
build_dataflow.log:[2021-12-22 04:03:49,450] Running step: step_resnet50_streamline [2/13]
build_dataflow.log:[2021-12-22 04:06:14,168] Running step: step_resnet50_convert_to_hls [3/13]
build_dataflow.log:[2021-12-22 04:07:15,738] Running step: step_create_dataflow_partition [4/13]
build_dataflow.log:[2021-12-22 04:07:16,438] Running step: step_apply_folding_config [5/13]
build_dataflow.log:[2021-12-22 04:07:16,932] Running step: step_generate_estimate_reports [6/13]
build_dataflow.log:[2021-12-22 04:07:17,559] Running step: step_hls_codegen [7/13]
build_dataflow.log:[2021-12-22 04:20:45,469] Running step: step_hls_ipgen [8/13]
build_dataflow.log:[2021-12-22 07:38:25,311] Running step: step_resnet50_set_fifo_depths [9/13]
build_dataflow.log:[2021-12-22 10:24:45,264] Running step: step_resnet50_slr_floorplan [10/13]
build_dataflow.log:[2021-12-22 10:25:04,196] Running step: step_synthesize_bitfile [11/13]
build_dataflow.log:[2021-12-23 10:49:20,827] Running step: step_make_pynq_driver [12/13]
build_dataflow.log:[2021-12-23 10:50:52,799] Running step: step_deployment_package [13/13]

==========================================

In comparison, the MobileNet model from finn-example looks good and the latency makes sense to me. Also it won't any generate errors during throughput_test. I think it is what a right bitfile will behavior.

The nw_metrics.txt of MobileNet running throughput_test

{'runtime[ms]': 1.8205642700195312, 'throughput[images/s]': 549.2802514405448, 'DRAM_in_bandwidth[Mb/s]': 82.68205768884232, 'DRAM_out_bandwidth[Mb/s]': 0.005492802514405448, 'fclk[mhz]': 249, 'batch_size': 1, 'fold_input[ms]': 0.0133514404296875, 'pack_input[ms]': 0.012874603271484375, 'copy_input_data_to_device[ms]': 0.22172927856445312, 'copy_output_data_from_device[ms]': 0.10204315185546875, 'unpack_output[ms]': 0.4482269287109375, 'unfold_output[ms]': 0.0050067901611328125}

Any clue for solving this is highly appreciated. Thanks!

==========================================

Environment and Setting
Vivado+Vitis - 2020.1
Ubuntu - 18.04
Device - Alveo U250

Lam