Failed to execute Ideal Dispatcher mode

[Vinay0203]

I am trying to execute fmatmul program in Ideal Dispatcher mode.

The steps i have taken is :
1.

cd apps
make bin/fmatmul.ideal

and the output got is :

cd fmatmul && if [ -d script ]; then python3 script/gen_data.py 128 128 128 > data.S ; else touch data.S; fi

sed -i s/"li t0, MSTATUS_FS | MSTATUS_XS$"/"li t0, MSTATUS_FS | MSTATUS_XS | MSTATUS_VS"/ /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S

git update-index --assume-unchanged /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -DPREALLOCATE=1 -DSPIKE=1 -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/env -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common -ffunction-sections -fdata-sections -c fmatmul/data.S -o fmatmul/data.S.o.spike

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -DPREALLOCATE=1 -DSPIKE=1 -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/env -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common -ffunction-sections -fdata-sections -c /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S -o /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S.o.spike

mkdir -p bin/

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -Iinclude -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -DPREALLOCATE=1 -DSPIKE=1 -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/env -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common -ffunction-sections -fdata-sections -o bin/fmatmul.spike fmatmul/kernel/fmatmul.c.o.spike fmatmul/main.c.o.spike fmatmul/data.S.o.spike /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S.o.spike /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/syscalls.c.o.spike common/util.c.o.spike -static -nostartfiles -lm -Wl,--gc-sections -nostdlib -T/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/test.ld -Wl,--gc-sections -DSPIKE

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/llvm-objdump --mattr=v -D bin/fmatmul.spike > bin/fmatmul.spike.dump

mkdir -p ideal_dispatcher/vtrace ideal_dispatcher/log ideal_dispatcher/temp

echo "run" | /home/vinay/Documents/ARA_new/ara/install/riscv-isa-sim-mod/bin/spike --isa=rv64gcv_zfh --varch="vlen:4096,elen:64" -d bin/fmatmul.spike 2> ideal_dispatcher/temp/fmatmul.temp 1> ideal_dispatcher/log/fmatmul.log

cd ideal_dispatcher && scripts/vtrace.sh temp/fmatmul.temp vtrace/fmatmul.vtrace

The SPIKE log was not clean.

mkdir -p bin/

cp bin/fmatmul.spike bin/fmatmul.ideal

cd hardware 
make simv app=fmatmul ideal_dispatcher=1

and the output got is :

Makefile:83: "Specified QuestaSim version (questa-2021.2) not found in PATH /home/vinay/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

build/verilator/Vara_tb_verilator  -l ram,/home/vinay/Documents/ARA_new/ara/apps/bin/fmatmul,elf

bash: line 1: build/verilator/Vara_tb_verilator: No such file or directory

make: *** [Makefile:232: simv] Error 127

3. I am in the up to date main branch and even failed to generate fst waveform for fmatmul after running this code :

# Compile fmatmul for Ara                                                                      
make -C apps bin/fmatmul        
# Verilate the design
make -C hardware verilate trace=1  
# Run fmatmul on Ara through Verilator
make -C hardware simv app=fmatmul trace=1 

and the output got is :

vinay@vinay:~/Documents/ARA_new/ara$ # Compile fmatmul for Ara                                                                      

make -C apps bin/fmatmul        

# Verilate the design

make -C hardware verilate trace=1 -B 

# Run fmatmul on Ara through Verilator

make -C hardware simv app=fmatmul trace=1 

make: Entering directory '/home/vinay/Documents/ARA_new/ara/apps'

cd fmatmul && if [ -d script ]; then python3 script/gen_data.py 128 128 128 > data.S ; else touch data.S; fi

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c fmatmul/data.S -o fmatmul/data.S.o

chmod +x /home/vinay/Documents/ARA_new/ara/apps/common/script/align_sections.sh

rm -f /home/vinay/Documents/ARA_new/ara/apps/common/link.ld && cp /home/vinay/Documents/ARA_new/ara/apps/common/arch.link.ld /home/vinay/Documents/ARA_new/ara/apps/common/link.ld

/home/vinay/Documents/ARA_new/ara/apps/common/script/align_sections.sh 4 /home/vinay/Documents/ARA_new/ara/apps/common/link.ld

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c fmatmul/main.c -o fmatmul/main.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c common/crt0.S -o common/crt0-llvm.S.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c common/printf.c -o common/printf-llvm.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c common/string.c -o common/string-llvm.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c common/serial.c -o common/serial-llvm.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -c common/util.c -o common/util-llvm.c.o

mkdir -p bin/

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -Iinclude -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf  -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument  -ffunction-sections -fdata-sections -o bin/fmatmul fmatmul/kernel/fmatmul.c.o fmatmul/main.c.o fmatmul/data.S.o common/crt0-llvm.S.o common/printf-llvm.c.o common/string-llvm.c.o common/serial-llvm.c.o common/util-llvm.c.o -static -nostartfiles -lm -Wl,--gc-sections -T/home/vinay/Documents/ARA_new/ara/apps/common/link.ld

ld.lld: warning: ignoring memory region assignment for non-allocatable section '.comment'

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/llvm-objdump --mattr=v -D bin/fmatmul > bin/fmatmul.dump

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/llvm-strip bin/fmatmul -S --strip-unneeded

rm common/serial-llvm.c.o common/printf-llvm.c.o common/crt0-llvm.S.o common/util-llvm.c.o common/string-llvm.c.o fmatmul/main.c.o

make: Leaving directory '/home/vinay/Documents/ARA_new/ara/apps'

make: Entering directory '/home/vinay/Documents/ARA_new/ara/hardware'

Makefile:83: "Specified QuestaSim version (questa-2021.2) not found in PATH /home/vinay/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

mkdir -p build

Bender already exists.

bender 0.27.3 available.

rm -rf build/verilator; mkdir -p build/verilator

/home/vinay/Documents/ARA_new/ara/hardware/../hardware/bender script verilator -t rtl -t cv64a6_imafdcv_sv39 -t tech_cells_generic_include_tc_sram -t tech_cells_generic_include_tc_clk -t ara_test -t cva6_test -t verilator --define NR_LANES=4 --define VLEN=4096 --define ARIANE_ACCELERATOR_PORT=1 --define COMMON_CELLS_ASSERTS_OFF > build/verilator/bender_script_default

/home/vinay/Documents/ARA_new/ara/install/verilator/bin/verilator -f build/verilator/bender_script_default           \

  -GNrLanes=4                                                         \

  -O3                                                                           \

  -Wno-fatal                                                                    \

  -Wno-PINCONNECTEMPTY                                                          \

  -Wno-BLKANDNBLK                                                               \

  -Wno-CASEINCOMPLETE                                                           \

  -Wno-CMPCONST                                                                 \

  -Wno-LATCH                                                                    \

  -Wno-LITENDIAN                                                                \

  -Wno-UNOPTFLAT                                                                \

  -Wno-UNPACKED                                                                 \

  -Wno-UNSIGNED                                                                 \

  -Wno-WIDTH                                                                    \

  -Wno-WIDTHCONCAT                                                              \

  -Wno-ENUMVALUE                                                                \

  -Wno-COMBDLY                                                                  \

  tb/verilator/waiver.vlt                                                       \

  --Mdir build/verilator                                                       \

  -Itb/dpi                                                                      \

  --compiler clang                                                              \

  -CFLAGS "-DTOPLEVEL_NAME=ara_tb_verilator"                                        \

  -CFLAGS "-DNR_LANES=4"                                              \

  -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp       \

  -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp \

  -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp \

  ""                                                             \

  -LDFLAGS "-lelf"                                                              \

  ""                                                              \

  --exe                                                                         \

  /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp/*.cc            \

  /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp/*.cc      \

  /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp/*.cc      \

  /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/ara_tb.cpp                                           \

  --cc                                                                          \

  --trace-fst -Wno-INSECURE                                     \

  --top-module ara_tb_verilator &&                                                  \

cd build/verilator && OBJCACHE='' make -j4 -f Vara_tb_verilator.mk

%Error: Verilator threw signal 9. Suggest trying --debug --gdbbt

%Error: Command Failed ulimit -s unlimited 2>/dev/null; exec /home/vinay/Documents/ARA_new/ara/install/verilator/bin/verilator_bin -f build/verilator/bender_script_default -GNrLanes=4 -O3 -Wno-fatal -Wno-PINCONNECTEMPTY -Wno-BLKANDNBLK -Wno-CASEINCOMPLETE -Wno-CMPCONST -Wno-LATCH -Wno-LITENDIAN -Wno-UNOPTFLAT -Wno-UNPACKED -Wno-UNSIGNED -Wno-WIDTH -Wno-WIDTHCONCAT -Wno-ENUMVALUE -Wno-COMBDLY tb/verilator/waiver.vlt --Mdir build/verilator -Itb/dpi --compiler clang -CFLAGS -DTOPLEVEL_NAME=ara_tb_verilator -CFLAGS -DNR_LANES=4 -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp  -LDFLAGS -lelf  --exe /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp/dpi_memutil.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp/sv_scoped.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp/verilator_memutil.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp/verilated_toplevel.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp/verilator_sim_ctrl.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/ara_tb.cpp --cc --trace-fst -Wno-INSECURE --top-module ara_tb_verilator

make: *** [Makefile:188: build/verilator/Vara_tb_verilator] Error 255

make: Leaving directory '/home/vinay/Documents/ARA_new/ara/hardware'

make: Entering directory '/home/vinay/Documents/ARA_new/ara/hardware'

Makefile:83: "Specified QuestaSim version (questa-2021.2) not found in PATH /home/vinay/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

build/verilator/Vara_tb_verilator -t -l ram,/home/vinay/Documents/ARA_new/ara/apps/bin/fmatmul,elf

bash: line 1: build/verilator/Vara_tb_verilator: No such file or directory

make: *** [Makefile:231: simv] Error 127

make: Leaving directory '/home/vinay/Documents/ARA_new/ara/hardware'

4. I am trying to achieve fst file only for the vector unit 'ARA' which does not include signals from CVA6 {or} some extra signals which are dedicated to ARA along with the CVA6 signals
   So if we get a trace file in Ideal Dispatcher mode is it the trace file only generated by ARA alone and only that signals? Am i thinking correctly?
   and what is the command modification to get trace file after solving this issue , is it just by giving command as :

make simv app=fmatmul ideal_dispatcher=1 trace=1

[mp-17]

Hello @Vinay0203,
Also other users reported issues with the Verilator traces. I will need some time to check this out since we use Verilator only in the CI with no traces and no ideal-dispatcher mode. If you have a QuestaSim license instead, it should work out of the box

[Vinay0203]

@mp-17

Please comment on the point 4. illustrated in previous message