Self build toolchains for RISC-V lacking __libc_fini_array entry

[EnkiSun86]

I'm trying to bootstrap and compile the toolchains for Vortex. I followed the README.vortex in the pocl source file pulled from https://github.com/vortexgpgpu/pocl/tree/vortex .

#### Compiling pocl on RiscV ####

## Dependencies:

- sudo apt-get -y install \
  binutils build-essential libtool texinfo \
  gzip zip unzip patchutils curl git \
  make cmake ninja-build automake bison flex gperf \
  grep sed gawk python bc \
  zlib1g-dev libexpat1-dev libmpc-dev \
  libglib2.0-dev libfdt-dev libpixman-1-dev

## Setting tools directory

export TOOLDIR=$HOME/tools

## Building RiscV GNU Toolchain (gcc, binutils, etc..)

- git clone --depth=1 --recursive https://github.com/riscv-collab/riscv-gnu-toolchain.git
- mkdir build && cd build
- export CPATH=$TOOLDIR/GNU/include
- export LIBRARY_PATH=$TOOLDIR/GNU/lib
- ../configure --prefix=$TOOLDIR/riscv32-gnu-toolchain --with-cmodel=medany --with-arch=rv32imf --with-abi=ilp32f
#../configure --prefix=$TOOLDIR/riscv64-gnu-toolchain --with-cmodel=medany --with-arch=rv64imafd --with-abi=lp64d
- make -j`nproc`
- make -j`nproc` build-qemu

## Building LLVM for Vortex

- git clone --recursive --branch vortex https://github.com/vortexgpgpu/llvm.git
- cd llvm
- mkdir build && cd build
- export LLVM_PREFIX=$TOOLDIR/llvm-vortex
- export RISCV_TOOLCHAIN_PATH=$TOOLDIR/riscv64-gnu-toolchain
- cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$LLVM_PREFIX -DLLVM_ENABLE_PROJECTS="clang;lld" -DBUILD_SHARED_LIBS=True -DLLVM_TARGETS_TO_BUILD="RISCV" -DLLVM_ABI_BREAKING_CHECKS=FORCE_OFF -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF -DDEFAULT_SYSROOT=$RISCV_TOOLCHAIN_PATH/riscv32-unknown-elf -DLLVM_DEFAULT_TARGET_TRIPLE="riscv32-unknown-elf" ../llvm
- make -j`nproc`
- make install

## Sanity test your new RISC-V LLVM

- echo -e '#include <stdio.h>\n int main(void) { printf("Hello world!\\n"); return 0; }' > hello.c
- clang hello.c
- qemu-riscv32 hello

## Building llvm-spirv
- git clone --depth=1 -b release/10.x  https://github.com/KhronosGroup/SPIRV-LLVM-Translator.git
- cd llvm-spirv
- mkdir build && cd build
- export LLVM_VORTEX=$TOOLDIR/llvm-vortex
- cmake .. -DLLVM_DIR=$LLVM_VORTEX -DCMAKE_INSTALL_PREFIX=$LLVM_VORTEX
- make llvm-spirv -j`nproc`
- make install
# manually copy over llvm-spirv binary
- cp ./tools/llvm-spirv/llvm-spirv $LLVM_VORTEX/bin

## build POCL compiler
- git clone --branch vortex --recursive https://github.com/vortexgpgpu/pocl
- cd pocl
- mkdir build_cc && cd build_cc
- export POCL_CC_PATH=$TOOLDIR/pocl/compiler
- export LLVM_VORTEX=$TOOLDIR/llvm-vortex
- cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$POCL_CC_PATH -DOCS_AVAILABLE=ON -DWITH_LLVM_CONFIG=$LLVM_VORTEX/bin/llvm-config -DENABLE_VORTEX=ON -DBUILD_TESTS=OFF -DPOCL_DEBUG_MESSAGES=ON -DENABLE_ICD=OFF -DCLANG_MARCH_FLAG:STRING=-mcpu= -DLLC_HOST_CPU=generic-rv32 ..
- make -j`nproc`
- make install

## build POCL runtime
- git clone --branch vortex --recursive https://github.com/vortexgpgpu/pocl
- cd pocl
- mkdir build_rt && cd build_rt
- export POCL_RT_PATH=$TOOLDIR/pocl/runtime
- export VORTEX_DRIVER_INC=$HOME/dev/vortex/runtime/include
- export VORTEX_DRIVER_LIB=$HOME/dev/vortex/build/runtime/stub/libvortex.so
- cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DHOST_DEVICE_BUILD_HASH=riscv32-unknown-unknown-elf -DCMAKE_INSTALL_PREFIX=$POCL_RT_PATH -DOCS_AVAILABLE=OFF -DENABLE_LLVM=OFF -DVORTEX_DRIVER_INC=$VORTEX_DRIVER_INC -DVORTEX_DRIVER_LIB=$VORTEX_DRIVER_LIB -DENABLE_VORTEX=ON -DBUILD_TESTS=OFF -DPOCL_DEBUG_MESSAGES=ON -DENABLE_ICD=OFF ..
- make -j`nproc`
- make install
- cp -r ../include $POCL_RT_PATH

And I built the source as follows:

• RiscV GNU Toolchain
• LLVM for Vortex
• llvm-spirv
• POCL compiler

Then I encountered a problem when test my new RISC-V LLVM as the README.vortex says. The error log showed as follows:

ld.lld: error: unable to find library -lclang_rt.builtins-riscv32
clang-16: error: ld.lld command failed with exit code 1 (use -v to see invocation)

As the error reported, I built compiler-rt in llvm standalone to get the libclang_rt.builtins-riscv32.a and rebuilt hello.c :

$ ~/share/vortex/toolchain/out/llvm-vortex/bin/clang -march=rv32imf -mabi=ilp32f hello.c

It returned with new errors:

ld.lld: error: undefined symbol: _close
>>> referenced by closer.c
>>>               libc_a-closer.o:(_close_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
>>> did you mean: fclose
>>> defined in: /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a(libc_a-fclose.o)

ld.lld: error: undefined symbol: _exit
>>> referenced by abort.c
>>>               libc_a-abort.o:(abort) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _fstat
>>> referenced by fstatr.c
>>>               libc_a-fstatr.o:(_fstat_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _isatty
>>> referenced by isattyr.c
>>>               libc_a-isattyr.o:(_isatty_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _lseek
>>> referenced by lseekr.c
>>>               libc_a-lseekr.o:(_lseek_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _read
>>> referenced by readr.c
>>>               libc_a-readr.o:(_read_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _kill
>>> referenced by signalr.c
>>>               libc_a-signalr.o:(_kill_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _getpid
>>> referenced by signalr.c
>>>               libc_a-signalr.o:(_getpid_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _sbrk
>>> referenced by sbrkr.c
>>>               libc_a-sbrkr.o:(_sbrk_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

ld.lld: error: undefined symbol: _write
>>> referenced by writer.c
>>>               libc_a-writer.o:(_write_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a

So I checked the prebuilt toolchains provided by the author of Vortex, and found a file folder named libc32 :

libc32/lib$ tree
.
├── crt1.o
├── crti.o
├── crtn.o
├── libc.a
├── libcrypt.a
├── libdl.a
├── libm.a
├── libpthread.a
├── libresolv.a
├── librt.a
├── libutil.a
├── libxnet.a
├── rcrt1.o
└── Scrt1.o

It seems to be a musl libc built standalone. But there is no document showing how to build it. So my first problem is how to bootstrap build this libc32 myself.

Since I do not know how to build this libc32, I use the prebuilt one instead and build hello.c as follows:

$ ~/share/vortex/toolchain/out/llvm-vortex/bin/clang -march=rv32imf -mabi=ilp32f -L/home/soc_szq/share/vortex/toolchain/out/libc32/lib hello.c

However, it still returns with the warning:

ld.lld: warning: cannot find entry symbol _start; not setting start address

So although it generated an a.out, the test with qemu returned a segmentation fault:

/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/bin/qemu-riscv32 a.out -v
Segmentation fault (core dumped)

Besides I skipped this test and went through the README.vortex to build pocl runtime. Since it needs a library named libvortex.so, I build vortex at first to get it. But build vortex with my self-build riscv32-gnu-toolchain returned with error:

make[3]: Entering directory '/home/soc_szq/share/vortex/toolchain/src/vortex/build/tests/kernel/conform'
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/bin/riscv32-unknown-elf-gcc -march=rv32imaf -mabi=ilp32f -O3 -mcmodel=medany -fno-exceptions -nostartfiles -nostdlib -fdata-sections -ffunction-sections -I/home/soc_szq/share/vortex/toolchain/src/vortex/kernel/include -I/home/soc_szq/share/vortex/toolchain/src/vortex/build/hw -DXLEN_32 -DNDEBUG /home/soc_szq/share/vortex/toolchain/src/vortex/tests/kernel/conform/main.cpp /home/soc_szq/share/vortex/toolchain/src/vortex/tests/kernel/conform/tests.cpp -Wl,-Bstatic,--gc-sections,-T,/home/soc_szq/share/vortex/toolchain/src/vortex/kernel/scripts/link32.ld,--defsym=STARTUP_ADDR=0x80000000 /home/soc_szq/share/vortex/toolchain/src/vortex/build/kernel/libvortex.a -L/home/soc_szq/share/vortex/toolchain/out/libc32/lib -lm -lc /home/soc_szq/share/vortex/toolchain/out/libcrt32/lib/baremetal/libclang_rt.builtins-riscv32.a -o conform.elf
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/lib/gcc/riscv32-unknown-elf/15.0.0/../../../../riscv32-unknown-elf/bin/ld: /home/soc_szq/share/vortex/toolchain/src/vortex/build/kernel/libvortex.a(vx_start.S.o): in function `_start':
(.init+0x64): undefined reference to `__libc_fini_array'
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/lib/gcc/riscv32-unknown-elf/15.0.0/../../../../riscv32-unknown-elf/bin/ld: (.init+0x70): undefined reference to `__libc_init_array'
collect2: error: ld returned 1 exit status
make[3]: *** [../common.mk:43: conform.elf] Error 1

This problem due to the newlib in riscv32-gnu-toolchain changed HAVE_INITFINI_ARRAY to _HAVE_INITFINI_ARRAY according to this commit:

commit 437c5c5085ff30b4a4960b2b53d06728c788361d
Author: Mike Frysinger <vapier@gentoo.org>
Date:   Mon Jan 17 22:20:20 2022 -0500

    newlib: internalize HAVE_INITFINI_ARRAY
   
    This define is only used by newlib internally, so stop exporting it
    as HAVE_INITFINI_ARRAY since this can conflict with defines packages
    use themselves.
   
    We don't really need to add _ to HAVE_INIT_FINI too since it isn't
    exported in newlib.h, but might as well be consistent here.
   
    We can't (easily) add this to newlib_cflags like HAVE_INIT_FINI is
    because this is based on a compile-time test in the top configure,
    not on plain shell code in configure.host.  We'd have to replicate
    the test in every subdir in order to have it passed down.

This led to the build of vx_syscalls.c lack of entry of __libc_init_array and __libc_fini_array. I fixed this by this change:

diff --git a/kernel/src/vx_syscalls.c b/kernel/src/vx_syscalls.c
index 6ff9fbb..ad0777b 100644
--- a/kernel/src/vx_syscalls.c
+++ b/kernel/src/vx_syscalls.c
@@ -65,7 +65,7 @@ void __init_tls(void) {
   memset(__thread_self + (size_t)__tbss_offset, 0, (size_t)__tbss_size);
 }
 
-#ifdef HAVE_INITFINI_ARRAY
+#ifdef _HAVE_INITFINI_ARRAY
 
 // These magic symbols are provided by the linker.
 extern void (*__preinit_array_start []) (void) __attribute__((weak));
@@ -96,7 +96,7 @@ void __libc_init_array (void) {
 }
 #endif
 
-#ifdef HAVE_INITFINI_ARRAY
+#ifdef _HAVE_INITFINI_ARRAY
 extern void (*__fini_array_start []) (void) __attribute__((weak));
 extern void (*__fini_array_end []) (void) __attribute__((weak));

[EnkiSun86]

The remaining puzzle is how to bootstrap build this libc32 myself.

[EnkiSun86]

I fixed the ld.lld: warning: cannot find entry symbol _start; not setting start address by linking the crt1.o in libc32 manually. However, the qemu-riscv32 still can not run the a.out since there are some Vortex-specific instructions in the a.out like vx_spilt. More effort is needed to modify the qemu to support the evaluation of Vortex