Avoid Unsafe usage where possible

[SirYwell]

Overview

Description

While reflections got restricted, we can still use them for everything in the unnamed module.
This allows us to get rid of almost all usages of Unsafe.
If Unsafe is ever removed in future Java releases, we are better prepared, using the proper replacements.

Submitter Checklist

Beta Give feedback

1. Make sure you are opening from a topic branch (/feature/fix/docs/ branch (right side)) and not your main branch.
2. Ensure that the pull request title represents the desired changelog entry.
3. New public fields and methods are annotated with @since TODO.
4. I read and followed the contribution guidelines.