document that extern statics may be bigger than the declared size

[RalfJung]

I am fairly sure we discussed at some point whether it's okay to access memory beyond the declared size of an extern static, and @nikic confirmed that this is fine. But I couldn't find any official place where we document this. So, let's add it to the Reference.

I wondered if we also need the converse on the UB page: it is UB to even start the program if an extern static ends up being backed by memory that is smaller or less-aligned than the declared type. What do you think?

Cc @rust-lang/opsem
Fixes rust-lang/unsafe-code-guidelines#259
Fixes rust-lang/unsafe-code-guidelines#554

[Jules-Bertholet]

Perhaps be more explicit about the need to use &raw (and avoid an intermediate reference)?

View changes since the review

[RalfJung]

I have removed the &raw requirement, anticipating #2338. I don't think we want subobject provenance to get in the way here, so this should be fine (if the static is big enough to read element N):

extern { static mut X: (); }

let ptr = &X as *const () as *const i32;
ptr.add(N).read();

This is a bad idea because of aliasing rules (the pointer gets invalidated on writes), but we already lint against that.

[DianaNites]

This would formally resolve/answer rust-lang/unsafe-code-guidelines#259 and maybe some of rust-lang/unsafe-code-guidelines#546 right?

[RalfJung]

The latter has already been resolved by #1657.

And this does resolve the former, yes. Thanks for digging that up.

[RalfJung]

This new kind of UB is kind of remarkable in that it does not require any code to trigger. But I don't see an alternative...

View changes since the review

[bjorn3]

For dynamic libraries that may be used by a PIE executable, the size given by the symbol must be exact given that the PIE executable will emit a copy relocation that copies a block with exactly the size the symbol had at link time to memory the executable image has reserved for this and redirect all accesses to the static to this copy. This way the executable can avoid GOT indirection, which is a slight perf win. And yes, this means adding elements to a static array in a dylib (or otherwise changing the size) is an ABI breaking change on Linux.

View changes since the review

[RalfJung]

Uh... I understand like maybe half of those words. (Can I have some 🥧 please? :D )
"Copying" sounds wrong, statics are places and if you copy them, well, you have two copies so that can't be right?

But it sounds like you are saying linkme and inventory are unsound? IIRC they rely on extern statics that are bigger than declared, filled in by the linker.

[bjorn3]

Basically when linking the executable, space is reserved for each static referenced from a dylib inside the executable based on symbol size. Then at runtime the dynamic linker will copy the contents of the static in the dylib to the reserved space in the executable and redirect all references to this static to the version in the executable. So there are two copies in memory, but only the one inside the executable is observable by the user. The copy inside the dylib is unused and would be safe to unmap from memory after the dynamic linker has copied it.

linkme and inventory don't work across dylib boundaries anyway as they need the section they emit to be contiguous in memory. They would only see items defined inside the same dylib/executable as the one where the access code happened to be codegened.

[RalfJung]

Thanks!

And what does that mean for crates like linkme/inventory?

[bjorn3]

linkme and inventory only see items defined in the same dylib/executable as the access code is codegened. So if you depend on them to have a global view, you will get misbehavior.

[bjorn3]

As I said the linker needs to know the size somehow to determine the amount of memory to reserve in a PIE executable and to emit the right sized copy relocation. Either it gets this size from the upstream dylib or in the case of raw-dylib it gets it from the generated import library where rustc sets the size based on the type of the static.

[RalfJung]

Ah right, raw-dylib was the thing where you don.t have a .so file to look up the "actual source of truth" for these things. I guess that makes sense.

(currently unstable for ELF)

Why is ELF relevant here?

[bjorn3]

Only ELF has copy relocations for PIE executables. The other object file formats don't implement this "optimization".

[RalfJung]

Ah, I see.

Not sure if we should make our UB-rules format-dependent, though.

[CAD97]

There are two cases here, IIUC:

• Rust is a dylib using the static. Access to the static goes through GOT lookup, independent of whether Rust is the definition site of the static or not. When linking to the dylib, the linker fixups must supply a memory region at least the declared size.
• Rust is the binary using a static from a dylib. Rust supplies the memory region for the static matching the declared size. At dynamic linking time (before main), the dynamic linker writes the initialization value to the provided memory region for the static.

The memory region of the static always matches the declaration exactly if Rustc is the one allocating it; it's only when Rust is linking to an external memory region that it may potentially be larger. The case where Rust allocates the static and dynamic linking initializes it could be declared morally equivalent to life-before-main w.r.t. the abstract machine, and thus the initialization going out of bounds of the static is an out-of-bounds access unsoundness. Notably, if the Rust code declares a larger static that doesn't require the entire static value to be initialized, it's perfectly fine for the linker to copy initialize a smaller value in that memory region.

In conclusion, I think ELF copy relocation doesn't preclude allowing Rust extern static from linking to a memory region that ends up being larger than declared. But it is another weird case of where the wrong signature in an unsafe extern block can lead to UB even if the symbols go unused by Rust. Which isn't pretty, but it is the reality of today's compiler/linker plumbing.

[joshlf]

What is the ground truth with respect to which "less mutable" is evaluated? I understand what "declaring an extern static with some... mutability" means, but what does the mutability of an "actual symbol" refer to?

View changes since the review

[RalfJung]

It refers to whether that symbol is in actually mutable memory. My understanding is that in ELF there are flags indicating which sections are mutable and which are not, and I presume other binary formats have something similar.

[CAD97]

Possible alternative wording instead of symbol: the linking-resolved memory region.

[RalfJung]

I guess this does make new commitments, we are officially allowing symbols-bigger-than-the-static-they-are-imported-with. So this should probably be FCP'd. I guess t-opsem is enough?

[CAD97]

We probably want to rope the linker-format experts in to the FCP as well at least, given potential limiting factors like ELF copy relocation could exist in other object formats. (Although I don't think ELF copy relocation precludes allowing this.)