64-bit build creates bad ISOs

[mborgerson]

I'm not sure why yet but building in 64b mode creates a reproducible bad ISO with the "Halo 2 Multiplayer Map Pack" and "Call of Duty: Big Red One" game files (which I've ripped from my discs, verified to work on my real Xbox).

When running H2:MMP with such a faulty ISO, the game will start but it will fail to install update the game executable and report "out of space" errors when trying to install the maps. In CoD:BRO, the game will report dirty disc errors before getting to the main menu. (Fixed in #52)

In both cases, building in 32-bit mode yields a fully-functional ISO. Suggest updating CI to build in 32-bit mode until this can be fixed.

Thanks to @Ernegien for helping me debug this issue!

[abaire]

This also triggers a crash in "Return To Castle Wolfenstein" (see thread in xemu-project/xemu#509)

[Shoegzer]

I found this issue after converting an ISO dump to XISO resulted in a bad dump. extract-iso was compiled with 64-bit linux - a 32-bit version can't be compiled without a 32-bit OS or chroot environment.

Others too are submitting more issue reports in xemu that have nothing to do with the emulator but are related to bad dumps, I'm sure that's distracting for its devs, and though I'm not sure if part of the reason for so many can be traced here it seems possible.

As a side note, I'm not sure what roadblocks exist for xemu to support full ISOs but that would be nice, even if 1:1 dumps take more hard drive space I think they're better than lossy alternatives in the long run with all things considered.

[mborgerson]

You can build a 32b executable @Shoegzer you just need gcc-multilib and to specify -m32

[Shoegzer]

Thanks @mborgerson. I tried gcc -m32 extract-xiso.c -o extract-xiso, but it threw an "unknown target" exception followed by further exceptions. How can the linux target be specified? Normally I'd do that with configure --target=linux.

[Shoegzer]

Update: here is the first part of the compile log in case it helps. I can add more if necessary, but I believe all the errors stem from lack of a specified target. I had previously been compiling with CMake but didn't here since the -m32 flag can't be used with CMake specifically and there doesn't appear to be anything supporting a 32-bit build in CMakeLists.

[abaire]

You can edit CMakeLists to add "-m32" to compile and link flags for the target. If you do a trivial search for "cmake" "32 bit" there are a number of examples.

@mborgerson given that we know 64-bit builds are still problematic, would it be reasonable to either force 32-bit in the makefile (maybe with a warning so developers seeking to fix 64 bit will be aware?) or update CI to do 32-bit releases for all platforms so there are fewer people doing incorrect builds and seeking support here/on Discord?

[dracc]

For the lazy: Adding

target_compile_options(extract-xiso PUBLIC -m32)
target_link_options(extract-xiso PUBLIC -m32)

to the CMakeLists.txt file works (for me) after installing gcc-multilib.

[mborgerson]

@mborgerson given that we know 64-bit builds are still problematic, would it be reasonable to either force 32-bit in the makefile (maybe with a warning so developers seeking to fix 64 bit will be aware?) or update CI to do 32-bit releases for all platforms so there are fewer people doing incorrect builds and seeking support here/on Discord?

@abaire I think so, yes (to both)

[Shoegzer]

Thanks, that's very helpful!

However, after adding those two lines, the following occurs during linking:

[100%] Linking C executable extract-xiso
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a when searching for -lgcc
/usr/bin/ld: cannot find -lgcc
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a when searching for -lgcc
/usr/bin/ld: cannot find -lgcc
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/extract-xiso.dir/build.make:97: extract-xiso] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/extract-xiso.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

gcc-multilib is installed:

$ apt list gcc-multilib
Listing... Done
gcc-multilib/focal,now 4:9.3.0-1ubuntu2 amd64 [installed]
gcc-multilib/focal 4:9.3.0-1ubuntu2 i386

[Shoegzer]

Update: After updating from Linux Mint 20.3 (Ubuntu 20.02 base) > 21 (Ubuntu 22.02 base), this now compiles successfully. Not sure of root cause, build-essential and all other deps existed in the prior setup, but in any event it works.

Ideally xemu would support full 1:1 dumps natively and avoid the requirement for this tool, but at least other projects can benefit from it too. Regardless, that's out of scope here.

[JayFoxRox]

@Shoegzer you likely don't need this tool. See #32 (comment).
Unfortunately #47 added support for redump ISOs, which means users are unaware of what kind of XISO they have (as they all extract now, even if the redump ISO shouldn't extract - in my opinion, they should be split in a separate step).

---

Back to topic please

[Shoegzer]

Thanks @JayFoxRox. Your point is well-taken that video/data "partitions" of archive-quality (i.e. redump) images can be split in ways that allow rejoining via fuse etc. to preserve integrity. Use cases vary of course, and dump quality matters more over time, though I think it's likely most people will eventually just keep the full redump image anyway as it's easy to validate a single-file dump against its corresponding redump hash in one shot. The Wii is a similar case, with its proprietary dvd format that Dolphin now internally validates against redump hashes, and more people are now ensuring their dumps are good. This helps the emulation testing process too, as issues related to bad dumps are either prevented or quickly identified.

Ideally xemu could read full dumps directly as an option (if video partition is detected, skip it and then read?), but unless/until that happens we'll maintain both redump for integrity and xiso for testing, and I'm sure some will use the solution you suggest as it's not a bad alternative that preserves integrity.

But yes, I agree this is totally off the topic of resolving the 64-bit build issue, so I'm happy to discuss elsewhere if you feel it holds merit.

[rapperskull]

I tried to rewrite "Halo 2 Multiplayer Map Pack" on Windows and both 32 and 64 bit produce the exact same ISO. Is it a problem only on Linux?

[PQCraft]

@rapperskull probably an issue with long. It's 32-bit on both 32-bit and 64-bit Windows, but on Linux and other Unix-like systems, it's 64-bit on 64-bit systems and 32-bit on 32-bit systems.