python312Packages.bitsandbytes: 0.43.1 -> 0.43.3 #343246

GaetanLepage · 2024-09-20T11:28:18Z

Description of changes

Changelog: https://github.com/TimDettmers/bitsandbytes/releases/tag/0.43.3

Things done

Add a 👍 reaction to pull requests you find important.

remexre · 2024-10-20T13:24:53Z

The CUDA-enabled version of this no longer builds when I do:

$ NIXPKGS_ALLOW_UNFREE=1 nix repl --impure
Welcome to Nix 2.18.5. Type :? for help.

nix-repl> :lf .
Added 18 variables.

nix-repl> :b outputs.legacyPackages.x86_64-linux.python3Packages.bitsandbytes.override { torch = outputs.legacyPackages.x86_64-linux.python3Packages.torchWithCuda; }
error: builder for '/nix/store/ylwx6as34nfygmxz5lwjvcqb15p7xd5q-python3.12-bitsandbytes-0.43.3.drv' failed with exit code 1;
       last 10 log lines:
       >
       >
       > Call Stack (most recent call first):
       >   /nix/store/yzi080r2c1zn2jzrhcfdv7dmr92yw07l-cmake-3.29.6/share/cmake-3.29/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
       >   /nix/store/yzi080r2c1zn2jzrhcfdv7dmr92yw07l-cmake-3.29.6/share/cmake-3.29/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
       >   /nix/store/yzi080r2c1zn2jzrhcfdv7dmr92yw07l-cmake-3.29.6/share/cmake-3.29/Modules/CMakeDetermineCUDACompiler.cmake:131 (CMAKE_DETERMINE_COMPILER_ID)
       >   CMakeLists.txt:74 (enable_language)
       >
       > 
       > -- Configuring incomplete, errors occurred!
       For full logs, run 'nix log /nix/store/ylwx6as34nfygmxz5lwjvcqb15p7xd5q-python3.12-bitsandbytes-0.43.3.drv'.

A fix is here: GaetanLepage/nixpkgs@2af0c9e...remexre:nixpkgs:bitsandbytes

GaetanLepage · 2024-10-20T16:47:16Z

A fix is here: GaetanLepage/nixpkgs@2af0c9e...remexre:nixpkgs:bitsandbytes

Thanks, the patch seems to work indeed.
I will wait for @SomeoneSerge's approval as he's much more knowledgeable than me on the CUDA stuff.

GaetanLepage · 2024-10-20T16:52:11Z

Well, even though it does build fine with cudaSupport = true, and the derivation is finalized successfully, it spits the following message:

Executing pythonImportsCheckPhase
Check whether the following modules can be imported: bitsandbytes
Could not load bitsandbytes native library: /nix/store/8x5i8xccb6h38iz22r8x3c4qg2pbrbg6-python3.12-bitsandbytes-0.44.1/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/nix/store/8x5i8xccb6h38iz22r8x3c4qg2pbrbg6-python3.12-bitsandbytes-0.44.1/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 104, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/8x5i8xccb6h38iz22r8x3c4qg2pbrbg6-python3.12-bitsandbytes-0.44.1/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 91, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/wfbjq35kxs6x83c3ncpfxdyl5gbhdx4h-python3-3.12.6/lib/python3.12/ctypes/__init__.py", line 460, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/wfbjq35kxs6x83c3ncpfxdyl5gbhdx4h-python3-3.12.6/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /nix/store/8x5i8xccb6h38iz22r8x3c4qg2pbrbg6-python3.12-bitsandbytes-0.44.1/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory

GaetanLepage · 2024-10-20T17:02:17Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 343246

`x86_64-linux`

✅ 4 packages built:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`aarch64-linux`

✅ 4 packages built:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`x86_64-darwin`

`aarch64-darwin`

samuela

diff lgtm

samuela · 2024-10-26T05:15:02Z

OSError: /nix/store/8x5i8xccb6h38iz22r8x3c4qg2pbrbg6-python3.12-bitsandbytes-0.44.1/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory

This looks like a job for LD_DEBUG=libs

GaetanLepage · 2024-10-26T10:24:41Z

Ok, I investigated some more.

Problem

When cudaSupport is enabled, the cuda variant of the library is built libbitsandbytes_cuda124.so instead of the native CPU one libbitsandbytes_cpu.so.
However, at runtime, they decide whether to load the cuda library or not depending on torch.cuda.is_available(). However, this will return False on any system where there is no GPU available.
Hence, it falls back to trying to load the native _cpu.so library, which has not been built and thus doesn't exist.

Relevant code

In this function, cuda_specs will evaluate to None and the whole if statement will not trigger.

Solution

The best solution I found was to:

ignore the get_cuda_specs() result and enter the if statement unconditionally.
bypass the call to get_cuda_bnb_library_path(cuda_specs) and hardcode the library path here (PACKAGE_DIR / "libbitsandbytes_cuda124.so").

Now everything builds and imports fine. No more error messages.

Result (with `cudaSupport = true`)

bitsandbytes> Running phase: pythonImportsCheckPhase
bitsandbytes> Executing pythonImportsCheckPhase
bitsandbytes> Check whether the following modules can be imported: bitsandbytes
┏━ Dependency Graph:
┃ ✔ python3.12-bitsandbytes-0.44.1 ⏱ 3m26s
┣━━━ Builds
┗━ ∑ ⏵ 0 │ ✔ 1 │ ⏸ 0 │ Finished at 12:21:35 after 3m29s
/nix/store/5n0nma25svsbw15x0qcj4sqyl8syrnyp-python3.12-bitsandbytes-0.44.1

GaetanLepage · 2024-10-26T10:34:05Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 343246

`x86_64-linux`

✅ 4 packages built:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`aarch64-linux`

✅ 4 packages built:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`x86_64-darwin`

`aarch64-darwin`

pkgs/development/python-modules/bitsandbytes/default.nix

SomeoneSerge · 2024-10-27T11:13:26Z

pkgs/development/python-modules/bitsandbytes/default.nix

+  preBuild =
+    if cudaSupport then
+      ''
+        export NVCC_APPEND_FLAGS="-I${cuda-native-redist}/include -L${cuda-native-redist}/lib"


cudaSetupHook sets NVCC_APPEND_FLAGS too. Use appendToVar or prependVar

I changed it to the same approach as in mistral-rs. Is it fine now ?

SomeoneSerge · 2024-10-27T11:16:03Z

pkgs/development/python-modules/bitsandbytes/default.nix

+    if cudaSupport then
+      ''
+        export NVCC_APPEND_FLAGS="-I${cuda-native-redist}/include -L${cuda-native-redist}/lib"
+        cmake -DCMAKE_CXX_FLAGS="-I${cuda-native-redist}/include" -DCOMPUTE_BACKEND=cuda -S .


This is normally done in confgiurePhase which pkgs.cmake in nativeBuildInputs would set automatically. Istead of specifying the -D flags in here, maybe move them to the nix attrset as cmakeFlags so the hook passes them on as a bash variable. If for some reason you choose not to use the hook but to call cmake yourself, reimplement the flagsArray=(...) ; concatTo ... functionality from the hook. The cmake hook respects dontUseCMake....Directory flag and you can pass -S . in cmakeFlags too

I was able to successfully rely on the cmake hook.
I still manually run make in the preBuild phase.

pkgs/development/python-modules/bitsandbytes/default.nix

SomeoneSerge · 2024-10-27T11:23:17Z

pkgs/development/python-modules/bitsandbytes/default.nix


-  postPatch =
-    ''
-      substituteInPlace Makefile --replace "/usr/bin/g++" "g++" --replace "lib64" "lib"
-      substituteInPlace bitsandbytes/cuda_setup/main.py  \
-        --replace "binary_path = package_dir / self.binary_name"  \
-                  "binary_path = Path('$out/${python.sitePackages}/${pname}')/self.binary_name"
-    ''
-    + lib.optionalString torch.cudaSupport ''
-      substituteInPlace bitsandbytes/cuda_setup/main.py  \
-        --replace "/usr/local/cuda/lib64" "${cuda-native-redist}/lib"
-    '';
+  # By default, which library is loaded depends on the result of `torch.cuda.is_available()`.
+  # When `cudaSupport` is enabled, bypass this check and load the cuda library unconditionnally.
+  # Indeed, in this case, only `libbitsandbytes_cuda124.so` is built. `libbitsandbytes_cpu.so` is not.
+  # Also, hardcode the path to the previously built library instead of relying on


Wow this really sounds like a bug in bitsanbytes, I'm surprised they're not reacting to the issue

FWIW on our side we could actually make their presumably-inteded behaviour work, only at double the cost: we could just make the cuda-version depend on the cpu-version.

The question is, is the xxxxx_cuXXX.so library usable in absence of GPUs?

I think that this current approach is fine. If you set cudaSupport = true, it means that you want to use cuda.

The question is, is the xxxxx_cuXXX.so library usable in absence of GPUs?

I don't think so.

I think that this current approach is fine. If you set cudaSupport = true, it means that you want to use cuda.

Well the upstream has implemented this (broken?) dynamic dispatching logic so I guess they intended it the other way. Anyway, implementing that logic from scratch is not in-scope here

Well the upstream has implemented this (broken?) dynamic dispatching logic so I guess they intended it the other way. Anyway, implementing that logic from scratch is not in-scope here

Yes, indeed.

pkgs/development/python-modules/bitsandbytes/default.nix

SomeoneSerge · 2024-10-29T19:37:27Z

pkgs/development/python-modules/bitsandbytes/default.nix

+    (lib.getLib libcusparse)
+    libcusparse.lib


drop the last? Btw getDev libcusparse has .lib in propagatedBuildInputs, I forget if that matters to symlinkJoin

Oh, well the last line libcusparse.lib is clearly a mistake and is not needing because I have added lib.getLib libcusparse above.
Now, having said that, you suggest also getting rid of that and only keeping lib.getDev libcusparse right ?

It works with lib.getDev libcusparse alone.

pkgs/development/python-modules/bitsandbytes/default.nix

Diff: bitsandbytes-foundation/bitsandbytes@refs/tags/0.43.1...0.44.1 Changelog: https://github.com/TimDettmers/bitsandbytes/releases/tag/0.44.1

GaetanLepage · 2024-10-30T09:09:57Z

Ok I was able to make it build in the end.
However, I am not sure that the cuda-native-redist and cuda-common-redist are as lean as they can be.
Can you double check my changes @SomeoneSerge please ?

SomeoneSerge · 2024-10-30T14:32:50Z

However, I am not sure that the cuda-native-redist and cuda-common-redist are as lean as they can be.

That matters fairly little when using symlinkJoin: the closure will be huge regardless.

GaetanLepage · 2024-10-30T14:51:48Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 343246

`x86_64-linux`

✅ 4 packages built:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`aarch64-linux`

✅ 4 packages built:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`x86_64-darwin`

⏩ 4 packages marked as broken and skipped:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

`aarch64-darwin`

⏩ 4 packages marked as broken and skipped:

python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python312Packages.bitsandbytes
python312Packages.bitsandbytes.dist

GaetanLepage marked this pull request as draft September 20, 2024 11:28

github-actions bot added the 6.topic: python label Sep 20, 2024

GaetanLepage force-pushed the bitsandbytes branch from d8c2f05 to 2af0c9e Compare September 20, 2024 11:33

GaetanLepage requested review from bcdarwin and SomeoneSerge September 20, 2024 11:33

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Sep 20, 2024

GaetanLepage force-pushed the bitsandbytes branch 2 times, most recently from 6d92bff to eee442b Compare October 20, 2024 16:40

GaetanLepage marked this pull request as ready for review October 20, 2024 16:47

GaetanLepage requested a review from samuela October 25, 2024 20:05

samuela approved these changes Oct 26, 2024

View reviewed changes

GaetanLepage force-pushed the bitsandbytes branch 2 times, most recently from d7f6384 to b1037b5 Compare October 26, 2024 10:27

GaetanLepage requested review from samuela and SuperSandro2000 October 26, 2024 13:52

SomeoneSerge reviewed Oct 27, 2024

View reviewed changes

pkgs/development/python-modules/bitsandbytes/default.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Oct 27, 2024

View reviewed changes

pkgs/development/python-modules/bitsandbytes/default.nix Show resolved Hide resolved

SomeoneSerge reviewed Oct 27, 2024

View reviewed changes

pkgs/development/python-modules/bitsandbytes/default.nix Show resolved Hide resolved

SomeoneSerge reviewed Oct 27, 2024

View reviewed changes

GaetanLepage force-pushed the bitsandbytes branch from b1037b5 to bf85561 Compare October 27, 2024 15:40

GaetanLepage force-pushed the bitsandbytes branch from bf85561 to 768730d Compare October 27, 2024 19:07

samuela reviewed Oct 27, 2024

View reviewed changes

pkgs/development/python-modules/bitsandbytes/default.nix Show resolved Hide resolved

SomeoneSerge reviewed Oct 29, 2024

View reviewed changes

pkgs/development/python-modules/bitsandbytes/default.nix Outdated Show resolved Hide resolved

SomeoneSerge approved these changes Oct 29, 2024

View reviewed changes

GaetanLepage force-pushed the bitsandbytes branch from 768730d to 08566de Compare October 29, 2024 22:34

python312Packages.bitsandbytes: 0.43.1 -> 0.44.1

4c4b30e

Diff: bitsandbytes-foundation/bitsandbytes@refs/tags/0.43.1...0.44.1 Changelog: https://github.com/TimDettmers/bitsandbytes/releases/tag/0.44.1

GaetanLepage force-pushed the bitsandbytes branch from 08566de to 4c4b30e Compare October 30, 2024 09:09

SomeoneSerge merged commit 67858a3 into NixOS:master Oct 30, 2024
27 of 28 checks passed

GaetanLepage deleted the bitsandbytes branch October 30, 2024 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python312Packages.bitsandbytes: 0.43.1 -> 0.43.3 #343246

python312Packages.bitsandbytes: 0.43.1 -> 0.43.3 #343246

GaetanLepage commented Sep 20, 2024 •

edited

Loading

remexre commented Oct 20, 2024

GaetanLepage commented Oct 20, 2024

GaetanLepage commented Oct 20, 2024

GaetanLepage commented Oct 20, 2024

samuela left a comment

samuela commented Oct 26, 2024

GaetanLepage commented Oct 26, 2024 •

edited

Loading

GaetanLepage commented Oct 26, 2024

SomeoneSerge Oct 27, 2024 •

edited

Loading

GaetanLepage Oct 27, 2024

SomeoneSerge Oct 27, 2024 •

edited

Loading

GaetanLepage Oct 27, 2024

SomeoneSerge Oct 27, 2024

SomeoneSerge Oct 27, 2024

GaetanLepage Oct 27, 2024

SomeoneSerge Oct 29, 2024

GaetanLepage Oct 29, 2024

SomeoneSerge Oct 29, 2024

GaetanLepage Oct 29, 2024

GaetanLepage Oct 29, 2024

GaetanLepage commented Oct 30, 2024

SomeoneSerge commented Oct 30, 2024

GaetanLepage commented Oct 30, 2024

python312Packages.bitsandbytes: 0.43.1 -> 0.43.3 #343246

python312Packages.bitsandbytes: 0.43.1 -> 0.43.3 #343246

Conversation

GaetanLepage commented Sep 20, 2024 • edited Loading

Description of changes

Things done

remexre commented Oct 20, 2024

GaetanLepage commented Oct 20, 2024

GaetanLepage commented Oct 20, 2024

GaetanLepage commented Oct 20, 2024

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

samuela left a comment

Choose a reason for hiding this comment

samuela commented Oct 26, 2024

GaetanLepage commented Oct 26, 2024 • edited Loading

Problem

Solution

Result (with cudaSupport = true)

GaetanLepage commented Oct 26, 2024

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

SomeoneSerge Oct 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SomeoneSerge Oct 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GaetanLepage commented Oct 30, 2024

SomeoneSerge commented Oct 30, 2024

GaetanLepage commented Oct 30, 2024

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

GaetanLepage commented Sep 20, 2024 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`

GaetanLepage commented Oct 26, 2024 •

edited

Loading

Result (with `cudaSupport = true`)

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`

SomeoneSerge Oct 27, 2024 •

edited

Loading

SomeoneSerge Oct 27, 2024 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`