Reuse Python bindings defined in one crate in another one with Python bindgings

[lycantropos]

TL/DR: is it possible to reuse pyo3 classes exposed in one library in another one?

like having one Rust crate with python bindings and after that use it as a dependency in another crate (which will have its own Python bindings) and reuse Python classes from the first one in it?

Here is the project with README to reproduce the error
https://github.com/lycantropos/pyo3-reuse-bindings

So how to do that? By some sort of linkage with installed dynamic library in site-packages?

[adamreichold]

I think the problem is that both extensions are separate DSO and therefore do not share their static variables. PyO3 uses static variables to cache type objects with a stable identify which is then compared in e.g. PyTypeInfo::is_exact_type_of.

This would imply that re-use is limited to statically linked Rust code (and possibly re-exporting Python end points), but not re-use as separately linked extensions.

(I am not sure if it would be viable to have the loader deduplicate these statics but it would certainly be complicated by the possibility of incompatible versions of PyO3 being used by different extension modules.)

[lycantropos]

is it documented somewhere? I see this file, but it's not that easy to understand to be honest

[adamreichold]

is it documented somewhere? I see this file, but it's not that easy to understand to be honest

The capsule API itself is documented here and the general pattern is worked out here.

I am also not sure whether this is really worth it for you (as I do not know the actual use case) as this is about maintaining a public API of base to be consumed by other native extensions and would be limited to a C-compatible FFI API surface. (For the same reasons why your two Rust-built extensions do not share statics: Rust has no stable ABI and links everything statically. Put differently, that both of your extensions are built using the same rustc and pyo3 versions is incidental from an API/ABI perspective.)

I suspect you might be better off either limiting the code re-use to the Rust side or limiting usage of base in derived to the Python-level API of base, e.g. using PyAny::setattr instead of accessing the Rust-level internals of MyClass.

[adamreichold]

Ah, something that might be good to make explicit: PyO3 does not really help you with such a capsule-based API besides wrapping the capsule API itself. It does not currently provide abstraction to provide or consume such an API. (I would say mainly because such an API is limited to C vocabulary by construction.)

[lycantropos]

I will stick to the simpler &PyAny + getattr/setattr approach for now then, it should work for my needs (working with ints, Fractions and floats), thanks

[adamreichold]

I will stick to the simpler &PyAny + getattr/setattr approach for now then,

Also have a look at the intern! macro introduced in PyO3 0.16.4 which might help you to get better performance out of an setattr/getattr-oriented approach.

[ChristopherRabotin]

I just found the following solution: use cargo vendor to vendor the other crate that has a python defined module. Then, in the definition of the module in the new crate, you'll need to make sure to import all of the classes from the other crate that you want.

As a more concrete example, I want to "re-export" the hifitime Python classes from a new demo project called pyarrowrs. The Python stuff in hifitime lives behind the python crate feature. So I need to make sure to enable that. I also need to make sure that the version of pyo3 is the same for both hifitime and my new project.

Note: none of what's here requires changing anything in the external crates.

Hence, after running cargo vendor, I've updated my Cargo.toml file for pyarrowrs to the following:

[package]
name = "pyarrowrs"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "pyarrowrs"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.17.3", features = ["extension-module"]}
hifitime = {version = "3.8.0", features = ["python"]}  # Enables the Python bindings in hifitime

Now, in lib.rs, I can just import the "prelude" of hifitime.

use hifitime::prelude::*;
use pyo3::prelude::*;

#[derive(Clone, Copy, Debug)]
#[repr(C)]
#[pyclass]
pub struct CustomDuration {
    pub(crate) centuries: i16,
    pub(crate) nanoseconds: u64,
}

#[pymodule]
fn pyarrowrs(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<Epoch>()?;
    m.add_class::<TimeScale>()?;
    m.add_class::<TimeSeries>()?;
    m.add_class::<Duration>()?;
    m.add_class::<Unit>()?;
    m.add_class::<CustomDuration>()?; // This is the only structure that is defined in this crate, everything else is from `hifitime::prelude`
    Ok(())
}

#[test]
fn test_main() {
    assert_eq!(1 + 1, 2);
}

Build a develop version with maturin:

╰$ maturin develop
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /home/chris/Workspace/pyarrowrs/.venv/bin/python
   Compiling pyo3-build-config v0.17.3
   Compiling pyo3-ffi v0.17.3
   Compiling pyo3 v0.17.3
   Compiling hifitime v3.8.0
   Compiling pyarrowrs v0.1.0 (/home/chris/Workspace/pyarrowrs)
    Finished dev [unoptimized + debuginfo] target(s) in 10.26s
⚠ Warning: Failed to set rpath for /home/chris/Workspace/pyarrowrs/target/debug/libpyarrowrs.so: Failed to execute 'patchelf', did you install it?
📦 Built wheel for CPython 3.10 to /tmp/.tmpooXXgf/pyarrowrs-0.1.0-cp310-cp310-linux_x86_64.whl
🛠  Installed pyarrowrs-0.1.0

And run!

╰$ python         
Python 3.10.9 (main, Dec 08 2022, 14:49:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrowrs
>>> dir(pyarrowrs)
['CustomDuration', 'Duration', 'Epoch', 'TimeScale', 'TimeSeries', 'Unit', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'pyarrowrs']
>>>

As you see, all of the classes from hifitime are also in this module.

[lycantropos]

the problem is if you have 2 separate modules exporting same classes -- how to make them the same class, so that like in your case we can import from hifitime & pyarrowrs so that

>>> import hifitime
>>> import pyarrowrs
>>> hifitime.Duration is pyarrows.Duration
True

[cmpute]

It might worth to check how other library implements this: pybind/pybind11#616, Cython's cimport

[adamreichold]

From a quick glance, it appears that at least the pybind11 example relies on the somwhat stable C++ ABI which is not available with Rust.