Passing Python function to different Rust thread

[marcpabst]

Hey all, I'm trying to adapt a Rust API for Python using PyO3 but I've hit a bit of a roadblock with threading.

I have an existing Rust API that looks something like this:

pub fn run_experiment<F>(&mut self, window_options: &WindowOptions, experiment_fn: F) -> ()
where
    F: FnOnce(Window) -> Result<(), SomeErrorType>
        + 'static
        + Send,

This function uses winit to create a new EventLoop that blocks the main thread, but crucially runs the experiment_fn function in its own separate thread. How would I make this API available through PyO3? I'd like to mimic the same signature in Python, i.e., doing something like this (wrapping WindowOptions in a wrapper type):

fn run_experiment(
    &mut self,
    window_options: &PyWindowOptions,
    experiment_fn: &PyFunction,
) {
    // create rust FnOnce(Window) -> Result<(), SomeErrorType> + 'static + Send,
    let rust_experiment_fn = move |window: Window| -> Result<
        (),
        SomeErrorType,
    > {
        experiment_fn.call0(); // call without args for now
        Ok(())
    };

    // run the experiment
    self.0.run_experiment(
        &window_options.0,
        rust_experiment_fn,
    );
}

But that gives me a "UnsafeCell<PyObject> cannot be shared between threads safely" error, as &PyFunction is not Send. Any pointers to what would be the best approach to implement this?

[birkenfeld]

You'll have to use GIL-independent objects, i.e. experiment_fn: Py<PyFunction>. In rust_experiment_fn, you then have to acquire the GIL before you can call experiment_fn.

[adamreichold]

Would using my unsafe variant be fine, given that the Rust function is blocking and I know that I will never run any Python code concurrently?

I don't think so because acquiring the GIL also sets some thread-local state, i.e. the main thread has the GIL and not the background thread which cannot run Python code even if the main thread is blocked and cannot do so either.

I suspect you will have to encapsulate things so that self.0 into the closure passed to allow_threads somehow.

[marcpabst]

Reading the docs was actually really helpful :)

this does the trick:

let mut self_wrapper = SendWrapper::new(self);
py.allow_threads(move || {
    self_wrapper
        .0
        .run_experiment(&window_options.0, rust_experiment_fn)
});

[adamreichold]

This is abusing a soundness loophole that we are currently trying to fix.

[davidhewitt]

(#3646 - although it's worth noting that unsafe { py.allow_threads().local_unconstrained() }.with(|| self.0.run_experiment(...)) will probably work, adding merit to that solution.)

[marcpabst]

This is abusing a soundness loophole that we are currently trying to fix.

One could claim that PyO3 is currently abusing Send :) I think I will stick with this for now and move to the new solution once that's stabilised.