Stable ABI builds and Py_XDECREF #500
-
|
Hey! I wanted to bring this to your attention. I came across this in nicholasjng/nanobind-bazel#20, where I enabled size optimizations by default, and subsequently got an abi3audit failure on Linux. A report can be found e.g. here. The reason why this wasn't raised on Mac (Windows is still failing for me, sigh) is presumably because clang inlines more aggressively, meaning that Python's reference counting APIs, which are all marked I raised this with the abi3audit author in pypa/abi3audit#83, who thinks that it is an abi3 violation to use
If there's anything I can help with, please let me know. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 21 replies
-
|
I don't see this as black and white as @woodruffw. When compiling for the limited API, Python aggressively hides internal data structures and functions. Things that one isn't supposed to use become invisible. Various other functions are emulated in terms of limited API exports. This isn't specific to There are good reasons to prefer these functions: going through the stable ABI has a real cost (many extra function calls in bindings). By using What I suspect is happening is that |
Beta Was this translation helpful? Give feedback.
-
|
One comment specifically to @nicholasjng (orthogonal to the |
Beta Was this translation helpful? Give feedback.
-
|
Is this the whole output of nm -U output, click to expand |
Beta Was this translation helpful? Give feedback.
-
|
Oh yes, you're on to something. For CMake: I'm not quite sure what's up here - I started setting |
Beta Was this translation helpful? Give feedback.
-
|
Maybe you're setting |
Beta Was this translation helpful? Give feedback.
-
|
I'm setting |
Beta Was this translation helpful? Give feedback.
-
|
EDIT: Found it, I was missing |
Beta Was this translation helpful? Give feedback.
-
|
I don't have a Linux machine on hand, but Console output:Does that help? |
Beta Was this translation helpful? Give feedback.
-
This argument was made on pypa/abi3audit#55, and I'm sympathetic to it. Then again, I don't have an official source from CPython stating that this is intentional: many non-stable inlineable functions might inline in stable ABI compatible ways in practice, but I'm not sure that CPython actually guarantees this or wants developers to build on this assumption (versus honoring it for compatibility reasons). At least, that's my read of that PEP discussion. That being said, if someone can find an official CPython source for |
Beta Was this translation helpful? Give feedback.
-
I think the authority here is @vstinner. So here is the question to him, in abbreviated form: When compiling for Is it this officially supported functionality? Or said differently, is it okay for a limited API / stable ABI extension to call |
Beta Was this translation helpful? Give feedback.
-
|
Py_XDECREF() is part of the limited C API and implemented as: static inline void Py_XDECREF(PyObject *op)
{
if (op != _Py_NULL) {
Py_DECREF(op);
}
}It's not part of the stable ABI, since it's compiled as a call to Py_DECREF() which calls _Py_DecRef() at the ABI level. At the end, only I'm not sure of what is your question. As soon as you use the limited C API, the generated binary uses the stable ABI and you're good. In short, Py_XDECREF() is part of the stable ABI, but it's implemented as |
Beta Was this translation helpful? Give feedback.
-
Sort of: the file itself says this:
Moreover, the contents have various attributes for indicating whether a member is a macro (which wouldn't be ABI relevant at all), whether a thing is
This is the ideal solution, but I don't think it's guaranteed to be sound: an optimizing compiler is allowed to perform local copying of functions so long as no addressing assumptions around the function are violated, meaning that there might be a local-only symbol for a non-stable-ABI function that |
Beta Was this translation helpful? Give feedback.
-
|
The Macros and inline functions are, of course, not part of the ABI. The functions that macros call, and struct members that macros access directly, are part of the ABI -- even if they're not part of the API (i.e. they shouldn't appear in your source code directly; for example, private “
Yes, it should. But it's not perfect :( |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @encukou! The tooling limitation is unfortunate but understandable, and I appreciate the additional context 🙂 So, it sounds like |
Beta Was this translation helpful? Give feedback.
-
|
@woodruffw I'm happy to help with implementing any of the resulting features for abi3audit if you're interested (since I kind of kicked off this to begin with). |
Beta Was this translation helpful? Give feedback.
-
|
@nicholasjng that would be great, thank you! I was thinking about this a bit, and I think the main thing it needs is a way to expose symbol visibility during the auditing phase. I'll open an issue on |
Beta Was this translation helpful? Give feedback.
Py_XDECREF() is part of the limited C API and implemented as:
It's not part of the stable ABI, since it's compiled as a call to Py_DECREF() which calls _Py_DecRef() at the ABI level.
At the end, only
_Py_DecRef()is part of the stable ABI.I'm not sure of what is your question. As soon as you use the limited C API, the generated binary uses the stable ABI and you're good.
In short, Py_XDECREF() is part of the stable ABI, but it's implemented as
if (ptr != 0)+ call to_Py_DecRef().