-
-
Notifications
You must be signed in to change notification settings - Fork 29.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C API] Make PyInterpreterState opaque: move it into the internal C API (internal/pycore_pystate.h) #80067
Comments
In November Victor created the Include/cpython directory and moved a decent amount of public (but not limited) API there. This included the PyInterpreterState struct. I'd like to move it into the "internal" headers since it is somewhat coupled to the internal runtime implementation. The alternative is extra complexity. I ran into this while working on another issue. Note that the docs indicate that all of the struct's fields are private (and I am not aware of any macros leaking fields into the stable ABI). So moving it should not break anything (yeah, right!), and certainly not the stable ABI (Victor's move would have done that already). |
FWIW, I was hoping to the same for PyThreadState but it looks like 6 of its fields are exposed in "stable" header files via the following macros: # Include/object.h I'm not sure how that factors into the stable ABI (PyThreadState wasn't ever guarded by Py_LIMITED_API). However, I didn't need to deal with it right now so I'm not going to go there. :) |
@victor, do you see any problems with doing this? It will help simplify other changes I'm working on. |
I'm quite sure that they are users of the PyInterpreterState structure outside CPython internals and stdlib, but I expect that the number is quite low. Since internal headers are now installed (I modified "make install" for that) (but need to define Py_BUILD_CORE), it might be acceptable to force users of this structure to opt-in for internal headers. Just make sure that we properly communicate on such backward incompatible changes:
The bpo-35810 also proposes a subtle backward incompatible change which makes me unhappy, but Stefan Behnel seems less scared than me, so maybe it will be fine. Maybe we need to organize a collective effort to better communicate on our backward incompatible C API changes. The capi-sig mailing list may be a good channel for that. I asked to test some popular C extensions to check that they are not broken. If it's the case, we should help them to be prepared for the new C API. |
Thanks, Victor! python-dev: https://mail.python.org/pipermail/python-dev/2019-February/156344.html Also, my PR already has a What's New entry in the porting section. :) |
This broke cffi:
I haven't investigated further but heads up. |
Just so you know, when we look at the changes to CPython, the easiest fix is to add these lines in cffi: #if PY_VERSION_HEX >= 0x03080000
# define Py_BUILD_CORE
# include "internal/pycore_pystate.h"
# undef Py_BUILD_CORE
#endif But if we're looking for a cleaner way to fix this, then cffi's needs could be covered by adding a general-purpose-storage dict to PyInterpreterState (like PyThreadState->dict, but per sub-interpreter instead of per thread), and an API function to get it. |
Well, that's not a surprise :-) It's a deliberate backward incompatible change. But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues. Can someone report the issue to cffi and post back the url to the bug here? |
The cffi issue is: https://bitbucket.org/cffi/cffi/issues/403/build-fails-on-38-dev-pyinterpreterstate Armin already went ahead and committed the change to make cffi include internal/pycore_pystate.h. I guess this might not be the ideal final outcome though :-). Without some fix though it's very difficult to test things on 3.8-dev, because cffi shows up in the dependency stacks of *large* portions of the ecosystem. |
@armin, thanks for fixing things on your end so quickly. :) Adding PyInterpreterState.dict and a public getter function (a la PyThreadState) is probably fine. I'm just not sure how that relates to the problem with cffi's use of the "builtins" field. If it would be useful in general then feel free to open a separate issue. I doubt there'd be much opposition if you presented a decent use case, since the cost would be low. In the case of PyInterpreterState.builtins specifically, if you need it we could add a simple PyInterpreterState_GetBuiltins(). However, couldn't you already use existing public API, e.g. |
On Mon, Feb 25, 2019 at 3:02 AM STINNER Victor <[email protected]> wrote:
Yeah, that was my bad. I jumped the gun a bit. |
Next incompatibility: python-hyper/brotlicffi#147 (which indirectly broke httpbin) |
(On closer inspection, that's actually be the same breakage as already mentioned above) However, what I'm not clear on is how this would affect projects that had *already* generated their cffi code, and include that in their sdist. Are all those sdists going to fail to build on Python 3.8 now? It's OK to require that extension modules be rebuilt for a new release, but breaking compatibility with a *code generator* that means a broad selection of projects are all going to fail to build is a much bigger problem. |
@nick which indirectly broke httpbin and this one is used by python-requests and we can't execute the tests of requests. |
@nick the C sources produced by cffi don't change. When they are compiled, they use Py_LIMITED_API so you can continue using a single compiled module version for any recent-enough Python 3.x. The required fix is only inside the cffi module itself. |
Oh, cool (both the fact the issue here is only with building cffi itself, and that cffi creates extension modules that build with PY_LIMITED_API). I've filed https://bugs.python.org/issue36124 to follow up on the PyInterpreter_GetDict API idea. |
Cool. Also, no bugfix release of cffi was planned, but I can make one if you think it might help for testing the current pre-release of CPython 3.8. |
Yep, I am interested because I would like to execute the tests of the major projects/libraries (django, flasks, pandas, requests, ...) and create issues for the maintainer. the sooner we get feedback, the sooner we can fix the bugs. |
Done. |
This change broke at least two projects:
It would be nice to document replacement solutions at:
|
Note: IMHO the "The PyInterpreterState struct has been moved into the “internal” header files (...)" entry should be documented in the "Changes in the C API" section, rather than in the "Changes in the Python API" section of: |
1 to the suggested "whatsnew" updates. If no one beats me to it I'll work on a PR soon. |
I created bpo-39947: "[C API] Make the PyThreadState structure opaque (move it to the internal C API)". |
Python 3.8 was released in October 2019 with this change (PyInterpreterState structure is now opaque). Fedora 32 is going to be released soon with Python 3.8 as /usr/bin/python. The change only broke very few projets: cffi (which indirectly broke brotlipy and httpbin), Blender, FreeBSD. Fixes are trivial:
I close the issue. Well done Eric! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: