Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add discussion of package formats, expanding "Wheel vs Egg" discussion #1397

Merged
merged 2 commits into from
Apr 3, 2024

Conversation

jeanas
Copy link
Contributor

@jeanas jeanas commented Nov 15, 2023

@jeanas
Copy link
Contributor Author

jeanas commented Nov 15, 2023

CI had a spurious failure:

(      contribute: line   77) broken    https://weblate.org/ - HTTPSConnectionPool(host='weblate.org', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8653733950>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

Can it be retried?

@jeanas
Copy link
Contributor Author

jeanas commented Dec 10, 2023

I have updated this PR with latest main and made some minor fixes.

Review would be appreciated.

@jeanas
Copy link
Contributor Author

jeanas commented Dec 19, 2023

Fixed the merge conflict.

Review would still be appreciated 🙂

Copy link
Contributor

@sinoroc sinoroc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Some comments/nitpicks, but they should not block getting this merged

source/glossary.rst Show resolved Hide resolved
source/discussions/package-formats.rst Show resolved Hide resolved
source/discussions/package-formats.rst Show resolved Hide resolved
source/discussions/package-formats.rst Show resolved Hide resolved
@chrysle chrysle added this pull request to the merge queue Apr 3, 2024
Merged via the queue into pypa:main with commit 091e45c Apr 3, 2024
5 checks passed
@chrysle
Copy link
Contributor

chrysle commented Apr 3, 2024

Thank you!

@zahlman
Copy link
Contributor

zahlman commented Nov 5, 2024

I want to dispute a point made in this discussion:

Wheel archives do not include .pyc files. Therefore, when the distribution only contains Python files (i.e. no compiled extensions), and is compatible with Python 2 and 3, it’s possible for a wheel to be “universal”, similar to an sdist.

This differs from the explanation offered for wheels specifically:

Wheel, being an installation format that is intended to work across multiple versions of Python, does not generally include .pyc files.

(Emphasis mine.)

There is nothing in the description there (nor in the original PEP), as far as I can tell, which forbids .pyc files in a wheel. In fact, by my understanding, it's perfectly valid to publish a wheel (as long as the wheel tags specify the Python version, since the bytecode will be specific to that version) which contains only .pyc representations of the Python code; or, indeed, where any Python code included is only a minimal wrapper for a native-code compiled library. (As far as I'm aware, there's no requirement for packages on PyPI to be open-source, either.)

If .egg files were required (I can't recall) to contain .pyc files (or maybe it's just that Setuptools was the only game in town, and it only made eggs with .pyc files?), and were thus necessarily specific to a Python version, this should be clarified (e.g.: "Wheel archives do not have to include .pyc files"). Otherwise the point should be removed entirely, since the distinction doesn't really exist.

@jeanas jeanas deleted the formats-discussion branch November 5, 2024 20:01
@jeanas
Copy link
Contributor Author

jeanas commented Nov 5, 2024

This is a discussion intended to be a helpful explanation for confused users, not a standard. Virtually nobody includes .pycs in wheels in practice (and I don't believe anybody would recommend that, I haven't tested if they actually work in pip), so this wording is fine IMHO.

@zahlman
Copy link
Contributor

zahlman commented Nov 5, 2024

(Sorry if you saw the double-post - I had some kind of glitch with the GitHub submission form.)

I'm pretty sure that wheels are intended to be allowed to include .pyc files based on other wording in PEP 427, even if it isn't common practice (all emphasis mine):

Wheel, being an installation format that is intended to work across multiple versions of Python, does not generally include .pyc files.

(same as in the other PyPA documentation)

All possible entries are hashed, including any generated files such as .pyc files, but not RECORD which cannot contain its own hash.

Wheel is an installation format; egg is importable. Wheel archives do not need to include .pyc and are less tied to a specific Python version or implementation.

If installers like Pip are meant to reject or filter such files then that really ought to be made explicit somewhere. But I would be surprised. Wheel is a "binary" distribution and open-source is not supposed to be a requirement to distribution Python code, to the best of my knowledge. (In my testing, Pip will happily unpack the .pyc files, just like any other files in the archive.)

@ncoghlan
Copy link
Member

ncoghlan commented Nov 8, 2024

Version specific wheels are definitely permitted to contain pyc files. They're usually omitted to make the archive smaller (since they can be losslessly recreated on the target system), but only universal wheels necessarily omit them.

So I think what @jeanas has written here is correct for the most typical case (public distribution on PyPI), but could reasonably be updated (in a new PR) with a footnote (or something along those lines) pointing out the wheel archives are permitted to include compiled Python files, they just usually don't (for the reasons stated).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants