-
Notifications
You must be signed in to change notification settings - Fork 883
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Canonical guide to specifying dependencies? #685
Comments
I also stumbled this weekend on the issue that basically nowhere in our docs we describe why you want to use a My current assumption is: nothing really changed except that you put your (mostly unpinned) dependencies to On top of that, for "deployable applications" (for lack of a better term), you still want to maintain a separate If that's correct, I could write a document explaining that in a nice way. |
That's correct, yes. Extracting the essential argument from https://caremad.io/posts/2013/07/setup-vs-requirement/ and adapting it to clearly reflect the difference between abstract vs concrete requirements in a world where we have pyproject.toml is worthwhile IMO. |
After reading install_requires vs requirements files Discussion and the abstract vs. concrete argument, I agree that there is a difference in how libraries and "deployable applications" should pin versions. However, I find that many of the stated reasons to justify separate If a "deployable application" cares only about concrete dependencies it would be nice to avoid the duplication. What are the downsides of defining concrete (pinned) dependencies exclusively in a |
I agree that the content of requirements.txt should be moved to pyproject.toml Either as an extra section or with a way to specify both, non-pinned and pinned versions for a dependency, e.g. dependencies = [ Also see and |
I have been stuck for this issue for a little long. |
What your deployable app usually cares about is not just pinned deps. Well, partially yes. But what you typically need are these things:
You can generate such constraint files using pip-tools (pip-compile) and they are supported by pip natively. What many people don't realize, though, is that we typically have different envs where we need those constraints and some envs may need additional deps. And what we're trying to make reproducible is not necessarily just an app but all the (virtual)envs where it runs. To answer the question regarding the downsides of pinned deps in |
Do you really need two separate sets of dependencies given we already must be relying on semantic versioning to make this work anyways? Just specify the exact version of each dependency you used in your project and when a conflict occurs in someone else's project trying to use your project with the same transitive dependency the tools can try to resolve automatically by using a potentially newer version only if the newer version doesn't change the API (given it's semver). Using |
Agreed that additional clarity could be useful here as specifying dependencies in With the general migration to
Precisely this^ In the cases I was dealing with, the solution was not to duplicate the |
I think the fundamental issue here is Python guidance has historically been to define your dependencies in a flexible way such that upgrades occur automatically. The install_requires docs state:
Having loose requirements is only a good idea if you like your CI build failing suddenly, even if you've made no changes (because some dependency lib released a new version). That is to say, never. Builds being repeatable is priority 1. The "optimization" of using newer versions of libs automatically is asking for trouble. Each version bump should be tested. For security, automatic updates seems most compelling. However, this feature can be handled by automated tooling instead, which can detect security vulnerabilities in dependencies and a bot will even create a pull request for you to test before merging (GitHub does this for you automatically for example). There is one important case when automatic upgrades makes sense, but it is not Pinned versions should be used for both libs and applications, though this only works for libs if the build tooling used by end-users have the flexibility to override for transitive dep conflicts. I think deps should be specified only in one place, probably pyproject.toml, with explicit pinned versions only. Tooling should handle the rest. |
…tion to more closely align with de facto PyPA standards (see discussion: pypa/packaging.python.org#685)
We should probably have a canonical guide to installing dependencies on packaging.python.org, especially since PEP 508 has changed things like how to link to a git repository. I get a lot of questions like, "Why are my dependency links not working now?" and since I don't really use those I can just tell people, "Oh something about that changed, here's an inscrutable PEP that maybe would help"?
This could be done incrementally (and can be broken into smaller parts, providing there is "clearing house" page that links to the other pages), but it would be nice to clearly cover:
requirements.txt
file and what can be declared ininstall_requires
constraints.txt
files (which I think really reduces the need forrequirements.txt
.Some of this is defined in various PEPs, but my understanding is that the canonical and current documentation should live on packaging.python.org.
There should also probably be a "hat note" (Wikipedia terminology) pointing to any such page from managing application dependencies to this page.
The text was updated successfully, but these errors were encountered: