-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow running scripts with dependencies using pipx #913
Comments
Similar request in the past: #562. If we’re doing that I think a comment (or maybe docstring?) would be better than a global |
Thanks for the link. I did a search but didn't spot that request. I agree a comment is better than I'll start work on a PR for this, as the comments on #562 seemed favourable. |
I wrote a reply to @pfmoore in jaraco/pip-run#52 (comment), but I'm following up here so that I can track this discussion :) In that comment, I suggested making a plugin system for For example Conda used to support (at least, from my reading around — I never used it) specifying package dependencies inside a Jupyter Notebook. This meant that Conda could provision an environment in which to run the notebook from the notebook itself. It would be fun to support that with pipx, e.g. Let me be clear, I'm not saying that everyone should go around creating notebook binary applications. But rather, Hatch's plugin system has been fantastically useful as a package author, and I can see how generalising it here would be handy. |
Personally, I think this should start small - let's not over-engineer things to start with. A plugin system would be a far bigger change, and should probably tie in to much more than just this command (plugins for venv creation, for installing dependencies, for Python interpreter discovery, ...) I'm not at all sure anything like that is needed in pipx, and I don't want to start down that route with this change. Making this functionality plugin-based can be done as a follow-up PR1 if people are interested.
I personally hate that style. It would also be incredibly complex to parse (particularly in combination with the other annotations) and would mean that the information could be scattered throughout the code (imports can go anywhere). It would also suggest that a construct like
would only install colorama on Windows, which isn't something we can implement. So no, I'm a strong -1 on this idea. Footnotes
|
Right, there are two sides to this; the end user experience, and the means of actually delivering something from a development perspective. With that in mind
Agreed. I think using comments would be a better solution than requiring the definition to be valid Python, though. It would be nice if tools that are provisioning the environment didn't need to evaluate Python code (even via # dependencies = ["numpy"]
import numpy I note that your original comment suggests a custom syntax, which I'm also neutral on. The benefit of using TOML is that we don't need any additional parsing logic beyond import tomllib
leading_comments = []
for line in file:
if not leading_comments and line.startswith("#!"):
continue
if not line.startswith("#"):
break
leading_comments.append(line.lstrip("#"))
try:
metadata = tomllib.loads("\n".join(leading_comments))
except ...:
return
dependencies = metadata['dependencies'] Another thought that I've had is that there could be scope for this to work with shebangs, e.g. #!/usr/bin/env pipx-run
# dependencies = ["numpy"]
import numpy This would probably require a distinct entrypoint in order to pass the appropriate arguments to `pipx, but it would be fantastic to be able to implement executables from Python scripts like this. |
Parsing: I assumed Good point about the OS check. Using the normal environment markers would seem reasonable:
But was just an idea. Keeping things simple for sure is a good approach in general. 👍 |
I think we're talking past each other here. What I am proposing (see my original post) is to recognise a block of comment lines in the source code. The first line must be The requirement specifiers will be passed to pip to install. There is no standard for embedding dependency specifications in a Python source code file. If there were, I'd expect pipx to follow that. But in the absence of a standard, this is purely a syntax recognised by pipx, and while I'm happy to make it easy for other tools to parse, that's as far as it seems worth going to me. I will note that pip-run defines embedded requirements using a |
Probably, seems to happen fairly easily on these kinds of forums :)
Agreed; permissive where sensible.
Right, and I agree with the spirit of this :) I'm suggesting that imposing a constraint like "this needs to be valid markup for X" where "X" already has a parser means that one can be very explicit about what is supported. If we don't need to impose our own format, doing something like TOML, YAML (no stdlib parser), JSON, or INI would mean that we don't automatically make it harder for other tools to read this block without good reason. If |
OK, I see what you're getting at. But we don't need anything more than a list of strings here. And we have to remove the I'll be frank here - reading the requirements from the script is the least of the difficulties here, so I really don't want to spend much time debating a syntax. I'm going to go with what I stated above for now. We can experiment with alternative syntaxes in future PRs - or if my initial approach turns out to be too simplistic. |
Life's too short to beat around the bush ;) It was fun discussing this, good luck with your efforts! |
Thanks for your comments, they made me think about some issues that I might not have spotted otherwise (validating requirements) so the PR will definitely be better for this conversation! |
I think Paul's suggestion is good enough as any language can parse the proposal out via a regex and/or some simple string searching, none of which is Python-specific. Effectively if Paul's proposal can be parsed out with nothing more than the |
Yes, I see the arguments against scope creep. I was thinking about embedding a subset of Given that I can name only one (two?) additional metadata fields that are truly useful, I don't think I can make a strong case for embedding I ended up making a PoC of what I was thinking of here. It's terrible code (a real hack job), but for me it cemented the benefit of being able to write I'm quite excited about this; |
I have a working proof of concept. There's one significant issue I need to resolve before submitting an initial PR, which is that pipx assumes that an environment will have a "main package" (this is checked in I think I need to refactor to allow (temporary) environments without a main package, but I'd appreciate any insights the @pypa/pipx-committers (or anyone else!) might have on how deeply embedded the assumption is that an environment has a main package. I'll do the research myself, but any pointers on where to look for potential problems would be very helpful! |
FYI I'm watching this closely as I have been toying with the idea of a |
You may be wondering, like I did, what shebang line to use in a pipx script. Here are two options. If your system supports
If your system doesn't support
(The space after |
That's handy, I didn't know about If the script is named
So the shebang that you suggested wouldn't be enough, you would still need to add the |
Both shebang lines work for me with pipx 1.3.3. You can test them using the following examples. The #! /usr/bin/env -S pipx run
print("Hey") #! /bin/sh
"exec" "/usr/bin/env" "python3" "-m" "pipx" "run" "$0"
print("Hey") |
I was running pipx 1.2.0 (from Ubuntu 23.10's repositories). It looks like the newer versions of pipx do not require the |
Oh, I see. I didn't realize this was the case because I didn't try to run scripts with pipx before #916. You're welcome! |
For those who found this issue in their favourite search engine: PEP 723 was accepted as final, and the Python Packaging User Guide now documents the official spec for inline script metadata. It's slightly different from the initial syntax in the first post of this GitHub issue. It looks like this:
The newer versions of pipx support this metadata. You can run |
@gaborbernat I'm not sure why this GitHub issue was reopened. I only meant to provide a news update to those who came across this issue, I didn't mean anyone to reopen this issue. |
How would this feature be useful?
At the moment, to run a Python script that depends on 3rd party packages, it is necessary to manually create a virtual environment, populate it with the dependencies, and then run the script. And then delete the environment (or keep it if you think you're likely to re-run the script and remember to delete it later).
pipx run
allows you to run packages in a temporary venv with all dependencies set up, but it doesn't work for scripts. It allows you to run scripts, but that doesn't use any of the venv mechanisms, just running the script with the system Python interpreter.Describe the solution you'd like
If a script needs dependencies, use a temporary virtualenv, the same as for packages, and run the script in that.
To determine if a script needs dependencies, a very simplistic approach would be to check if
--pip-args
was specified. So invocation would be something like:Even better would be to parse the script, looking for an embedded list of dependencies to install. This would require agreeing on a format for the list. My initial proposal would be something simple, like:
The specifiers could be any requirement spec acceptable to pip, so
numpy
, orclick>=7.0
, or even a URL.Describe alternatives you've considered
The
pip-run
command offers similar functionality, but it recreates the environment every time, which makes for very slow runtimes. It also has (IMO) a clumsy UI for specifying the packages to install and the script invocation.Possible Issues
Script dependencies could potentially change more frequently than package dependencies do. So there's a potential for the cached environment to no longer match the declared requirements. This is also a problem with a package, though, so it's possibly something we can live with. Maybe having an option to re-create the cached environment would be enough to alleviate this issue?
Implementation
I would be happy to create a PR implementing this feature, if the view is that it would be a good idea.
The text was updated successfully, but these errors were encountered: