Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python kernels in the system prefix without an absolute interpretor path fail to launch if started by jupyter_client in a virtual environment #949

Open
joouha opened this issue May 24, 2023 · 4 comments

Comments

@joouha
Copy link
Contributor

joouha commented May 24, 2023

I've received several reports from users where a Python kernel fails to launch when started by a Jupyter client running in a virtual environment (e.g. joouha/euporie#25, joouha/euporie#75), and I've tracked down the cause to these lines of code in jupyter_client.


Let's say I have ipykernel installed in my system Python environment, and I want to run this kernel from a client (which uses jupyter_client) installed in a virtual environment:

$ cat /usr/share/jupyter/kernels/python3/kernel.json
{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

$ python -m venv venv

$ ./venv/bin/pip install jupyter-console

$ ./venv/bin/pip uninstall ipykernel 

$ ./venv/bin/jupyter-console --kernel=python3

/home/josiah/venv/bin/python: No module named ipykernel_launcher

jupyter_client replaces the python command in the kernelspec argv line with the path to the interpreter on which jupyter_client is running. This is implemented here:

https://github.com/jupyter/jupyter_client/blob/ed7f9c27b0ae054e2b87d581f6252d240f2e6686/jupyter_client/manager.py#LL273C1-L284C36

I'm really not sure this is expected or desirable behaviour. There is a comment stating the following:

if the current process is in an env and has been launched by abspath without activating the env, python on PATH may not be sys.executable, but it should be.

but I don't think this assumption is correct, and I don't understand what problem was this originally implemented to solve.

@joouha
Copy link
Contributor Author

joouha commented May 26, 2023

This is also relevant:

https://github.com/ipython/ipykernel/blob/f7e5a8fb5c37d81c55b9ea785168c8a19d110eb4/hatch_build.py#L23

When the ipykernel wheel is built, the packaged kernelspec has just python as the first argument of the kernel launch command.

If the user installs the kernel themselves (i.e. python -m ipykernel install --user, the kernelspec is created with an absolute interpreter path.

@fecet
Copy link
Contributor

fecet commented Jun 26, 2023

You have to add envs manually like

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "env":{
     "PATH": PATH after activate venv.
 },
 "display_name": "Python 3.10 (ddip)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

@joouha
Copy link
Contributor Author

joouha commented Jun 29, 2023

Thanks for your reply, but I think you have misunderstood this issue I described:

I'm writing a new Jupyter client which I want users to run from a virtual environment (using pipx). My client does not depend on ipykernel.

Typically the user will have ipykernel (and a corresponding kernelspec) installed in their system prefix.

When the user tries to launch a kernel from the system kernelspec it fails, because jupyter_client incorrectly edits the kernelspec, substituting the python argv command with the absolute path of the interpreter in the client's venv.

Since ipykernel is not installed in the pipx venv, launching the kernel fails.

@joouha
Copy link
Contributor Author

joouha commented Jul 25, 2023

I think in the case of a non-absolute python command, cmd[0] should be set to sys._base_executable, unless the client is running in the same prefix that the connection_file is located, in which case cmd[0] could reasonably be set to sys.executable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants