Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Notebook execution started hanging in jupyter_client 8 #922

Closed
gibsondan opened this issue Jan 27, 2023 · 5 comments · Fixed by #925
Closed

Notebook execution started hanging in jupyter_client 8 #922

gibsondan opened this issue Jan 27, 2023 · 5 comments · Fixed by #925
Labels
bug

Comments

@gibsondan
Copy link

gibsondan commented Jan 27, 2023 •
edited
Loading

Hello,

Starting last week, a test in the Dagster papermill / jupyter integration started hanging. The hang coincided with a number of jupyter-related releases, but we were able to isolate the problem to the jupyter-client 8 release - downgrading to the previous release made the problem go away.

To reproduce the problem:

  • Check out the dagster github repo at tag 1.1.13, with a python 3.8 venv
  • Install tox
  • In the python_modules/libraries/dagstermill folder, run:
  • tox -vv -e py38 -- -k test_hello_world -x -s

The test will hang while executing a notebook, but the hang will go away if you force the previous version of jupyter-client to be installed instead.

Code in Dagster that executes the notebook: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/factory.py#L194-L213

Console output during the hang (this shows forever):

Executing:  50%|█████     | 1/2 [00:03<00:03,  3.18s/cell]

Stack trace of the notebook while execution is hanging:

Thread 0x10D504600 (idle): "MainThread"
    select (selectors.py:558)
    _run_once (asyncio/base_events.py:1823)
    run_forever (asyncio/base_events.py:570)
    run_until_complete (asyncio/base_events.py:603)
    just_run (nbclient/util.py:57)
    wrapped (nbclient/util.py:78)
    execute (papermill/clientwrap.py:46)
    execute_managed_notebook (dagstermill/engine.py:75)
    execute_notebook (papermill/engines.py:343)
    execute_notebook_with_engine (papermill/engines.py:49)
    execute_notebook (papermill/execute.py:107)
    _t_fn (dagstermill/asset_factory.py:86)
    iterate_with_context (dagster/_utils/__init__.py:457)
    _yield_compute_results (dagster/_core/execution/plan/compute.py:145)
    execute_core_compute (dagster/_core/execution/plan/compute.py:177)
    _step_output_error_checked_user_event_sequence (dagster/_core/execution/plan/execute_step.py:94)
    core_dagster_event_sequence_for_step (dagster/_core/execution/plan/execute_step.py:382)
    dagster_event_sequence_for_step (dagster/_core/execution/plan/execute_plan.py:265)
    inner_plan_execution_iterator (dagster/_core/execution/plan/execute_plan.py:114)
    __iter__ (dagster/_core/execution/api.py:1103)
    execute (dagster/_core/executor/multiprocess.py:93)
    _execute_command_in_child_process (dagster/_core/executor/child_process_executor.py:79)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:315)
    _main (multiprocessing/spawn.py:129)
    spawn_main (multiprocessing/spawn.py:116)
    <module> (<string>:1)

Let me know if any other information would be helpful.
@cscheid
Copy link

cscheid commented Jan 27, 2023

I wonder if this is the same issue that we're seeing with nbclient jupyter_client 8: jupyter/nbclient#272

@edublancas
Copy link

edublancas commented Jan 27, 2023 •
edited
Loading

Our CI also started failing, and we narrowed it down to jupyter_client 8 (we run some of our examples with papermill, which uses jupyter_client).

we didn't try downgrading to jupyter_client 7 since we already built an in-house notebook executor, and migrating to it fixed the issues.

@idomic idomic mentioned this issue Jan 27, 2023
Open
@blink1073
Copy link
Member

Ah, this was a gap in testing for nbclient, because we've been testing it against main as a downstream project.

@blink1073 blink1073 added the bug label Jan 27, 2023
@blink1073
Copy link
Member

I'm closing this in favor of jupyter/nbclient#272.

@blink1073
Copy link
Member

Actually the error is in this repo, I'm investigating in #925

@blink1073 blink1073 reopened this Jan 28, 2023
@blink1073 blink1073 mentioned this issue Jan 28, 2023
Merged
@cscheid cscheid mentioned this issue Jan 29, 2023
Closed
4 tasks
blink1073 added a commit that referenced this issue Jan 30, 2023
@blink1073
Add papermill downstream check and fix kernel client replies (#925)
666eab0 
Fixes #922
This was referenced Jan 30, 2023
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add papermill downstream check and fix kernel client replies blink1073/jupyter_client
4 participants
@cscheid @edublancas @blink1073 @gibsondan