Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

warnings: log_vc_info - unclosed file #5989

Open
oliver-sanders opened this issue Feb 22, 2024 · 5 comments
Open

warnings: log_vc_info - unclosed file #5989

oliver-sanders opened this issue Feb 22, 2024 · 5 comments
Labels
small
Milestone
8.x

Comments

@oliver-sanders
Copy link
Member

Address a Python warning:

cylc-flow/cylc/flow/install_plugins/log_vc_info.py:185: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='UTF-8'>
  continue
ResourceWarning: Enable tracemalloc to get the object allocation traceback
cylc-flow/cylc/flow/install_plugins/log_vc_info.py:185: ResourceWarning: unclosed file <_io.TextIOWrapper name=7 encoding='UTF-8'>
  continue
ResourceWarning: Enable tracemalloc to get the object allocation traceback
cylc-flow/cylc/flow/install_plugins/log_vc_info.py:185: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='UTF-8'>
  continue
ResourceWarning: Enable tracemalloc to get the object allocation traceback
cylc-flow/cylc/flow/install_plugins/log_vc_info.py:185: ResourceWarning: unclosed file <_io.TextIOWrapper name=7 encoding='UTF-8'>
  continue

To see this warning, run the tests or try adding:

import warnings
warnings.filterwarnings('once')
@oliver-sanders oliver-sanders added this to the cylc-8.x milestone Feb 22, 2024
@MetRonnie
Copy link
Member

I've narrowed this down to pipe_poller (by commenting out the first branch here and seeing the warnings disappear):

cylc-flow/cylc/flow/install_plugins/log_vc_info.py

Lines 257 to 260 in 8b2feae

if stdout == PIPE:
out, err = pipe_poller(proc, proc.stdout, proc.stderr)
else:
out, err = proc.communicate()

cylc-flow/cylc/flow/pipe_poller.py

Lines 29 to 30 in 8b2feae

def pipe_poller(proc, *files, chunk_size=4096):
"""Read from a process without hitting buffer issues.

@oliver-sanders
Copy link
Member Author

Good spot, this may have the same underlying cause as #5990

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Mar 18, 2024 •
edited
Loading

Doesn't make sense to me, the pipe_poller does not open any files, it just reads from already open files. It wouldn't make sense for the pipe_poller to close the files, it might not even be provided with all of them.

Is proc.communicate closing proc.out and proc.err? If so, then we just need to add that into the pipe_poller branch. Or possible proc.wait if that also closes the files?

@MetRonnie
Copy link
Member

Is proc.communicate closing proc.out and proc.err?

Yes looks like it. When it reaches the end of the file it closes it - https://github.com/python/cpython/blob/ecd9946c9d1df7cd1aa15d08d72bbcc0899d272d/Lib/subprocess.py#L1750-L1754

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Mar 18, 2024 •
edited
Loading

Ok, then we can bung a proc.communicate after or possibly even inside the proc_poller bit to resolve this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
small
Projects
None yet
Milestone
8.x
Development

No branches or pull requests
2 participants
@oliver-sanders @MetRonnie