Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Symlinking for log/job #6168

Merged
merged 3 commits into from
Jul 31, 2024
Merged

Allow Symlinking for log/job #6168

merged 3 commits into from
Jul 31, 2024

Conversation

wxtim
Copy link
Member

@wxtim wxtim commented Jun 25, 2024 •
edited
Loading

closes #6145

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at Allow symlinking for log/job cylc-doc#748.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@wxtim wxtim mentioned this pull request Jun 25, 2024
Open
1 task
@wxtim wxtim changed the title Allow the Symlinking for log/job Allow Symlinking for log/job Jun 26, 2024
@wxtim wxtim self-assigned this Jun 26, 2024
@wxtim wxtim added this to the 8.4.0 milestone Jun 26, 2024
@wxtim wxtim added the config change Involves a change to global or workflow config label Jun 26, 2024
oliver-sanders
oliver-sanders requested changes Jun 26, 2024
View reviewed changes
Copy link
Member

@oliver-sanders oliver-sanders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cylc clean needs to be updated to reflect this change.

Try this config:

[install]
    [[symlink dirs]]
        [[[localhost]]]
            log = $SCRATCH
            log/job = $DATADIR
$ cylc vip x
$ sleep 10
$ cylc stop x
$ cylc clean x -y
$ tree $HOME/cylc-run/x $SCRATCH/cylc-run/x $DATADIR/cylc-run/x

I get the log/job directory left behind on $DATADIR with this configuration.

Also worth trying this out with remote platforms to make sure clean works in that context too.

@wxtim
Copy link
Member Author

wxtim commented Jun 27, 2024

cylc clean needs to be updated to reflect this change.

Mildly irked that there isn't a single source of truth about the symlink dirs, but I ended up with circular import issues trying to do that.

Also worth trying this out with remote platforms to make sure clean works in that context too.

Yes - have done.

@oliver-sanders
Copy link
Member

Mildly irked that there isn't a single source of truth about the symlink dirs

There was before!

Try moving your new dict in with the previous one. Circular import issues can be solved.

@oliver-sanders
Copy link
Member

Worth looking at testing cylc clean with log/job symlink dirs setup.

I would have thought tests/functional/cylc-clean/00-basic.t would have noticed the dirs left behind by cylc clean, but it didn't seem to?

oliver-sanders
oliver-sanders reviewed Jul 8, 2024
View reviewed changes
tests/functional/cylc-clean/00-basic.t Outdated Show resolved Hide resolved
cylc/flow/workflow_files.py Outdated Show resolved Hide resolved
@wxtim wxtim marked this pull request as draft July 19, 2024 14:22
@wxtim
Allow the Symlinking for log/job
3ff5b4e 
response to review: ensure cylc clean works.
@wxtim
Provided a single source of truth for symlinkable dirs
44e846a 
Ensure that test actually creates symlinks to be cleaned.
@wxtim wxtim marked this pull request as ready for review July 22, 2024 14:38
oliver-sanders
oliver-sanders approved these changes Jul 31, 2024
View reviewed changes
Copy link
Member

@oliver-sanders oliver-sanders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

hjoliver
hjoliver approved these changes Jul 31, 2024
View reviewed changes
Copy link
Member

@hjoliver hjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

@hjoliver hjoliver merged commit 2143d66 into cylc:master Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hjoliver hjoliver hjoliver approved these changes

@oliver-sanders oliver-sanders oliver-sanders approved these changes

@markgrahamdawson markgrahamdawson Awaiting requested review from markgrahamdawson

Assignees

@wxtim wxtim

Labels
config change Involves a change to global or workflow config
Projects
None yet
Milestone
8.4.0
Development

Successfully merging this pull request may close these issues.

symlink dirs: support symlinking log/job
3 participants
@wxtim @oliver-sanders @hjoliver