Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix localhost platform matching #5343

Merged
merged 8 commits into from
Apr 16, 2023

Conversation

wxtim
Copy link
Member

@wxtim wxtim commented Feb 2, 2023

Closes #5342

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg and conda-environment.yml.
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@wxtim wxtim self-assigned this Feb 2, 2023
@wxtim wxtim changed the base branch from master to 8.1.x February 2, 2023 10:52
@wxtim wxtim force-pushed the fix_localhost_platform_matching branch from 954418d to 92c0242 Compare February 2, 2023 10:54
@wxtim wxtim added the bug Something is wrong :( label Feb 2, 2023
@wxtim wxtim added this to the cylc-8.1.2 milestone Feb 2, 2023
@wxtim wxtim force-pushed the fix_localhost_platform_matching branch from 92c0242 to 44a57fa Compare February 2, 2023 12:47
Copy link
Contributor

@datamel datamel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hasn't fixed the problem for me unfortunately. Here is my test:

# global.cylc
[platforms]
    [[localhost]]
    [[localhost_test]]
        hosts = <a remote platform>
        install target = <remote platform install target>

The play a workflow e.g:

# flow.cylc
[scheduling]
    [[graph]]
        R1 = """
        hello => goodbye
    """

[runtime]
    [[root]]
        script = "sleep 1"
        platform = localhost_test
    [[hello]]
    [[goodbye]]

hello and goodbye both run locally, not respecting the settings in global.cylc.

As suggested, changing localhost_test to localhxst_test works fine.
I'll have a look to see if I can spot the problem.

@wxtim
Copy link
Member Author

wxtim commented Feb 15, 2023

This hasn't fixed the problem for me unfortunately. Here is my test:

# global.cylc
[platforms]
    [[localhost]]
    [[localhost_test]]
        hosts = <a remote platform>
        install target = <remote platform install target>

The play a workflow e.g:

# flow.cylc
[scheduling]
    [[graph]]
        R1 = """
        hello => goodbye
    """

[runtime]
    [[root]]
        script = "sleep 1"
        platform = localhost_test
    [[hello]]
    [[goodbye]]

hello and goodbye both run locally, not respecting the settings in global.cylc.

As suggested, changing localhost_test to localhxst_test works fine. I'll have a look to see if I can spot the problem.

I can't reproduce this.

@oliver-sanders
Copy link
Member

There are other uses of this interface with host_check=True so this error might be able to crop up in other places?

I'm not sure I understand the logic of host_check in subshell_eval, seems a little strange:

if host_check is True:
if is_remote_host(command):
return command
else:
return 'localhost'
else:
return command

Return localhost if the host is localhost???

I would have thought the subshell_eval should look something like this:

match = command_pattern.match(command) 
if not match:
    # this isn't a command, no subshell evaluation to perform
    return command

Although note that later on we do:

command = os.path.expandvars(command)

Which might need to be moved further up if we intend to support configuring the platform by an environment variable (e.g. platform = $MY_PLATFORM) which I don't think we do?

@oliver-sanders oliver-sanders modified the milestones: cylc-8.1.2, cylc-8.1.3 Feb 16, 2023
Copy link
Contributor

@datamel datamel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't reproduce this.

Sorry @wxtim my remote testing was broken. I have run this with a fresh environment and this is now working perfectly.

I will hold off approval as I think Oliver has made some suggestions about host checking which I think sound sensible to address in this pr.

@wxtim
Copy link
Member Author

wxtim commented Feb 17, 2023

Which might need to be moved further up if we intend to support configuring the platform by an environment variable (e.g. platform = $MY_PLATFORM) which I don't think we do?

No.

@wxtim
Copy link
Member Author

wxtim commented Feb 17, 2023

I'm not sure I understand the logic of host_check in subshell_eval,

This function can be called to find the value of a platform or a host. If it's a host_check indicates we've called it from the compatibility mode part of the code, and that it should retain Cylc 7 behavior - that is to run host check on command, which should, confusingly, be an evaluated host name at this point. Since a platform isn't a host this check makes no sense if the command is getting a value for platform.

I can't say for sure without looking more whether the Cylc 7 behavior is sane. But surely we'd have noticed if unreachable hosts caused jobs to run on local?

cylc/flow/task_job_mgr.py Outdated Show resolved Hide resolved
@wxtim
Copy link
Member Author

wxtim commented Mar 13, 2023

I'm not sure I understand the logic of host_check in subshell_eval, seems a little strange:

Only being run if subshell eval is supposed to be returning a host.

As for why it's doing even this, I'm afraid that's going into the depths of time/git.

@MetRonnie
Copy link
Member

I have put up a draft PR to improve the readability of the subshell eval code, let me know thoughts: wxtim#53

@wxtim wxtim force-pushed the fix_localhost_platform_matching branch from 9df1a9f to 83484ef Compare March 20, 2023 08:33
CHANGES.md Outdated Show resolved Hide resolved
@oliver-sanders oliver-sanders removed their request for review March 23, 2023 14:51
@wxtim wxtim requested review from MetRonnie and removed request for datamel March 24, 2023 12:44
@wxtim
Copy link
Member Author

wxtim commented Mar 24, 2023

@MetRonnie & @hjoliver - Poke

@hjoliver
Copy link
Member

hjoliver commented Apr 1, 2023

I think this is still awaiting merge of @MetRonnie 's side PR to your branch @wxtim ?

Tested as working however. (And I agree the affected code is a bit confusing).

@wxtim
Copy link
Member Author

wxtim commented Apr 11, 2023

I think this is still awaiting merge of @MetRonnie 's side PR to your branch @wxtim ?

Tested as working however. (And I agree the affected code is a bit confusing).

Yes, but I'm assuming that @MetRonnie has marked it draft for a reason

wxtim and others added 2 commits April 11, 2023 12:41
@MetRonnie MetRonnie removed the request for review from datamel April 14, 2023 10:56
Copy link
Member

@hjoliver hjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@hjoliver hjoliver merged commit da8516b into cylc:8.1.x Apr 16, 2023
@wxtim wxtim deleted the fix_localhost_platform_matching branch April 17, 2023 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :(
Projects
None yet
Development

Successfully merging this pull request may close these issues.

platform name can't start with "localhost"?
5 participants