Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history
commit ad8231a8f22dd3cd8d887774474f97df2c83e429
Merge: acc0ca3 6fc3c58
Author: Hilary James Oliver <[email protected]>
Date:   Mon Mar 20 10:27:19 2023  1300

    Merge branch 'master' into initial-final-graph

commit acc0ca3
Author: Hilary Oliver <[email protected]>
Date:   Wed Oct 26 12:00:00 2022  1300

    Style fix.

commit a7170fb
Author: Hilary Oliver <[email protected]>
Date:   Wed Oct 26 11:53:31 2022  1300

    Update change log.

commit 9024065
Merge: 858a8c1 b0ff549
Author: Hilary Oliver <[email protected]>
Date:   Wed Oct 26 11:32:26 2022  1300

    Merge branch 'master' into initial-final-graph

commit 858a8c1
Author: Hilary James Oliver <[email protected]>
Date:   Mon Sep 26 23:27:01 2022  1300

    Fix alpha/omega graph runahead release.

commit b581ab2
Author: Hilary James Oliver <[email protected]>
Date:   Mon Sep 26 23:26:33 2022  1300

    Add new func tests.

commit 8d09900
Author: Hilary James Oliver <[email protected]>
Date:   Mon Sep 26 15:53:42 2022  1300

    Revert graph sorting change.

commit 75c1763
Author: Hilary James Oliver <[email protected]>
Date:   Fri Sep 23 22:44:08 2022  1200

    Adapt integration tests.

commit 1b685a0
Author: Hilary James Oliver <[email protected]>
Date:   Fri Sep 23 15:55:57 2022  1200

    Tidy up.

commit 7a62450
Author: Hilary James Oliver <[email protected]>
Date:   Fri Sep 23 13:20:51 2022  1200

    Switch to alpha and omega.

commit 129c2e3
Author: Hilary James Oliver <[email protected]>
Date:   Fri Sep 23 10:32:54 2022  1200

    Implicit integer ICP if only nocycle graphs.

commit a53ac82
Author: Hilary James Oliver <[email protected]>
Date:   Thu Sep 22 16:23:16 2022  1200

    startup and shutdown graphs; needs tidying
  • Loading branch information
hjoliver committed Apr 26, 2023
1 parent 15346be commit 2eb2cde
Show file tree
Hide file tree
Showing 25 changed files with 629 additions and 87 deletions.
14 changes: 10 additions & 4 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 77,11 @@ from working.
-------------------------------------------------------------------------------
## __cylc-8.2.0 (<span actions:bind='release-date'>Coming Soon</span>)__

### Enhancements

[#5090](https://github.com/cylc/cylc-flow/pull/5090) - Implement initial and
final graphs, distinct from the main cycling graph.

### Fixes
[#5328](https://github.com/cylc/cylc-flow/pull/5328) -
Efficiency improvements to reduce task management overheads on the Scheduler.
Expand Down Expand Up @@ -126,19 131,20 @@ workflows with many-to-many dependencies (e.g. `<a> => <b>`).

### Enhancements

[#5184](https://github.com/cylc/cylc-flow/pull/5184) - Scan for active
runs of the same workflow at install time.

[#5032](https://github.com/cylc/cylc-flow/pull/5032) - Set a default limit of

[#5229](https://github.com/cylc/cylc-flow/pull/5229) -
- Added a single command to validate a previously run workflow against changes
to its source and reinstall a workflow.
- Allows Cylc commands (including validate, list, view, config, and graph) to load template variables
configured by `cylc install` and `cylc play`.

[#5184](https://github.com/cylc/cylc-flow/pull/5184) - scan for active
runs of the same workflow at install time.

[#5121](https://github.com/cylc/cylc-flow/pull/5121) - Added a single
command to validate, install and play a workflow.

[#5032](https://github.com/cylc/cylc-flow/pull/5032) - set a default limit of
100 for the "default" queue.

[#5055](https://github.com/cylc/cylc-flow/pull/5055) and
Expand Down
44 changes: 33 additions & 11 deletions cylc/flow/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 54,11 @@
get_sequence, get_sequence_cls, init_cyclers, get_dump_format,
INTEGER_CYCLING_TYPE, ISO8601_CYCLING_TYPE
)
from cylc.flow.cycling.nocycle import (
NocycleSequence,
NOCYCLE_SEQ_ALPHA,
NOCYCLE_SEQ_OMEGA
)
from cylc.flow.id import Tokens
from cylc.flow.cycling.integer import IntegerInterval
from cylc.flow.cycling.iso8601 import ingest_time, ISO8601Interval
Expand Down Expand Up @@ -269,6 274,7 @@ def __init__(
self.start_point: 'PointBase'
self.stop_point: Optional['PointBase'] = None
self.final_point: Optional['PointBase'] = None
self.nocycle_sequences: Set['NocycleSequence'] = set()
self.sequences: List['SequenceBase'] = []
self.actual_first_point: Optional['PointBase'] = None
self._start_point_for_actual_first_point: Optional['PointBase'] = None
Expand Down Expand Up @@ -608,9 614,16 @@ def prelim_process_graph(self) -> None:
if (
'cycling mode' not in self.cfg['scheduling'] and
self.cfg['scheduling'].get('initial cycle point', '1') == '1' and
all(item in ['graph', '1', 'R1'] for item in graphdict)
all(
item in [
'graph', '1', 'R1',
str(NOCYCLE_SEQ_ALPHA),
str(NOCYCLE_SEQ_OMEGA)
]
for item in graphdict
)
):
# Pure acyclic graph, assume integer cycling mode with '1' cycle
# Non-cycling graph, assume integer cycling mode with '1' cycle
self.cfg['scheduling']['cycling mode'] = INTEGER_CYCLING_TYPE
for key in ('initial cycle point', 'final cycle point'):
if key not in self.cfg['scheduling']:
Expand Down Expand Up @@ -2076,15 2089,24 @@ def load_graph(self):
try:
seq = get_sequence(section, icp, fcp)
except (AttributeError, TypeError, ValueError, CylcError) as exc:
if cylc.flow.flags.verbosity > 1:
traceback.print_exc()
msg = 'Cannot process recurrence %s' % section
msg = ' (initial cycle point=%s)' % icp
msg = ' (final cycle point=%s)' % fcp
if isinstance(exc, CylcError):
msg = ' %s' % exc.args[0]
raise WorkflowConfigError(msg)
self.sequences.append(seq)
try:
seq = NocycleSequence(section)
except ValueError:

Check warning on line 2094 in cylc/flow/config.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/config.py#L2092-L2094

Added lines #L2092 - L2094 were not covered by tests
if cylc.flow.flags.verbosity > 1:
traceback.print_exc()
msg = (

Check warning on line 2097 in cylc/flow/config.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/config.py#L2096-L2097

Added lines #L2096 - L2097 were not covered by tests
f"Cannot process recurrence {section}"
f" (initial cycle point={icp})"
f" (final cycle point={fcp})"
)
if isinstance(exc, CylcError):
msg = ' %s' % exc.args[0]
raise WorkflowConfigError(msg)

Check warning on line 2104 in cylc/flow/config.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/config.py#L2103-L2104

Added lines #L2103 - L2104 were not covered by tests
else:
self.nocycle_sequences.add(seq)

Check warning on line 2106 in cylc/flow/config.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/config.py#L2106

Added line #L2106 was not covered by tests
else:
self.sequences.append(seq)

parser = GraphParser(
family_map,
self.parameters,
Expand Down
6 changes: 5 additions & 1 deletion cylc/flow/cycling/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 345,11 @@ def TYPE_SORT_KEY(self) -> int:
@classmethod
@abstractmethod # Note: stacked decorator not strictly enforced in Py2.x
def get_async_expr(cls, start_point=0):
"""Express a one-off sequence at the initial cycle point."""
"""Express a one-off sequence at the initial cycle point.
Note "async" has nothing to do with asyncio. It was a (bad)
name for one-off (non-cycling) graphs in early Cylc versions.
"""
pass

@abstractmethod
Expand Down
7 changes: 6 additions & 1 deletion cylc/flow/cycling/integer.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 21,12 @@
import re

from cylc.flow.cycling import (
PointBase, IntervalBase, SequenceBase, ExclusionBase, parse_exclusion, cmp
PointBase,
IntervalBase,
SequenceBase,
ExclusionBase,
parse_exclusion,
cmp
)
from cylc.flow.exceptions import (
CylcMissingContextPointError,
Expand Down
159 changes: 159 additions & 0 deletions cylc/flow/cycling/nocycle.py
Original file line number Diff line number Diff line change
@@ -0,0 1,159 @@
# THIS FILE IS PART OF THE CYLC WORKFLOW ENGINE.
# Copyright (C) NIWA & British Crown (Met Office) & Contributors.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

"""
Cycling logic for isolated non-cycling startup and shutdown graphs.
"""

from cylc.flow.cycling import PointBase, SequenceBase, cmp

# TODO: scheduler check DB to be sure alpha and omega sections have run or not.

# cycle point values
NOCYCLE_PT_ALPHA = "alpha"
NOCYCLE_PT_OMEGA = "omega"

NOCYCLE_POINTS = (
NOCYCLE_PT_ALPHA,
NOCYCLE_PT_OMEGA
)

CYCLER_TYPE_NOCYCLE = "nocycle"
CYCLER_TYPE_SORT_KEY_NOCYCLE = 1


class NocyclePoint(PointBase):
"""A string-valued point."""

TYPE = CYCLER_TYPE_NOCYCLE
TYPE_SORT_KEY = CYCLER_TYPE_SORT_KEY_NOCYCLE

__slots__ = ('value')

def __init__(self, value: str) -> None:
if value not in [NOCYCLE_PT_ALPHA, NOCYCLE_PT_OMEGA]:
raise ValueError(f"Illegal Nocycle value {value}")

Check warning on line 48 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L48

Added line #L48 was not covered by tests
self.value = value

def __hash__(self):
return hash(self.value)

Check warning on line 52 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L52

Added line #L52 was not covered by tests

def __eq__(self, other):
return str(other) == self.value

Check warning on line 55 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L55

Added line #L55 was not covered by tests

def __le__(self, other):
"""less than or equal only if equal."""
return str(other) == self.value

Check warning on line 59 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L59

Added line #L59 was not covered by tests

def __lt__(self, other):
"""never less than."""
return False

Check warning on line 63 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L63

Added line #L63 was not covered by tests

def __gt__(self, other):
"""never greater than."""
return False

Check warning on line 67 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L67

Added line #L67 was not covered by tests

def __str__(self):
return self.value

def _cmp(self, other):
return cmp(int(self), int(other))

Check warning on line 73 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L73

Added line #L73 was not covered by tests

def add(self, other):
# NOT USED
return None

Check warning on line 77 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L77

Added line #L77 was not covered by tests

def sub(self, other):
# NOT USED
return None

Check warning on line 81 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L81

Added line #L81 was not covered by tests


class NocycleSequence(SequenceBase):
"""A single point sequence."""

def __init__(self, dep_section, p_context_start=None, p_context_stop=None):
"""Workflow cycling context is ignored."""
self.point = NocyclePoint(dep_section)

def __hash__(self):
return hash(str(self.point))

def is_valid(self, point):
"""Is point on-sequence and in-bounds?"""
return str(point) == self.point

Check warning on line 96 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L96

Added line #L96 was not covered by tests

def get_first_point(self, point):
"""First point is the only point"""
return self.point

Check warning on line 100 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L100

Added line #L100 was not covered by tests

def get_next_point(self, point):
"""There is no next point"""
return None

Check warning on line 104 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L104

Added line #L104 was not covered by tests

def get_next_point_on_sequence(self, point):
"""There is no next point"""
return None

Check warning on line 108 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L108

Added line #L108 was not covered by tests

def __eq__(self, other):
try:
return other.point == self.point
except AttributeError:

Check warning on line 113 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L111-L113

Added lines #L111 - L113 were not covered by tests
# (other is not a nocycle sequence)
return False

Check warning on line 115 in cylc/flow/cycling/nocycle.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/cycling/nocycle.py#L115

Added line #L115 was not covered by tests

def __str__(self):
return str(self.point)

def TYPE(self):
raise NotImplementedError

def TYPE_SORT_KEY(self):
raise NotImplementedError

def get_async_expr(cls, start_point=0):
raise NotImplementedError

def get_interval(self):
"""Return the cycling interval of this sequence."""
raise NotImplementedError

def get_offset(self):
"""Deprecated: return the offset used for this sequence."""
raise NotImplementedError

def set_offset(self, i_offset):
"""Deprecated: alter state to offset the entire sequence."""
raise NotImplementedError

def is_on_sequence(self, point):
"""Is point on-sequence, disregarding bounds?"""
raise NotImplementedError

def get_prev_point(self, point):
"""Return the previous point < point, or None if out of bounds."""
raise NotImplementedError

def get_nearest_prev_point(self, point):
"""Return the largest point < some arbitrary point."""
raise NotImplementedError

def get_stop_point(self):
"""Return the last point in this sequence, or None if unbounded."""
raise NotImplementedError


NOCYCLE_SEQ_ALPHA = NocycleSequence(NOCYCLE_PT_ALPHA)
NOCYCLE_SEQ_OMEGA = NocycleSequence(NOCYCLE_PT_OMEGA)
Loading

0 comments on commit 2eb2cde

Please sign in to comment.