# Objective

- Make running CI locally relatively less painful by allowing
continuation after failure
- Fix #12206 

## Solution

Previously, `ci` would halt after encounting a failure (or shortly
thereafter). For instance, if you ran `cargo run -p ci` and a single
test within a CI check failed somewhere in the middle, you would never
even reach many of the other tests within the check and certainly none
of the CI checks that came afterward. This was annoying; if I am fixing
issues with CI tests, I want to just see all of the issues up-front
instead of having to rerun CI every time I fix something to see the next
error.

Furthermore, it is not infrequent (because of subtle
configuration/system differences) to encounter spurious CI failures
locally; previously, when these occurred, they would make the process of
running CI largely pointless, since you would have to push your branch
in order to see your actual CI failures from the automated testing
service.

Now, when running `ci` locally, we have a couple new tools to ameliorate
these and provide a smoother experience:
- Firstly, there is a new optional flag `--keep-going` that can be
passed while calling `ci` (e.g. `cargo run -p ci -- doc --keep-going`).
It has the following effects:
- It causes the `--keep-going` flag to be passed to the script's `cargo
doc` and `cargo check` invocations, so that they do not stop when they
encounter an error in a single module; instead, they keep going (!) and
find errors subsequently.
- It causes the `--no-fail-fast` flag to be passed to the script's
`cargo test` invocations, to similar effect.
- Finally, it causes `ci` itself not to abort after a single check
fails; instead, it will run every check that was invoked.

Thus, for instance, `cargo run -p ci -- --keep-going` will run every CI
check even if it encounters intermediate failures, and every such check
will itself be run to completion.

- Secondly, we now allow multiple ordinary arguments to be passed to
`ci`. For instance, `cargo -p ci -- doc test` now executes both the
'doc' checks and the 'test' checks. This allows the caller to run the
tests they care about with fewer invocations of `ci`.

As of this PR, automated testing will remain unchanged. 

---

## Changelog

- tools/ci/src/main.rs refactored into staging and execution steps,
since the previous control flow did not naturally support continuing
after failure.
- Added "--keep-going" flag to `ci`.
- Added support for invoking `ci` with multiple arguments.

---

## Discussion

### Design considerations

I had originally split this into separate flags that controlled:
1. whether `--keep-going`/`--no-fail-fast` would be passed to the
constituent commands
2. whether `ci` would continue after a component test failed

However, I decided to merge these two behaviors, since I think that, if
you're in the business of seeing all the errors, you'll probably want to
actually see all of the errors. One slightly annoying thing, however,
about the new behavior with `--keep-going`, is that you will sometimes
find yourself wishing that the script would pause or something, since it
tends to fill the screen with a lot of junk. I have found that sending
stdout to /dev/null helps quite a bit, but I don't think `cargo fmt` or
`cargo clippy` actually write to stderr, so you need to be cognizant of
that (and perhaps invoke the lints separately).

~~Next, I'll mention that I feel somewhat strongly that the current
behavior of `ci` for automated testing should remain the same, since its
job is more like detecting that errors exist rather than finding all of
them.~~ (I was convinced this could have value.) Orthogonally, there is
the question of whether or not we might want to make this (or something
similar) actually the default behavior and make the automated test
script invoke some optional flags — it doesn't have to type with its
hands, after all. I'm not against that, but I don't really want to rock
the boat much more with this PR, since anyone who looks at the diff
might already be a little incensed.

# Objective

- Make running CI locally relatively less painful by allowing
continuation after failure
- Fix bevyengine#12206 

## Solution

Previously, `ci` would halt after encounting a failure (or shortly
thereafter). For instance, if you ran `cargo run -p ci` and a single
test within a CI check failed somewhere in the middle, you would never
even reach many of the other tests within the check and certainly none
of the CI checks that came afterward. This was annoying; if I am fixing
issues with CI tests, I want to just see all of the issues up-front
instead of having to rerun CI every time I fix something to see the next
error.

Furthermore, it is not infrequent (because of subtle
configuration/system differences) to encounter spurious CI failures
locally; previously, when these occurred, they would make the process of
running CI largely pointless, since you would have to push your branch
in order to see your actual CI failures from the automated testing
service.

Now, when running `ci` locally, we have a couple new tools to ameliorate
these and provide a smoother experience:
- Firstly, there is a new optional flag `--keep-going` that can be
passed while calling `ci` (e.g. `cargo run -p ci -- doc --keep-going`).
It has the following effects:
- It causes the `--keep-going` flag to be passed to the script's `cargo
doc` and `cargo check` invocations, so that they do not stop when they
encounter an error in a single module; instead, they keep going (!) and
find errors subsequently.
- It causes the `--no-fail-fast` flag to be passed to the script's
`cargo test` invocations, to similar effect.
- Finally, it causes `ci` itself not to abort after a single check
fails; instead, it will run every check that was invoked.

Thus, for instance, `cargo run -p ci -- --keep-going` will run every CI
check even if it encounters intermediate failures, and every such check
will itself be run to completion.

- Secondly, we now allow multiple ordinary arguments to be passed to
`ci`. For instance, `cargo -p ci -- doc test` now executes both the
'doc' checks and the 'test' checks. This allows the caller to run the
tests they care about with fewer invocations of `ci`.

As of this PR, automated testing will remain unchanged. 

---

## Changelog

- tools/ci/src/main.rs refactored into staging and execution steps,
since the previous control flow did not naturally support continuing
after failure.
- Added "--keep-going" flag to `ci`.
- Added support for invoking `ci` with multiple arguments.

---

## Discussion

### Design considerations

I had originally split this into separate flags that controlled:
1. whether `--keep-going`/`--no-fail-fast` would be passed to the
constituent commands
2. whether `ci` would continue after a component test failed

However, I decided to merge these two behaviors, since I think that, if
you're in the business of seeing all the errors, you'll probably want to
actually see all of the errors. One slightly annoying thing, however,
about the new behavior with `--keep-going`, is that you will sometimes
find yourself wishing that the script would pause or something, since it
tends to fill the screen with a lot of junk. I have found that sending
stdout to /dev/null helps quite a bit, but I don't think `cargo fmt` or
`cargo clippy` actually write to stderr, so you need to be cognizant of
that (and perhaps invoke the lints separately).

~~Next, I'll mention that I feel somewhat strongly that the current
behavior of `ci` for automated testing should remain the same, since its
job is more like detecting that errors exist rather than finding all of
them.~~ (I was convinced this could have value.) Orthogonally, there is
the question of whether or not we might want to make this (or something
similar) actually the default behavior and make the automated test
script invoke some optional flags — it doesn't have to type with its
hands, after all. I'm not against that, but I don't really want to rock
the boat much more with this PR, since anyone who looks at the diff
might already be a little incensed.

# Objective

- Make running CI locally relatively less painful by allowing
continuation after failure
- Fix bevyengine#12206 

## Solution

Previously, `ci` would halt after encounting a failure (or shortly
thereafter). For instance, if you ran `cargo run -p ci` and a single
test within a CI check failed somewhere in the middle, you would never
even reach many of the other tests within the check and certainly none
of the CI checks that came afterward. This was annoying; if I am fixing
issues with CI tests, I want to just see all of the issues up-front
instead of having to rerun CI every time I fix something to see the next
error.

Furthermore, it is not infrequent (because of subtle
configuration/system differences) to encounter spurious CI failures
locally; previously, when these occurred, they would make the process of
running CI largely pointless, since you would have to push your branch
in order to see your actual CI failures from the automated testing
service.

Now, when running `ci` locally, we have a couple new tools to ameliorate
these and provide a smoother experience:
- Firstly, there is a new optional flag `--keep-going` that can be
passed while calling `ci` (e.g. `cargo run -p ci -- doc --keep-going`).
It has the following effects:
- It causes the `--keep-going` flag to be passed to the script's `cargo
doc` and `cargo check` invocations, so that they do not stop when they
encounter an error in a single module; instead, they keep going (!) and
find errors subsequently.
- It causes the `--no-fail-fast` flag to be passed to the script's
`cargo test` invocations, to similar effect.
- Finally, it causes `ci` itself not to abort after a single check
fails; instead, it will run every check that was invoked.

Thus, for instance, `cargo run -p ci -- --keep-going` will run every CI
check even if it encounters intermediate failures, and every such check
will itself be run to completion.

- Secondly, we now allow multiple ordinary arguments to be passed to
`ci`. For instance, `cargo -p ci -- doc test` now executes both the
'doc' checks and the 'test' checks. This allows the caller to run the
tests they care about with fewer invocations of `ci`.

As of this PR, automated testing will remain unchanged. 

---

## Changelog

- tools/ci/src/main.rs refactored into staging and execution steps,
since the previous control flow did not naturally support continuing
after failure.
- Added "--keep-going" flag to `ci`.
- Added support for invoking `ci` with multiple arguments.

---

## Discussion

### Design considerations

I had originally split this into separate flags that controlled:
1. whether `--keep-going`/`--no-fail-fast` would be passed to the
constituent commands
2. whether `ci` would continue after a component test failed

However, I decided to merge these two behaviors, since I think that, if
you're in the business of seeing all the errors, you'll probably want to
actually see all of the errors. One slightly annoying thing, however,
about the new behavior with `--keep-going`, is that you will sometimes
find yourself wishing that the script would pause or something, since it
tends to fill the screen with a lot of junk. I have found that sending
stdout to /dev/null helps quite a bit, but I don't think `cargo fmt` or
`cargo clippy` actually write to stderr, so you need to be cognizant of
that (and perhaps invoke the lints separately).

~~Next, I'll mention that I feel somewhat strongly that the current
behavior of `ci` for automated testing should remain the same, since its
job is more like detecting that errors exist rather than finding all of
them.~~ (I was convinced this could have value.) Orthogonally, there is
the question of whether or not we might want to make this (or something
similar) actually the default behavior and make the automated test
script invoke some optional flags — it doesn't have to type with its
hands, after all. I'm not against that, but I don't really want to rock
the boat much more with this PR, since anyone who looks at the diff
might already be a little incensed.