…nur-ozkan

Bootstrap command refactoring: refactor `BootstrapCommand` (step 1)

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors `BoostrapCommand` to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of `BootstrapCommand` are complicated primarily because of different ways of handling output. There are commands that:
1) Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode `PrintAll`). Note that this error message attempts to print stdout/stderr of the command when `-v` is enabled, but that will always be empty, since this mode uses `.status()` and not `.output()`.
2) Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode `PrintOutput`)
3) Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode `PrintFailure`). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:
1) Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
2) Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):
- Remove `run` and `run_cmd` and implement everything in terms of `run_tracked` and rename `run_tracked` to `run`
- Implement the refactoring specified above (change how output modes work)
- Modify `BootstrapCmd` so that it stores `Command` instead of `&mut Command` and remove all the annoying `BootstrapCmd::from` by changing `Command::new` to `BootstrapCmd::new`
- Refactor the rest of command executions not currently using `BootstrapCmd` that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.
- Handle the most complex cases, such as output streaming. That will probably need to be handled separately.
- Refactor the rest of commands that cannot access builder (e.g. `Config::parse`) by introducing a new command context that will be passed to these places, and then stored in `Builder`. Move certain fields (such as `fail_fast`) from `Builder` to the command context.
- Handle the co-operation of `Builder`, `Build`, `Config` and command context. There are some fields and logic used during command execution that are distributed amongst `Builder/Build/Config`, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).
- Refactor logging of commands, so that it is either logged to a file or printed in a nice hierarchical way that cooperates with the `Step` debug hierarchical output.
- Implement profiling of commands (add command durations to the command log, print a log of slowest commands and their execution counts at the end of bootstrap execution, perhaps store command executions to `metrics.json`).
- Implement caching of commands.
- Implement testing of commands through snapshot tests/mocking.

Best reviewed commit by commit.

r? `@onur-ozkan`

…iaskrgr

Rollup of 3 pull requests

Successful merges:

 - rust-lang#116113 ( Generalize `{Rc,Arc}::make_mut()` to unsized types.)
 - rust-lang#126686 (Add `#[rustc_dump_{predicates,item_bounds}]`)
 - rust-lang#126731 (Bootstrap command refactoring: refactor `BootstrapCommand` (step 1))

r? `@ghost`
`@rustbot` modify labels: rollup

…nur-ozkan

Bootstrap command refactoring: refactor `BootstrapCommand` (step 1)

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors `BoostrapCommand` to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of `BootstrapCommand` are complicated primarily because of different ways of handling output. There are commands that:
1) Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode `PrintAll`). Note that this error message attempts to print stdout/stderr of the command when `-v` is enabled, but that will always be empty, since this mode uses `.status()` and not `.output()`.
2) Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode `PrintOutput`)
3) Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode `PrintFailure`). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:
1) Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
2) Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):
- Remove `run` and `run_cmd` and implement everything in terms of `run_tracked` and rename `run_tracked` to `run`
- Implement the refactoring specified above (change how output modes work)
- Modify `BootstrapCmd` so that it stores `Command` instead of `&mut Command` and remove all the annoying `BootstrapCmd::from` by changing `Command::new` to `BootstrapCmd::new`
- Refactor the rest of command executions not currently using `BootstrapCmd` that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.
- Handle the most complex cases, such as output streaming. That will probably need to be handled separately.
- Refactor the rest of commands that cannot access builder (e.g. `Config::parse`) by introducing a new command context that will be passed to these places, and then stored in `Builder`. Move certain fields (such as `fail_fast`) from `Builder` to the command context.
- Handle the co-operation of `Builder`, `Build`, `Config` and command context. There are some fields and logic used during command execution that are distributed amongst `Builder/Build/Config`, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).
- Refactor logging of commands, so that it is either logged to a file or printed in a nice hierarchical way that cooperates with the `Step` debug hierarchical output.
- Implement profiling of commands (add command durations to the command log, print a log of slowest commands and their execution counts at the end of bootstrap execution, perhaps store command executions to `metrics.json`).
- Implement caching of commands.
- Implement testing of commands through snapshot tests/mocking.

Best reviewed commit by commit.

r? ```@onur-ozkan```

…llaumeGomez

Rollup of 5 pull requests

Successful merges:

 - rust-lang#126555 (Add `f16` inline ASM support for 32-bit ARM)
 - rust-lang#126686 (Add `#[rustc_dump_{predicates,item_bounds}]`)
 - rust-lang#126723 (Fix `...` in multline code-skips in suggestions)
 - rust-lang#126731 (Bootstrap command refactoring: refactor `BootstrapCommand` (step 1))
 - rust-lang#126823 (Migrate `run-make/inline-always-many-cgu` to `rmake.rs`)

r? `@ghost`
`@rustbot` modify labels: rollup

Rollup merge of rust-lang#126731 - Kobzol:bootstrap-cmd-refactor, r=onur-ozkan

Bootstrap command refactoring: refactor `BootstrapCommand` (step 1)

This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.

It refactors `BoostrapCommand` to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).

The existing usages of `BootstrapCommand` are complicated primarily because of different ways of handling output. There are commands that:
1) Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode `PrintAll`). Note that this error message attempts to print stdout/stderr of the command when `-v` is enabled, but that will always be empty, since this mode uses `.status()` and not `.output()`.
2) Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode `PrintOutput`)
3) Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode `PrintFailure`). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.

The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.

It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:
1) Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
2) Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).

Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.

Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):
- Remove `run` and `run_cmd` and implement everything in terms of `run_tracked` and rename `run_tracked` to `run`
- Implement the refactoring specified above (change how output modes work)
- Modify `BootstrapCmd` so that it stores `Command` instead of `&mut Command` and remove all the annoying `BootstrapCmd::from` by changing `Command::new` to `BootstrapCmd::new`
- Refactor the rest of command executions not currently using `BootstrapCmd` that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.
- Handle the most complex cases, such as output streaming. That will probably need to be handled separately.
- Refactor the rest of commands that cannot access builder (e.g. `Config::parse`) by introducing a new command context that will be passed to these places, and then stored in `Builder`. Move certain fields (such as `fail_fast`) from `Builder` to the command context.
- Handle the co-operation of `Builder`, `Build`, `Config` and command context. There are some fields and logic used during command execution that are distributed amongst `Builder/Build/Config`, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).
- Refactor logging of commands, so that it is either logged to a file or printed in a nice hierarchical way that cooperates with the `Step` debug hierarchical output.
- Implement profiling of commands (add command durations to the command log, print a log of slowest commands and their execution counts at the end of bootstrap execution, perhaps store command executions to `metrics.json`).
- Implement caching of commands.
- Implement testing of commands through snapshot tests/mocking.

Best reviewed commit by commit.

r? ``@onur-ozkan``

…nur-ozkan

Bootstrap command refactoring: consolidate output modes (step 3)

This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (`Print` vs `CaptureAll` vs `CaptureStdout`) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of `Command` to `BootstrapCommand`, most notably the git helpers and many usages of the `output` function.

The last commit was added because the third commit made two variants of the `Tool` enum unused (no idea why, but it seems to have been a false positive that they were used before).

It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further.

As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue.

As always, best reviewed commit by commit.

Tracking issue: rust-lang#126819

r? `@onur-ozkan`

…try>

Bootstrap command refactoring: consolidate output modes (step 3)

This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (`Print` vs `CaptureAll` vs `CaptureStdout`) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of `Command` to `BootstrapCommand`, most notably the git helpers and many usages of the `output` function.

The last commit was added because the third commit made two variants of the `Tool` enum unused (no idea why, but it seems to have been a false positive that they were used before).

It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further.

As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue.

As always, best reviewed commit by commit.

Tracking issue: rust-lang#126819

r? `@onur-ozkan`

try-job: aarch64-apple

…nur-ozkan

Bootstrap command refactoring: consolidate output modes (step 3)

This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (`Print` vs `CaptureAll` vs `CaptureStdout`) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of `Command` to `BootstrapCommand`, most notably the git helpers and many usages of the `output` function.

The last commit was added because the third commit made two variants of the `Tool` enum unused (no idea why, but it seems to have been a false positive that they were used before).

It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further.

As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue.

As always, best reviewed commit by commit.

Tracking issue: rust-lang#126819

r? `@onur-ozkan`

try-job: aarch64-apple

Update Rust toolchain from nightly-2024-06-22 to nightly-2024-06-23
without any other source changes.
This is an automatically generated pull request. If any of the CI checks
fail, manual intervention is required. In such a case, review the
changes at https://github.com/rust-lang/rust from
rust-lang@c1b336c
up to
rust-lang@3cb521a.
The log for this commit range is:
rust-lang@3cb521a434 Auto merge of
rust-lang#126761 - GuillaumeGomez:unsafe_extern_blocks, r=spastorino
rust-lang@a0f01c3c10 Auto merge of
rust-lang#126838 - matthiaskrgr:rollup-qkop22o, r=matthiaskrgr
rust-lang@dc9a08f535 Rollup merge of
rust-lang#126552 - fee1-dead-contrib:rmfx, r=compiler-errors
rust-lang@162120b4fa Rollup merge of
rust-lang#126318 - Kobzol:bootstrap-perf, r=onur-ozkan
rust-lang@f3ced9d540 Rollup merge of
rust-lang#126140 - eduardosm:stabilize-fs_try_exists, r=Amanieu
rust-lang@f944afe380 Auto merge of
rust-lang#116113 - kpreid:arcmut, r=dtolnay
rust-lang@88c3db57e4 Generalize
`{Rc,Arc}::make_mut()` to unsized types.
rust-lang@a9a4830d25 Replace
`WriteCloneIntoRaw` with `CloneToUninit`.
rust-lang@ec201b8650 Add
`core::clone::CloneToUninit`.
rust-lang@81da6a6d40 Make `effects` an
incomplete feature
rust-lang@ac47dbad50 Auto merge of
rust-lang#126824 - GuillaumeGomez:rollup-sybv8o7, r=GuillaumeGomez
rust-lang@d265538016 Rollup merge of
rust-lang#126823 - GuillaumeGomez:migrate-run-make-inline-always-many-cgu,
r=Kobzol
rust-lang@25bcc7d130 Rollup merge of
rust-lang#126731 - Kobzol:bootstrap-cmd-refactor, r=onur-ozkan
rust-lang@399c5cabdd Rollup merge of
rust-lang#126723 - estebank:dot-dot-dot, r=Nadrieril
rust-lang@3ed2cd74b5 Rollup merge of
rust-lang#126686 - fmease:dump-preds-n-item-bounds, r=compiler-errors
rust-lang@07e8b3ac01 Rollup merge of
rust-lang#126555 - beetrees:f16-inline-asm-arm, r=Amanieu
rust-lang@d03d6c0fea Auto merge of
rust-lang#126750 - scottmcm:less-unlikely, r=jhpratt
rust-lang@e7dfd4a913 Migrate
`run-make/inline-always-many-cgu` to `rmake.rs`
rust-lang@d9962bb4d8 Make `read_dir`
method take a mutable callback
rust-lang@f1b0d54ca9 Auto merge of
rust-lang#126816 - weihanglo:update-cargo, r=weihanglo
rust-lang@0bd58d8122 Apply review
comments.
rust-lang@250586cb2e Wrap std `Output` in
`CommandOutput`
rust-lang@f0aceed540 Auto merge of
rust-lang#126817 - workingjubilee:rollup-0rg0k55, r=workingjubilee
rust-lang@38bd7a0fcb Add
`#[rustc_dump_{predicates,item_bounds}]`
rust-lang@1916b3d57f Rollup merge of
rust-lang#126811 - compiler-errors:tidy-ftl, r=estebank
rust-lang@539090e5cd Rollup merge of
rust-lang#126809 - estebank:wording-tweak, r=oli-obk
rust-lang@b9ab6c3501 Rollup merge of
rust-lang#126798 - miguelfrde:master, r=tmandry
rust-lang@9498d5cf2f Rollup merge of
rust-lang#126787 - Strophox:get-bytes, r=RalfJung
rust-lang@1f9793f1aa Rollup merge of
rust-lang#126722 - adwinwhite:ptr_fn_abi, r=celinval
rust-lang@84b0922565 Rollup merge of
rust-lang#126712 - Oneirical:bootest-constestllation, r=jieyouxu
rust-lang@e7956cd994 Rollup merge of
rust-lang#126530 - beetrees:f16-inline-asm-riscv, r=Amanieu
rust-lang@10e1f5d212 Auto merge of
rust-lang#124101 - the8472:pidfd-methods, r=cuviper
rust-lang@2c65a24b8c Update cargo
rust-lang@fcae62649e Auto merge of
rust-lang#126758 - spastorino:avoid-safe-outside-unsafe-blocks, r=compiler-errors
rust-lang@ffd72b1700 Fix remaining cases
rust-lang@ea681ef281 Add a tidy rule to
make sure that diagnostics don't end in periods
rust-lang@8abf149bde to extract a pidfd
we must consume the child
rust-lang@0787c7308c Add PidFd::{kill,
wait, try_wait}
rust-lang@5d5892e966 Remove stray `.`
from error message
rust-lang@d94a40516e
[fuchsia-test-runner] Remove usage of kw_only
rust-lang@771e44ebd3 Add `f16` inline ASM
support for RISC-V
rust-lang@753fb070bb Add `f16` inline ASM
support for 32-bit ARM
rust-lang@22831ed117 Do not allow safe
usafe on static and fn items
rust-lang@a6a83d3d4e bless tests
rust-lang@b512bf6f77 add as_ptr to trait
AllocBytes, fix 2 impls; add pub fn get_bytes_unchecked_raw in
allocation.rs; add pub fn get_alloc_bytes_unchecked_raw[_mut] in
memory.rs
rust-lang@02aaea1803 update intrinsic
const param counting
rust-lang@3b14b756d8 Remove
`feature(effects)` from the standard library
rust-lang@a314f7363a Stop using
`unlikely` in `strict_*` methods
rust-lang@225796a2df Add method to get
`FnAbi` of function pointer
rust-lang@630c3adb14 Add regression test
for `unsafe_extern_blocks`
rust-lang@bb9a3ef90c Implement
`unsafe_extern_blocks` feature in rustdoc
rust-lang@3c0a4bc915 rewrite
crate-name-priority to rmake
rust-lang@bc12972bcd Slightly refactor
the dumping of HIR analysis data
rust-lang@3fe4d134dd Appease `clippy`
rust-lang@c15293407f Remove unused import
rust-lang@5c4318d02c Implement `run_cmd`
in terms of `run_tracked`
rust-lang@0de7b92cc6 Remove
`run_delaying_failure`
rust-lang@e933cfb13c Remove
`run_quiet_delaying_failure`
rust-lang@949e667d3f Remove `run_quiet`
rust-lang@a12f541a18 Implement new
command execution logic
rust-lang@9fd7784b97 Fix `...` in
multline code-skips in suggestions
rust-lang@f22b5afa6a rewrite
error-writing-dependencies to rmake
rust-lang@75ee1d74a9 rewrite
relocation-model to rmake
rust-lang@87d2e61428 Add `x perf` command
for profiling the compiler using `rustc-perf`
rust-lang@fd44aca2aa Copy `rustc-fake`
binary when building the `rustc-perf` tool
rust-lang@9e0b76201b Add `RustcPerf`
bootstrap tool
rust-lang@9ec178df0b Add `cargo_args` to
`ToolBuild`
rust-lang@6a04dfe78c Rename
`std::fs::try_exists` to `std::fs::exists` and stabilize fs_try_exists

Co-authored-by: celinval <35149715 [email protected]>