Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable parallel rustc front end in nightly builds #117435

Merged
merged 4 commits into from
Nov 6, 2023

Conversation

SparrowLii
Copy link
Member

@SparrowLii SparrowLii commented Oct 31, 2023

Refers to the MCP, this pr does:

  1. Enable the parallel front end in nightly builds, and keep the default number of threads as 1. Then users can use the parallel rustc front end via -Z threads=n option.

  2. Set it up to serial front end for beta/stable builds via bootstrap.

  3. Switch over the alt builders from parallel rustc to serial, so we have artifacts without parallel to test against the artifacts with parallel.

r? @oli-obk

cc @cjgillot @nnethercote @bjorn3 @Kobzol

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Oct 31, 2023
@rustbot
Copy link
Collaborator

rustbot commented Oct 31, 2023

This PR modifies src/bootstrap/src/core/config. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

@rustbot
Copy link
Collaborator

rustbot commented Oct 31, 2023

This PR changes config.example.toml. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

@SparrowLii SparrowLii changed the title enable parallel rustc in nightly builds enable parallel rustc front end in nightly builds Oct 31, 2023
Copy link
Contributor

@oli-obk oli-obk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm gonna let people chime in on this for a day or two. If I don't approve by Thursday myself, r=me

@lqd
Copy link
Member

lqd commented Nov 1, 2023

Do we need to land #115220 prior to this PR? Or the fact that it applies to more than a single thread is enough not to (in my mind people are immediately going to try >1 threads).

@oli-obk
Copy link
Contributor

oli-obk commented Nov 1, 2023

More than one thread is deadlock prone and anyone opting into it with the -Z flag has to accept that I think

@SparrowLii
Copy link
Member Author

SparrowLii commented Nov 3, 2023

@oli-obk I think we can merge now

@bors r=oli-obk rollup=never

@bors
Copy link
Contributor

bors commented Nov 3, 2023

📌 Commit 248dd14 has been approved by oli-obk

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 3, 2023
src/ci/run.sh Show resolved Hide resolved
@matthiaskrgr
Copy link
Member

@bors rollup=never

@bors
Copy link
Contributor

bors commented Nov 3, 2023

⌛ Testing commit 248dd14 with merge bb4d1c3...

bors added a commit to rust-lang-ci/rust that referenced this pull request Nov 3, 2023
enable parallel rustc front end in nightly builds

Refers to the [MCP](rust-lang/compiler-team#681), this pr does:
1. Enable the parallel front end in nightly builds, and keep the default number of threads as 1. Then users can use the parallel rustc front end via -Z threads=n option.

2. Set it up to serial front end for beta/stable builds via bootstrap.

3. Switch over the alt builders from parallel rustc to serial, so we have artifacts without parallel to test against the artifacts with parallel.

r? `@oli-obk`

cc `@cjgillot` `@nnethercote` `@bjorn3` `@Kobzol`
@bors

This comment was marked as resolved.

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 3, 2023
@rustbot rustbot assigned petrochenkov and unassigned oli-obk Nov 6, 2023
@davidtwco
Copy link
Member

r? @davidtwco

@bors r=oli-obk,davidtwco

@bors
Copy link
Contributor

bors commented Nov 6, 2023

📌 Commit f2a40e9 has been approved by oli-obk,davidtwco

It is now in the queue for this repository.

@rustbot rustbot assigned davidtwco and unassigned petrochenkov Nov 6, 2023
@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 6, 2023
@bors
Copy link
Contributor

bors commented Nov 6, 2023

⌛ Testing commit f2a40e9 with merge f9b6446...

@bors
Copy link
Contributor

bors commented Nov 6, 2023

☀️ Test successful - checks-actions
Approved by: oli-obk,davidtwco
Pushing f9b6446 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 6, 2023
@bors bors merged commit f9b6446 into rust-lang:master Nov 6, 2023
12 checks passed
@rustbot rustbot added this to the 1.75.0 milestone Nov 6, 2023
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (f9b6446): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
2.9% [0.4%, 5.1%] 209
Regressions ❌
(secondary)
2.7% [0.4%, 9.1%] 219
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 2.9% [0.4%, 5.1%] 209

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
1.2% [0.4%, 2.6%] 28
Regressions ❌
(secondary)
2.5% [0.5%, 6.8%] 61
Improvements ✅
(primary)
-1.6% [-2.2%, -0.9%] 2
Improvements ✅
(secondary)
-2.2% [-2.5%, -2.0%] 3
All ❌✅ (primary) 1.0% [-2.2%, 2.6%] 30

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
2.5% [0.4%, 5.2%] 186
Regressions ❌
(secondary)
3.3% [1.3%, 8.3%] 119
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 2.5% [0.4%, 5.2%] 186

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 636.284s -> 661.435s (3.95%)
Artifact size: 304.53 MiB -> 308.99 MiB (1.46%)

@rustbot rustbot added the perf-regression Performance regression. label Nov 6, 2023
bors-ferrocene bot added a commit to ferrocene/ferrocene that referenced this pull request Nov 7, 2023
89: Automated pull from upstream `master` r=Dajamante a=github-actions[bot]


This PR pulls the following changes from the upstream repository:

* rust-lang/rust#117006
* rust-lang/rust#117511
* rust-lang/rust#117641
  * rust-lang/rust#117637
  * rust-lang/rust#117631
  * rust-lang/rust#117516
  * rust-lang/rust#117190
* rust-lang/rust#117292
* rust-lang/rust#117603
* rust-lang/rust#116988
* rust-lang/rust#117630
  * rust-lang/rust#117615
  * rust-lang/rust#117613
  * rust-lang/rust#117592
* rust-lang/rust#117578
* rust-lang/rust#117435
* rust-lang/rust#117607



90: bump serde and serde_derive r=tshepang a=Dajamante

Trying to get around the failure seen in #86 

Co-authored-by: Ralf Jung <[email protected]>
Co-authored-by: Esteban Küber <[email protected]>
Co-authored-by: SparrowLii <[email protected]>
Co-authored-by: Matthias Krüger <[email protected]>
Co-authored-by: Gurinder Singh <[email protected]>
Co-authored-by: Michael Goulet <[email protected]>
Co-authored-by: Thom Chiovoloni <[email protected]>
Co-authored-by: klensy <[email protected]>
Co-authored-by: Jack Huey <31162821 [email protected]>
Co-authored-by: bjorn3 <17426603 [email protected]>
Co-authored-by: hkalbasi <[email protected]>
Co-authored-by: bors <[email protected]>
Co-authored-by: Sven Marnach <[email protected]>
Co-authored-by: Rémy Rakic <remy.rakic [email protected]>
Co-authored-by: aissata <[email protected]>
@pnkfelix
Copy link
Member

pnkfelix commented Nov 7, 2023

  • I am not sure that wg-parallel-rustc was anticipating there being this much of a regression here.
  • asked for followup in zulip for wg-parallel-rustc
  • not marking as triaged

@SparrowLii
Copy link
Member Author

SparrowLii commented Nov 8, 2023

@pnkfelix Sorry for not explaining in the MCP. We decided in the working group meeting early this year that wall-time is a better choice to measuring the perf result of parallel front end. Therefore, the measurement we choose is wall-time non-relevent. (You may also need to exclude secondary Categories). In this case, the regression of #117435 should be 1.65%

@pnkfelix
Copy link
Member

pnkfelix commented Nov 8, 2023

  • after followup in zulip for wg-parallel-rustc, they clarified that their expectation of a 1.65% regression was based on wall-time that includes so-called "non-relevant results". (Our default rustc-perf presentation filters out the non-relevant results, which will boost outliers and thus inflate the true mean effect.)
  • concluded that this was anticipated, and marking as triaged

@rustbot label: perf-regression-triaged

@rustbot rustbot added the perf-regression-triaged The performance regression has been triaged. label Nov 8, 2023
@SparrowLii
Copy link
Member Author

Thanks!

@@ -77,7 77,7 @@ const LLD_FILE_NAMES: &[&str] = &["ld.lld", "ld64.lld", "lld-link", "wasm-ld"];
///
/// If you make any major changes (such as adding new values or changing default values), please
/// ensure that the associated PR ID is added to the end of this list.
pub const CONFIG_CHANGE_HISTORY: &[usize] = &[115898, 116998];
pub const CONFIG_CHANGE_HISTORY: &[usize] = &[115898, 116998, 117435];
Copy link
Member

@RalfJung RalfJung Nov 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you do this, please explain in the PR description what we have to do to update our configs. I was sent here by bootstrap ("WARNING: there have been changes to x.py since you last updated") and now I have no idea what I'm supposed to do.

celinval added a commit to celinval/rust-dev that referenced this pull request Jun 4, 2024
Update Rust toolchain from nightly-2023-11-06 to nightly-2023-11-07
without any other source changes.
This is an automatically generated pull request. If any of the CI checks
fail, manual intervention is required. In such a case, review the
changes at https://github.com/rust-lang/rust from
rust-lang@fee5518
up to
rust-lang@189d6c7.
The log for this commit range is:
rust-lang@189d6c71f3 Auto merge of
rust-lang#117641 - matthiaskrgr:rollup-f9c12td, r=matthiaskrgr
rust-lang@9efe60b1eb Rollup merge of
rust-lang#117637 - lqd:trivial-bounds-with-binder-vars, r=compiler-errors
rust-lang@2a1f8bccee Rollup merge of
rust-lang#117631 - smarnach:error-request-doc-fix, r=ChrisDenton
rust-lang@8a34fea835 Rollup merge of
rust-lang#117516 - matthiaskrgr:test_effects_113375_oob, r=fee1-dead
rust-lang@1683df7d3b Rollup merge of
rust-lang#117190 - matthiaskrgr:test_effects_113381, r=fee1-dead
rust-lang@fb61292105 Auto merge of
rust-lang#117292 - estebank:issue-80446, r=davidtwco
rust-lang@4b7aacaa4f Silence redundant
error on typo resulting on binop
rust-lang@aea82b268a Auto merge of
rust-lang#117603 - HKalbasi:make-feature-additive, r=Nilstrieb
rust-lang@58351ae03f add test for trivial
bound not holding in `soa-derive`
rust-lang@2beca157c9 check binders with
bound vars for global bounds that don't hold
rust-lang@b049093560 Auto merge of
rust-lang#116988 - RalfJung:null, r=WaffleLapkin
rust-lang@e1fcecb1b9 Auto merge of
rust-lang#117630 - matthiaskrgr:rollup-v0d5p3f, r=matthiaskrgr
rust-lang@3a096e96fa Documentation
cleanup for core::error::Request.
rust-lang@b88d62e222 Rollup merge of
rust-lang#117615 - bjorn3:misc_changes, r=davidtwco
rust-lang@ec35c03143 Rollup merge of
rust-lang#117613 - jackh726:vacation, r=compiler-errors
rust-lang@60dca87cff Rollup merge of
rust-lang#117592 - thomcc:env-span-wrong, r=davidtwco
rust-lang@6bf2fb3679 Auto merge of
rust-lang#117578 - compiler-errors:derive-encode-in-rustc_type_ir, r=davidtwco
rust-lang@f9b644636f Auto merge of
rust-lang#117435 - SparrowLii:nightly_parallel, r=oli-obk,davidtwco
rust-lang@fcca978aa0 Auto merge of
rust-lang#117607 - klensy:clang-17.0.4, r=Mark-Simulacrum
rust-lang@152a4e90d1 Auto merge of
rust-lang#117585 - dnbln:feat/move-kw-span, r=cjgillot
rust-lang@f2a40e99ff use portable
AtomicU64 for powerPC and MIPS
rust-lang@7a892ab8d8 Auto merge of
rust-lang#117576 - the8472:fix-io-copy-vec, r=Mark-Simulacrum
rust-lang@c8a25eddfe Make the randomize
feature of rustc_abi additive
rust-lang@ec29a02071 Update doc comment
for CodegenBackend::link
rust-lang@ba82056a14 Use the actual
computed crate name for -Zprint-vtable-sizes
rust-lang@1a1b10fa63 Don't steal the
parse query when using --pretty
rust-lang@6e33e89337 Remove from vacation
and compiler review group
rust-lang@38ff91c453 bump clang version
for dist-x86_64-linux from 17.0.2 to 17.0.4
rust-lang@86fca873ba Use the correct span
when emitting the `env!` result
rust-lang@c077147200 fix clippy author
and failing test
rust-lang@876f698790 Add the
vis.visit_capture_by() in noop_visit_expr
rust-lang@54ce0346c0 add `fn
visit_capture_by` to MutVisitor and fix pprust-expr-roundtrip.rs
rust-lang@df85b28b72 fixes for rustfmt  
ast visitor
rust-lang@a6b41aa6ba fmt
rust-lang@241a654c07 Fix remaining uses
of `CaptureBy::Value`
rust-lang@8de489918b feat(hir): Store the
`Span` of the `move` keyword
rust-lang@bee7b5889e Derive
TyEncodable/TyDecodable implementations that are parameterized over
interner
rust-lang@78aa5e511c detect EOF earlier
rust-lang@8d8f06b277 avoid excessive
initialization when copying to a Vec
rust-lang@666cad9a5f add test for rust-lang#113375
rust-lang@384f7382ba add test for rust-lang#113381
rust-lang@248dd14fa5 update
config.example.toml
rust-lang@bf5fb7614b update bootstrap
change history
rust-lang@ab8101d019 enable parallel
rustc in nightly builds
rust-lang@231f93524d Detect misparsed
binop caused by missing semi
rust-lang@98d54da1ee document that the
null pointer has the 0 address

---------

Co-authored-by: celinval <[email protected]>
Co-authored-by: Zyad Hassan <[email protected]>
Co-authored-by: Zyad Hassan <88045115 [email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. perf-regression Performance regression. perf-regression-triaged The performance regression has been triaged. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet