Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel formatting to increase speed #6095

Open
wants to merge 23 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add some comments
  • Loading branch information
MarcusGrass committed Feb 26, 2024
commit 2d8c7aa1225cd3aa2e028b1fe451e702dbd95236
4 changes: 2 additions & 2 deletions Progress.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ either the printing facilities need to be reinvented (no), or some compatibility

## Changes

### Output facilities

In practice there are four printing facilities used:

1. Regular `stdout`, pretty easy, replace `println` with printing into a buffer.
Expand All @@ -24,5 +26,3 @@ In practice there are four printing facilities used:
4. Term stderr, this is done by `rustc_error` and the most complex to integrate.

Additionally, these four facilities can't be separated, since they have to preserve order between each other.


14 changes: 10 additions & 4 deletions src/bin/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -329,14 +329,17 @@ fn format(
}

let parallelism = std::thread::available_parallelism().unwrap_or(NonZeroUsize::MIN);
// Use a channel + map to get 'next-completed' thread, rather than
// waiting on the chronologically first handle to join, if there are more files
// than available parallelism.
let (send, recv) = std::sync::mpsc::channel();
let mut handles: HashMap<i32, JoinHandle<_>> = HashMap::new();

let mut exit_code = 0;
let mut outstanding = 0;
// If the thread panics, the channel will just be dropped,
// so keep track of the spinning threads
// so keep track of the spinning threads to get a stacktrace from it later
let mut id = 0;
let mut handles: HashMap<i32, JoinHandle<_>> = HashMap::new();
let check = options.check;
let color = config.color();
for file in files {
Expand Down Expand Up @@ -373,7 +376,7 @@ fn format(
let _ = s.send(ThreadedFileOutput {
session_result: Err(e),
id: my_id,
exit_code,
exit_code: 1,
printer,
});
return Ok::<_, std::io::Error>(());
Expand Down Expand Up @@ -431,13 +434,16 @@ fn format(
}
// Drop sender, or this will deadlock
drop(send);

// Drain running threads
while let Ok(thread_out) = recv.recv() {
let exit = join_thread_reporting_back(&mut handles, thread_out)?;
if exit != 0 {
exit_code = exit;
}
}
// These have errors

// All successful threads have been removed from `handles` only errors are left
for (_id, jh) in handles {
match jh.join() {
Ok(res) => {
Expand Down