`rustc` inside `make -j2` warns (regression from 1.75.0) #120515

ojeda · 2024-01-30T23:28:27Z

Non-recursive calls of rustc from Make in beta warn about the jobserver being there but not being usable. Thus users with calls to rustc will start to see warnings in their builds unless they start marking the calls as recursive.

Make fills MAKEFLAGS in all cases in 4.3, i.e. even for non-recursive cases (and in 4.4 only in some cases it passes an invalid pair). Is rustc going to try to always use the jobserver if it appears to be there even if -Zthreads is 1? The warning is nice to detect mistakenly non-recursive calls (and the Make manual asks for that), though, but it should probably be not emitted when rustc is behaving as sequential, i.e. users are not seeing the advantage, right?

Otherwise, if we want to consider this an early warning for projects to update their Makefiles, then I think the diagnostic text should provide a way to disable it easily without changes to the build system so that projects have time to adapt.

a:
	echo  'fn main() {}' | rustup run stable rustc -
	echo  'fn main() {}' | rustup run beta   rustc -
b:
	 echo 'fn main() {}' | rustup run stable rustc -
	 echo 'fn main() {}' | rustup run beta   rustc -

$ make -j2 a
echo  'fn main() {}' | rustup run stable rustc -
echo  'fn main() {}' | rustup run beta   rustc -
warning: failed to connect to jobserver from environment variable `MAKEFLAGS=" -j2 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured

$ make -j2 b
echo 'fn main() {}' | rustup run stable rustc -
echo 'fn main() {}' | rustup run beta   rustc -

Warning added at #113730
Cc #113349
Cc @belovdv @nnethercote @oksbsb @bjorn3 @SparrowLii
@rustbot modify labels: regression-from-stable-to-beta -regression-untriaged

The text was updated successfully, but these errors were encountered:

bjorn3 · 2024-01-31T08:12:32Z

Rustc uses parallelism for both frontend parallelism (currently disabled by default, configurable using -Zthreads=) and backend parallelism (doing optimizations on multiple codegen backends at the same time). Backend parallelism has been supported for years and is on by default. Rustc looks up the jobserver very early in the compilation process to ensure that if MAKEFLAGS references a non-existent fd rustc hasn't accidentally opened this fd before checking if it actually exists, which would cause rustc to read/write an arbitrary file, corrupting it. AFAIK there is no way to prevent it from trying to look up the jobserver other than unsetting MAKEFLAGS. You also would want rustc to respect the jobserver once rustc is no longer forced to use a single codegen unit through the use of --emit obj.

ojeda · 2024-01-31T10:59:44Z

Right, I should have added -Ccodegen-units=1 (we do that explicitly on top of --emit=obj and others).

The point is that similar users may wonder what rustc is trying to parallelize now (at upgrade time) when it was requested to behave sequentially.

I understand that it is simpler (and more flexible for the future) to tell users that they should always let rustc use the jobserver even when they pass -Ccodegen-units=1, future -Cthreads=1 and possibly other sources of parallelism in the future (which is why the warning is nice), but in that case, I would suggest mentioning this requirement and the jobserver in the user-facing rustc docs and adding a note in the diagnostic to those docs. It is already mentioned in Cargo's reference, but I don't see it in the rustc book.

To be clear, it is not an issue for us -- I will just pass the jobserver.

the8472 · 2024-01-31T11:37:47Z

per-rustc parallelism seems like a red herring to me. In principle rustc could yield its implicit token to the jobserver even when it's running single-threaded.
The issue is that the environment instructs the process via MAKEFLAGS that there exists a jobserver which is passed via inherited file descriptors. This is verified at startup and found to be a lie, which can result in safety/correctness bugs.

A more robust way to do this is to use the named-fifo protocol where the environment variables can't go out of sync with descriptor inheritance. Support for that was added last year to jobserver-rs. But this requires gnu make 4.4

Passing the jobserver is not a requirement. Removing --jobserver-auth works too. It only warns when the environment variable and the file descriptors go out of sync.

ojeda · 2024-01-31T16:38:57Z

If I understand correctly, you are saying rustc could always take advantage of the jobserver. That is fine; however, what I am saying is that it should be documented that rustc may precisely do that, and thus recommend that users call rustc from Make as recursive.

Ideally, rustc should also describe what it is supposed to happen if things like -Cthreads/-Ccodegen-units are passed and the jobserver is also there, like Make suggests:

If you have a command-line argument controlling the parallel operation of your tool, consider whether your tool should detect situations where both the jobserver and the command-line argument are specified, and how it should react.

Passing the jobserver is not a requirement. Removing --jobserver-auth works too.

That is still a requirement: "mark it as recursive (recommended) or workaround make by removing the flag". In other words, what normal users should do is actually always call rustc as recursive if they want to support 4.3 and simple pipe mode in 4.4.

It only warns when the environment variable and the file descriptors go out of sync.

I guess it depends on what you include in "out of sync". Note that Make fills MAKEFLAGS with the jobserver regardless of whether it is marked as recursive or not (so the child will not be able to connnect). And in 4.4 in pipe mode it will even fill the variable twice in order to pass explicitly invalid descriptors if it considers the invocation to be non-recursive.

By the way, now that I take a closer look, I see other details that could use some tweaking:

rustc warns when finding an unknown jobserver style, which the Make manual suggests to avoid (since new kinds may be introduced in the future):

If your tool does not recognize the format of the --jobserver-auth string, it should assume the jobserver is using a different style and it cannot connect.
rustc seems to use the first --jobserver-auth it finds, instead of the last, as Make asks. So, for instance, in the invalid descriptor case, I see it trying to open the first pair instead of the invalid pair:

Be aware that the MAKEFLAGS variable may contain multiple instances of the --jobserver-auth= option. Only the last instance is relevant.

I should probably open another issue for that. These may be issues for jobserver, but others are on rustc's side, e.g. it should go with the default client if it gets CannotParse.

the8472 · 2024-01-31T16:43:18Z

And in 4.4 in pipe mode it will even fill the variable twice in order to pass explicitly invalid descriptors if it considers the invocation to be non-recursive.

What values does it use? Are they can't-possibly-be-valid values, like -1? If so we should handle that as if they were not set.

ojeda · 2024-01-31T16:52:10Z

It uses (-2, -2). Yeah, you could explicitly handle those. Sadly, that only happens in 4.4 in pipe mode [*], which is why users likely want to call rustc as recursive if they intend to support 4.3:

a:
	@echo 'fn main() {}' | rustc -

$ make-4.4.1 -j2 --jobserver-style=pipe
warning: failed to connect to jobserver from environment variable `MAKEFLAGS=" -j2 --jobserver-auth=3,4 --jobserver-auth=-2,-2"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured

That test also shows that it attempts to use the first flag, not the last, too (it tries to open 3, not -2).

[*] Note that 4.4 only does it if it "considers" it non-recursive, e.g. if you have another command marked as recursive in the same recipe, the non-recursive will still be considered recursive... So you will again get the supposedly valid ones. It is quite confusing.

ojeda · 2024-01-31T17:49:22Z

I just filled the other two issues since they are separate.

I am happy to open the documentation one if we all agree it should be so (or we can reuse this one if you prefer).

petrochenkov · 2024-01-31T20:23:12Z

It uses (-2, -2).

IIRC, @belovdv planned to support the negative file descriptor case in jobserver-rs as make docs suggest (i.e. ignore, not warn), but the plans didn't materialize yet.

weihanglo · 2024-02-01T14:31:27Z

Are we planning to fix this and do beta backport?

petrochenkov · 2024-02-01T20:17:09Z

I think we can release a new version of jobserver-rs with the fixes, and then backport the version bump in rustc.

ojeda · 2024-02-01T20:32:18Z

Two are on rustc's side: this one (for which the solution may be documenting that rustc may always use the jobserver, and thus recommend users to always enable it & mark as recursive, if others agree) and #120532 (since it is about how the error is handled in rustc's side).

apiraino · 2024-02-02T22:11:44Z

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize P-medium

petrochenkov · 2024-02-08T10:58:58Z

@weihanglo
I've released the new version of jobserver-rs, it can be updated in rustc and cargo now.

ojeda · 2024-02-08T12:23:47Z

Thanks Vadim!

@petrochenkov

Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>

@petrochenkov

Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>

@petrochenkov

Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>

@petrochenkov

Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>

`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>

@petrochenkov

Explicitly document that the jobserver may be used by `rustc` and show the warning to increase the chances that this document is found when searching for solutions online. In particular, add a section about the interaction with build systems, which is intended to contain recommendations on how to integrate `rustc` with different built systems. For GNU Make, recommend using the ` ` indicator. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). For CMake, recommend using `JOB_SERVER_AWARE` and show a workaround using `$(MAKE)` for earlier versions (when using the Makefile generator). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>

`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>

@petrochenkov

Explicitly document that the jobserver may be used by `rustc` and show the warning to increase the chances that this document is found when searching for solutions online. In particular, add a section about the interaction with build systems, which is intended to contain recommendations on how to integrate `rustc` with different built systems. For GNU Make, recommend using the ` ` indicator. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). For CMake, recommend using `JOB_SERVER_AWARE` and show a workaround using `$(MAKE)` for earlier versions (when using the Makefile generator). From: rust-lang#120515 Cc: @petrochenkov @belovdv @weihanglo @bjorn3 Signed-off-by: Miguel Ojeda <[email protected]>

rustc: document the jobserver Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang#120515 Cc: `@petrochenkov` `@belovdv` `@weihanglo` `@bjorn3` --- v2: To be able to use tab characters for the Make examples, add `` support to `tidy`. v3: Added "Integration with build systems" section to hold the GNU Make one. Added "by clearing the `MAKEFLAGS` variable". Added "aforementioned" so that it is clear we are talking about the warning above. v4: Added CMake subsection. Added a note that `rustc` may be affected by other flags, e.g. `CARGO_MAKEFLAGS`. v5: Added that `rustc` will choose the number of jobs if a jobserver is not passed.

`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>

rustc: document the jobserver Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang/rust#120515 Cc: `@petrochenkov` `@belovdv` `@weihanglo` `@bjorn3` --- v2: To be able to use tab characters for the Make examples, add `` support to `tidy`. v3: Added "Integration with build systems" section to hold the GNU Make one. Added "by clearing the `MAKEFLAGS` variable". Added "aforementioned" so that it is clear we are talking about the warning above. v4: Added CMake subsection. Added a note that `rustc` may be affected by other flags, e.g. `CARGO_MAKEFLAGS`. v5: Added that `rustc` will choose the number of jobs if a jobserver is not passed.

ojeda · 2024-05-03T16:04:38Z

With #121564 merged (1.80), i.e. the documentation solution (https://doc.rust-lang.org/nightly/rustc/jobserver.html), I believe this one can be closed (the other issues we mentioned have their own issue).

One further improvement could perhaps be to add a note: see ... (or similar) in the diagnostic to point users to that new documentation.

`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>

rustc: document the jobserver Explicitly document that the jobserver may be used by `rustc`, as well as recommend the ` ` indicator for integration of `rustc` into GNU Make. In particular, show the warning to increase the chances that this document is found when searching for solutions online. In addition, add a note about the issue with GNU Make 4.3 since it is important that users realize they should do this even if they do not expect parallelism from `rustc`. Finally, show how to workaround the issue of `$(shell ...)` calls in recursive Make (which e.g. was needed for the Linux kernel). The GNU Make 4.4 case under `--jobserver-style=pipe` is not added since it got fixed after Rust 1.76.0 already (i.e. `rustc` will not warn if it finds the negative file descriptors). From: rust-lang/rust#120515 Cc: `@petrochenkov` `@belovdv` `@weihanglo` `@bjorn3` --- v2: To be able to use tab characters for the Make examples, add `` support to `tidy`. v3: Added "Integration with build systems" section to hold the GNU Make one. Added "by clearing the `MAKEFLAGS` variable". Added "aforementioned" so that it is clear we are talking about the warning above. v4: Added CMake subsection. Added a note that `rustc` may be affected by other flags, e.g. `CARGO_MAKEFLAGS`. v5: Added that `rustc` will choose the number of jobs if a jobserver is not passed.

`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]>

`rustc` (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call `rustc` with `-Ccodegen-units=1` (and `-Zthreads` is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by `rustc` [1] when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and `--print sysroot` do, but `--version` does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), `rustc` will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style). One solution is to remove the jobserver from `MAKEFLAGS`. However, we can mark the lines with calls to `rustc` (and Cargo) as recursive, which looks simpler. This is being documented as a recommendation in `rustc` [2] and allows us to be ready for the time we may use parallelism inside `rustc` (potentially now, if a user passes `-Zthreads`). Thus do so. Similarly, do the same for `rustdoc` and `cargo` calls. Finally, there is one case that the solution does not cover, which is the `$(shell ...)` call we have. Thus, for that one, set an empty `MAKEFLAGS` environment variable. Link: rust-lang/rust#120515 [1] Acked-by: Masahiro Yamada <[email protected]> Link: rust-lang/rust#121564 [2] Link: https://lore.kernel.org/r/[email protected] [ Reworded to add link to PR documenting the recommendation. ] Signed-off-by: Miguel Ojeda <[email protected]> (cherry picked from commit f1e6a71 https://github.com/AsahiLinux/linux) Signed-off-by: John Cabaj <[email protected]>

ojeda added C-bug Category: This is a bug. regression-untriaged Untriaged performance or correctness regression. labels Jan 30, 2024

ojeda mentioned this issue Jan 30, 2024

rustc wanted features & bugfixes Rust-for-Linux/linux#355

Open

41 tasks

saethlin added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Jan 30, 2024

This was referenced Jan 31, 2024

The last --jobserver-auth flag should be used rust-lang/jobserver-rs#66

Closed

rustc warns when finding an unknown jobserver style #120532

Open

belovdv mentioned this issue Feb 1, 2024

disable jobserver on unix, if file descriptors are negative rust-lang/jobserver-rs#68

Merged

rustbot added P-medium Medium priority and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Feb 2, 2024

weihanglo mentioned this issue Feb 8, 2024

chore: update to jobserver 0.1.28 rust-lang/cargo#13419

Merged

petrochenkov mentioned this issue Feb 9, 2024

Update jobserver-rs to 0.1.28 #120846

Merged

ojeda mentioned this issue Feb 24, 2024

rustc: document the jobserver #121564

Merged

Mark-Simulacrum added regression-from-stable-to-stable Performance or correctness regression from one stable version to another. and removed regression-from-stable-to-beta Performance or correctness regression from stable to beta. labels Mar 15, 2024

Mark-Simulacrum added this to the 1.76.0 milestone Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`rustc` inside `make -j2` warns (regression from 1.75.0) #120515

`rustc` inside `make -j2` warns (regression from 1.75.0) #120515

ojeda commented Jan 30, 2024

bjorn3 commented Jan 31, 2024

ojeda commented Jan 31, 2024 •

edited

Loading

the8472 commented Jan 31, 2024 •

edited

Loading

ojeda commented Jan 31, 2024

the8472 commented Jan 31, 2024

ojeda commented Jan 31, 2024 •

edited

Loading

ojeda commented Jan 31, 2024

petrochenkov commented Jan 31, 2024

weihanglo commented Feb 1, 2024

petrochenkov commented Feb 1, 2024

ojeda commented Feb 1, 2024

apiraino commented Feb 2, 2024

petrochenkov commented Feb 8, 2024

ojeda commented Feb 8, 2024

ojeda commented May 3, 2024

rustc inside make -j2 warns (regression from 1.75.0) #120515

rustc inside make -j2 warns (regression from 1.75.0) #120515

Comments

ojeda commented Jan 30, 2024

bjorn3 commented Jan 31, 2024

ojeda commented Jan 31, 2024 • edited Loading

the8472 commented Jan 31, 2024 • edited Loading

ojeda commented Jan 31, 2024

the8472 commented Jan 31, 2024

ojeda commented Jan 31, 2024 • edited Loading

ojeda commented Jan 31, 2024

petrochenkov commented Jan 31, 2024

weihanglo commented Feb 1, 2024

petrochenkov commented Feb 1, 2024

ojeda commented Feb 1, 2024

apiraino commented Feb 2, 2024

petrochenkov commented Feb 8, 2024

ojeda commented Feb 8, 2024

ojeda commented May 3, 2024

`rustc` inside `make -j2` warns (regression from 1.75.0) #120515

`rustc` inside `make -j2` warns (regression from 1.75.0) #120515

ojeda commented Jan 31, 2024 •

edited

Loading

the8472 commented Jan 31, 2024 •

edited

Loading

ojeda commented Jan 31, 2024 •

edited

Loading