Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example #113026

jieyouxu · 2023-06-25T09:48:09Z

Preface

See issue #40713: Switch run-make tests from Makefiles to rust for more context.

Basic Description of `run-make` V2

run-make V2 aims to eliminate the dependency on make and Makefiles for building run-make-style tests. Makefiles are replaced by recipes (rmake.rs). The current implementation runs run-make V2 tests in 3 steps:

We build the support library run_make_support which the rmake.rs recipes depend on as a tool lib.
We build the recipe rmake.rs and link in the support library.
We run the recipe to build and run the tests.

rmake.rs is basically a replacement for Makefile, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example run-make tests ported over to the new infrastructure.

Example `run-make` V2 test

// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}

Follow Up Work

Adjust rustc-dev-guide docs

rustbot · 2023-06-25T09:48:17Z

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

bjorn3 · 2023-06-26T09:54:35Z

src/tools/run-make-support/src/lib.rs

+ let mut cmd = $crate::xshell::cmd!(sh, "{rustc}")
+ .arg("--out-dir")
+ .arg($scx.tmpdir())
+ .arg("-L")
+ .arg($scx.tmpdir());
+ for arg in args.split_whitespace() {
+ cmd = cmd.arg(arg);
+ }


Maybe do:

Suggested change

let mut cmd = $crate::xshell::cmd!(sh, "{rustc}")

.arg("--out-dir")

.arg($scx.tmpdir())

.arg("-L")

.arg($scx.tmpdir());

for arg in args.split_whitespace() {

cmd = cmd.arg(arg);

}

let mut cmd = $crate::xshell::cmd!(sh, concat_str!("{rustc} ", $args))

.arg("--out-dir")

.arg($scx.tmpdir())

.arg("-L")

.arg($scx.tmpdir());

This should allow interpolations the same way as xshell itself and should correctly handle quoting arguments. I haven't tested it though.

A bit unfortunate but cmd!() only accepts a literal, can concat_str! output a literal? I don't think so right? And yeah quoting arguments is a problem with the current incorrect behavior of split_whitespace.

I could force the input to be a &[], since

let args = ["hello", "world"]; let c = cmd!(sh, "echo {args...}");

works I think. This however forces the use site to become something like

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]); let output = rustc!( scx, [ "--emit=metadata", "--extern", &format!("stable={}/libstable.rmeta", scx.tmpdir()), "main.rs" ] );

Of course. cmd!() would see the concat_str!() before expanding. @matklad would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance? Such that this could be cmd!(sh, $args) where sh is Shell::new().wrapper_command(rustc_path).

While a proper fix to this requires a change to xshell, I will temporarily use the iterable form for the recipes since they're at least more correct than split_whitespace.

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance?

This would technically work, but I don't think that's an obvious API with a clear use-case. My answer would be, if you want to control the first arg, to use something like

cmd!(sh, "{rustc} --emit=metadata --crate-type=lib")

Like shells, xshell supports interpolation of the first argumetn (or at least it should support it).

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

(Note that passing-as-an-array here is a temporary band-aid fix, the intention is to avoid needing an array literal.) That is, ideally, the API would look something more like

let _output = rustc!("--emit=metadata --crate-type=lib stable.rs");

so the test writer doesn't have to bother with an array literal.

src/tools/run-make-support/src/lib.rs

src/bootstrap/tool.rs

src/tools/compiletest/src/runtest.rs

src/tools/run-make-support/src/lib.rs

Cargo.lock

matklad · 2023-06-26T15:03:45Z

src/tools/run-make-support/src/lib.rs

+ let mut cmd = $crate::xshell::cmd!(sh, "{rustc}")
+ .arg("--out-dir")
+ .arg($scx.tmpdir())
+ .arg("-L")
+ .arg($scx.tmpdir());
+ for arg in args.split_whitespace() {
+ cmd = cmd.arg(arg);
+ }


let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance?

This would technically work, but I don't think that's an obvious API with a clear use-case. My answer would be, if you want to control the first arg, to use something like

cmd!(sh, "{rustc} --emit=metadata --crate-type=lib")

Like shells, xshell supports interpolation of the first argumetn (or at least it should support it).

jieyouxu · 2023-06-27T03:29:04Z

I changed the implementation to rely on shell-words instead to split the input string into words that can be passed to Command::arg. The API now looks something like

extern crate run_make_support;

use run_make_support::{rustc, run, run_fail};

fn main() {
    let _output = rustc("a.rs --cfg x -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = rustc("b.rs -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = run("b");
    let _output = rustc("a.rs --cfg y -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = run_fail("b");
}

This unfortunately isn't as nice as having straight up string interpolation like xshell does, e.g. in CURRENT_RUSTC_VERSION:

extern crate run_make_support;

use run_make_support::{rustc, aux_build};

fn main() {
    let _output = aux_build("stable.rs");
    let output = rustc(format!("--emit=metadata --extern stable={}/libstable.rmeta main.rs", env!("TMPDIR")));

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}

rustbot · 2023-06-27T05:40:22Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

matklad · 2023-06-27T12:26:07Z

It might be interesting to combine "split-on-whitespace" idea with builder-style interface. This is the style of testing employed by Cargo:

https://github.com/rust-lang/cargo/blob/5febbe5587b74108165f748e79a4f8badbdf5e0e/tests/testsuite/build.rs#L4168-L4169

So,

let tmpdir = env!("TMPDIR");
let output = rustc("main.rs --emit=metadata")
    .arg(format!("--extern=stable={tempdir}/libstable.rmeta"))
    .stderr_utf8_lossy()?
;

Thinking more about this, I think "specific builders" are probably a better fit here than "generic shell".

I expect these tests to be relatively few, so we don't have to optimize suuuper hard for readability.

At the same time, I expect most of these tests to be similar, and to benefit from abstracting common functionality. Abstracting through a builder is easier than abstracting through macro-based DSL.

bors · 2023-06-30T06:56:49Z

☔ The latest upstream changes (presumably #113162) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum

r? @bjorn3 - this looks good to me in broad strokes, but looks like you've done a bunch of review here already.

src/bootstrap/tool.rs

src/tools/compiletest/src/lib.rs

jieyouxu · 2024-02-29T13:12:10Z

tools.mk special cases Windows and uses PATH=PATH:$TARGET_RPATH_DIR for executables instead of just using $TARGET_RPATH_ENV for non-Windows platforms...

jieyouxu · 2024-02-29T14:00:24Z

DLL_NOT_FOUND but am not sure which DLL is not found... One definite bug is PATH=$1;$2 seperator is ; on Windows and not : like *nixes.

bjorn3 · 2024-02-29T14:51:34Z

src/tools/run-make-support/src/lib.rs

 target_rpath_env_path.push_str(&std::env::var("TARGET_RPATH_ENV").unwrap());
- target_rpath_env_path.push(':');
+ target_rpath_env_path.push(path_sep(&target));


It is the host that matters here, right? You can probably use std::env::join_paths here.

Yep, that did it!

jieyouxu · 2024-02-29T16:32:45Z

Seems to finally work for msvc!
@rustbot ready

bjorn3 · 2024-03-01T15:18:47Z

@bors r

bors · 2024-03-01T15:18:50Z

📌 Commit 48e9f92 has been approved by bjorn3

It is now in the queue for this repository.

bors · 2024-03-01T16:44:00Z

⌛ Testing commit 48e9f92 with merge 17edace...

bors · 2024-03-01T19:05:33Z

☀️ Test successful - checks-actions
Approved by: bjorn3
Pushing 17edace to master...

rust-timer · 2024-03-01T20:57:06Z

Finished benchmarking commit (17edace): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.7%	[1.7%, 3.5%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 651.67s -> 651.533s (-0.02%)
Artifact size: 311.10 MiB -> 311.14 MiB (0.01%)

Pkgsrc changes: * Adapt checksums and patches, some have beene intregrated upstream. Upstream chnages: Version 1.78.0 (2024-05-02) =========================== Language -------- - [Stabilize `#[cfg(target_abi = ...)]`] (rust-lang/rust#119590) - [Stabilize the `#[diagnostic]` namespace and `#[diagnostic::on_unimplemented]` attribute] (rust-lang/rust#119888) - [Make async-fn-in-trait implementable with concrete signatures] (rust-lang/rust#120103) - [Make matching on NaN a hard error, and remove the rest of `illegal_floating_point_literal_pattern`] (rust-lang/rust#116284) - [static mut: allow mutable reference to arbitrary types, not just slices and arrays] (rust-lang/rust#117614) - [Extend `invalid_reference_casting` to include references casting to bigger memory layout] (rust-lang/rust#118983) - [Add `non_contiguous_range_endpoints` lint for singleton gaps after exclusive ranges] (rust-lang/rust#118879) - [Add `wasm_c_abi` lint for use of older wasm-bindgen versions] (rust-lang/rust#117918) This lint currently only works when using Cargo. - [Update `indirect_structural_match` and `pointer_structural_match` lints to match RFC] (rust-lang/rust#120423) - [Make non-`PartialEq`-typed consts as patterns a hard error] (rust-lang/rust#120805) - [Split `refining_impl_trait` lint into `_reachable`, `_internal` variants] (rust-lang/rust#121720) - [Remove unnecessary type inference when using associated types inside of higher ranked `where`-bounds] (rust-lang/rust#119849) - [Weaken eager detection of cyclic types during type inference] (rust-lang/rust#119989) - [`trait Trait: Auto {}`: allow upcasting from `dyn Trait` to `dyn Auto`] (rust-lang/rust#119338) Compiler -------- - [Made `INVALID_DOC_ATTRIBUTES` lint deny by default] (rust-lang/rust#111505) - [Increase accuracy of redundant `use` checking] (rust-lang/rust#117772) - [Suggest moving definition if non-found macro_rules! is defined later] (rust-lang/rust#121130) - [Lower transmutes from int to pointer type as gep on null] (rust-lang/rust#121282) Target changes: - [Windows tier 1 targets now require at least Windows 10] (rust-lang/rust#115141) - [Enable CMPXCHG16B, SSE3, SAHF/LAHF and 128-bit Atomics in tier 1 Windows] (rust-lang/rust#120820) - [Add `wasm32-wasip1` tier 2 (without host tools) target] (rust-lang/rust#120468) - [Add `wasm32-wasip2` tier 3 target] (rust-lang/rust#119616) - [Rename `wasm32-wasi-preview1-threads` to `wasm32-wasip1-threads`] (rust-lang/rust#122170) - [Add `arm64ec-pc-windows-msvc` tier 3 target] (rust-lang/rust#119199) - [Add `armv8r-none-eabihf` tier 3 target for the Cortex-R52] (rust-lang/rust#110482) - [Add `loongarch64-unknown-linux-musl` tier 3 target] (rust-lang/rust#121832) Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. Libraries --------- - [Bump Unicode to version 15.1.0, regenerate tables] (rust-lang/rust#120777) - [Make align_offset, align_to well-behaved in all cases] (rust-lang/rust#121201) - [PartialEq, PartialOrd: document expectations for transitive chains] (rust-lang/rust#115386) - [Optimize away poison guards when std is built with panic=abort] (rust-lang/rust#100603) - [Replace pthread `RwLock` with custom implementation] (rust-lang/rust#110211) - [Implement unwind safety for Condvar on all platforms] (rust-lang/rust#121768) - [Add ASCII fast-path for `char::is_grapheme_extended`] (rust-lang/rust#121138) Stabilized APIs --------------- - [`impl Read for &Stdin`] (https://doc.rust-lang.org/stable/std/io/struct.Stdin.html#impl-Read-for-&Stdin) - [Accept non `'static` lifetimes for several `std::error::Error` related implementations] (rust-lang/rust#113833) - [Make `impl<Fd: AsFd>` impl take `?Sized`] (rust-lang/rust#114655) - [`impl From<TryReserveError> for io::Error`] (https://doc.rust-lang.org/stable/std/io/struct.Error.html#impl-From-for-Error) These APIs are now stable in const contexts: - [`Barrier::new()`] (https://doc.rust-lang.org/stable/std/sync/struct.Barrier.html#method.new) Cargo ----- - [Stabilize lockfile v4](rust-lang/cargo#12852) - [Respect `rust-version` when generating lockfile] (rust-lang/cargo#12861) - [Control `--charset` via auto-detecting config value] (rust-lang/cargo#13337) - [Support `target.<triple>.rustdocflags` officially] (rust-lang/cargo#13197) - [Stabilize global cache data tracking] (rust-lang/cargo#13492) Misc ---- - [rustdoc: add `--test-builder-wrapper` arg to support wrappers such as RUSTC_WRAPPER when building doctests] (rust-lang/rust#114651) Compatibility Notes ------------------- - [Many unsafe precondition checks now run for user code with debug assertions enabled] (rust-lang/rust#120863) This change helps users catch undefined behavior in their code, though the details of how much is checked are generally not stable. - [riscv only supports split_debuginfo=off for now] (rust-lang/rust#120518) - [Consistently check bounds on hidden types of `impl Trait`] (rust-lang/rust#121679) - [Change equality of higher ranked types to not rely on subtyping] (rust-lang/rust#118247) - [When called, additionally check bounds on normalized function return type] (rust-lang/rust#118882) - [Expand coverage for `arithmetic_overflow` lint] (rust-lang/rust#119432) Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Update to LLVM 18](rust-lang/rust#120055) - [Build `rustc` with 1CGU on `x86_64-pc-windows-msvc`] (rust-lang/rust#112267) - [Build `rustc` with 1CGU on `x86_64-apple-darwin`] (rust-lang/rust#112268) - [Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example] (rust-lang/rust#113026) - [Windows: Implement condvar, mutex and rwlock using futex] (rust-lang/rust#121956)

rustbot assigned Mark-Simulacrum Jun 25, 2023

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 25, 2023

This comment has been minimized.

Sign in to view