`strip` binary is `PATH` dependent which breaks builds in MacOS #123114

n8henrie · 2024-03-27T00:21:45Z

I use MacOS frequently, and I've been bitten multiple times by issues related to the fact that one must have /usr/bin/strip first on PATH, prior to any strip command that may be installed by homebrew, nix, or a custom command.

As I also use Linux quite a bit, I prefer having the GNU version of utilities like grep, sed, and find, and I keep these first in my PATH so that my scripts keep the behavior I'm accustomed to. However, this leads to the wrong strip also being first on PATH, which breaks things.

Rust's current behavior with strip leaves me in a bit of a tough spot. In particular with the recent change to strip by default for release builds, I've found that cargo install foo unexpectedly crashes and burns if my PATH doesn't lead with /usr/bin:....

It seems like this behavior might be coming from

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Line 1102 in 47ecded

fn strip_symbols_with_external_utility<'a>(

If so, it seems like it would be trivial to prefix PATH with /usr/bin (on MacOS specifically) in this function to work around this difficult behavior in a way that shouldn't affect other OSes and that should be universally correct on MacOS.

Would such a change be reasonable to consider? If so, I'd happily work on a PR.

The text was updated successfully, but these errors were encountered:

workingjubilee · 2024-03-27T01:21:16Z

@n8henrie Hello! Has a bug been filed with Homebrew about this? It would be most appropriate for the Homebrew package manager to not distribute a flawed version of binutils rather than us working around them distributing a flawed version of binutils, and this is surely causing problems for other systems programming languages (or program build systems) on macOS that try to make use of the available utilities instead of shipping their own version.

My understanding is that such a bug would be filed at https://github.com/Homebrew/homebrew-core/ right?

Likewise with Nix, I believe one would file this bug with nixpkgs, right? That would be over here: https://github.com/NixOS/nixpkgs

Working around the fact that lots of people on macOS are going to hit this problem is still on the table, though I would (personally, slightly) prefer a solution that doesn't involve overriding an explicit command for "where should we first look for binaries" (i.e. $PATH). I realize that often it is manipulated programmatically but it seems equally likely to cause problems if we simply ignore it by prefixing a different path each time.

workingjubilee · 2024-03-27T01:24:42Z

Also if I'm misunderstanding what you mean by "wrong strip" then please correct me! It just sounds like "wrong" here means "one that is going to invent some other behavior, or perform it inadequately, rather than correctly performing the requested debuginfo stripping on macOS".

n8henrie · 2024-03-27T01:39:16Z

Hi, thanks for the quick response!

I'm pretty sure that the strip binaries provided by nix and homebrew work fine when used as a standalone tool, this is a rust / cargo specific bug -- I wouldn't be surprised if it has something to do with code signing (like many, many bugs on MacOS).

I'll do a little more testing to make sure I'm not misremembering and come back with more info.

n8henrie · 2024-03-27T01:41:53Z

Also, there is still the issue of "a user happens to have a custom utility named strip on their path" -- it would be nice if this didn't break cargo in unexpected (and potentially difficult to track down) ways. But I suppose that would affect more than just MacOS.

workingjubilee · 2024-03-27T01:45:45Z

🤔 I see! I am curious how it is "supposed" to be used then...? I am not aware of our usage of strip being anything interesting.

n8henrie · 2024-03-27T02:44:57Z

Demonstrating the issue:

$ # nix's strip is first on my PATH
$ type -a strip
strip is /etc/profiles/per-user/n8henrie/bin/strip
strip is /run/current-system/sw/bin/strip
strip is /usr/bin/strip
$ 
$ sw_vers
ProductName:            macOS
ProductVersion:         14.3.1
BuildVersion:           23D60
$ arch
arm64
$
$ cargo init --name foo
     Created binary (application) package
$
$ # stable seems to work fine
$ cargo run
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished dev [unoptimized   debuginfo] target(s) in 0.33s
     Running `target/debug/foo`
Hello, world!
$ cargo run --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished release [optimized] target(s) in 0.12s
     Running `target/release/foo`
Hello, world!
$ cargo --version
cargo 1.76.0 (c84b36747 2024-01-18)
$
$ # only nightly with --release shows the issue
$ cargo  nightly --version
cargo 1.78.0-nightly (2fe739fcf 2024-03-15)
$ cargo  nightly run
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished `dev` profile [unoptimized   debuginfo] target(s) in 0.45s
     Running `target/debug/foo`
Hello, world!
$ cargo  nightly run --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished `release` profile [optimized] target(s) in 0.14s
     Running `target/release/foo`
Killed: 9
$ cargo clean
     Removed 64 files, 2.5MiB total
$ # putting /usr/bin/strip ahead of the rest of the PATH seems to fix the issue
$ PATH=$(getconf PATH):~/.cargo/bin cargo  nightly run --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished `release` profile [optimized] target(s) in 0.55s
     Running `target/release/foo`
Hello, world!

nix's strip seems to work fine by itself (compare the size pre / post):

$ cargo build
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished dev [unoptimized   debuginfo] target(s) in 0.35s
$ stat target/debug/foo
  File: target/debug/foo
  Size: 403592          Blocks: 792        IO Block: 4096   regular file
Device: 1,25    Inode: 618156267   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:42:52.455995870 -0600
Modify: 2024-03-26 20:42:52.456156287 -0600
Change: 2024-03-26 20:42:52.463580716 -0600
 Birth: 2024-03-26 20:42:52.455995870 -0600
$ /etc/profiles/per-user/n8henrie/bin/strip target/debug/foo
$ echo $?
0
$ stat target/debug/foo
  File: target/debug/foo
  Size: 274800          Blocks: 640        IO Block: 4096   regular file
Device: 1,25    Inode: 618156267   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:43:16.342834724 -0600
Modify: 2024-03-26 20:43:16.084531123 -0600
Change: 2024-03-26 20:43:16.084551332 -0600
 Birth: 2024-03-26 20:42:52.455995870 -0600

n8henrie · 2024-03-27T02:46:32Z

The same with the MacOS strip, for comparison:

$ stat target/debug/foo
  File: target/debug/foo
  Size: 403592          Blocks: 792        IO Block: 4096   regular file
Device: 1,25    Inode: 618157322   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:45:37.877717578 -0600
Modify: 2024-03-26 20:45:37.877877328 -0600
Change: 2024-03-26 20:45:37.885154127 -0600
 Birth: 2024-03-26 20:45:37.877717578 -0600
$ /usr/bin/strip target/debug/foo
$ stat target/debug/foo
  File: target/debug/foo
  Size: 313904          Blocks: 616        IO Block: 4096   regular file
Device: 1,25    Inode: 618157396   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:45:55.569270038 -0600
Modify: 2024-03-26 20:45:55.569539246 -0600
Change: 2024-03-26 20:45:55.569747038 -0600
 Birth: 2024-03-26 20:45:55.569270038 -0600

workingjubilee · 2024-03-27T03:18:50Z

@n8henrie Er, sorry, I'm a bit confused by your demo. Did you verify the post-stripped binaries, using the variant binutils as a standalone tool, actually run? The problem seems to be when you use cargo run and not cargo build, after all.

workingjubilee · 2024-03-27T03:41:59Z

The Nix binutils distribution for macOS seems to be aware of, and deliberately tries to compensate for, the strip problem: https://github.com/NixOS/nixpkgs/blob/b07adff97cbe1f994715b987385ea94b20edbf22/pkgs/os-specific/darwin/binutils/default.nix#L16-L38

I wonder why it isn't working...?

n8henrie · 2024-03-27T12:18:59Z

Er, sorry, I'm a bit confused by your demo.

Sorry! Let me know how I can clarify. This has bitten me 3 or so different times over the last 5 years, so I've also accumulated a few preconceived notions about what's going on, I may be jumping around or making unwarranted assumptions.

The problem seems to be when you use cargo run and not cargo build, after all.

Yes, this "hello world" build fine with cargo nightly build --release, but the issue often crops up at build-time if there are build scripts (build.rs) involved (which obviously need to "run" at build-time):

$ cat build.rs
fn main() {
    println!("cargo:rerun-if-changed=src/main.rs");
}
$
$ cargo  nightly build --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL)
error: failed to run custom build command for `foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL)`

Caused by:
  process didn't exit successfully: `/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL/target/release/build/foo-f1d47ccc88c86fe0/build-script-build` (signal: 9, SIGKILL: kill)

Did you verify the post-stripped binaries, using the variant binutils as a standalone tool, actually run?

Somehow that hadn't occurred to me (SMH) and you're right:

$ cargo build
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL)
    Finished dev [unoptimized   debuginfo] target(s) in 0.25s
$ /etc/profiles/per-user/n8henrie/bin/strip target/debug/foo
$ target/debug/foo
Killed: 9

I get the same behavior running it on a "hello world" in C.

Looks like I need to do some more digging! Will close for now, thanks for your time!

workingjubilee · 2024-03-27T21:59:27Z

I think some confusion is inevitable here!

If you can produce a .nix that replicates the configuration that rearranges your PATH and produces the bad binaries, that would be lovely, but you're by no means obligated to.

n8henrie · 2024-03-27T22:59:05Z

I can work on a repro -- do you have access to a darwin machine?

Here's a thread I made on the same issue last year: https://users.rust-lang.org/t/nix-installed-strip-is-crashing-cargo-install/97019

rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Mar 27, 2024

jieyouxu added O-macos Operating system: macOS T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. C-discussion Category: Discussion or questions that doesn't represent real issues. labels Mar 27, 2024

workingjubilee added the A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) label Mar 27, 2024

n8henrie closed this as completed Mar 27, 2024

saethlin removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 27, 2024

n8henrie mentioned this issue Mar 27, 2024

Rust 1.77 will clash harder with GNU binutils on Darwin Nix NixOS/nixpkgs#299606

Open

workingjubilee mentioned this issue Apr 1, 2024

Document variance in strip rust-lang/cargo#13677

Open

LeviticusNelson mentioned this issue Jun 7, 2024

Cannot install on macOS 14.5 Decodetalkers/neocmakelsp#61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`strip` binary is `PATH` dependent which breaks builds in MacOS #123114

`strip` binary is `PATH` dependent which breaks builds in MacOS #123114

n8henrie commented Mar 27, 2024 •

edited

Loading

workingjubilee commented Mar 27, 2024 •

edited

Loading

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

n8henrie commented Mar 27, 2024

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

n8henrie commented Mar 27, 2024

workingjubilee commented Mar 27, 2024 •

edited

Loading

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

strip binary is PATH dependent which breaks builds in MacOS #123114

strip binary is PATH dependent which breaks builds in MacOS #123114

Comments

n8henrie commented Mar 27, 2024 • edited Loading

workingjubilee commented Mar 27, 2024 • edited Loading

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

n8henrie commented Mar 27, 2024

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

n8henrie commented Mar 27, 2024

workingjubilee commented Mar 27, 2024 • edited Loading

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

workingjubilee commented Mar 27, 2024

n8henrie commented Mar 27, 2024

`strip` binary is `PATH` dependent which breaks builds in MacOS #123114

`strip` binary is `PATH` dependent which breaks builds in MacOS #123114

n8henrie commented Mar 27, 2024 •

edited

Loading

workingjubilee commented Mar 27, 2024 •

edited

Loading

workingjubilee commented Mar 27, 2024 •

edited

Loading