Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebuild triggered after touching a (git submodule) file under the registry/src directory #10223

Closed
Ben-PH opened this issue Dec 21, 2021 · 5 comments
Labels
A-rebuild-detection Area: rebuild detection and fingerprinting C-bug Category: bug

Comments

@Ben-PH
Copy link

Ben-PH commented Dec 21, 2021

Problem

Initial presentation of bug

In our CI pipeline, one of our dependencies (specifically rocksdb) will run a rebuild, despite being cached.

Steps

I am using rocksdb as a representative

  1. Make minimalist hello, world! executable
  2. Add rocksdb = "0.17.0" as a dependency.
  3. install clang or llvm as per the crate requirements
  4. cargo build
    5.a. rm -rf ~/.cargo/registry/src
    5.b. (alternate rebuild trigger) touch /path/to/registry/src/directory/librocksdb-sys-6.20.3/rocksdb
  5. run cargo build and observe the rebuild being triggered.

To validate this is unique, I followed the steps with other executables using the rm option, without any issues.

Possible Solution(s)

Notes

The re-build appears to be triggered by this line of code

The CI setup is following the advice outlined here

Version

cargo 1.56.0 (4ed5d137b 2021-10-04)
release: 1.56.0
commit-hash: 4ed5d137baff5eccf1bae5a7b2ae4b57efad4a7d
commit-date: 2021-10-04
@Ben-PH Ben-PH added the C-bug Category: bug label Dec 21, 2021
@alexcrichton
Copy link
Member

You can try seeing why Cargo is rebuilding with:

export CARGO_LOG=cargo::core::compiler::fingerprint=info

and that should print out why the rebuild is happening

@Ben-PH
Copy link
Author

Ben-PH commented Dec 29, 2021

[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint] stale: changed "/home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/librocksdb-sys-6.20.3/rocksdb/"
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint]           (vs) "/home/ben/rust/repro/target/debug/build/librocksdb-sys-66a9853c3d7cfb95/output"
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint]                FileTime { seconds: 1640771696, nanos: 604901852 } != FileTime { seconds: 1640771857, nanos: 395909002 }
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint] fingerprint error for repro v0.1.0 (/home/ben/rust/repro)/Build/TargetInner { name: "repro", doc: true, ..: with_path("/home/ben/rust/repro/src/main.rs", Edition2021) }
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint] fingerprint error for rocksdb v0.17.0/Build/TargetInner { ..: lib_target("rocksdb", ["lib"], "/home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.17.0/src/lib.rs", Edition2018) }
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint] fingerprint error for librocksdb-sys v6.20.3/Build/TargetInner { ..: lib_target("librocksdb-sys", ["lib"], "/home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/librocksdb-sys-6.20.3/src/lib.rs", Edition2018) }
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint] fingerprint error for librocksdb-sys v6.20.3/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/librocksdb-sys-6.20.3/build.rs", Edition2018) }
[2021-12-29T09:57:37Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated

@Ben-PH
Copy link
Author

Ben-PH commented Dec 29, 2021

the following in our ci.yml file prevents the rebuild:

    - name: Zero mtimes of deps
      run: |
        set -euxo pipefail
        CARGO_DIR="${CARGO_HOME:-$HOME/.cargo}"
        for source in "$CARGO_DIR"/registry/cache/*/; do
          pushd "$source"
          source=$(basename -- "$PWD")
          mkdir -p "$CARGO_DIR/registry/src/$source"
          for crate in *.crate; do
            crate=${crate%.crate}
            mkdir "$CARGO_DIR/registry/src/$source/$crate" || continue
            tar --strip-components=1 -xzf "$crate.crate" -C "$CARGO_DIR/registry/src/$source/$crate"
            echo -n ok > "$CARGO_DIR/registry/src/$source/$crate/.cargo-ok"
          done
          popd
        done
        find "$CARGO_DIR/registry/src" -exec touch -cfht 197001010000 {}  
        ```

@alexcrichton
Copy link
Member

Does the rocksdb build script print out a rerun-if-changed pointing at the directory? If so that may be the cause of this.

@ehuss ehuss added the A-rebuild-detection Area: rebuild detection and fingerprinting label Jan 17, 2022
@epage
Copy link
Contributor

epage commented Nov 3, 2023

#11613 has us no longer checking the mtime for "immutable" sources. It looks like that should resolve this, so I'm closing this. If there is a reason to reconsider, let us know!

@epage epage closed this as not planned Won't fix, can't repro, duplicate, stale Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-rebuild-detection Area: rebuild detection and fingerprinting C-bug Category: bug
Projects
None yet
Development

No branches or pull requests

4 participants