Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CI caching by skipping mtime checks for paths in $CARGO_HOME #11613

Conversation

arriven
Copy link
Contributor

@arriven arriven commented Jan 22, 2023

Skip mtime checks for paths pointing into $CARGO_HOME to avoid rebuilds when only caching $CARGO_HOME/registry/{index, cache} and $CARGO_HOME/git/db and some of the dependencies have rerun-if-changed=directory in their build.rs

I considered skipping mtime checking only on $CARGO_HOME/registry/src but it looks like other functionality (like downloading a newer version of dependency) is unaffected by this and this way we also cover the same issue with git based dependencies (except the part where cargo is forced to re-fetch submodules if $CARGO_HOME/git/checkouts is missing) and it is more in line with the discussion in #9455

Fix #11083

Credit @weihanglo for the test (I did add a case of checking that dependency update still triggers a rebuild but they are the original author of the rest of the test)

@rustbot
Copy link
Collaborator

rustbot commented Jan 22, 2023

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @epage (or someone else) soon.

Please see the contribution instructions for more information.

@rustbot rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 22, 2023
@arriven arriven force-pushed the fix/11083-rerun-if-changed-published-sources-dir branch from 53e1129 to 111a4a6 Compare January 22, 2023 16:58
Comment on lines 4874 to 4880
[COMPILING] mylib-sys [..]
[RUNNING] `rustc --crate-name build_script_build [..]
[RUNNING] `[..]build-script-build[..]`
[RUNNING] `rustc --crate-name mylib_sys [..]
[CHECKING] foo [..]
[RUNNING] `rustc --crate-name foo [..]
[FINISHED] [..]",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the test looks good but one way to test the test is if you have one commit with the test showing the old behavior and then the commit with the fix also updates the test.

Not required but something you can do to be extra sure and communicate that to reviewers.

Copy link
Contributor

@epage epage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides the feedback on the comment, this looks good!

@arriven arriven force-pushed the fix/11083-rerun-if-changed-published-sources-dir branch from 111a4a6 to 9e01c8b Compare January 24, 2023 08:27
@arriven arriven requested a review from epage January 24, 2023 08:28
@epage epage changed the title skip mtime checks for paths in $CARGO_HOME Improve CI caching by skipping mtime checks for paths in $CARGO_HOME Jan 24, 2023
@epage
Copy link
Contributor

epage commented Jan 24, 2023

@bors r

@bors
Copy link
Collaborator

bors commented Jan 24, 2023

📌 Commit 9e01c8b has been approved by epage

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 24, 2023
@bors
Copy link
Collaborator

bors commented Jan 24, 2023

⌛ Testing commit 9e01c8b with merge 99f841c...

@bors
Copy link
Collaborator

bors commented Jan 24, 2023

☀️ Test successful - checks-actions
Approved by: epage
Pushing 99f841c to master...

@bors bors merged commit 99f841c into rust-lang:master Jan 24, 2023
@arriven arriven deleted the fix/11083-rerun-if-changed-published-sources-dir branch January 24, 2023 20:07
bors pushed a commit to rust-lang-ci/rust that referenced this pull request Jan 25, 2023
11 commits in 985d561f0bb9b76ec043a2b12511790ec7a2b954..3c5af6bed9a1a243a693e8e22ee2486bd5b82a6c
2023-01-20 14:39:28  0000 to 2023-01-24 15:48:15  0000

- Add a note about verifying your email address on crates.io (rust-lang/cargo#11620)
- Improve CI caching by skipping mtime checks for paths in $CARGO_HOME (rust-lang/cargo#11613)
- test: Update for clap 4.1.3 (rust-lang/cargo#11619)
- Fix unused attribute on Windows. (rust-lang/cargo#11614)
- [Doc]: Added links to the `Target` section of the glossary for occurences of `target triple` (rust-lang/cargo#11603)
- feat: stabilize auto fix note (rust-lang/cargo#11558)
- Clarify the difference between CARGO_CRATE_NAME and CARGO_PKG_NAME (rust-lang/cargo#11576)
- Temporarily pin libgit2-sys. (rust-lang/cargo#11609)
- Disable network SSH tests on windows. (rust-lang/cargo#11610)
- fix(toml): Add `default-features` to `TomlWorkspaceDependency` (rust-lang/cargo#11409)
- doc(contrib): remove rls in release process (rust-lang/cargo#11601)
bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 25, 2023
Update cargo

11 commits in 985d561f0bb9b76ec043a2b12511790ec7a2b954..3c5af6bed9a1a243a693e8e22ee2486bd5b82a6c 2023-01-20 14:39:28  0000 to 2023-01-24 15:48:15  0000

- Add a note about verifying your email address on crates.io (rust-lang/cargo#11620)
- Improve CI caching by skipping mtime checks for paths in $CARGO_HOME (rust-lang/cargo#11613)
- test: Update for clap 4.1.3 (rust-lang/cargo#11619)
- Fix unused attribute on Windows. (rust-lang/cargo#11614)
- [Doc]: Added links to the `Target` section of the glossary for occurences of `target triple` (rust-lang/cargo#11603)
- feat: stabilize auto fix note (rust-lang/cargo#11558)
- Clarify the difference between CARGO_CRATE_NAME and CARGO_PKG_NAME (rust-lang/cargo#11576)
- Temporarily pin libgit2-sys. (rust-lang/cargo#11609)
- Disable network SSH tests on windows. (rust-lang/cargo#11610)
- fix(toml): Add `default-features` to `TomlWorkspaceDependency` (rust-lang/cargo#11409)
- doc(contrib): remove rls in release process (rust-lang/cargo#11601)

r? `@ghost`
@ehuss ehuss added this to the 1.69.0 milestone Jan 28, 2023
@tamird
Copy link
Contributor

tamird commented Jul 17, 2023

Perhaps an unintended effect of this change is that $CARGO_HOME/bin is also ignored, and that can be pretty surprising. See aya-rs/aya#668 for the nasty workaround that was required.

@weihanglo
Copy link
Member

Yep. The proposed fix is here #12090 (comment). Just no one yet picked it up.

@tamird
Copy link
Contributor

tamird commented Jul 17, 2023

Looks like #12369 is working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cargo should skip rerun-if-changed paths to files in published crate sources
7 participants