Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unify cargo check and clippy commands #631

Merged
merged 3 commits into from
Nov 3, 2023

Conversation

OmarTawfik
Copy link
Collaborator

@OmarTawfik OmarTawfik commented Nov 2, 2023

So that all targets/features/packages are checked in the same way.
Additionally, I'm removing rust-analyzer.checkOnSave from settings, as I suggest leaving this up to individual user preferences. Running checks for a large workspace is expensive.

Copy link

changeset-bot bot commented Nov 2, 2023

⚠️ No Changeset found

Latest commit: 42c39cd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@OmarTawfik OmarTawfik marked this pull request as ready for review November 2, 2023 05:13
@OmarTawfik OmarTawfik requested a review from a team as a code owner November 2, 2023 05:13
Copy link
Contributor

@Xanewok Xanewok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks! Since we're changing this code, I'd like to also fix the --offline flag for the check runs.

pub fn get_check_command(subcommand: impl AsRef<str>) -> Result<Command> {
let mut command = Command::new("cargo")
.arg(subcommand.as_ref())
.flag("--offline")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this should be --frozen, if I understand your initial intention of making sure the lockfile is preserved and enforced.

This bit me twice already, when we added some dependencies but my local infra check/test failed because we specify --offline and I actually needed to download new dependencies.

Copy link
Collaborator Author

@OmarTawfik OmarTawfik Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these are two separate concerns:

On one hand, infra setup cargo is responsible for doing that .. locally, it will cargo fetch to download any new deps .. but in CI, it uses uses cargo fetch --locked to make sure Cargo.lock is up to date. We don't use cargo fetch --frozen, because the cache is rarely up to date, and we don't want to fail for that.

On the other hand, infra check cargo or infra test cargo ] use --offline (both locally and CI) for different reasons; Cargo will periodically try to update its manifest/local cache from internet, blocking any local builds/tests. This periodically means 1min interruptions for local development, and even blocking development completely if you don't have internet (working offline). --offline prevents that, and makes sure it is always reusing the dependencies already downloaded by infra setup ..

In reality, I never really ran into the problem of missing dependencies, because if I added things in my VS Code, Rust Analyzer will automatically run cargo check on file save (without --locked), which will attempt to download them regardless. I wonder if it is different in your case? isn't it not installed as soon as you save the new Cargo.toml file?

In any case, I added these comments to the source code for clarity!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't so focused so much on using --frozen specifically but rather not using --offline unconditionally for check/test locally (we could also use --locked - I understood that enforcing same Cargo.lock is the key here).

The CI case is clear cut - we always expect to run these 4 commands sequentially, so we always run setup first, which fetches the dependencies. One could argue that --offline is too much and just --locked would suffice to make sure the committed Cargo.lock is up to date, but let's leave that.

However, in the local case, I would expect that the subcommands are independent of each other, with the notable exception of having to run setup first.

After running it once, I shouldn't be expected to always run setup before running check/test/lint because it adds overhead and somewhat defeats the purpose of having an ergonomic build system facade.

In reality, I never really ran into the problem of missing dependencies (...)

I just mentioned that this happened to me twice already.

I'm not always running a background check and/or VS Code, so relying on the accidental dependency fetch done by Rust Analyzer is not really reliable, and also goes against the another argument of being blocked by accidental Cargo build, as each build (this includes RA check build) locks the target directory and slows both editor analysis and the infra call.

Cargo will periodically try to update its manifest/local cache from internet

Could you elaborate on that? I don't think Cargo attempts to hit the network again and/or download the registry once it already resolves the lockfile and downloads the relevant packages, unless asked to (e.g. update).

If the index download performance is a concern, maybe we should be using Cargo's sparse-registry feature? See the link for more details, but long story short, this allows to download the index over a HTTP over resolving it via git, which speeds up the initial crates.io index fetch tremendously.

If we want to add an explicit support for --offline, I think this is better done as an extra flag to the infra rather than doing this unconditionally, as explained per above.

Copy link
Collaborator Author

@OmarTawfik OmarTawfik Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on that? I don't think Cargo attempts to hit the network again and/or download the registry once it already resolves the lockfile and downloads the relevant packages, unless asked to (e.g. update).

Even cargo check or cargo test can routinely update its offline copy of crates.io index (IIRC, fetching from this repo), which can take up to a minute to complete. I'm not sure which exact conditions that trigger it.

this happened to me twice already.
I'm not always running a background check and/or VS Code

I see. In that case, I will remove -offline for now, as it defeats the purpose if it actually blocks local development. I'll look into implementing a better solution when/if that comes up again. Will also look up sparse-registry! thanks for the link.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed!

crates/infra/utils/src/cargo/workspace.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@Xanewok Xanewok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate the change, thank you!

@OmarTawfik OmarTawfik added this pull request to the merge queue Nov 3, 2023
Merged via the queue into NomicFoundation:main with commit 36be89c Nov 3, 2023
1 check passed
@OmarTawfik OmarTawfik deleted the clippy-fix branch November 3, 2023 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants