Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel's execution log should include remote cache timeouts #7084

Closed
ob opened this issue Jan 10, 2019 · 6 comments
Closed

Bazel's execution log should include remote cache timeouts #7084

ob opened this issue Jan 10, 2019 · 6 comments
Labels
P2 We'll consider working on this in future. (Assignee optional) stale Issues or PRs that are stale (no activity for 30 days) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: feature request

Comments

@ob
Copy link
Contributor

ob commented Jan 10, 2019 •
edited
Loading

When debugging remote cache misses, the ability to compare two executions following the instructions in this page is invaluable. Unfortunately, I've seen cases where the only difference is that an action that in a fully-cached execution is this:

@@ -224,8  224,7 @@
     hash_function_name: "SHA-256"
   }
 }
-runner: "remote cache hit"
-remote_cache_hit: true
 runner: "local"
 
 
 ---------------------------------------------------------

This doesn't tell me why Bazel chose to run the action locally instead of fetching it from the remote cache. I know it's in the remote cache so I suspect it was due to a timeout, network error, or something like that.

Bazel should add the reason why it chose to run the action locally to the execution log.
@jin jin added team-Local-Exec Issues and PRs for the Execution (Local) team and removed team-Execution labels Jan 14, 2019
@jmmv
Copy link
Contributor

jmmv commented Jan 18, 2019

I understand that you expected the action to run remotely but it fell back to local execution for some reason? If so, this is probably a bug for the remote execution team.

@jmmv jmmv added team-Remote-Exec Issues and PRs for the Execution (Remote) team and removed team-Local-Exec Issues and PRs for the Execution (Local) team labels Jan 18, 2019
@ob ob mentioned this issue Jan 19, 2019
Open
@ob
Copy link
Contributor Author

ob commented Jan 20, 2019

I understand that you expected the action to run remotely but it fell back to local execution for some reason? If so, this is probably a bug for the remote execution team.

Note that I'm not using remote execution, only remote caching. I expected the action to be downloaded from the cache since I knew it was fully cached (I've been debugging cache misses so this was a re-run of a build that had been fully cached).

I am guessing that the remote cache took too long to reply and the bazel client decided to run the action locally, but I don't know. Having some information about why bazel decided to run the action locally (like for instance mentioning that the remote cache timed out or had an error) would be really useful.

@buchgr buchgr added P2 We'll consider working on this in future. (Assignee optional) type: feature request and removed untriaged labels Jan 21, 2019
@igorgatis
Copy link

I'm facing the same issue. @ob any recommendations?

@ob
Copy link
Contributor Author

ob commented Mar 27, 2021

I ended up debugging this by a combination of logging in the remote cache and sniffing the protocol... definitively not ideal.

@github-actions
Copy link

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1 years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

@github-actions github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label May 10, 2023
@github-actions
Copy link

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P2 We'll consider working on this in future. (Assignee optional) stale Issues or PRs that are stale (no activity for 30 days) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ob @jin @jmmv @igorgatis @buchgr @iirina