[naga xtask] Run validation jobs in parallel, using jobserver. #4902

jimblandy · 2023-12-19T21:58:04Z

Run all snapshot validation tasks in parallel, using the jobserver crate to limit concurrency.

Checklist

Run cargo fmt.
Run cargo clippy.
Run cargo xtask test to run tests.
Add change to CHANGELOG.md. See simple instructions inside file.

jimblandy · 2023-12-19T22:14:33Z

Combined with #4871, this reduces the time required to validate the Naga snapshot outputs from 21s to 0.6s on my Linux build machine (omitting Metal and HLSL FXC validation, since I can't run those on Linux).

jimblandy · 2023-12-19T22:19:16Z

These commits can be landed without squashing. They should be a bit easier to review individually.

ErichDonGubler

This mostly LGTM. There's enough misc. issues that I'd like to force their discussion before actually approving, tho.

naga/xtask/src/validate.rs

naga/xtask/src/main.rs

naga/xtask/src/validate.rs

jimblandy · 2023-12-26T01:29:26Z

@ErichDonGubler Thanks for the review! I believe all comments have been resolved one way or another. Please take a look.

ErichDonGubler

@jimblandy:

@ErichDonGubler Thanks for the review! I believe all comments have been resolved one way or another. Please take a look.

Look taken! I've got some continued discussion in the threads here, but none of it is blocking IMO. Ship it!

Add an `all` method to `ValidateSubcommand`, so that we can just use matches elsewhere, and let the compile catch missing or duplicated cases for us.

…225e1744fd7. r=webgpu-reviewers,supply-chain-reviewers,teoxoy Changelog: * #4865 fix present mode for wgl By xiaopengli89 in gfx-rs/wgpu#4865 * #4873 Bump ctor from 0.2.5 to 0.2.6 By dependabot[bot] in gfx-rs/wgpu#4873 * #4874 Bump syn from 2.0.40 to 2.0.41 By dependabot[bot] in gfx-rs/wgpu#4874 * #4869 [naga wgsl-out] Include the `f` suffix on `f32` literals. By jimblandy in gfx-rs/wgpu#4869 * #4850 [naga wgsl-in] Support abstract operands to binary operators. By jimblandy in gfx-rs/wgpu#4850 * #4850 [naga wgsl-in] Support abstract operands to binary operators. By jimblandy in gfx-rs/wgpu#4850 * #4850 [naga wgsl-in] Support abstract operands to binary operators. By jimblandy in gfx-rs/wgpu#4850 * #4850 [naga wgsl-in] Support abstract operands to binary operators. By jimblandy in gfx-rs/wgpu#4850 * #4850 [naga wgsl-in] Support abstract operands to binary operators. By jimblandy in gfx-rs/wgpu#4850 * #4870 [naga wgsl] Let unary operators accept and produce abstract types. By jimblandy in gfx-rs/wgpu#4870 * #4867 Reintroduce buffer snatching Part 1 By nical in gfx-rs/wgpu#4867 * #4882 Bump zerocopy from 0.7.26 to 0.7.31 By dependabot[bot] in gfx-rs/wgpu#4882 * #4878 Buffer snatching part 2 - Refactor create_buffer By nical in gfx-rs/wgpu#4878 * #4815 Add `wgpu` crate features for backends By daxpedda in gfx-rs/wgpu#4815 * #4887 Allow clippy::pattern_type_mismatch By nical in gfx-rs/wgpu#4887 * #4886 Document wgpu & wgpu-core features By Wumpf in gfx-rs/wgpu#4886 * #4826 validation: More detailed on incompatible BGL By scoopr in gfx-rs/wgpu#4826 * #4888 Web: add support for more `RawWindowHandle` variants By daxpedda in gfx-rs/wgpu#4888 * #4890 Bump thiserror from 1.0.50 to 1.0.51 By dependabot[bot] in gfx-rs/wgpu#4890 * #4880 Simplify `ResourceMaps` By nical in gfx-rs/wgpu#4880 * #4891 Make the naga version in trunk as high as the latest published one By nical in gfx-rs/wgpu#4891 * #4893 Avoid allocating memory every time we might log a label By nical in gfx-rs/wgpu#4893 * #4894 Remove some locks in BindGroup By nical in gfx-rs/wgpu#4894 * #4862 Ensure that DeviceLostCallbackC is always called exactly once By bradwerth in gfx-rs/wgpu#4862 * #4900 Support Device fence sharing with dx12 on Windows By sotaroikeda in gfx-rs/wgpu#4900 * #4903 Bump tokio from 1.35.0 to 1.35.1 By dependabot[bot] in gfx-rs/wgpu#4903 * #4895 Check that raw buffers and raw bind groups are valid By nical in gfx-rs/wgpu#4895 * #4901 fix: docs By miaobuao in gfx-rs/wgpu#4901 * #4892 Simplify some code around buffer unmapping By nical in gfx-rs/wgpu#4892 * #4896 Buffer snatching By nical in gfx-rs/wgpu#4896 * #4851 Eagerly release GPU resources when we lose the device. By bradwerth in gfx-rs/wgpu#4851 * #4906 Use nightly for docs By cwfitzgerald in gfx-rs/wgpu#4906 * #4908 Bump syn from 2.0.41 to 2.0.42 By dependabot[bot] in gfx-rs/wgpu#4908 * #4909 Bump profiling from 1.0.12 to 1.0.13 By dependabot[bot] in gfx-rs/wgpu#4909 * #4910 Bump anyhow from 1.0.75 to 1.0.76 By dependabot[bot] in gfx-rs/wgpu#4910 * #4913 Remove id32 Feature By cwfitzgerald in gfx-rs/wgpu#4913 * #4914 Add BGL Deduplication Index Test By cwfitzgerald in gfx-rs/wgpu#4914 * #4921 Fix typo "layout pipeline layout" -> "pipeline layout" By HactarCE in gfx-rs/wgpu#4921 * #4922 Bump winit from 0.29.4 to 0.29.5 By dependabot[bot] in gfx-rs/wgpu#4922 * #4924 Inline `document-features` usage, remove dep. By ErichDonGubler in gfx-rs/wgpu#4924 * #4871 Speed up Naga's `cargo xtask validate wgsl` from 12s to 0.8s By jimblandy in gfx-rs/wgpu#4871 * #4871 Speed up Naga's `cargo xtask validate wgsl` from 12s to 0.8s By jimblandy in gfx-rs/wgpu#4871 * #4871 Speed up Naga's `cargo xtask validate wgsl` from 12s to 0.8s By jimblandy in gfx-rs/wgpu#4871 * #4930 Bump winit from 0.29.5 to 0.29.6 By dependabot[bot] in gfx-rs/wgpu#4930 * #4929 Bump web-time from 0.2.3 to 0.2.4 By dependabot[bot] in gfx-rs/wgpu#4929 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4902 [naga xtask] Run validation jobs in parallel, using jobserver. By jimblandy in gfx-rs/wgpu#4902 * #4940 Align `wgpu_types::CompositeAlphaMode` serde serializations to spec By littledivy in gfx-rs/wgpu#4940 * #4936 Bump anyhow from 1.0.76 to 1.0.77 By dependabot[bot] in gfx-rs/wgpu#4936 * #4933 Bump thiserror from 1.0.51 to 1.0.52 By dependabot[bot] in gfx-rs/wgpu#4933 * #4932 Bump syn from 2.0.42 to 2.0.43 By dependabot[bot] in gfx-rs/wgpu#4932 Differential Revision: https://phabricator.services.mozilla.com/D197519

jimblandy added area: infrastructure Testing, building, coordinating issues naga Shader Translator labels Dec 19, 2023

jimblandy requested a review from ErichDonGubler December 19, 2023 21:58

jimblandy self-assigned this Dec 19, 2023

jimblandy requested a review from a team as a code owner December 19, 2023 21:58

jimblandy force-pushed the naga-xtask-jobserver branch from fa7f9c3 to 60f1c69 Compare December 19, 2023 22:18

ErichDonGubler requested changes Dec 21, 2023

View reviewed changes

jimblandy added 6 commits December 25, 2023 15:13

[naga xtask] Move validation subcommands into their own module.

5348608

[naga xtask] Break validation subcommands into functions.

25e4535

[naga xtask] Produce output only when an error occurs.

d33df26

[naga xtask] Collect validation jobs in a Vec before running them.

d5541e2

[naga xtask] Remove ack_visiting function.

e7899c1

[naga xtask] Use indicatif crate to report validation progress.

04fe2a6

jimblandy force-pushed the naga-xtask-jobserver branch from 60f1c69 to af8f90f Compare December 25, 2023 23:16

This comment was marked as resolved.

Sign in to view

jimblandy added 2 commits December 25, 2023 17:18

[naga xtask] Run validation jobs in parallel, using jobserver.

cd6262d

[naga xtask] Add validate all subcommand.

517cde9

jimblandy force-pushed the naga-xtask-jobserver branch from af8f90f to 517cde9 Compare December 26, 2023 01:18

jimblandy added 2 commits December 25, 2023 17:21

[naga xtask] Use log::error in preference to eprintln.

4ab26d2

[naga xtask] Use anyhow::ensure! where appropriate.

b22b565

jimblandy requested a review from ErichDonGubler December 26, 2023 01:28

ErichDonGubler approved these changes Dec 26, 2023

View reviewed changes

[naga xtask] Add and use ValidateSubcommand::all.

efb563e

Add an `all` method to `ValidateSubcommand`, so that we can just use matches elsewhere, and let the compile catch missing or duplicated cases for us.

jimblandy mentioned this pull request Dec 27, 2023

Naga xtask should be less noisy #4938

Open

jimblandy merged commit e45a86f into gfx-rs:trunk Dec 27, 2023
27 checks passed

jimblandy deleted the naga-xtask-jobserver branch March 31, 2024 18:05

ErichDonGubler mentioned this pull request May 3, 2024

[naga xtask] Minor simplifications. #4883

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[naga xtask] Run validation jobs in parallel, using jobserver. #4902

[naga xtask] Run validation jobs in parallel, using jobserver. #4902

jimblandy commented Dec 19, 2023 •

edited

Loading

jimblandy commented Dec 19, 2023

jimblandy commented Dec 19, 2023

ErichDonGubler left a comment

This comment was marked as resolved.

jimblandy commented Dec 26, 2023

ErichDonGubler left a comment

[naga xtask] Run validation jobs in parallel, using jobserver. #4902

[naga xtask] Run validation jobs in parallel, using jobserver. #4902

Conversation

jimblandy commented Dec 19, 2023 • edited Loading

jimblandy commented Dec 19, 2023

jimblandy commented Dec 19, 2023

ErichDonGubler left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

jimblandy commented Dec 26, 2023

ErichDonGubler left a comment

Choose a reason for hiding this comment

jimblandy commented Dec 19, 2023 •

edited

Loading