-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
separate the build runner process from the configure process #20981
Comments
This breaking change may also be an opportune time to rename |
As you've already stated on discord, this will break certain use cases of custom build steps. I'd consider #20935 a blocker for this as currently we need to fall back to custom build steps to achieve re-running the step when something in the directory changes. Uses cases here are asset bundling for games/websites/... Also projects like https://github.com/zig-osdev/disk-image-step will suffer from this because of two reasons:
|
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This moves multiversion from release.zig to build.zig, so that you can just build a multiversion binary normally, eg: ``` $ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux $ ./tigerbeetle multiversion ./tigerbeetle multiversioning.exe_path=./tigerbeetle multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 } ... ``` You can pass `-Dmultiversion=latest` to fetch the latest release from GitHub. Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image (this _is_ confusing flag name, but I don't see a simple non-ambigious solution here). As a bonus, multiversion build now should work on any host, and not on x86_64 linux only. Implementation wise, the bulk of work happens in `build_multiversion.zig` program, which is invoked from build.zig as a custom Run step. Why not encode all that logic direcctly into build.zig? The main reason is adrewrk, the most famous Zig troll. His favorite way of trolling Zig developers is by regularly breaking the API of build.zig: ziglang/zig#14498 The next planned trolling is to separate "configure" and "make" phases of `build.zig` into physically separate processes, and remove support for custom steps: ziglang/zig#20981 In this PR, we are going to troll adrewrk _back_ by fixing our code even before it gets broken, and by making our custom step a Run step instead. That being said, it I have split `gh release download` into a separate step, so that is properly cached.
This might be a silly question based on my very limited understanding of the build runner, so apologies in advance if that's the case. Would it be possible to achieve the goals of
by just doing #14941 and then leaning on linking? That is, compile the build runner and the build script separately so that the build runner object is almost always a cache hit, and then link the two together to create the final build binary. Any communication between the two would then happen via Again, I'm not familiar enough with the build runner internals to know if this even makes sense. If it does, though, it seems worth considering since we wouldn't have to open the can of worms that is serialization of build state, and what that means for custom steps. |
This is a prerequisite to #14286.
Currently, the build runner process starts from the main() function inside
lib/compiler/build_runner.zig
and is compiled alongside the user'sbuild.zig
logic. The build runner invokes the user'sbuild.zig
logic, which is the "configure phase". After thebuild
function returns, the build runner proceeds to execute the build graph, which is the "make" phase.This issue is to separate the one compilation into two:
build.zig
logic and thebuild.zig
logic of its dependencies.The primary motivation for this is to make
zig build
faster, both in the sense that it will take less time to compile, because only the user'sbuild.zig
logic is being recompiled when it changes, and in the sense that the build runner could be built with optimizations enabled, which is starting to become more valuable now that we have introduced--watch
and--fuzz
.--debug-runner-optimize=[mode]
should be introduced to allow switching the build runner to be compiled in a different optimization mode; this will be helpful when working on a patch to the build runner itself.The primary challenge for implementing this will be separating out state from
std.Build.Step
that corresponds to configure phase and that corresponds to make phase. It will be a welcome change, because it will make the API a lot smaller and prevent users from having access to state that is only meant for use by the build runner. The build runner will probably have its own version of eachstd.Build.Step
with different fields and logic.The other major thing that needs to be done to make this work is serialization of the build graph. The output of the configure process needs to be communicated to the runner process, ideally by using stdout alone.
This constitutes a breaking change. In particular, custom build steps will stop working. Users should switch to using Run steps instead.
If we can't get this fully done in this release cycle, #14941 will be a good first step along the way.
The text was updated successfully, but these errors were encountered: