build.ccache.global_storage policy is not working as expected #5793

for2years · 2024-11-06T08:50:52Z

Xmake Version

v2.9.6

Operating System Version and Architecture

ubuntu 2204

Describe Bug

I cloned two identical projects on the same machine and set set_policy("build.ccache.global_storage", true) for both.

However, after compilation the another repo, the cache hit rate is 0.

Expected Behavior

These two identical projects should share the same ccache in ~/.xmake/.build_cache.

Project Configuration

none

Additional Information and Error Logs

build cache stats:
cache directory: /root/.xmake/.build_cache
cache hit rate: 0%
cache hit: 0
cache hit total time: 0.000s
cache miss: 453
cache miss total time: 0.209s
new cached files: 453
remote cache hit: 0
remote new cached files: 0
preprocess failed: 0
compile fallback count: 3
compile total time: 5406.370s

The text was updated successfully, but these errors were encountered:

waruqi · 2024-11-06T09:35:16Z

it works for me. you can debug here.

xmake/xmake/modules/private/cache/build_cache.lua

Lines 131 to 133 in a71785c

    
           if not cachedir and project.policy("build.ccache.global_storage") then 
        
               cachedir = path.join(global.directory(), ".build_cache") 
        
           end

ruki-2:test ruki$ xmake -r
[ 50%]: cache compiling.release src/main.cpp
ccachedir
ccachedir2 /Users/ruki/.xmake/.build_cache
[ 75%]: linking.release test
[100%]: build ok, spent 0.772s
ruki-2:test ruki$ tree ~/.xmake/.build_cache/
/Users/ruki/.xmake/.build_cache/
└── 42
    └── 42f0b0efe4f63b66f9833ec1252d169d

add_rules("mode.debug", "mode.release")
set_policy("build.ccache.global_storage", true)

target("test")
    set_kind("binary")
    add_files("src/*.cpp")

for2years · 2024-11-06T09:40:28Z

if not cachedir and project.policy("build.ccache.global_storage") then
cachedir = path.join(global.directory(), ".build_cache")
end

set_policy("build.ccache.global_storage", true) do indeed redirect the ccache directory to ~/.xmake/.build_cache.

However, my question is that for two same projects in different directory but same machine, the cache hit rate should not be 0 after compiling them sequentially.

for2years · 2024-11-06T09:44:43Z

I want to use this policy because I cloned the same project in different directories on the same machine, like ~/A/my_project and ~/B/my_project, so I expect them to share the same cache.

waruqi · 2024-11-06T09:51:56Z

it still works for me.

ruki-2:tmp ruki$ cp -r test test2
ruki-2:test2 ruki$ xmake -r
[ 50%]: cache compiling.release src/main.cpp
ccachedir
ccachedir2 /Users/ruki/.xmake/.build_cache
[ 75%]: linking.release test
[100%]: build ok, spent 1.775s
ruki-2:test2 ruki$ tree ~/.xmake/.build_cache/
/Users/ruki/.xmake/.build_cache/
└── 42
    └── 42f0b0efe4f63b66f9833ec1252d169d

build cache stats:
cache directory: /Users/ruki/.xmake/.build_cache
cache hit rate: 100%
cache hit: 1
cache hit total time: 0.002s
cache miss: 0
cache miss total time: 0.000s
new cached files: 0
remote cache hit: 0
remote new cached files: 0
preprocess failed: 0
compile fallback count: 0
compile total time: 0.000s

ruki-2:test2 ruki$ cp -r test ../test3
cp: test: No such file or directory
ruki-2:test2 ruki$ cd ^C
ruki-2:test2 ruki$ cd ..
ruki-2:tmp ruki$ cp -r test test3
ruki-2:tmp ruki$ cd test3
ruki-2:test3 ruki$ xmake -rvD
checking for Xcode directory ... /Applications/Xcode.app
checking for Codesign Identity of Xcode ... Apple Development: [email protected] (T3NA4MRVPU)
checking for SDK version of Xcode for macosx (x86_64) ... 14.0
checking for Minimal target version of Xcode for macosx (x86_64) ... 14.0
checkinfo: cannot runv(zig version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(zig version), No such file or directory
checking for zig ... no
checking for nim ... /usr/local/bin/nim
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr
/bin/clang ... ok
checking for the c   compiler (cxx) ... clang
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr
/bin/clang ... ok
checking for flags (-fPIC) ... ok
> clang "-fPIC" "-Qunused-arguments" "-target" "x86_64-apple-macos14.0" "-isysroot" "/Applicati
ons/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (-fvisibility-inlines-hidden) ... ok
> clang "-fvisibility-inlines-hidden" "-Qunused-arguments" "-target" "x86_64-apple-macos14.0" "
-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
/MacOSX14.0.sdk"
checking for flags (-O3) ... ok
> clang "-O3" "-Qunused-arguments" "-target" "x86_64-apple-macos14.0" "-isysroot" "/Application
s/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (-DNDEBUG) ... ok
> clang "-DNDEBUG" "-Qunused-arguments" "-target" "x86_64-apple-macos14.0" "-isysroot" "/Applic
ations/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
[ 50%]: cache compiling.release src/main.cpp
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -c
 -Qunused-arguments -target x86_64-apple-macos14.0 -isysroot /Applications/Xcode.app/Contents/D
eveloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibility=hidden -fvisibili
ty-inlines-hidden -O3 -DNDEBUG -o build/.objs/test/macosx/x86_64/release/src/main.cpp.o src/mai
n.cpp
checking for flags (-MMD -MF) ... ok
> clang "-MMD" "-MF" "/dev/null" "-Qunused-arguments" "-target" "x86_64-apple-macos14.0" "-isys
root" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacO
SX14.0.sdk"
checking for flags (-fdiagnostics-color=always) ... ok
> clang "-fdiagnostics-color=always" "-Qunused-arguments" "-target" "x86_64-apple-macos14.0" "-
isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/
MacOSX14.0.sdk"
ccachedir
ccachedir2 /Users/ruki/.xmake/.build_cache
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr
/bin/clang   ... ok
checking for the linker (ld) ... clang  
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr
/bin/clang   ... ok
checking for flags (-fPIC) ... ok
> clang   "-fPIC" "-target" "x86_64-apple-macos14.0" "-isysroot" "/Applications/Xcode.app/Conte
nts/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk" "-lz" "-target" "x86_64-
apple-macos14.0" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platf
orm/Developer/SDKs/MacOSX14.0.sdk" "-lz"
[ 75%]: linking.release test
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang  
-o build/macosx/x86_64/release/test build/.objs/test/macosx/x86_64/release/src/main.cpp.o -targ
et x86_64-apple-macos14.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX
.platform/Developer/SDKs/MacOSX14.0.sdk -lz -Wl,-x -Wl,-dead_strip

build cache stats:
cache directory: /Users/ruki/.xmake/.build_cache
cache hit rate: 100%
cache hit: 1
cache hit total time: 0.002s
cache miss: 0
cache miss total time: 0.000s
new cached files: 0
remote cache hit: 0
remote new cached files: 0
preprocess failed: 0
compile fallback count: 0
compile total time: 0.000s

[100%]: build ok, spent 0.925s

for2years · 2024-11-06T10:03:23Z

ok thanks, I try to debug it.

for2years · 2024-11-07T03:02:01Z

The reason is that I used add_includedirs(os.projectdir()) which caused the compile parameters to be inconsistent.
I replace it with add_includedirs(".") and fix it.

Another issue is that if my target is a proto-target to generate *.h and *.cc according to *.proto files like:

target("my_proto_target")
    set_kind("object")
    add_rules("protobuf.cpp")
    add_files("proto/*.proto", {proto_public = true})
    add_packages("protobuf-cpp", {public = true})

It does not generate any cache files, rusulting in a cache hit rate of 0, what should I do to reuse the cache for proto files?

waruqi · 2024-11-07T03:07:18Z

same cache policy. you can debug proto files compilation flags.

And this issue should have been resolved. build.ccache.global_storage policy has no any problem.

for2years · 2024-11-07T03:08:15Z

ok thanks anyway~

for2years · 2024-11-07T03:10:11Z

you can debug proto files compilation flags.

Do you have any suggestions to help me debug the no-cache-problem for proto files. I don't know where to start actually.

waruqi · 2024-11-07T03:16:50Z

xmake/xmake/modules/core/tools/gcc.lua

Line 812 in 8693657

cppinfo = build_cache.build(program, argv, {envs = self:runenvs(),

xmake/xmake/modules/private/cache/build_cache.lua

Line 277 in 8693657

local cachekey = cachekey(program, cppinfo, opt.envs)

check the cache key and cppinfo

for2years · 2024-11-07T03:17:57Z

ok I will try it, thanks

for2years added the bug label Nov 6, 2024

for2years closed this as completed Nov 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build.ccache.global_storage policy is not working as expected #5793

build.ccache.global_storage policy is not working as expected #5793

for2years commented Nov 6, 2024

waruqi commented Nov 6, 2024

for2years commented Nov 6, 2024 •

edited

Loading

for2years commented Nov 6, 2024

waruqi commented Nov 6, 2024 •

edited

Loading

for2years commented Nov 6, 2024

for2years commented Nov 7, 2024 •

edited

Loading

waruqi commented Nov 7, 2024

for2years commented Nov 7, 2024

for2years commented Nov 7, 2024

waruqi commented Nov 7, 2024

for2years commented Nov 7, 2024

build.ccache.global_storage policy is not working as expected #5793

build.ccache.global_storage policy is not working as expected #5793

Comments

for2years commented Nov 6, 2024

Xmake Version

Operating System Version and Architecture

Describe Bug

Expected Behavior

Project Configuration

Additional Information and Error Logs

waruqi commented Nov 6, 2024

for2years commented Nov 6, 2024 • edited Loading

for2years commented Nov 6, 2024

waruqi commented Nov 6, 2024 • edited Loading

for2years commented Nov 6, 2024

for2years commented Nov 7, 2024 • edited Loading

waruqi commented Nov 7, 2024

for2years commented Nov 7, 2024

for2years commented Nov 7, 2024

waruqi commented Nov 7, 2024

for2years commented Nov 7, 2024

for2years commented Nov 6, 2024 •

edited

Loading

waruqi commented Nov 6, 2024 •

edited

Loading

for2years commented Nov 7, 2024 •

edited

Loading