Can I purge old and outdated cache files? #13291
Comments
|
There is a This is an example of current large cache: The same project with a single build locally: |
|
Yep I know about this problem. MaxAge does cleanup old cache entries, but no longer used files are not removed from disk... |
|
Perhaps we can include cache outdated information in output stats |
You can set On short term we want to fix the cache issue by removing all unused files from the cache directory after writing the cache. That sounds easy but actually it a bit tricky since, due to lazy deserialization of the cache, we don't know all files that are currently in use. We might change the cache format a little bit to know these details. |
|
I'm on the same boat, using filesystem caches to speedup our CI builds. @sokra looking @ the code, I see the file serialization middleware lib/serialization/FileMiddleware is used only fore What would be a good approach for a short term fix? I have a naive approach to fix this locally passing down a https://github.com/webpack/webpack/blob/main/lib/serialization/FileMiddleware.js#L472-L474 Maybe the idea is for serialization middlewares to be more generic in the long run, so creating/deleting files specifically for cache might be too specific or not ideal? However, in the short time WDYT of this PR? #13875 Either way, any guidance is welcome. |
|
ping @sokra (cc @otakustay) I have an implementation here #13875 I'd appreciate any revie/comments |
|
Friendly Ping/reminder! :D |
|
@sokra is it still an issue? |
|
Yes |
|
In filesystem cache, the unused cache items are now splited to a new cache content file partially, perhaps we need a config to remove it, just like maxGenerations:1 works in memory cache. |
|
there is an opened PR #14661 |
|
This issue had no activity for at least three months. It's subject to automatic issue closing if there is no activity in the next 15 days. |
|
I have this issue also with the filesystem cache. I am using yarn as our package manager, and I get a .cache folder inside my root .yarn folder which starts to accumulate webpack cache files inside a "client-development" folder. It can grow (in development using the hot reloader) to several gigabytes within even an hour of frequent work. In one case, it crashed my computer because it ate all but a couple hundred megabytes of disk space when I wasn't paying attention and deleting the files in it. I am sure I might be missing some config to make this work correctly - but the performance increase for rebuilds is worth it. 😄 My webpack cache config is just this: |
|
This was from some basic react app editing with HMR on (as you can see the files are massive and they keep getting generated): |
|
Issue was closed because of inactivity. If you think this is still a valid issue, please file a new issue with additional information. |
|
Is there any progress on this issue? |
|
As an workaround, I wrote a simple webpack plugin. https://github.com/Gavin-Gong/purge-cache-webpack-plugin npm i purge-cache-webpack-plugin -D// webpack config
/** @type {import('webpack').Configuration}*/
const config = {
cache: {
type: "filesystem", // only working with filesystem cache
maxAge: 1000 * 60,
},
plugins: [
new PurgeCacheWebpackPlugin(),
// ...
],
// ...
}; |
|
Any update ? |
github-project-automation
bot
moved this from Priority - Low
to Ready for Merge
in webpack 5/6
|
This is marked as I can only see one PR referencing this issue, which is still opened. Is there any update on this? I am sorry if I missed anything. Thank you |
|
This issue Is still present on Webpack 5.91.0. |
Feature request
On out CI system we upload webpack's cache to a storage after each build and download them on the next build to speed up build.
After about 6 months, we find there is a 800MB cache directory with more than 100
.packfiles in this cache, causing slower and slower download time.What is the expected behavior?
We'd like to see an API to purge outdated cache content from these files, we can then run purge monthly, a slow purge process is acceptable since this runs only once per month.
What is motivation or use case for adding/changing the behavior?
Try to limit cache size in long term.
How should this be implemented in your opinion?
Maybe we should have a
{purge: true}incacheconfigurationAre you willing to work on this yourself?
Not able to do that
The text was updated successfully, but these errors were encountered: