Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MultiCompiler didn't release memory after one compiler is finished. #15521

Open
Edge00 opened this issue Mar 12, 2022 · 15 comments
Open

MultiCompiler didn't release memory after one compiler is finished. #15521

Edge00 opened this issue Mar 12, 2022 · 15 comments
Labels
blocked enhancement webpack-5 X0: need more information

Comments

@Edge00
Copy link

Edge00 commented Mar 12, 2022 •
edited
Loading

Bug report

What is the current behavior?
I use MultiCompiler configs.
If i run them one by one, Webpack can build them all.

const config = {}
webpack(config)

If i run them together, OOM appeares.

const configs = [config1, config2, config3, config4]
webpack(configs)

If the current behavior is a bug, please provide the steps to reproduce.
MultiCompiler uses more memory than single compiler.

What is the expected behavior?
Each compiler should work separately. when one is finished, who's memory should be released.

Other relevant information:
webpack version: 5.70.0
Node.js version: 14.17.1
Operating System: macOS 11.6
Additional tools: none

Part of configuration
entries (each compiler): 2 159 230
cache: dsiabled
mode: 'production'
loaders: babel / postcss / vue-laoder
minimizer: terser / css-minimizer-webpack-plugin
devtool: false
target: ['web', 'es5']
@vankop
Copy link
Member

vankop commented Mar 12, 2022

in multicompiler mode next compilation could depends on previous one https://github.com/webpack/webpack/blob/main/schemas/WebpackOptions.json#L396 . but maybe this could be improved.

@Edge00
Copy link
Author

Edge00 commented Mar 12, 2022

in multicompiler mode next compilation could depends on previous one

@vankop Is there any way to stop such behavior because in my scenario, there is no dependencies between multicompilers

@vankop
Copy link
Member

vankop commented Mar 12, 2022

not sure.. lets keep tracking issue for now.

@Edge00
Copy link
Author

Edge00 commented Mar 12, 2022

The strange thing is even all assets are emited. OOM still appeares at last.

@alexander-akait
Copy link
Member

@Edge00 Can you provide OOM message?

@Edge00
Copy link
Author

Edge00 commented Mar 13, 2022

@alexander-akait Sure

<--- Last few GCs --->

[94438:0x1048db000]   239481 ms: Scavenge (reduce) 3886.2 (4100.5) -> 3886.0 (4103.8) MB, 11.5 / 0.0 ms  (average mu = 0.090, current mu = 0.033) allocation failure
[94438:0x1048db000]   239494 ms: Scavenge (reduce) 3887.0 (4108.3) -> 3886.9 (4109.5) MB, 11.9 / 0.0 ms  (average mu = 0.090, current mu = 0.033) allocation failure
[94438:0x1048db000]   239510 ms: Scavenge (reduce) 3888.1 (4102.8) -> 3888.0 (4106.0) MB, 12.2 / 0.0 ms  (average mu = 0.090, current mu = 0.033) allocation failure


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10130d5e5 node::Abort() (.cold.1) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 2: 0x1000b2289 node::Abort() [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 3: 0x1000b23ef node::OnFatalError(char const*, char const*) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 4: 0x1001f68c7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 5: 0x1001f6863 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 6: 0x1003a47e5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 7: 0x1003a628a v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 8: 0x1003a19b5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 9: 0x10039f2e0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
10: 0x1003ad9ea v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
11: 0x1003ada71 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
12: 0x100376cce v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
13: 0x10078f54d v8::internal::IncrementalStringBuilder::Extend() [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
14: 0x1004bd866 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
15: 0x1004bf3dd v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
16: 0x1004c37c4 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
17: 0x1004c07ee v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
18: 0x1004ba503 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
19: 0x1002a0253 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
20: 0x100a82099 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]

@alexander-akait
Copy link
Member

@Edge00 Cache enabled? What is mode? If you can will be great to provide configuration

@Edge00
Copy link
Author

Edge00 commented Mar 13, 2022

@alexander-akait Sorry. The configuration are generated by some complicated logic. I supplemented some configuration information on the issue description. If you need more information please let me know. Thank you!

@webpack-bot
Copy link
Contributor

This issue had no activity for at least three months.

It's subject to automatic issue closing if there is no activity in the next 15 days.

@webpack-bot
Copy link
Contributor

Issue was closed because of inactivity.

If you think this is still a valid issue, please file a new issue with additional information.

@alexander-akait
Copy link
Member

@vankop Still valid?

@vankop
Copy link
Member

vankop commented Jun 29, 2022

lets implement compiler.runOnce that will run and close compilers. ( maybe use in in webpack-cli then )

@vankop vankop reopened this Jun 29, 2022
@vankop
Copy link
Member

vankop commented Jun 30, 2022

first of all we need to solve memory leak issue with module graph.. otherwise all modules will be in this WeakMap until compiler. _cleanupLastCompilation will not be called..

@vankop vankop added the blocked label Jun 30, 2022
@bigbossx
Copy link

bigbossx commented Aug 2, 2022 •
edited
Loading

first of all we need to solve memory leak issue with module graph.. otherwise all modules will be in this WeakMap until compiler. _cleanupLastCompilation will not be called..

the same problem we have too. I record mutli times momory useage in chrome devtool after compiler.close trigger.

image

@TheLarkInn
Copy link
Member

Do you think that you can provide a reproducible repo which portrays this memory leak @Edge00. We have interesting in working on this but hard to reproduce without a small example to profile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocked enhancement webpack-5 X0: need more information
Projects
webpack 5/6
Status: Priority - Low
Milestone
No milestone
Development

No branches or pull requests
6 participants
@TheLarkInn @alexander-akait @vankop @Edge00 @webpack-bot @bigbossx