Reduce memory consumption during aggregate state rebuild #368
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I tried benchmarking both solutions and the time it takes is around 60% of the old implementation.
Even bigger difference is in the amount of memory being allocated and not released during the rebuild. With old implementation it gradualy rises into gigabytes. With new implementation it stays around 100-200 megabytes. I am not sure if the Stream is not used correctly, or if there's some bug in the Stream internals or if it's just my special case, but it could be useful if somebody could do some benchmarks with his data to prove my suspicion.
Before change:
After change:
I tested this on around
100_000
events stored as JSONB in PostgreSQL.Benchmarking machine:
I didn't write new test(s). I assume, this is covered by
event_persistence_test.exs
.