Chunked downloads #373
Chunked downloads #373
Conversation
Before this change, whenever the user would download an attachment, the client (Pantalaimon) would store all the data in memory while processing it. This causes significant memory usage spikes when downloading multiple large files at once. The high memory usage is a major problem when running Pantalaimon in a memory-constrained setting. This change adds a function for lazily decrypting files (an analogue to the one for lazily encrypting files, which already exists), and a new client function for using the chunked data.
|
@PaarthShah why no movement on this? |
Mainly just that this PR was from before I became a maintainer and I'd not been keeping track of it :^) Adding myself for review for later |
There was a problem hiding this comment.
After reviewing this, I think the only real change that's required is that we need to preserve the non-streaming versions of these tests; other than that, things look quite good to me!
There was a problem hiding this comment.
Do the existing attachment_test.py tests suffice? I don't quite remember, but that might be why I simply replaced the tests in this file instead of creating a new one.
|
Just curious if I could help this get merged by chance? |
@chookity-pokk by all means; if you want to fork/resubmit the PR, go for it |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #373 /- ##
==========================================
- Coverage 89.71% 89.64% -0.07%
==========================================
Files 42 42
Lines 8090 8142 52
==========================================
Hits 7258 7299 41
- Misses 832 843 11 ☔ View full report in Codecov by Sentry. |
|
@PaarthShah Sweet, sounds good. Would you mind just responding to @codemonium's question above about test coverage? |
|
@chookity-pokk I'm not sure I see what the question is (did you mean the post by the bot?): can you link it? |
I believe it was in reply to this discussion: #373 (comment) |
Before this change, whenever the user would download an attachment, the client (Pantalaimon) would store all the data in memory while processing it. This causes significant memory usage spikes when downloading multiple large files at once. The high memory usage is a major problem when running Pantalaimon in a memory-constrained setting.
This change adds a function for lazily decrypting files (an analogue to the one for lazily encrypting files, which already exists), and a new client function for using the chunked data.
See also matrix-org/pantalaimon#149