hll memory estimate in MB instead of Bytes #312
Conversation
jshencode
requested review from
lxning,
shz117 and
voyager-dw
as code owners
| @@ -30,7 30,7 @@ import ( | |||
| ) | |||
|
|
|||
| const ( | |||
| hllQueryRequiredMemoryInMB = 10 * 1024 | |||
| hllQueryRequiredMemoryInBytes = 10 * (1 << 30) | |||
There was a problem hiding this comment.
so originally we only reserved 10K for HLL? and now 10GB, isn't a little too bigger?
There was a problem hiding this comment.
yeah we have been reserved 10K.
10GB is to make sure each card only run one hll at a time since we cannot estimate hll effectively. We have agreed on 10G at the first place, but i think at some point the code got mixed up, MB changed to bytes. In production we have enough machines to spread out the large contract queries so that we don't see the problem. but in staging we can actually see the problem
There was a problem hiding this comment.
sure, we better do 2 observation:
- how many real gpu memory allocated when running weekly hll contractor in production/staging
- how it impacts production query when hll runs with 10GB reserved. Now as we under-reserve the memory, so other queries may still able to run. After we bump to 10G, other queries in the same machine will be totally blocked even the GPU might have spare space.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #312 /- ##
========================================
Coverage 71.88% 71.89%
========================================
Files 166 166
Lines 23252 23367 115
========================================
Hits 16714 16799 85
- Misses 5246 5275 29
- Partials 1292 1293 1 ☔ View full report in Codecov by Sentry. |
|
Jian Shen seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
this drastically under estimate memory usage for hll query and cause out of memory situation during massive hll query hits, such as weekly hll contracts