You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When upgrading the grpcio python package in the system processes that are grpc clients from 1.57.0 to 1.57.1, memory utilization should not change drastically.
What did you see instead?
For some users, memory consumption in 1.57.1 grows continuously for the life of the process. Downgrading to 1.57.0 bring memory consumption to normal.
Anything else we should know about your project / environment?
A Dagster deployment involves a GRPC server process that loads users Dagster code artifacts and two system processes (a webserver and a daemon) that communicate with the GRPC server. These system processes regularly communicate with the GRPC server to fetch various piece of information. We use a threaded grpc server and the client system processes make blocking requests, not asyncio.
A previous report #36117 was closed under the assumption that the issue was related to a resolved cpython asyncio bug python/cpython#111246. Since we observe this in a non asyncio set-up I don't believe that is an accurate assessment.
What version of gRPC and what language are you using?
grpcio
1.57.0
and1.57.1
What operating system (Linux, Windows,...) and version?
various
What runtime / compiler are you using (e.g. python version or version of gcc)
python 3.X (various)
What did you do?
This issue represents reports from Dagster users dagster-io/dagster#18997 .
What did you expect to see?
When upgrading the
grpcio
python package in the system processes that are grpc clients from1.57.0
to1.57.1
, memory utilization should not change drastically.What did you see instead?
For some users, memory consumption in
1.57.1
grows continuously for the life of the process. Downgrading to1.57.0
bring memory consumption to normal.Anything else we should know about your project / environment?
A Dagster deployment involves a GRPC server process that loads users Dagster code artifacts and two system processes (a webserver and a daemon) that communicate with the GRPC server. These system processes regularly communicate with the GRPC server to fetch various piece of information. We use a threaded grpc server and the client system processes make blocking requests, not
asyncio
.A previous report #36117 was closed under the assumption that the issue was related to a resolved cpython
asyncio
bug python/cpython#111246. Since we observe this in a nonasyncio
set-up I don't believe that is an accurate assessment.When viewing the changes between
1.57.0
and1.57.1
v1.57.0...v1.57.1 the only meaningful change is #34557 a backport of #34549 that introduced a new heap allocation https://github.com/grpc/grpc/pull/34549/files#r1341637876 . It seems extremely likely that this change is related to the observed changes in memory utilization.Well outside my area of expertise, but cross referencing https://github.com/abseil/abseil-cpp/blob/master/absl/strings/cord.h#L214-L252 and trying to make some guesses:
The text was updated successfully, but these errors were encountered: