The values of memsample in the Celery worker stats don’t change when I run a Celery task that consumes memory. According to the documentation, memsample should return the current memory usage value of the worker.

In my case, I have a single worker. The task goes into the worker, and the task is successful, but the memsample values don’t change when the task is in progress; they remain the same every time.

Command i used to fetch the celery memsample

celery -A <project-name> inspect memsample -d <worker-id>

Code:

import time

from celery import group, shared_task


@shared_task
def large_list():
    """Return a list of 10^5 elements, each element is an integer from 0 to 10^5-1"""
    large_list1 = [i for i in range(10**6)]
    time.sleep(2)
    large_list1 = [i for i in range(10**6)]
    time.sleep(2)
    large_list2 = [i for i in range(10**6)]
    time.sleep(2)
    large_list3 = [i for i in range(10**6)]
    time.sleep(2)
    large_list4 = [i for i in range(10**6)]
    time.sleep(2)
    return large_list1, large_list2, large_list3, large_list4


def caller():
    print("BEGIN: inside the caller fucntion")
    group_task = []
    for i in range(10):
        group_task.append(large_list.si().set(queue="eq.BASIC.MISC.1")) #Custom queue configuration

    group(group_task).apply_async()
    print("END: inside the caller fucntion")

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
• I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version:

software -> celery:5.4.0 (opalescent) kombu:5.3.7 py:3.10.14
            billiard:4.2.0 redis:4.6.0
platform -> system:Linux arch:64bit, ELF
            kernel version:6.6.26-linuxkit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://****redis:6379/

deprecated_settings: None
CELERY_TASK_SERIALIZER: 'json'
CELERY_ACCEPT_CONTENT: ['pickle', 'json']
CELERY_RESULT_SERIALIZER: 'json'
CELERY_ENABLE_UTC: True
CELERY_RESULT_BACKEND: 'redis://ever-redis:6379/'
CELERY_COMPRESSION: 'lzma'
BROKER_URL: 'redis://default:********@host.docker.internal:6379//'
BROKER_TRANSPORT: 'redis'
CELERY_SEND_TASK_SENT_EVENT: 'true'
CELERYD_MAX_TASKS_PER_CHILD: 100
BROKER_LOGIN_METHOD: 'NONE'

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior