How to determine current memory usage of the pod in k8s inside the python program?-CodePudding

I have my python program running on k8s pod. The program reads data in pandas dataframe from the db & do some data transformation operations. In one of the operation, data size grows exponentially causing pod out of memory error. Since its pandas, all data is held in the memory. To avoid this, I am required to write the data to the disk if memory consumption crosses the threshold. And to do this, I need to know the current pod memory utilization inside my python program so that as soon as threshold reaches, say 90%, I start dumping data to disk & freeing memory. Basically I need to know pod memory utilization inside my python program. Any suggestion ? Thanks in advance.

CodePudding user response：

The easiest way to check memory utilization is to use psutil library.

I created a simple kubernetes pod which is using python image. Then, I got a shell to it, using kubectl exec command, and I installed psutil library...

user@shell:~/python-testing $ kubectl exec -it my-test-6d4468964d-67jlx -c my-python -- sh
# pip3 install psutil
Collecting psutil
  Downloading psutil-5.8.0.tar.gz (470 kB)
     |████████████████████████████████| 470 kB 4.6 MB/s 
Building wheels for collected packages: psutil
  Building wheel for psutil (setup.py) ... done
  Created wheel for psutil: filename=psutil-5.8.0-cp310-cp310-linux_x86_64.whl size=279671 sha256=510df7c715b76b8d7534b72f5fca306cfd1190b59ed318bb77cfa7a4ff76acc8
  Stored in directory: /root/.cache/pip/wheels/12/a3/6d/615295409067d58a62a069d30d296d61d3ac132605e3a9555c
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

...and I run following commands to check memory utilization:

# python3 
Python 3.10.0 (default, Nov 17 2021, 15:26:39) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.virtual_memory().percent
16.0

After that I decided to generate some memory usage (about 70%):

>>> psutil.virtual_memory().percent
72.1

So it seems that it is working correctly.

It also works fine when one pod has multiple containers as they are sharing common memory, e.g. one container is consuming 70% of the total memory of the pod, so on the second container I can see that memory utilization is ~ 70%.

To sum up, you need to include something like this in your python code:

import psutil

memory_usage = psutil.virtual_memory().percent
print(memory_usage)

Check also more examples of the psutil commands in this answer.

CodePudding user response：

In container, you need to get cgroup's memory.

 ls /sys/fs/cgroup/memory/

In container: cat /sys/fs/cgroup/memory/memory.usage_in_bytes to get memory usage.

Or in host: /sys/fs/cgroup/memory/docker/[containerId]/memory.usage_in_bytes

https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt