File retention mechanism in a large data storage-CodePudding

recently I faced performance problem with mp4 files retention. I have kind a recorder which saves 1 min long mp4 files from multiple RTSP streams. Those files are stored on external drive in file tree like this:

./recordings/{camera_name}/{YYYY-MM-DD}/{HH-MM}.mp4

Apart from video files, there are many other files on this drive which are not considered (unless they have mp4 extension), as they took much less space.

Assumption of file retention is as follows. Every minute, python script that is responsible for recording, check for external drive fulfillment level. If the level is above 80%, it performs a scan of the whole drive, and look for .mp4 files. When scanning is done, it sorts a list of files by its creation date, and deletes the number of the oldest files which is equal to the cameras number.

The part of the code, which is responsible for files retention, is shown below.

total, used, free = shutil.disk_usage("/home")
used_percent = int(used / total * 100)
if used_percent > 80:
    logging.info("SSD usage %s. Looking for the oldest files", used_percent)
    try:
        oldest_files = sorted(
            (
                os.path.join(dirname, filename)
                for dirname, dirnames, filenames in os.walk('/home')
                for filename in filenames
                if filename.endswith(".mp4")
            ),
            key=lambda fn: os.stat(fn).st_mtime,
        )[:len(camera_devices)]
        logging.info("Removing %s", oldest_files)
        for oldest_file in oldest_files:
            os.remove(oldest_file)
            logging.info("%s removed", oldest_file)
    except ValueError as e:
        # no files to delete
        pass

(/home is external drive mount point)

The problem is that this mechanism used to work as a charm, when I used 256 or 512 GB SSD. Now I have a need of larger space (more cameras and longer storage time), and it takes a lot of time to create files list on larger SSD (from 2 to 5 TB now and maybe 8 TB in the future). The scanning process takes a lot more than 1 min, what could be resolved by performing it more rarely, and extending the length of "to delete" files list. The real problem is, that the process uses a lot of CPU load (by I/O ops) itself. The performance drop is visible is the whole system. Other applications, like some simple computer vision algorithms, works slower, and CPU load can even cause kernel panic.

The HW I work on is Nvidia Jetson Nano and Xavier NX. Both devices have problem with performance as I described above.

The question is if you know some algorithms or out of the box software for file retention that will work on the case I described. Or maybe there is a way to rewrite my code, to let it be more reliable and perform?

CodePudding user response：

A quick optimization would be not to bother checking file creation time and trusting the filename.

total, used, free = shutil.disk_usage("/home")
used_percent = int(used / total * 100)
if used_percent > 80:
    logging.info("SSD usage %s. Looking for the oldest files", used_percent)
    try:
        files = []
        for dirname, dirnames, filenames in os.walk('/home/recordings'):
            for filename in filenames:
                files.push((
                    name := os.path.join(dirname, filename),
                    datetime.strptime(
                        re.search(r'\d{4}-\d{2}-\d{2}\/\d{2}-\d{2}', name)[0],
                        "%Y-%m-%d/%H-%M"
                        ))
        oldest_files = files.sort(key=lambda e: e[1])[:len(camera_devices)]
        logging.info("Removing %s", oldest_files)
        for oldest_file in oldest_files:
            os.remove(oldest_file)
            # logging.info("%s removed", oldest_file)
        logging.info("Removed")
    except ValueError as e:
        # no files to delete
        pass

CodePudding user response：

You could use the subprocess module to list all the mp4 files directly, without having to loop through all the files in the directory.

import subprocess as sb
oldest_files = sb.getoutput("dir /b /s .\home\*.mp4").split("\n")).sort(lambda fn: os.stat(fn).st_mtime,)[:len(camera_devices)]