How can I limit the number of parallel child processes using C code in FreeBSD-CodePudding

I want to limit N processes ( introduced as parameters ) at the same time, but there is always more than N files to process. One process per file. Only N processes are being processes at one time. I know what the program must do but don't know how to make it happen.

I am sorry if I don't explain too well, I will answer all details needed. Using C code in FreeBSD

for (int i = 3; i < argc; i  ) {
    fflush(NULL);
    if((pid_son = fork()) < 0){
        printf("Error");
        exit(-1);
    }
    else if(pid_son == 0){

    }
}

CodePudding user response：

Because you asked how to limit processes with FreeBSD OS by passing an argument to your program, one answer is to use the cpuset utility with the same parameter as the one given to your program :

$ NPROC=4
$ cpuset -l 0-${NPROC} myProgram ${NPROC}

Another solution is to use rctl. And another solution is to simply use ulimit -u ${NPROC} ... .

I must point out that argc you used is not an argument, but the number of arguments given to your program. If you want to access an argument, you have to use argv. But if I understand correctly what you want to do, it seems you use argc as the number of files given as arguments. In this case, I'm not sure this is the best way to achieve your goal, because the argument could be wrong, unreadable, etc... so the number of processes.

If you want to limit the number of process programmatically under FreeBSD, there's many a way to do it: through FreeBSD cpuset, or simply using POSIX pthread_affinity_np functionalities.

In this case, you have to provide a question to SO explaining what you really want to do. In other cases, the web will provide a lot of example on how to use it.

CodePudding user response：

It isn't clear whether you want a limit enforced by the o/s, nor what your code will do if it runs into that limit (if a fork() call fails). However, you can arrange to run commands until you reach a limit, and then wait for one of those to finish before launching another.

Here is some C code that should work on most POSIX systems. It is available in my SOQ (Stack Overflow Questions) repository on GitHub as file numproc19.c in the src/so-1974-7644 sub-directory.

`numproc19.c` — with comments removed

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <string.h>

enum { MAX_KIDS = 4 };
enum { DEF_TASKS = 20 };

static _Noreturn void be_childish(int tasknum)
{
    srand(getpid());
    struct timespec nap = { .tv_sec = rand() % 5, .tv_nsec = (rand() % 1000) * 1000000 };
    int millisecs = nap.tv_nsec / 1000000;
    printf("PID ] (PPID ]): Task - - dozing %d.%.3d\n",
            getpid(), getppid(), tasknum, (int)nap.tv_sec, millisecs);
    nanosleep(&nap, 0);
    printf("PID ] (PPID ]): Task - - done\n", getpid(), getppid(), tasknum);
    exit(tasknum);
}

static size_t dead_kid(pid_t corpse, size_t nkids, pid_t *kids)
{
    for (size_t i = 0; i < nkids; i  )
    {
        if (kids[i] == corpse)
        {
            kids[i] = kids[--nkids];
            return nkids;
        }
    }
    printf("PID ] exited but was not a known child\n", corpse);
    return nkids;
}

static int cmp_pid(const void *vp1, const void *vp2)
{
    pid_t v1 = *(pid_t *)vp1;
    pid_t v2 = *(pid_t *)vp2;
    return (v1 > v2) - (v1 < v2);
}

static void print_kids(size_t nkids, pid_t *kids)
{
    qsort(kids, nkids, sizeof(kids[0]), cmp_pid);
    printf("Kids (%zu):", nkids);
    for (size_t i = 0; i < nkids; i  )
        printf(" ]", kids[i]);
    putchar('\n');
}

int main(void)
{
    pid_t kids[MAX_KIDS];
    size_t nkids = 0;

    setvbuf(stdout, NULL, _IOLBF, 0);

    for (size_t task = 0; task < DEF_TASKS; task  )
    {
        pid_t pid = fork();
        if (pid < 0)
        {
            fprintf(stderr, "failed to fork(): (%d) %s\n", errno, strerror(errno));
            exit(EXIT_FAILURE);
        }
        if (pid == 0)
            be_childish(task);
        kids[nkids  ] = pid;

        printf("Kid: ]; Number of kids: %2zu\n", pid, nkids);
        print_kids(nkids, kids);
        if (nkids >= MAX_KIDS)
        {
            int status;
            int corpse = waitpid(-1, &status, 0);
            if (corpse < 0)
                break;
            printf("Child ] exited with status 0x%.4X\n", corpse, status);
            nkids = dead_kid(corpse, nkids, kids);
        }
    }

    int corpse;
    int status;
    while (nkids > 0 && (corpse = waitpid(-1, &status, 0)) > 0)
    {
        printf("Child ] exited with status 0x%.4X\n", corpse, status);
        nkids = dead_kid(corpse, nkids, kids);
        print_kids(nkids, kids);
    }

    return 0;
}

The be_childish() function does whatever real work is required. Here, it reports on its existence and sleeps for a random time rough in the range 0-5 seconds. In production code, it would either run a function in the program or a command that does whatever is required.

The main() function keeps track of which processes it has launched and once it has reached the limit, it waits for a process to finish. It recognizes that it might have children it doesn't know about and handles them cleanly (reports on their termination and continues unphased by their presence).

Sample run

Kid: 23402; Number of kids:  1
Kids (1): 23402
Kid: 23403; Number of kids:  2
Kids (2): 23402 23403
PID 23402 (PPID 23401): Task  0 - dozing 4.632
Kid: 23404; Number of kids:  3
Kids (3): 23402 23403 23404
PID 23403 (PPID 23401): Task  1 - dozing 1.881
Kid: 23405; Number of kids:  4
Kids (4): 23402 23403 23404 23405
PID 23404 (PPID 23401): Task  2 - dozing 3.130
PID 23405 (PPID 23401): Task  3 - dozing 0.379
PID 23405 (PPID 23401): Task  3 - done
Child 23405 exited with status 0x0300
Kid: 23406; Number of kids:  4
Kids (4): 23402 23403 23404 23406
PID 23406 (PPID 23401): Task  4 - dozing 2.628
PID 23403 (PPID 23401): Task  1 - done
Child 23403 exited with status 0x0100
Kid: 23407; Number of kids:  4
Kids (4): 23402 23404 23406 23407
PID 23407 (PPID 23401): Task  5 - dozing 4.877
PID 23406 (PPID 23401): Task  4 - done
Child 23406 exited with status 0x0400
Kid: 23408; Number of kids:  4
Kids (4): 23402 23404 23407 23408
PID 23408 (PPID 23401): Task  6 - dozing 1.479
PID 23404 (PPID 23401): Task  2 - done
Child 23404 exited with status 0x0200
Kid: 23409; Number of kids:  4
Kids (4): 23402 23407 23408 23409
PID 23409 (PPID 23401): Task  7 - dozing 3.728
PID 23408 (PPID 23401): Task  6 - done
Child 23408 exited with status 0x0600
Kid: 23410; Number of kids:  4
Kids (4): 23402 23407 23409 23410
PID 23410 (PPID 23401): Task  8 - dozing 0.977
PID 23402 (PPID 23401): Task  0 - done
Child 23402 exited with status 0x0000
Kid: 23411; Number of kids:  4
Kids (4): 23407 23409 23410 23411
PID 23411 (PPID 23401): Task  9 - dozing 2.226
PID 23410 (PPID 23401): Task  8 - done
Child 23410 exited with status 0x0800
Kid: 23412; Number of kids:  4
Kids (4): 23407 23409 23411 23412
PID 23412 (PPID 23401): Task 10 - dozing 4.475
PID 23407 (PPID 23401): Task  5 - done
Child 23407 exited with status 0x0500
Kid: 23413; Number of kids:  4
Kids (4): 23409 23411 23412 23413
PID 23413 (PPID 23401): Task 11 - dozing 1.724
PID 23411 (PPID 23401): Task  9 - done
PID 23409 (PPID 23401): Task  7 - done
Child 23411 exited with status 0x0900
Kid: 23414; Number of kids:  4
Kids (4): 23409 23412 23413 23414
Child 23409 exited with status 0x0700
Kid: 23415; Number of kids:  4
Kids (4): 23412 23413 23414 23415
PID 23414 (PPID 23401): Task 12 - dozing 3.973
PID 23415 (PPID 23401): Task 13 - dozing 0.222
PID 23415 (PPID 23401): Task 13 - done
Child 23415 exited with status 0x0D00
Kid: 23416; Number of kids:  4
Kids (4): 23412 23413 23414 23416
PID 23416 (PPID 23401): Task 14 - dozing 2.824
PID 23413 (PPID 23401): Task 11 - done
Child 23413 exited with status 0x0B00
Kid: 23418; Number of kids:  4
Kids (4): 23412 23414 23416 23418
PID 23418 (PPID 23401): Task 15 - dozing 1.322
PID 23418 (PPID 23401): Task 15 - done
Child 23418 exited with status 0x0F00
Kid: 23419; Number of kids:  4
Kids (4): 23412 23414 23416 23419
PID 23419 (PPID 23401): Task 16 - dozing 3.571
PID 23416 (PPID 23401): Task 14 - done
Child 23416 exited with status 0x0E00
Kid: 23420; Number of kids:  4
Kids (4): 23412 23414 23419 23420
PID 23420 (PPID 23401): Task 17 - dozing 0.820
PID 23412 (PPID 23401): Task 10 - done
Child 23412 exited with status 0x0A00
Kid: 23421; Number of kids:  4
Kids (4): 23414 23419 23420 23421
PID 23421 (PPID 23401): Task 18 - dozing 2.069
PID 23420 (PPID 23401): Task 17 - done
Child 23420 exited with status 0x1100
Kid: 23422; Number of kids:  4
Kids (4): 23414 23419 23421 23422
PID 23422 (PPID 23401): Task 19 - dozing 4.318
PID 23414 (PPID 23401): Task 12 - done
Child 23414 exited with status 0x0C00
PID 23421 (PPID 23401): Task 18 - done
Child 23421 exited with status 0x1200
Kids (2): 23419 23422
PID 23419 (PPID 23401): Task 16 - done
Child 23419 exited with status 0x1000
Kids (1): 23422
PID 23422 (PPID 23401): Task 19 - done
Child 23422 exited with status 0x1300
Kids (0):

numproc19.c — with comments removed

Sample run

`numproc19.c` — with comments removed