Application crashes if (own) dll try to allocate a few MB of RAM-CodePudding

I'm not a good c programmer and I'm not very familiar creating DLLs for other application. My main programming language is Java so I'm having a little trouble with coding c and compiling with MinGW. However, as we need some small portion of "native" code for our main application (which is coded in Cobol) I was needed to try to do this with c . In general my DLL has a small footprint of memory usage as I'm doing nothing special with it. This DLL is mainly there to communicate with an external process using sockets.

As far as I can tell the DLL itself is fine and my problem must be more "specific" and is not a general c coding problem. Somewhere in my code I try to allocate some "large" amount of RAM with char buf[size]; and later read file content into this buffer.

size can be a few kilobyte up to a few megabyte (not more than 10MB). If the size is getting somewhere near to 1MB our main application crashes after above mentioned statement (I pinpoint this down to that statement using logging outputs). If I compile my DLL as an EXE file and test the function directly (using Windows console) all is going fine and no crash happens. Even if I use several more MB for this buffer there is no problem with it. This leads me to suspect that there must be some sort of upper limit on memory usage using my DLL which causes the main application to crash hardly if my dll try to grab more memory as it may be allowed to.

So my question is: Could it be that there is some sort of upper limit a DLL can/may use? And if yes, is there some kind of compiling switch or option I need to use while generating my DLL to overcome this limitation? Our main application is using several other DLLs which using way more memory as my DLL try to use so this is not a limitation of our main application.

Here is the whole function which try to read the file:

/**
 * Read file content as Base64 Data
 */
char* readFileAsBase64(const char* filename) {
    FILE *file;
    if(!fileExists(filename)) {
        return NULL;
    }
    long size = (long)fileSize(filename);
    file = fopen(filename, "rb");
    char ch;
    // Crach happens here if 'size' is too large
    char buf[size];
    /* copy the file */
    for(long i = 0; i<size; i  ) {
        ch = fgetc(file);
        if(ferror(file)) {
            return NULL;
        }
        if(!feof(file)) {
            buf[i] = ch;
        }
    }

    if(fclose(file)==EOF) {
        return NULL;
    }

    return b64_encode(buf, size);
}

CodePudding user response：

Yes, that's a stack allocation (and not legal C either).

Simple fix is to use a std::string

#include <string>

...
std::string buf
buf.resize(size);
...
return b64_encode(buf.data(), size);

No other code changes needed. Although you could make further changes to use std::string throughout your code, instead of using raw pointers.