I'm writing some C code that gets compiled on both a Linux machine and a Windows machine (specifically, I am writing C code that is used within some R functions, so not a standalone C program). When compiling this code, I get "stack smashing detected" on my Linux machine, but no error is thrown when I compile it on a Windows machine. I've identified where the error came about (writing to an array beyond its length), and fixed it, but now I am curious:
Could somebody explain why C code compiled on a Windows machine (using the standard C compiler that comes with the R devtools) would allow stack smashing to go un-noted, while the compiler on my Linux machine would error out. Is it just a difference in the settings of the compiler? Wouldn't it be better to have some protection from this even on a Windows machine? I've already fixed the issue, so this question is just born out of curiosity!
CodePudding user response：
It's a combination of compiler settings and differences in the mechanisms used on the two operating systems for this purpose.
On Linux GCC/clang you have the "stack protector" (-fstack-protector), while on MSVC you have "GuardStack" (/GS). The underlying strategy to detect stack buffer overflows is similar, but the implementation and binary instrumentation mechanism is different. Depending on compiler version and settings, those options might be enabled or not, and might be able to detect errors only under certain conditions rather than others.
using the standard C compiler that comes with the R devtools
I am not familiar with the compiler you use, but I would inspect its configuration to understand whether GuardStack is being enabled or not.
Could somebody explain why C code compiled on a Windows machine (using the standard C compiler that comes with the R devtools) would allow stack smashing to go un-noted, while the compiler on my Linux machine would error out.
There are two reasonable explanations:
You are not enabling GuardStack in your Windows build, while on your Linux build the stack protector is enabled by default.
You have GuardStack enabled, but the system is not able to detect the stack smashing at runtime on Windows because either (A) it is not happening at all (i.e. alignment requirements might generate a buffer larger than it seems from the source and you do not exceed its size) or (B) it is happening, but not being detected. You could for example be overflowing the buffer, but not enough to break the stack canary value or the saved return address, thus remaining undetected.
In any case, the key point to understand is that both -fstack-protector and /GS usually only detect large enough buffer overflows that go past the local variables in the current stack frame and potentially destroy the saved frame pointer or return address. They do not detect any kind of buffer overflow, and do not perform strict bounds checks. You can read more about when/how this detection is made going through the documentation links I provided above.