MSDN says
Integer valued arguments in the leftmost four positions are passed in left-to-right order in RCX, RDX, R8, and R9, respectively. Space is allocated on the call stack as a shadow store for callees to save those registers. Remaining arguments get pushed on the stack in right-to-left order.
So, I'm trying to call the CreateFileW function, and this is my code:
sub rsp, 20h ; Allocate 32 bytes because 4 registers 8 byte each
mov rcx, offset filename ; lpFileName
mov rdx, GENERIC_READ or GENERIC_WRITE ; dwDesiredAccess
mov r8, FILE_SHARE_DELETE ; dwShareMode
xor r9, r9 ; LpSecurityAttributes
;__And right-to-left order remaining arguments__
push 0 ; hTemplateFile
push FILE_ATTRIBUTE_NORMAL ;dwFlagsAndAttributes
push CREATE_ALWAYS ; dwCreationDisposition
call CreateFileW
It assembles, but does not work and win64dbg causes next error:
00000057 (ERROR_INVALID_PARAMETER)
The parameters are 100% ok because it works with the Invoke macro, only the generated code is different.
mov rcx,src.403000 ;name
mov edx,C0000000 ;GENERIC_READ or GENERIC_WRITE
mov r8d,4 ;FILE_SHARE_DELETE
xor r9d,r9d ;0
mov qword ptr ss:[rbp-20],2; ;CREATE_ALWAYS
mov qword ptr ss:[rbp-18],80 ;FILE_ATTRIBUTE_NORMAL
and qword ptr ss:[rbp-10],0 ;0
call qword ptr ds:[<&CreateFileW>]
So my question is why it uses the RBP register instead of push and does not allocate 32 bytes for "shadow-store"?
Notes
Since 64-bit MASM by Microsoft no longer has an invoke directive I am using a Russian MASM64 SDK project that has an invoke macro. That project is loosely based on the MASM32 SDK.
CodePudding user response：
If you want to push args, you have to do it before sub rsp,20h. (Which doesn't work well because normally you only want one sub rsp,20h for the whole function, not one for each call). And you'd have to count correctly to have RSP == 0 after the last push. You normally don't want to change RSP except in the function prologue/epilogue in Windows x64, except for alloca type of things.
Stack args go above the shadow space, so if the function were to dump its register args to their "home space" in the shadow space, it would have a contiguous array of args. (Variadic functions like printf will actually do that; normal functions not unless it's a debug build.)
Use a debugger to look at the contents of stack memory (above RSP) right before the call for your way vs. the normal way of doing mov stores to stack arg space above the shadow space.
Note that sub rsp,20h isn't big enough to reserve space for shadow space and stack args, so the "invoke macro" code you show must be reserving more space at the start of this function.
Why it use RBP register instead of push and does not allocate 32 bytes for "shadow-store"?
It uses RBP because that's a normal way to access stack space if you've already spent instructions to set up RBP as a frame pointer.
It would be even clearer and easier to see what's going on with addressing modes relative to RSP, like [rsp 20h] to access the first slot above the shadow space, where you'd want to store the first stack arg.
That would be necessary if you've allocated a variable amount of space so the distance from RBP to just above the shadow space isn't known, but you might do it that way anyway, just for clarity and ease of getting the offsets correct. But if a compiler or clever macros can calculate correct offsets, and you've already spent the instructions to set up RBP as a frame pointer, then it's slightly more efficient to use it because it saves one byte in the machine code ([rsp constant] requires a SIB byte to encode.)
Regular MASM doesn't have an invoke in 64-bit mode. I don't know what you're using. Maybe you need to manually reserve stack space, or maybe it does that for you at the top of the function but you left that out. Michael Petch says MASM64 comes with an invoke macro that does add a sub rsp to your code.