Home > Back-end >  In assembly, how to handle Windows API with a return value that's documented as less than nativ
In assembly, how to handle Windows API with a return value that's documented as less than nativ

Time:10-08

For many years, ntohs (32-bit version) had a return value that was zero extended into the high 16 bits (word). However, after a recent update of Windows 10, it sometimes returns garbage in the upper word.

For example, passing argument 0xF00D to

call        ntohs

the return values are

  • EAX = 00F00DF0 (32-bit code)
  • RAX = 0000000000000DF0 (64-bit code)

The fix in MASM code is to zero extend the return register using movzx (identical to MSVC)

call        ntohs
movzx       eax, ax

Question

Is it correct that any Windows API with a return value that's documented as less than native register width, there's no guarantee of the upper bits being zeroed out i.e., the return value in RAX = 0000000000000DF0 (64-bit) was most likely coincidental?

CodePudding user response:

At least for x64, the upper bits are undefined and the caller is responsible for doing zero- or sign-extension if needed. If your code relied on upper bits being zero, I'm afraid it was buggy all along, and you have just been lucky until now.

https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160

The state of unused bits in the value returned in RAX or XMM0 is undefined.

  • Related