Undefined Symbol rax,rbx,rcx,rdx in MASM Visual Studio-CodePudding

Following is the code:

.386

.model flat, stdcall

.stack 4096

    ExitProcess PROTO, dwExitCode:DWORD

    .DATA

    .code

Main proc

mov rax,0

mov rbx,0

mov rcx,0

mov rdx,0

       mov al, 0F0H           ;using 8-bit register

    mov bx, 1012H            ;using 16-bit register

    mov ecx, 15284H          ;using 32-bit register
   
    mov rdx, 3578815H        ;using 64-bit register


INVOKE ExitProcess, 0

main ENDP

END main

Whenever i run this, it says :undefined symbol rax, rbx, rcx,rdx

But it works when I use:
.386
.model flat, stdcall
.stack 4096
    ExitProcess PROTO, dwExitCode:DWORD
    .DATA
    .code
Main proc

mov eax,0

mov ebx,0

mov ecx,0

mov edx,0

       mov al, 0F0H           ;using 8-bit register

    mov bx, 1012H            ;using 16-bit register

    mov ecx, 15284H          ;using 32-bit register  
 
    mov edx, 3578815H        ;using 64-bit register  

INVOKE ExitProcess, 0

main ENDP

END main

I'm using x86 configuration. x64 doesn't work for me. Could someone tell me why rax,rbx,rcx and rdx arent working?

CodePudding user response：

For Visual Studio, a 64 bit build would use ML64.EXE instead of MASM.EXE. Don't use .386 directive. There is no .model directive for ML64.EXE, since the 64 bit calling convention is fixed.

CodePudding user response：

You can't use 64-bit registers in 32-bit code.

In 16-bit mode, 32-bit registers are usable in machine code with 66h and 67h prefixes. So 16 and 32-bit code can both use instructions like movzx ax, byte ptr [ecx] (assuming the code is running on a 386 or newer, of course, hence .model 386 telling the assembler that it should let you write code like that).

But for x86-64, there wasn't an unused byte left for use as a prefix in 32-bit mode, so the AMD64 architects had to do it differently. The new features of x86-64 are only available in 64-bit mode, not in 32-bit compat mode.

The actual mechanism they picked was to remove the 1-byte encodings for inc reg and dec reg, and reuse those 0x4? bytes as REX prefixes, including a Width bit to distinguish add eax, ecx from add rax, rcx, and 3 extra bits for register numbers, allowing access to r8-r15 for instructions like add r8d, [r9 r10].

As https://wiki.osdev.org/X86-64_Instruction_Encoding#REX_prefix says, The REX prefix is only available in long mode.

Fun fact: If you try to execute 64-bit machine code with the CPU in 32-bit mode (e.g. from using NASM bits 64 in the source and assembling with nasm -fwin32), the REX prefixes will decode as inc or dec instructions.

You can play fun tricks like making machine code that runs 3 different ways depending on what mode it executes in. For example, x86-32 / x86-64 polyglot machine-code fragment that detects 64bit mode at run-time?. Also a 3-way codegolf answer shows disassembly of the same code 3 different ways.