Why wont the write mode change for writing to graphics memory in VGA mode?-CodePudding

I am trying to draw to the screen in VGA graphics in DOSBOX using NASM. My code is able to write to a black screen perfectly fine however if I edit a pixel that is already a certain color, the output color ends up being the previous color ORed with my new color. For example, if I fill a red screen then fill it again as blue it the final screen will not be blue but it will be a magenta.

Here is my code, it should fill the screen blue then green but instead fills the screen blue then ORs the blue memory with green resulting in a teal filled screen. I was able to see this happening in DOSBOX by slowing down the cpu speed.

org 100h
section .text
start:
    mov ax, 4F02h
    mov bx, 102h
    int 10h       ; enter 800x600 16 color VGA mode
    push 0x01     ; blue
    call fill_screen
    push 0x2      ; green
    call fill_screen
    jmp $

fill_screen:
    ; multiply screen width by height then divide by 8 to get number of times to repeat write operation, push to stack
    mov ax, [screen_width]
    mov dx, [screen_height]
    mul dx
    mov cx, 8
    div cx
    push ax

    ; point es to video memory
    mov ax, 0A000h
    mov es, ax

    pop cx  ; times to repeat write operation
    pop dx  ; return address
    pop ax  ; color 
    push dx ; put return address back on stack
    
    mov dx, 03c4h ; register selection port
    shl ax, 8     ; color is one byte, it was in al but we need it in ah so shift left one byte
    mov al, 02h   ; map mask
    out dx, ax    ; write all the bitplanes
    xor di, di    ; video memory pointer to start
    mov ax, 0ffh  ; write to every pixel
    rep stosb
    ret

section .data
    screen_width dw 320h
    screen_height dw 258h

Here in section 5 it says that you can change the write mode to replace, or, xor, or and by writing to port 03CFh. They include the sample code to switch to XOR mode:

mov ax,1803h
mov dx,03CEh
out dx,ax

When I add this after line 6 (int 10h) nothing happens and the output of the code is still teal. Since blue OR green is the same as blue XOR green I changed the green input to magenta (make line 9 push 0x5). The output is now magenta meaning the memory is in OR mode still and not in XOR mode.

Here is the final testing code:

org 100h
section .text
start:
    mov ax, 4F02h
    mov bx, 102h
    int 10h       ; enter 800x600 16 color VGA mode

    ; switch to XOR write mode (?)
    mov ax, 1803h
    mov dx, 03CEh
    out dx, ax

    push 0x01 ; blue
    call fill_screen
    push 0x5 ; magenta
    call fill_screen
    jmp $

fill_screen:
    ; multiply screen width by height then divide by 8 to get number of times to repeat write operation, push to stack
    mov ax, [screen_width]
    mov dx, [screen_height]
    mul dx
    mov cx, 8
    div cx
    push ax

    ; point es to video memory
    mov ax, 0A000h
    mov es, ax

    pop cx  ; times to repeat write operation
    pop dx  ; return address
    pop ax  ; color 
    push dx ; put return address back on stack

    mov dx, 03c4h ; register selection port
    shl ax, 8     ; color is one byte, it was in al but we need it in ah so shift left one byte
    mov al, 02h   ; map mask
    out dx, ax    ; write all the bitplanes
    xor di, di    ; video memory pointer to start
    mov ax, 0ffh  ; write to every pixel
    rep stosb
    ret

section .data
    screen_width dw 320h
    screen_height dw 258h

How can I fix this so I enter a different write mode? Is there a different better way to write to video memory that forgoes writing modes that I should use instead? How would I enter replace mode?

CodePudding user response：

Some reference about VGA hardware: https://wiki.osdev.org/VGA_Hardware, https://web.stanford.edu/class/cs140/projects/pintos/specs/freevga/vga/vgamem.htm

First, the XOR mode (and others) do not XOR with the data already present in video memory. They XOR with the latch registers, which are loaded when reading from video memory. The latch registers keep their value until you read another value from video memory. If you don't read from video memory, then the latch registers have an undefined value.

To stay in normal mode (NOP mode), just set the mode to 0:

mov ax,0003h
mov dx,03CEh
out dx,ax

Also, when writing to 03c4h/reg 2, you set a mask which will be applied to all the writes to video memory, filtering each bit you write. Setting the mask to 0Fh will allow writing all the bits as expected. If you don't use a mask value 0Fh (which is your case, you fill it with your color instead), then the 0 bits of the mask will be taken from the latch registers, which you don't set since you don't read.

To write colors with no masking, set the mask to 0Fh, and write your actual color instead of FFh:

mov dx,03C4h
mov ax,0F02h
out dx,ax
mov es:[di], <color> ; example write

If you wish to use the XOR and mask operations for each pixel, then you need to correctly fill the latches before each write by doing a dummy read:

mov bl, es:[di] ; ignore bl if not needed
mov es:[di], <color> ; example write with VGA operations

Note that the alternative, using bl and doing the xor/mask yourself, is not much more complicated.