Home > Blockchain >  Rust Win32 FFI: User-mode data execution prevention (DEP) violation
Rust Win32 FFI: User-mode data execution prevention (DEP) violation

Time:04-30

I'm trying to pass a ID3D11Device instance from Rust to a C FFI Library (FFMPEG).

I made this sample code:

pub fn create_d3d11_device(&mut self, device: &mut Box<windows::Win32::Graphics::Direct3D11::ID3D11Device>, context: &mut Box<windows::Win32::Graphics::Direct3D11::ID3D11DeviceContext>) {
            let av_device : Box<AVBufferRef> = self.alloc(HwDeviceType::D3d11va);
            unsafe {
                let device_context = Box::from_raw(av_device.data as *mut AVHWDeviceContext);
                let mut d3d11_device_context = Box::from_raw(device_context.hwctx as *mut AVD3D11VADeviceContext);
                d3d11_device_context.device = device.as_mut() as *mut _;
                d3d11_device_context.device_context = context.as_mut() as *mut _;
                let avp = Box::into_raw(av_device);
                av_hwdevice_ctx_init(avp);
                self.av_hwdevice = Some(Box::from_raw(avp));
            }
        }

On the Rust side the Device does work, but on the C side, when FFMEPG calls ID3D11DeviceContext_QueryInterface the app crashes with the following error: Exception 0xc0000005 encountered at address 0x7ff9fb99ad38: User-mode data execution prevention (DEP) violation at location 0x7ff9fb99ad38

The address is actually the pointer for the lpVtbl of QueryInterface, like seen here:

The disassembly of the address also looks correct (this is done on an another debugging session):

(lldb) disassemble --start-address 0x00007ffffdf3ad38
    0x7ffffdf3ad38: addb   %ah, 0x7ffffd(%rdi,%riz,8)
    0x7ffffdf3ad3f: addb   %al, (%rax)
    0x7ffffdf3ad41: movabsl -0x591fffff80000219,            
  • Related