I'm trying to pass a ID3D11Device instance from Rust to a C FFI Library (FFMPEG).
I made this sample code:
pub fn create_d3d11_device(&mut self, device: &mut Box<windows::Win32::Graphics::Direct3D11::ID3D11Device>, context: &mut Box<windows::Win32::Graphics::Direct3D11::ID3D11DeviceContext>) {
let av_device : Box<AVBufferRef> = self.alloc(HwDeviceType::D3d11va);
unsafe {
let device_context = Box::from_raw(av_device.data as *mut AVHWDeviceContext);
let mut d3d11_device_context = Box::from_raw(device_context.hwctx as *mut AVD3D11VADeviceContext);
d3d11_device_context.device = device.as_mut() as *mut _;
d3d11_device_context.device_context = context.as_mut() as *mut _;
let avp = Box::into_raw(av_device);
av_hwdevice_ctx_init(avp);
self.av_hwdevice = Some(Box::from_raw(avp));
}
}
On the Rust side the Device does work, but on the C side, when FFMEPG calls ID3D11DeviceContext_QueryInterface
the app crashes with the following error: Exception 0xc0000005 encountered at address 0x7ff9fb99ad38: User-mode data execution prevention (DEP) violation at location 0x7ff9fb99ad38
The address is actually the pointer for the lpVtbl of QueryInterface, like seen here:
The disassembly of the address also looks correct (this is done on an another debugging session):
(lldb) disassemble --start-address 0x00007ffffdf3ad38
0x7ffffdf3ad38: addb %ah, 0x7ffffd(%rdi,%riz,8)
0x7ffffdf3ad3f: addb %al, (%rax)
0x7ffffdf3ad41: movabsl -0x591fffff80000219,