VisualByteOrg

How to set up a JIT debugger to inspect the runtime memory of Rust code?

1. First, enable a Just-In-Time debugger; for instance, using WinDbg.

2. You can also enable and disable the JIT debugger form the windows registry.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug  

(For 32-bit apps on 64-bit machines) HKEY_LOCAL_MACHINE\Software\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug

3. Insert an int3 assembly instruction in Rust code.

use std::arch::asm;

#[derive(Debug)]
pub struct MenuItem {
    pub id: i32,
    pub name: String,
}

fn main() {
    unsafe {
        asm!("int 3");
    }
    let item: MenuItem = MenuItem { id: 1, name: "TEST".to_string() };

    print!("item is {:?}\n", item);

    let v = vec!['a';10];
    let v_ptr = v.as_ptr();
    print!("v_addr is {:p}\n", v_ptr);

    let x = 42;
    let x_addr = &x as *const i32;
    print!("x_addr is {:p}\n\n", x_addr);

    let mut vec_obj = Vec::<u8>::new();
    vec_obj.resize(16, b'b');
    print!("vec_ptr is {:p}\n", vec_obj.as_ptr());

    let box_vec = Box::new(vec_obj);
    let box_vec_ptr = box_vec.as_ptr();
    
    print!("box_vec_ptr is {:p}\n", box_vec_ptr);
}

4. Running the compiled executale application with cargo run will launch the JIT debugger.

5. The process from triggering an exception with int3 to launching the JIT debugger is as follows:

int3 → CPU exception → Kernel converts to EXCEPTION_BREAKPOINT → Dispatched to user-mode SEH/VEH → If unhandled, invokes UnhandledExceptionFilter → Reads AeDebug registry entry → Launches JIT debugger → Debugger attaches and handles the exception.

6. The Key Terminology Breakdown:  

int3 : x86 instruction triggering a software breakpoint.  

CPU exception : Hardware-level interrupt (trap) for breakpoint (#BP).  

EXCEPTION_BREAKPOINT : Windows-specific exception code (0x80000003).  

SEH/VEH : Structured Exception Handling (per-thread) / Vectored Exception Handling (global).  

UnhandledExceptionFilter : Final Windows API for unhandled exceptions.  

AeDebug registry entry : Registry key (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug) defining JIT debugger settings.

JIT debugger : Just-In-Time debugger (e.g., WinDbg, Visual Studio) configured via Debugger registry value.