Rust Hook

With the accelerating advancement of Rust language, more and more tech companies are recompiling their programs using Rust, especially client security software. However, there are few documents available that explain how to hook any functions using Rust. Therefore, I have written this blog to help you understand how to hook programs using the retour-rs crate for Rust.

In the C language, two hook libraries, detour and MinHook, are commonly used and both have been implemented in Rust. However, I encountered some problems when trying to use MinHook, and the detour-rs crate have not been maintained for a long time. Therefore, I decided to use the retour-rs crate, which is a reconstructed version of detour using the Rust language. Both detour-rs and retour-rs have Generic Hook and Static Hook methods. If you want to use Static Hook, you have to install nightly Rust. In this blog, I will explain how to hook programs in various ways using these two types of hooks.

Environment

Static Hook feature require you to install the nightly rust toolchain, and lots of practical methods can only run in the nightly Rust, such as Backtrace. You can use the following commands to install the nightly Rust.

rustup install nightly
rustup default nightly
rustc --version

And you can install retour-rs crate easily by:

cargo add retour

InlineHook

There is no denying that InlineHook is the most common hook method. I will introduce how to use Generic and Static to hook API and functions.

GenericHook

I will give the easiest GenericHook demo to Hook Windows API “LoadLibraryA”, which can print the path of loaded file and ret_val.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


type type_LoadLibraryA = extern "system" fn(PCSTR) -> HINSTANCE;
static hook_LoadLibraryA: Lazy<GenericDetour<type_LoadLibraryA>> = Lazy::new(|| {
    let ori: type_LoadLibraryA = unsafe{std::mem::transmute(utils::win::get_proc_address("kernel32.dll", "LoadLibraryA").unwrap())};
    return unsafe { GenericDetour::new(ori, hooked_LoadLibraryA).unwrap() };
  });

extern "system" fn hooked_LoadLibraryA(lpFileName: PCSTR) -> HINSTANCE {
    let file_name = unsafe { CStr::from_ptr(lpFileName.as_ptr() as _) };
    
    let ret_val = hook_LoadLibraryA.call(lpFileName);
    println!(
        "hooked_LoadLibraryA lpFileName = {:?} ret_val = {:#X}",
        file_name, ret_val.0
    );
    return ret_val;
}

pub fn register_hook() -> anyhow::Result<()> {
    unsafe{ hook_LoadLibraryA.enable()?; }
    Ok(())
}

If you want to hook a specific address instead of an API, you just need replace ori to a specific address. Sure, you need cast this address number to a function type, same as hooked_function.

StaticHook

The “ReadMe” of retour-rs introduces an easy method for hooking functions. There is an easiest demo which can hook two functions using static_detour! macro, which will be expanded to StaticHook instance. Sure, this feature requires the nightly version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


use retour::static_detour;

static_detour! {
  static Test: /* extern "X" */ fn(i32) -> i32;
}

fn add5(val: i32) -> i32 {
  val + 5
}

fn add10(val: i32) -> i32 {
  val + 10
}

pub fn register_hook() -> anyhow::Result<()> {
    unsafe { Test.initialize(add5, add10)? };
    unsafe { Test.enable()? };

    assert_eq!(add5(1), 11);
    assert_eq!(Test.call(1), 6);
    Ok(())
}

Obviously, this demo is easier than GenericHook demo. However, if you want to hook any address rather than two functions you wrote with the same type, you must do some extra work.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


type type_LoadLibraryA = extern "system" fn(PCSTR) -> HINSTANCE;
static_detour! {
    static HookLoadLibraryA: extern "system" fn(PCSTR) -> HINSTANCE;
}

fn hooked_LoadLibraryA(lpFileName: PCSTR) -> HINSTANCE {
    let file_name = unsafe { CStr::from_ptr(lpFileName.as_ptr() as _) };
    let ret_val = HookLoadLibraryA.call(lpFileName);
    println!(
        "hooked_LoadLibraryA lpFileName = {:?} ret_val = {:#X}",
        file_name, ret_val.0
    );
    return ret_val;
}

pub fn register_hook() -> anyhow::Result<(), > {

    unsafe{ HookLoadLibraryA.initialize(get_proc_address::<type_LoadLibraryA>("kernel32.dll", "LoadLibraryA").unwrap(), hooked_LoadLibraryA)?;}
    unsafe{ HookLoadLibraryA.enable()?; }

    Ok(())
}

pub fn get_proc_address<F: AnyFn>(module_name: &str, func_name: &str) -> anyhow::Result<F> {
    let module = CString::new(module_name)?;
    let func = CString::new(func_name)?;
    let library_handle = unsafe { LoadLibraryA(PCSTR(module.as_ptr() as _)) }?;
    let address = unsafe { GetProcAddress(library_handle, PCSTR(func.as_ptr() as _)).ok_or(anyhow!("GetProcAddress failed : {} : {}", module_name, func_name))? };

    return Ok(unsafe{<F as AnyFn>::from_ptr(std::mem::transmute(address))});
}

To use StaticHook, we have to first define a function type like GenericHook, and then initalize a StaticHook instance using static_detour! macro. And it is important to ensure that two parameters of StaticHook.initialize are the same. However, this can be challenging in certain scenarios, particularly obtaining the address with usize type from get_proc_address. So I implemented a trait to help me cast the ret_val’s type.

With StaticHook, you can easily hook any address to your functions by replacing the API address with a specific address casting its type, like using GenericHook. As shown in the example above, we found that StaticHook is not much simpler than GenericHook if we want to hook an API or a specific address.

VirtualTableHook

In C++, classes that implement polymorphim usually contain virtual functions, which form a virtual table, that stores the address of each virtual function. Every object of this class contains a pointer to its virtual table, which is called vtable pointer. VTable Hook has the advantage that we don’t need to modify the page protect attributes compared with replacing the virtual function address from VTable, because the page stored the vtable pointer is writable.

I have implemented a simple function that allows us to hook any class’s vtable pointer and returns the original vtable pointer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


pub fn read<T>(address: usize) -> T {
    let value_ptr = address as *const T;
    let value = unsafe{std::ptr::read(value_ptr)};
    value
}

pub fn vtb_hook(classptr: usize , hooked_func: usize, index: usize) -> anyhow::Result<usize>
{
    let vtable: usize = read(classptr);

    let mut i: usize = 0;
    let methods = loop{
        let func_ptr: usize = read(vtable + i*8);
        if func_ptr == 0 {
            break i;
        }
        i+=1;
    };
    // println!("method:{}", methods);

    if methods < index {
        panic!("vtb_hook method > index : class_ptr 0x{:X} : method : {}", classptr, methods);
    }

    let vtable_size = methods*ALLIGN_SIZE;
    let mut my_vtable = alloc_memory(None, vtable_size, false)?;
    memcpy(my_vtable, vtable, vtable_size);
    // println!("classptr: 0x{:X}", classptr);
    // println!("vtable: 0x{:X}", vtable);
    // println!("my_vtable: 0x{:X}", my_vtable);

    let orig_func = read(vtable + ALLIGN_SIZE*index as usize);
    write(my_vtable + index*ALLIGN_SIZE, hooked_func);
    write(classptr, my_vtable);
    // println!("orig_func: {:X}", orig_func);
    Ok(orig_func)
}