Problem description

At runtime, I am given a list of addresses of functions (in the same process). Each time any of them is called, I need to log its address.

My attempt

If there was just one function (with help of a hooking library like subhook) I could create a hook:

create_hook(function_to_be_hooked, intermediate)

intermediate(args...):
  log("function with address {&function_to_be_hooked} got called")
  remove_hook(function_to_be_hooked)
  ret = function_to_be_hooked(args...)
  create_hook(function_to_be_hooked, intermediate)
  return ret

This approach does not trivially extend. I could add any number of functions at compile-time, but I only know how many I need at runtime. If I hook multiple functions with the same intermediate, it doesn't know who called it.

Details

It seems like this problem should be solved by a hooking library. I am using C/C and Linux and the only options seem to be subhook and funchook, but none of them seem to support this functionality.

CodePudding user response：

This should be fairly doable with assembly language manually, like if you were modifying a hook library. The machine code that overwrites the start of the original function can set a register or global variable before jumping to (or calling) the hook. Using call would push a unique return address which the hook likely wouldn't want to actually return to. (So it unbalances the return-address predictor stack, unless the hook uses ret with a modified return address, or it uses some prefixes as padding to make the call hook or call [rel hook_ptr] or whatever end at an instruction boundary of the original code so it can ret.)

Like mov al, imm8 if the function isn't variadic in the x86-64 System V calling convention, or mov r11b, imm8 in x86-64. Or mov ah, imm8 would work in x86-64 SysV without disturbing the AL= # of XMM args for a variadic function and still only be 2 bytes. Or use push imm8.

If the hook function itself was written in asm, it would be straightforward for it to look for a register, and extra stack arg, or just a return address from a call, as an extra arg without disturbing its ability to find the args for the hooked function. If it's written in C, looking in a global (or thread-local) variable avoids needing a custom calling convention.

---

But with existing hook libraries, assuming you're right they don't pass an int id

Using that library interface, it seems you'd need to generate an unknown number of unique things that are callable as a function pointer? That's not something ISO C can do. It can be strictly ahead-of-time compiled, not needing to generate any new machine code at run-time. It's compatible with a strict Harvard architecture.

You could define a huge array of function pointers to hook1(), hook2(), etc. which each look for their own piece of side data in another struct member of that array. Enough hook functions that however many you need at run-time, you'll already have enough. Each one can hard-code the array element it should access for its unique string.

You could use some C preprocessor macros to define some large more-than-enough number of hooks, and separately get an array initialized with structs containing function pointers to them. Some CPP tricks may allow iterating over names so you don't have to manually write out define_hook(0) define_hook(1) ... define_hook(MAX_HOOKS-1). Or maybe have a counter as a CPP macro that gets #defined to a new higher value.

Unused hooks would be sitting in memory and in your executable on disk, but wouldn't ever be called so they wouldn't be hot in cache. Ones that didn't share a page with any other code wouldn't ever need to get paged in to RAM at all. Same for later parts of the array of pointers and side-data. It's inelegant and clunky, and doesn't allow an unbounded number, but if you can reasonably say that 1024 or 8000 "should be enough for everyone", then this can work.

---

Another way also has many downsides, different but worse than the above. Especially that it requires calling the rest of your program from the bottom of a recursion (not just calling an init function that returns normally), and uses a lot of stack space. (You might ulimit -s to bump up your stack size limit over Linux's usual 8MiB.) Also it requires GNU extensions.

GNU C nested functions can make new callable entities with, making "trampoline" machine code on the stack when you take the address of a nested function. This would your stack executable, so there's a security hardening downside. There'd be one copy of the actual machine code for the nested function, but n copies of trampoline code that sets up a pointer to the right stack frame. And n instances of a local variable that you can arrange to have different values.

So you could use a recursive function that went through your array of hooks like foo(counter 1, hooks 1), and have the hook be a nested function that reads counter. Or instead of a counter, it can be a char* or whatever you like; you just set it in this invocation of the function.

This is pretty nasty (the hook machine code and data is all on the stack) and uses potentially a lot of stack space for the rest of your program. You can't return from this recursion or your hooks will break. So the recursion base-case will have to be (tail) calling a function that implements the rest of your program, not returning to your ultimate caller until the program is ending.

---

C has some std:: callable objects, like std::function = std::bind of a member function of a specific object, but they're not type-compatible with function pointers.

You can't pass a std::function * pointer to a function expecting a bare void (*fptr)(void) function pointer; making that happen would potentially require the library to allocate some executable memory and generate machine code in it. But ISO C is designed to be strictly ahead-of-time compilable, so they don't support that.

std::function<void(void)> f = std::bind(&Class::member, hooks[i]); compiles, but the resulting std::function<void(void)> object can't convert to a void (*)() function pointer. (https://godbolt.org/z/TnYM6MYTP). The caller needs to know it's invoking a std::function<void()> object, not a function pointer. There is no new machine code, just data, when you do this.

CodePudding user response：

My instinct is to follow a debugger path.

You would need

• a uin8_t * -> uint8_t map,
• a trap handler, and
• a single step handler

In broad stokes,

• When you get a request to monitor a function, add its address, and the byte pointed by it to the map. Patch the pointed-to byte with int3.

• The trap handler shall get an offending address from the exception frame, and log it. Then It shall unpatch the byte with the value from the map, set a single-step flag in the PSW (again, in the exception frame), and return. That will execute the instruction, and raise a single-step exception.

• The single-step handler shall re-patch int3, and return to let the program run until it hits int3 again.

In POSIX, the exception frame is pointed by uap argument to a sigaction handler.

PROS:

• No bloated binary
• No compile-time instrumentation

CONS:

• Tricky to implement correctly. Remapping text segment writable; invalidating I-cache; perhaps something more.
• Huge performance penalty; a no-go in real-time system.