Some compilers, says old gcc or egcs, apply ABI-breaking optimization for static functions within single file, like passing arguments or returning results with arbitrary registers.

Consider some source code like:

// Original foobar.c
// This example targets MIPS o32 ABI.

// Shared subroutine
// Compiler decided to use $16, $17 to pass a0 and a1 to minimize stack usage and move between registers.
static void __bar(int a0, int a1) {
    // Something very complicated
}

// ...

void foo(int a0, int a1) {
    // ...

    /*
    This call was compiled to something like:
        ori $16, $0, 0x1
        jal __bar
        ori $17, $0, 0x1
    */
    __bar(1, 1); 

    // ...
}

// ...

Suppose someone want to restore / reimplement foobar.c from the compiled assembly without access to the original source.
One would probably like to decompile / rewrite some part first, says start from foo() or other standard functions. However, in order to test the correctness of the implementation, one must deal with calls to non-standard ABI routines.
A trivial way is to workaround with global register variables provided by gcc / clang:

// Restoration of foobar.c

// void __bar(int asm("s0"), int asm("s1"))
// External function in assembly, says foobar.s, which is from compiled original foobar.c.
void __bar();
volatile register int s0 asm ("s0"); // $16 = s0
volatile register int s1 asm ("s1"); // $17 = s1

// ...

void foo(int a0, int a1) {
    // ...

    // __bar(1, 1);
    s0 = 1; s1 = 1;
    __bar();

    // ...
}

// ...

---

The question is:

1. Does gcc / clang supports customize calling convention for some specific functions?
2. Are there any way to deal with non-standard ABI calls more elegantly?

CodePudding user response：

1. Does gcc / clang supports customize calling convention for some specific functions?

The best you can do is opt in to one of the specific supported calling conventions, e.g. one of these for x86. If the static function in question does not conform to any of them, then you're stuck.

2. Are there any way to deal with non-standard ABI calls more elegantly?

Nothing truly elegant. If none of the supported calling conventions apply, you're stuck with either:

1. Reversing & rebuilding the whole thing (so it can compile as normal without relying on original binaries), or at least enough of it that you're replacing ABI conforming functions and all their dependencies completely, or
2. Calling it from assembly, explicitly passing the arguments per the non-standard calling conventions of the compiled function.

#2 is the basis for the most elegant solution, which is basically to write a wrapper function in assembly that receives the arguments and returns the values according to the ABI, and otherwise does nothing but rearrange them to pass to the non-standard function it wraps (and possibly fix up the return value if it's not returning according to normal rules). You write the wrapper(s) once, and now the rest of your code can be written in C, calling the wrapper functions which adhere to the ABI and being blissfully unaware of the weirdness under the covers.

Similarly, if you're trying to replace the existing non-standard function with another, you'd write the non-conforming wrapper in assembly, then write your replacement function in plain C and have the wrapper call it, and swap in your wrapper in your hacked together mix of the original binary and the new code.