What are some effective approaches to debug the actual process of loading an ELF binary on a Linux system? I am on Fedora 36 and have created an ELF file. However, when I try to run the ELF file from bash by entering ./myElfFile, I get an error like cannot execute binary file: Exec format error. What I am trying to determine is WHICH specific exec format error, and I'd like to do this using a debugger or some other dynamic tracing. I once had to do this on Windows and I ended up using WinDbg in kernel debug mode, and something called GFlags, and I turned on "loader snaps" which allowed me to watch the entire portable executable loading process. Eventually, I found the checks inside of NtCreateSection and its callees. How would I do something like this in the Linux/ELF world? I want to find out at which step of the loading process it is failing, and precisely which check fails.

CodePudding user response：

What I am trying to determine is WHICH specific exec format error, and I'd like to do this using a debugger or some other dynamic tracing.

The problem (invalid ELF file) is most likely detected by the kernel. Because of that, your program never starts up, and no debugger or any other user-space tool will help.

There are a few approaches you can try:

• If myElfFile started as valid file and was modified / patched, examine differences between the output from readelf -l ... on the working and non-working copies, and see if you can spot a violation of some required invariants (e.g. does .p_offset % pagesize == .p_vaddr % pagesize ?).
• Use readelf -a myElfFile and see if readelf prints any warnings. If it does, you can debug readelf to figure out what it's complaining about -- it may or may not be the same thing which causes the kernel to reject your binary as well.
• Update your question with the output from readelf -l myElfFile and hope that someone else points out what's wrong with it.
• Use user-mode Linux to debug the kernel while it is trying to exec the file.