MacPorts /opt/local/bin/python3 can't find _libiconv symbol unless run under shell-CodePudding

I just updated macOS to Monterey 12.5 and updated MacPorts as described here. I have MacPorts python38, python39 and python310 installed. Each exhibits the same new bad behavior.

I have a C program that writes a small Python script into a file and then runs a Python interpreter as a child process, specifying the full pathnames for both the desired Python interpreter and that script file. (This is for self-tests, and it has worked well for years. I'm interested in resolving the problem, not in alternative structures.)

If I run that same command under the shell, either by hand or via std::system(), the script runs as I expect, as it always used to.

But when I directly execute the command under my C program, bypassing the shell, the Python interpeter fails to load with:

dyld[54748]: Symbol not found: (_libiconv)
  Referenced from: '/opt/local/lib/libintl.8.dylib'
  Expected in: '/usr/lib/libiconv.2.dylib'

I must emphasize that this used to work until this week's macOS update. It's not the C library; I get the same behavior from two different library implementations.

You might reasonably think that the new MacPorts libintl.8.dylib contains a bad reference to /usr/lib/libiconv.2.dylib, which in fact no longer exists. But:

$ otool -L /opt/local/lib/libintl.8.dylib
/opt/local/lib/libintl.8.dylib:
    /opt/local/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
    /opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1853.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

So maybe the bad reference would be in /opt/local/lib/libiconv.2.dylib ?

$ otool -L /opt/local/lib/libiconv.2.dylib
/opt/local/lib/libiconv.2.dylib:
    /opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

I don't even understand the mechanism of the error here, never mind how to resolve it. Help very much appreciated!

CodePudding user response：

The only situation where macOS' loader would search /usr/lib/libiconv.2.dylib for a symbol that /opt/local/lib/libintl.8.dylib expects in /opt/local/lib/libiconv.2.dylib is when you have DYLD_LIBRARY_PATH set to include /usr/lib. This variable causes dyld (the macOS loader) to ignore the absolute path of /opt/local/lib/libiconv.2.dylib and instead search for files called libiconv.2.dylib in all paths specified in DYLD_LIBRARY_PATH.

Since libraries are referenced using absolute paths on macOS, setting DYLD_LIBRARY_PATH is usually not necessary. If it is necessary, you should try setting DYLD_FALLBACK_LIBRARY_PATH first, since it does not cause the issue you are seeing.

Ideally, you would avoid the need to set the environment variable by modifying whatever binary requires you to set it. To do that, change the binary to reference its libraries using absolute paths. Use install_name_tool's -change option to fix an existing binary, or relink the binary after fixing the offending library's ID load command (either using install_name_tool -id or by setting the -install_name linker option to an absolute path when linking the library).

The reason why this starts working when run through a Shell is probably macOS' System Integrity Protection. Apple ships all binaries in /usr/bin with a flag that causes the loader to ignore and strip all DYLD_* environment variables, so that they always run using the libraries Apple intended them to use. Your shell is likely either /bin/sh or /bin/zsh and also has this mode bit set, which means any DYLD_LIBRARY_PATH variable that was set when executing the shell was removed from the environment. Note that the same mechanism also prevents you from seeing DYLD_LIBRARY_PATH when you run env (because that's /usr/bin/env also has the bit). Use set | grep DYLD in a running shell instead.

Finally, the reason why /usr/lib/libiconv.2.dylib and /opt/local/lib/libiconv.2.dylib contain different symbols is because Apple sets a define that causes symbols to not have the lib prefix (i.e., the symbol is called _iconv in /usr/lib/libiconv.2.dylib) and MacPorts just uses the configuration that you'd get by default from upstream. A #define in a header file makes this different transparent to users. I'm not sure why the iconv developers have chosen to provide this mechanism. I can only speculate that this is some kind of mechanism that ensures the headers that were used to compile match the library that is being linked and loaded.