Why is Python not finding something on the system path?-CodePudding

I have compiled yajl in a Linux container, which puts the compiled files in /usr/local/lib. I've also added this folder to the system path, so I think I should be able to reference the compiled libyajl.so file without specifying the full path to the file. But when I try to pip install yajl-py it fails, and its failing when it tries to run this line of code:

cdll.LoadLibrary('libyajl.so')

I am getting the error:

OSError: libyajl.so: cannot open shared object file: No such file or directory

To confirm I have things set up correctly I print the path:

import os
print(os.environ["PATH"])

And I get:

/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/lib

And to confirm the LoadLibrary function would work if the file was found I run:

from ctypes import cdll
cdll.LoadLibrary('/usr/local/lib/libyajl.so')

And I get the expected:

<CDLL '/usr/local/lib/libyajl.so', handle 560efac0f110 at 0x7f61ded49ed0>

I'm a bit lost on what is going wrong here. Any ideas?

CodePudding user response：

This happens because PATH is only used to find executables, not libraries.

To specify additional search paths for libraries, use LD_LIBRARY_PATH.

CodePudding user response：

Short Explanation

The short answer because you're on a Linux distribution - rebuild your /etc/ld.so.cache with...

sudo ldconfig

And then try your code that uses ctypes.cdll.LoadLibrary again, because libraries in /usr/local/lib are normally indexed and discoverable by ctypes.

Longer Explanation

Fundamentally, ctypes.cdll.LoadLibrary uses the system call dlopen(), which will attempt to find libraries in the following order:

(ELF only) If the calling object (i.e., the shared library or executable from which dlopen() is called) contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag, then the directories listed in the DT_RPATH tag are searched.

If, at the time that the program was started, the environment variable LD_LIBRARY_PATH was defined to contain a colon-separated list of directories, then these are searched. (As a security measure, this variable is ignored for set-user-ID and set-group-ID programs.) o

(ELF only) If the calling object contains a DT_RUNPATH tag, then the directories listed in that tag are searched. o

The cache file /etc/ld.so.cache (maintained by ldconfig(8)) is checked to see whether it contains an entry for filename. o

The directories /lib and /usr/lib are searched (in that order).

So you can see that LD_LIBRARY_PATH is what would be searched, and not PATH.

However, your problem is probably not that you didn't have LD_LIBRARY_PATH configured, because dlopen also looks in /etc/ld.so.cache, which is a special file that is a cache of known shared objects on the system, and on any typical Linux distribution I know of, shared libraries in /usr/local/lib are included in this cache.

And how this cache is rebuilt is via the tool ldconfig, which is (usually) configured with the file /etc/ld.so.conf. On my test machine (a pristine Debian Linux install), the /etc/ld.so.conf looks like this:

# Multiarch support
/usr/local/lib/aarch64-linux-gnu
/lib/aarch64-linux-gnu
/usr/lib/aarch64-linux-gnu
/usr/lib/aarch64-linux-gnu/libfakeroot
# libc default configuration
/usr/local/lib

So you can see that the ld.so.cache will be compiled from libraries in this directory.

So, let me demonstrate what happens on a pristine Linux install:

# docker run -it debian

apt update && apt install cmake git python3 python3-pip
git clone https://github.com/lloyd/yajl
cd yajl
./configure && make && make install


# build stuff
...
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libyajl.so.2.1.1
-- Installing: /usr/local/lib/libyajl.so.2
-- Installing: /usr/local/lib/libyajl.so
-- Installing: /usr/local/lib/libyajl_s.a
-- Installing: /usr/local/include/yajl/yajl_parse.h
-- Installing: /usr/local/include/yajl/yajl_gen.h
-- Installing: /usr/local/include/yajl/yajl_common.h
-- Installing: /usr/local/include/yajl/yajl_tree.h
-- Installing: /usr/local/include/yajl/yajl_version.h
-- Installing: /usr/local/share/pkgconfig/yajl.pc
-- Installing: /usr/local/bin/json_reformat
-- Set runtime path of "/usr/local/bin/json_reformat" to ""
-- Installing: /usr/local/bin/json_verify
-- Set runtime path of "/usr/local/bin/json_verify" to ""
make[1]: Leaving directory '/root/yajl/build'

So now I have /usr/local/lib/libyajl.so (and a number of others), but Python doesn't find it:

root@11f76d842b80:~/yajl# python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.cdll.LoadLibrary("libyajl.so")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libyajl.so: cannot open shared object file: No such file or directory

So, you can now do one of two things - one is to manually set LD_LIBRARY_PATH before running python3, like so...

root@11f76d842b80:~/yajl# LD_LIBRARY_PATH="/usr/local/lib" python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.cdll.LoadLibrary("libyajl.so")
<CDLL 'libyajl.so', handle 2c32ba0 at 0xffffa1699910>

And this lets ctypes find the library.

However, why do that when you could also just rebuild your ld.so.cache? This is usually rebuilt on system reboots (or other events, not sure what every distribution does), but you can also manually force it by running ldconfig.

So in my Debian container where ctypes can't find libyajl.so yet, I do this...

# you might need 'sudo ldconfig'
root@11f76d842b80:~/yajl# ldconfig

root@11f76d842b80:~/yajl# python3

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.cdll.LoadLibrary("libyajl.so")
<CDLL 'libyajl.so', handle 4040a30 at 0xffffa0d24ee0>

And now everything's right and libyajl.so has been indexed.