Making sure the correct libraries are loaded at run-time

Once I cross-compiled the OpenSSL libraries and moved them onto my Raspberry Pi, I needed to use them to compile my own program that makes use of them. This is when I became aware of the fact that telling the program linker in your Makefile which libraries to link against is not enough for them to actually be loaded into memory when you run your program!

I now had two shared librariesĀ and in my /usr/local/lib folder, which I had copied over from my computer. After successfully linking the program against them by telling make where to find them with the -L option, I ran the program.

However, I noticed very quickly that it was not running properly. Specifically, I was getting an error that I’d already come across before when I had tried to use the 1.0.1x version of OpenSSL – some ciphers I needed were missing. I then noticed by running ldd on the executable that the two libraries I had linked against in my Makefile were not the libraries that were being loaded at run-time!

ldd showed me unmistakably that it knew two other libraries with the very same names but at a different path, specifically and in /usr/lib/arm-linux-gnueabihf, and those were being loaded at run time, not the ones I had cross-compiled and linked against.

Why?! What can I say, I confess to my own ignorance :) After some googling I became aware of the existence of two different kinds of linker, the program linker and the dynamic linker, which perform very different jobs. The program linker (such as ld) is the one make calls at linking time, and it uses the shared libraries you tell it to with the -L -l options. The dynamic linker, though, has a completely different job. The whole point of using dynamically linked libraries as opposed to statically linked libraries is, of course, to be able to share the same code across several running processes that use it, instead of copying it into each one of them. This makes for much smaller executables and much better use of memory. It is the dynamic linker who is responsible for identifying the shared library that is to be loaded when you run your program.

1. The “external” solution with LD_LIBRARY_PATH

So. The dynamic linker is directing your program to the wrong library. How do you point it to the correct one? The first solution doesn’t concern the ELF executable itself, and it achieves the goal of directing the dynamic linker to the correct libraries by setting the environment variable LD_LIBRARY_PATH:

$ export LD_LIBRARY_PATH=/path/to/folder1:/path/to/folder2

The dynamic linker will look for the required libraries in the specified folder(s) first and, if it finds them, that’s it! Otherwise it will proceed to look in the “standard locations”. This works, but requires you to set the environment variable each time, either manually or with a wrapper script. Which is why the next solution may be more practical.

2. The “internal” solution by setting DT_RUNPATH

By running readelf -d on your executable you can have a look at its .dynamic section, which includes information the program linker writes for the dynamic linker, so that the required libraries may be found and linked properly. You can see that, unless you take one of the steps discussed below, the shared libraries are marked as NEEDED and identified solely by their sonames:

$ readelf -d myprog
Dynamic section at offset 0x270 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []

It is up to the dynamic linker to locate the actual files corresponding to the sonames, which can be a bit of a problem if, for any reason, you happen to have two different libraries with equal file names and equal sonames.

At compile time

Fortunately, the program linker can leave the dynamic linker “a note” in this .dynamic section, pointing it not just to the soname of the libraries, but also to their actual location! This is done by setting the DT_RPATH orĀ DT_RUNPATH attribute. For this, use the -rpath=/path/to/folder1:/path/to/folder2/ option of ld, which you can have gcc pass on for you with the -Wl,-rpath=/path/to/folder1/:/path/to/folder2/ option. You see that you can provide a colon-separated list of the locations the libraries are to be loaded from.

$ gcc -Wall -Wl,-rpath=/path/to/folder1:/path/to/folder2 yourprog.c \
-o yourprog -L/path/to/folder1 -L/path/to/folder2 -lyourlib1 -lyourlib2

Use -Wl,-rpath=/path/to/folder1/:/path/to/folder2,--enable-new-dtags to create a DT_RUNPATH rather than a DT_RPATH attribute, the difference being that the former can be used together with LD_LIBRARY_PATH (first LD_LIBRARY_PATH is searched, then DT_RUNPATH).

The result?

$ readelf -d myprog

Dynamic section at offset 0xde8 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []
 0x000000000000001d (RUNPATH)            Library runpath: [/path/to/folder]

Run ldd or lddtree on the executable to convince yourself that the correct libraries are going to be loaded!

Can’t recompile?

You can’t or don’t want to recompile the executable? You can change the DT_RPATH or DT_RUNPATH attribute with the patchelf utility:

$ patchelf --set-rpath /path/to/folder1:/path/to/folder2 yourprog

If you’re going to be using the libraries as “default” libraries for a number of programs, these are probably not the right solutions for you, and I would suggest both of us go and have a look at ldconfig!