Skywalker13

Diary of an ex-GeeXboX developer…

Relocatable binaries with rpath on Linux

Posted at — Jan 15, 2017

Portable binaries on Linux

The rpath functionality is something that exists only on unix-like OS. It’s very powerful in order to manage the dynamic loading as you want. The goal here is not to explain rpath; just use your favorite search engine for that. Here I will told you how to use rpath in order to provide binaries and libraries without a dependency on the system paths like /usr/lib and without the use of LD_LIBRARY_PATH or DYLD_LIBRARY_PATH which can be very dangerous and even a bit limited on Darwin (see SIP).

On Linux

It’s simple but it’s a bit more limited that on Darwin. In most of cases you have your libraries in lib and the main executable in bin. It means that the main executable must look for your libraries in ../lib.

In order to add a relative path, you must link with -rpath. This attribute must provides one or more paths where the dynamic libraries must be searched. For relative path, it’s possible to use a placeholder named $ORIGIN which is the location of the main executable. Otherwise, you must pass an absolute location.

You can pass this attribute with the CFLAGS=-Wl for the linker. For example:

CFLAGS='-Wl,-rpath=$ORIGIN/../lib'

Note that the use of [$] can be a bit problematic because it can be expanded by the shell or by make. It must be escaped by \$ORIGIN with the shell and $$ORIGIN with make.

You can have more than one path just by doing like this: -Wl,-rpath=$ORIGIN/../lib:/opt/my/lib.

See this example with the build of curl (I use readelf in order to see the sections in the executable):

$ readelf -d ./usr/bin/curl

Dynamic section at offset 0x28dd0 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]
 [...]]

An rpath value was added by the linker in the RUNPATH section. The same must be done for the libraries, for example with libcurl.so.4:

$ readelf -d ./usr/lib/libcurl.so.4

Dynamic section at offset 0x68d20 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libidn.so.11]
 0x0000000000000001 (NEEDED)             Shared library: [librtmp.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [liblber-2.4.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libldap_r-2.4.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libcurl.so.4]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]
 [...]]

You can check that the loader is using the right paths with ldd:

$ ldd ./usr/bin/curl
        linux-vdso.so.1 (0x00007ffccb787000)
        libcurl.so.4 => /home/schroeterm/usr/bin/../lib/libcurl.so.4 (0x00007fb817e5f000)
        libssl.so.1.1 => /home/schroeterm/usr/bin/../lib/libssl.so.1.1 (0x00007fb817bea000)
        libcrypto.so.1.1 => /home/schroeterm/usr/bin/../lib/libcrypto.so.1.1 (0x00007fb81774e000)
        libz.so.1 => /home/schroeterm/usr/bin/../lib/libz.so.1 (0x00007fb817531000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb817148000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007fb816f14000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fb816cf7000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fb816ae8000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fb816897000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb816691000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb816474000)
        /lib64/ld-linux-x86-64.so.2 (0x000055a978e49000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fb8160dc000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fb815ea7000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fb815c70000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fb8159ed000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fb8157d4000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fb8155b9000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fb815354000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fb815141000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fb814f38000)

You can see with ldd that a lot of libraries are still pointing on the system paths. The reason is that I’ve just rebuilt libcurl, libssl, libcrypto and libz. I’m sure at least for these libraries that it’s using my own versions and not the versions provided by my system. I can move the bin and lib directory together without breaking something because I’m using a relative rpath.

If you are using LD_LIBRARY_PATH, you should consider to rebuild everything properly with rpath because you can break some libraries otherwise. If I try ldd on /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (which is a dependency of my curl, you will see libz too; but it’s not the same):

$ ldd /usr/lib/x86_64-linux-gnu/libgnutls.so.30
        linux-vdso.so.1 (0x00007ffc915ec000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f702f95d000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f702f6f8000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007f702f4c4000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f702f2b1000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f702f07a000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f702ee43000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f702ebc0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f702e822000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f702e619000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f702e415000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f702e1f8000)
        /lib64/ld-linux-x86-64.so.2 (0x000055e562126000)

It means that ./usr/bin/curl depends of libz.so.1 => /home/schroeterm/usr/bin/../lib/libz.so.1 (0x00007fb817531000) and libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30. But libgnutls depends of libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f702f95d000).

You can see two libz files. One in my own tree, and a second one in the system tree. With rpath, everything is fine because each library uses the right dependencies.

But if I use LD_LIBRARY_PATH, I break the dependencies:

LD_LIBRARY_PATH=/home/schroeterm/usr/lib ldd /usr/lib/x86_64-linux-gnu/libgnutls.so.30
        linux-vdso.so.1 (0x00007ffec8df0000)
        libz.so.1 => /home/schroeterm/usr/lib/libz.so.1 (0x00007fb899560000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fb8992b2000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007fb89907e000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fb898e6b000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fb898c34000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fb8989fd000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fb89877a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb8983dc000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fb8981d3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb897fcf000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb897db2000)
        /lib64/ld-linux-x86-64.so.2 (0x000055684a461000)

In this case, libgnutls is using my own libz version and it’s not what I want. Most of time it works fine, but sometimes it breaks your runtime. Then I repeat, you should never use LD_LIBRARY_PATH (or similar).

patchelf

Maybe you can not rebuild the libraries, or maybe it’s too difficult to pass $ORIGIN by command line because it’s expanded by the shell and/or by make. It’s possible to patch the binaries after the linking by using patchelf.

Do not use chrpath, it’s too limited because when the new rpath is longer, it’s not possible to change the value.

To set the rpath with patchelf, it’s very simple:

patchelf --set-rpath '$ORIGIN/../lib' ./usr/bin/curl

On Darwin

In a future article…