Making sure the correct libraries are loaded at run-time

Once I cross-compiled the OpenSSL libraries and moved them onto my Raspberry Pi, I needed to use them to compile my own program that makes use of them. This is when I became aware of the fact that telling the program linker in your Makefile which libraries to link against is not enough for them to actually be loaded into memory when you run your program!

I now had two shared libraries and in my /usr/local/lib folder, which I had copied over from my computer. After successfully linking the program against them by telling make where to find them with the -L option, I ran the program.

However, I noticed very quickly that it was not running properly. Specifically, I was getting an error that I’d already come across before when I had tried to use the 1.0.1x version of OpenSSL – some ciphers I needed were missing. I then noticed by running ldd on the executable that the two libraries I had linked against in my Makefile were not the libraries that were being loaded at run-time!

ldd showed me unmistakably that it knew two other libraries with the very same names but at a different path, specifically and in /usr/lib/arm-linux-gnueabihf, and those were being loaded at run time, not the ones I had cross-compiled and linked against.

Why?! What can I say, I confess to my own ignorance :) After some googling I became aware of the existence of two different kinds of linker, the program linker and the dynamic linker, which perform very different jobs. The program linker (such as ld) is the one make calls at linking time, and it uses the shared libraries you tell it to with the -L -l options. The dynamic linker, though, has a completely different job. The whole point of using dynamically linked libraries as opposed to statically linked libraries is, of course, to be able to share the same code across several running processes that use it, instead of copying it into each one of them. This makes for much smaller executables and much better use of memory. It is the dynamic linker who is responsible for identifying the shared library that is to be loaded when you run your program.

1. The “external” solution with LD_LIBRARY_PATH

So. The dynamic linker is directing your program to the wrong library. How do you point it to the correct one? The first solution doesn’t concern the ELF executable itself, and it achieves the goal of directing the dynamic linker to the correct libraries by setting the environment variable LD_LIBRARY_PATH:

$ export LD_LIBRARY_PATH=/path/to/folder1:/path/to/folder2

The dynamic linker will look for the required libraries in the specified folder(s) first and, if it finds them, that’s it! Otherwise it will proceed to look in the “standard locations”. This works, but requires you to set the environment variable each time, either manually or with a wrapper script. Which is why the next solution may be more practical.

2. The “internal” solution by setting DT_RUNPATH

By running readelf -d on your executable you can have a look at its .dynamic section, which includes information the program linker writes for the dynamic linker, so that the required libraries may be found and linked properly. You can see that, unless you take one of the steps discussed below, the shared libraries are marked as NEEDED and identified solely by their sonames:

$ readelf -d myprog
Dynamic section at offset 0x270 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []

It is up to the dynamic linker to locate the actual files corresponding to the sonames, which can be a bit of a problem if, for any reason, you happen to have two different libraries with equal file names and equal sonames.

At compile time

Fortunately, the program linker can leave the dynamic linker “a note” in this .dynamic section, pointing it not just to the soname of the libraries, but also to their actual location! This is done by setting the DT_RPATH or DT_RUNPATH attribute. For this, use the -rpath=/path/to/folder1:/path/to/folder2/ option of ld, which you can have gcc pass on for you with the -Wl,-rpath=/path/to/folder1/:/path/to/folder2/ option. You see that you can provide a colon-separated list of the locations the libraries are to be loaded from.

$ gcc -Wall -Wl,-rpath=/path/to/folder1:/path/to/folder2 yourprog.c \
-o yourprog -L/path/to/folder1 -L/path/to/folder2 -lyourlib1 -lyourlib2

Use -Wl,-rpath=/path/to/folder1/:/path/to/folder2,--enable-new-dtags to create a DT_RUNPATH rather than a DT_RPATH attribute, the difference being that the former can be used together with LD_LIBRARY_PATH (first LD_LIBRARY_PATH is searched, then DT_RUNPATH).

The result?

$ readelf -d myprog

Dynamic section at offset 0xde8 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []
 0x0000000000000001 (NEEDED)             Shared library: []
 0x000000000000001d (RUNPATH)            Library runpath: [/path/to/folder]

Run ldd or lddtree on the executable to convince yourself that the correct libraries are going to be loaded!

Can’t recompile?

You can’t or don’t want to recompile the executable? You can change the DT_RPATH or DT_RUNPATH attribute with the patchelf utility:

$ patchelf --set-rpath /path/to/folder1:/path/to/folder2 yourprog

If you’re going to be using the libraries as “default” libraries for a number of programs, these are probably not the right solutions for you, and I would suggest both of us go and have a look at ldconfig!


Cross-compile OpenSSL for your Raspberry Pi

Have you tried to compile OpenSSL directly on your Raspberry Pi? Don’t! Save yourself a lot of time by compiling it on your desktop computer or laptop instead, then move it to your Pi!

1. Get the cross-compiler

First you need to get the cross-compiling tools on your computer. Those will allow you to compile for your Pi’s ARM architecture. The cross-compiling tools are part of the raspberrypi/tools repository on Github, which you’re going to have to clone to your computer. From whatever base folder you’ve chosen for this, type:

$ git clone --depth=1 pitools

The option --depth=1 specifies you only want to copy the latest revision of the repository. You can leave it out if you prefer to clone the whole thing instead.

You will now find two important folders in the arm-bcm2708 subfolder of your newly created pitools repository. The first is called gcc-linaro-arm-linux-gnueabihf-raspbian and contains the cross-compiling tools you must use on a 32-bit computer, the second is called gcc-linaro-arm-linux-gnueabihf-raspbian-x64 and contains predictably the tools for 64-bit systems. Note that the Raspberry Pi itself has a 32-bit processor, so we aren’t talking about the Pi now, but your computer, which is either 32 or 64-bit.

The cross-compiling tools themselves are in the respective bin subfolder. To put it all together, on a 64-bit system the path to the cross-compiling tools for your Raspberry Pi will be:


Export it as a variable for later:

$ export CROSSCOMP_DIR=/path/to/basefolder/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin

Have a look at the files in the bin folder. You’ll notice they all start with the arm-linux-gnueabihf- prefix and include a C compiler (gcc), a C++ compiler (g++), a linker (ld), etc.

2. Get the OpenSSL source

Assuming you want to compile and install the latest OpenSSL version on your Pi, type:

$ git clone

Again, choose a base folder for cloning the repository into. If you don’t want the latest OpenSSL version, go to and download the tar.gz file of the version you’re interested in, extract it and move the folder to the location of your choice.

3. Cross-compile the OpenSSL source

At this point you’re ready to cross-compile.

cd into the OpenSSL repository and export the following variable for convenience:

$ export INSTALL_DIR=/path/to/installation/folder

It should contain the path to the folder you want to install the cross-compiled OpenSSL to.

Now run the Configure script with the following options:

  • linux-generic32 : You’re compiling for a 32-bit Linux system, i.e. your Raspberry Pi
  • shared : To create static (.a) AND shared (.so) OpenSSL libraries
  • --prefix=$INSTALL_DIR : The path to the folder your cross-compiled OpenSSL will be installed in, as seen above
  • --openssldir=$INSTALL_DIR/openssl : This folder will contain certain configuration files
  • --cross-compile-prefix=$CROSSCOMP_DIR/arm-linux-gnueabihf- : This is the path to the cross-compiling tools we’ve seen above, and kind of the point of the whole thing. Notice the dash at the end!
user@host:/path/to/folder/openssl $ ./Configure linux-generic32 shared \
--prefix=$INSTALL_DIR --openssldir=$INSTALL_DIR/openssl \

After the configuration step is done, run the following commands:

make depend
make install

Your installation folder now contains the cross-compiled OpenSSL!

Specifically, the bin folder contains the openssl executable and the lib folder contains the OpenSSL libraries.

You can check the openssl executable in $INSTALL_DIR/bin to see that it really is a 32-bit Linux executable for the ARM platform :)

$ file $INSTALL_DIR/bin/openssl
openssl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.26,
BuildID[sha1]=60d55b25b044bf1f378eba2e7890c928f8552609, not stripped

4. Transfer the cross-compiled OpenSSL to your Pi

Now transfer the installation folder to your Pi, for example via scp, and check that the executable can effectively run, e.g. by cd‘ing into the folder and typing:

user@host:/path/to/folder/openssl/bin $ ./openssl help

Once compiling your programs on the Pi with the new OpenSSL libraries, make sure they are also the ones actually being loaded at run-time!

Use SSL/TLS within a different protocol with BIO pairs

In this post I’m going to explain to you how you can use the OpenSSL library to wrap the SSL/TLS protocol in a different protocol. I’m going to assume basic knowledge of the TLS protocol as well as the OpenSSL library, mainly: what is an SSL context (SSL_CTX), what is an SSL struct (SSL) and how you set them up for a regular TLS session over TCP/IP sockets.

One of the great things the OpenSSL library does is allow you to separate all that is related to the TLS protocol itself from any specific communication channel that TLS is going to be used over.

TLS being a cryptographic protocol for the Internet, it is typically used by applications communicating through TCP/IP sockets. In this case, OpenSSL allows you to set up a TLS connection fairly easily by dealing with the sockets for you right from the beginning. Once you’ve set up your SSL struct and called SSL_accept (on the server) or SSL_connect (on the client), all the necessary TLS handshake messages are sent back and forth for you over the socket you provided. Any plaintext data that you subsequently want to send over to the other side simply needs a call to SSL_write. This function will cause the data to be encrypted and sent for you. Whatever data you receive, a call to SSL_read will cause it to be read from the socket, decrypted and provided to you in plaintext (fig. 1).

Fig. 1 OpenSSL takes care of encryption/decryption and sending/receiving over the network for you.

This is all good if you want to use OpenSSL to establish a regular TLS connection between two hosts over TCP/IP sockets, as one probably does most of the time. However, you can still use OpenSSL to perform TLS regardless of how the TLS data is going to be sent between the client and the server.

Possibly the TLS data has to be exchanged over something other than TCP/IP sockets. Or, like me, you may be working with TCP/IP sockets but need to encapsulate the TLS data stream within a different protocol, which means that you can’t let OpenSSL take care of everything for you like above, or you wouldn’t have any control over the TLS data stream that is sent and received.

In such cases, the idea is the following:

Instead of having OpenSSL encrypt-and-send as well as receive-and-decrypt the data for you over a socket, you separate the encryption/decryption step from the sending/receiving step by having OpenSSL write to and read from memory buffers.

OpenSSL will take your application’s data, encrypt it, and put it in a memory buffer, from which your application can take it and do whatever it needs to do in order to send it over to the other host. Messages generated autonomously by OpenSSL, such as the TLS handshake messages, will be written to that memory buffer, too. OpenSSL will also read encrypted data coming from the other host, which your application will have put in another memory buffer, decrypt it and, unless it’s TLS protocol stuff, provide it to your application. So it’s your application’s job to transport the data from a memory buffer to the network and vice versa, after adding or removing whatever headers may be required in your particular situation (fig. 2).

Fig. 2 You can use OpenSSL to encrypt and decrypt TLS data as necessary, taking care yourself of how the data is then sent over the network through custom functions as appropriate.

BIO pairs

OpenSSL achieves the separation between the TLS data stream and the medium and/or protocol it is carried over by using abstract I/O entities called BIO’s, of which there are several kinds serving different purposes, which we’re not going to look into here. See the dreadful OpenSSL documentation if you want to learn more about them.

For our purposes we’re going to be using a pair of memory BIO’s working together as a BIO pair. From the OpenSSL documentation:

A memory BIO is a source/sink BIO which uses memory for its I/O. Data written to a memory BIO is stored in a BUF_MEM structure which is extended as appropriate to accommodate the stored data. […] Any data written to a memory BIO can be recalled by reading from it. Unless the memory BIO is read only any data read from it is deleted from the BIO. [1]


A BIO pair is a pair of source/sink BIOs where data written to either half of the pair is buffered and can be read from the other half. Both halves must usually by handled by the same application thread since no locking is done on the internal data structures. […] One typical use of BIO pairs is to place TLS/SSL I/O under application control, this can be used when the application wishes to use a non standard transport for TLS/SSL or the normal socket routines are inappropriate. [2]

The Steps

These are the necessary steps you’re going to have to implement for a data flow like the one in fig. 2:

  1. Create two memory BIO’s. These are going to be your interface to the OpenSSL engine on the encrypted side of the communication, i.e. the side that contains encrypted data which has to be sent or has just been received over the network (right side of fig. 2).
  2. Create an SSL struct. The SSL struct embodies the TLS session that is going to be established between the two hosts. It is also going to be your interface to the OpenSSL engine on the plaintext side of the communication, which contains the application data that needs to be encrypted and sent, as well as the received data that has been decrypted and needs to be processed by your application (right side of fig. 2)..
  3. Connect the memory BIO’s to the SSL struct. This is the step that builds the essential connection between the plaintext and the encrypted side of the communication: what is written to OpenSSL with one interface, will be able to be retrieved from the other interface after the appropriate processing by OpenSSL.
  4. Set up special_send and special_recv functions to transport the TLS data between the network and the memory BIO’s. Once encrypted data is available for sending in the appropriate memory BIO, it needs to be sent over the network with some special_send function. If encrypted data has been received over the network, special_recv needs to pass it on to OpenSSL for decryption via the appropriate memory BIO. These functions are where the TLS data gets wrapped in or unwrapped from the additional protocol, meaning that any additional, non-TLS headers must be prepended to or removed from the TLS data in these functions.

Let’s have a look at a sketch of each of these steps with some code. I’m only going to provide a basic logical sequence of function calls, since a proper implementation obviously requires you to make your own code structure decisions.

1. Creating the memory BIO’s

Remember that the memory BIO’s represent your application’s interface to the OpenSSL engine on the encrypted side of the communication. Specifically, rbio will be read from to collect the TLS data coming from OpenSSL, whereas wbio will be written to to pass the received TLS data over to OpenSSL. We also set both BIO’s up to return -1 as opposed to EOF in case they’re empty.

#include <openssl/ssl.h>
#include <openssl/err.h>
BIO *rbio = BIO_new(BIO_s_mem()); /* For reading from with BIO_read() */
BIO *wbio = BIO_new(BIO_s_mem()); /* For writing to with BIO_write() */
BIO_set_mem_eof_return(rbio, -1);
BIO_set_mem_eof_return(wbio, -1);

2. Creating the SSL struct

Creating the SSL struct is a matter of a single function call, provided your application has already set up an appropriate SSL_CTX struct containing the TLS parameters. I’m assuming here that you already know how to do this, or that you will find out with some more Googling.

SSL_CTX *ctx = setup_ctx();
if (ctx == NULL) {
SSL *ssl = SSL_new(ctx);
if (ssl == NULL) {

3. Connecting the memory BIO’s to the SSL struct

At this point we link both sides, the “plaintext” and the “encrypted”, by calling SSL_set_bio. As mentioned above, ssl will be the interface to OpenSSL on the plaintext side, whereas wbio and rbio will be the interface on the encrypted side.

SSL_set_bio(ssl, wbio, rbio);
SSL_set_connect_state(ssl); /* If server: SSL_set_accept_state(ssl) */

An important note. Here I’ve defined “write BIO” as the BIO that is written to by BIO_write, “read BIO” accordingly as the BIO that is read from by BIO_read. This is the way I’ve come to look at it while trying to make sense of the whole construction. This, however, causes the order in my function call SSL_set_bio(ssl, wbio, rbio) to differ from the signature of SSL_set_bio, which is SSL_set_bio(SSL *ssl, BIO *rbio, BIO *wbio). I’m sure you can turn my logic around and find a way in which my “read BIO” can be conceived of as a “write BIO”, and vice versa – possibly the “read BIO” is the one a call to SSL_read will read from… I’ll be consistent here and stick to my logic, but bear that in mind when you consult the OpenSSL documentation.

4. Transporting data to and from the BIO’s

Outgoing data. Your application needs to:

  1. Write the application data placed in buffer buf1 to OpenSSL for encryption using SSL_write.
  2. When the corresponding TLS data is available, read it from the read memory BIO rbio into buffer buf2 by calling BIO_read.
  3. From there, in the function special_send add to the TLS data whatever headers you may need to add before sending, then send it over to the other side over the network (here represented by file descriptor fd).
int fd; /* File descriptor for the network, e.g. socket */
int flags; /* Flags controlling the behaviour of special_send */
int numwritten, numread;
int to_write_len, max_read_len, data_len;
unsigned char buf1[BUF_SIZE];
unsigned char buf2[BUF_SIZE];
/* 1 */
numwritten = SSL_write(ssl, buf1, to_write_len);
/* 2 */
if (BIO_ctrl_pending(rbio) > 0) {
	numread = BIO_read(rbio, buf2, max_read_len);
	/* 3 */
	numwritten = special_send(fd, buf2, data_len, flags);

Incoming data. Similarly, your application needs to:

  1. Read the data from the network into buffer buf3 with special_read, stripping it of any additional headers belonging to a different protocol, so that the “clean” TLS data is placed in buf3.
  2. Write the TLS data to the write memory BIO wbio for decryption (and possibly processing by OpenSSL) with BIO_write.
  3. Call SSL_read to read the decrypted data from OpenSSL into buffer buf4, which now contains the decrypted data ready for use by your application.
unsigned char buf3[BUF_SIZE];
unsigned char buf4[BUF_SIZE];
/* 1 */
numread = special_recv(fd, buf3, max_read_len, flags);
/* 2 */
numwritten = BIO_write(wbio, buf3, to_write_len);
/* 3 */
if (BIO_ctrl_pending(wbio) < 0) {
	numread = SSL_read(ssl, buf4, max_read_len);
	/* The decrypted data is now in buf4 */

Note that to_write_len, max_read_len and data_len are mere placeholders for variables actually specifying how many bytes of data are to be read/written by each of the functions used above. What their values need to be in each function call and how they are calculated is up to your implementation to decide. Also, I’m using a dummy signature for special_send and special_recv along the lines of send and recv.

Note the use of BIO_ctrl_pending(BIO *bio) to ensure that data is available for reading.

At this point you’ve seen the logical sequence of function calls that allow you to write plaintext data to OpenSSL and retrieve the corresponding TLS data from a memory buffer, instead of it being sent straight over a TCP/IP socket. And you’ve seen the logical sequence of function calls allowing you to retrieve plaintext data from the encapsulated TLS data you’ve received over the network. It’s up to you now to decide how these logical sequences are to be expanded in your code as well as add all other necessary steps in order for your code to work! :)

Prepending headers to the TLS data

Before I go, let me add a few words on how to prepend additional headers to your TLS data before sending it over the network. The keyword here is flat data structure. More specifically, you’re going to have to build one buffer containing all bytes to be sent, from the very first of your extra header to the very last of the TLS data, all adjacent.

When you call special_send(fd, buf2, data_len, flags), you have data_len bytes of TLS data in buffer buf2 that you need to pass to fd for sending, but before you do that, you need to add a certain header to the TLS data according to the protocol you’re implementing. For simplicity’s sake, assume you have a function unsigned char *create_header() that returns a pointer to a buffer of constant size HEADER_SIZE containing your header. In your special_send function, then, you could do something like this:

int special_send(int fd, unsigned char *buf, size_t data_len, int flags) {
	unsigned char *header = create_header();
	/* Create flat buffer of appropriate size */
	unsigned char *tmp = malloc(HEADER_SIZE + data_len);

	/* Write the header at the beginning of the tmp buffer */
	memcpy((void *)tmp, (void *)header, HEADER_SIZE);

	/* Now write the TLS data right after it */
	memcpy((void *)tmp + HEADER_SIZE, (void *)buf, data_len);
	/* Now do the actual sending */
	int numsent = send(fd, tmp, HEADER_SIZE + data_len, flags);
	return numsent;

Whatever data you need to send, whatever the buffers you take it from, you always do the same: copy the right amount of bytes from whatever buffer to the next location in your flat buffer tmpin the right order. Once you’re done, send it.