Use SSL/TLS within a different protocol with BIO pairs

In this post I’m going to explain to you how you can use the OpenSSL library to wrap the SSL/TLS protocol in a different protocol. I’m going to assume basic knowledge of the TLS protocol as well as the OpenSSL library, mainly: what is an SSL context (SSL_CTX), what is an SSL struct (SSL) and how you set them up for a regular TLS session over TCP/IP sockets.

One of the great things the OpenSSL library does is allow you to separate all that is related to the TLS protocol itself from any specific communication channel that TLS is going to be used over.

TLS being a cryptographic protocol for the Internet, it is typically used by applications communicating through TCP/IP sockets. In this case, OpenSSL allows you to set up a TLS connection fairly easily by dealing with the sockets for you right from the beginning. Once you’ve set up your SSL struct and called SSL_accept (on the server) or SSL_connect (on the client), all the necessary TLS handshake messages are sent back and forth for you over the socket you provided. Any plaintext data that you subsequently want to send over to the other side simply needs a call to SSL_write. This function will cause the data to be encrypted and sent for you. Whatever data you receive, a call to SSL_read will cause it to be read from the socket, decrypted and provided to you in plaintext (fig. 1).

Fig. 1 OpenSSL takes care of encryption/decryption and sending/receiving over the network for you.

This is all good if you want to use OpenSSL to establish a regular TLS connection between two hosts over TCP/IP sockets, as one probably does most of the time. However, you can still use OpenSSL to perform TLS regardless of how the TLS data is going to be sent between the client and the server.

Possibly the TLS data has to be exchanged over something other than TCP/IP sockets. Or, like me, you may be working with TCP/IP sockets but need to encapsulate the TLS data stream within a different protocol, which means that you can’t let OpenSSL take care of everything for you like above, or you wouldn’t have any control over the TLS data stream that is sent and received.

In such cases, the idea is the following:

Instead of having OpenSSL encrypt-and-send as well as receive-and-decrypt the data for you over a socket, you separate the encryption/decryption step from the sending/receiving step by having OpenSSL write to and read from memory buffers.

OpenSSL will take your application’s data, encrypt it, and put it in a memory buffer, from which your application can take it and do whatever it needs to do in order to send it over to the other host. Messages generated autonomously by OpenSSL, such as the TLS handshake messages, will be written to that memory buffer, too. OpenSSL will also read encrypted data coming from the other host, which your application will have put in another memory buffer, decrypt it and, unless it’s TLS protocol stuff, provide it to your application. So it’s your application’s job to transport the data from a memory buffer to the network and vice versa, after adding or removing whatever headers may be required in your particular situation (fig. 2).

Fig. 2 You can use OpenSSL to encrypt and decrypt TLS data as necessary, taking care yourself of how the data is then sent over the network through custom functions as appropriate.

BIO pairs

OpenSSL achieves the separation between the TLS data stream and the medium and/or protocol it is carried over by using abstract I/O entities called BIO’s, of which there are several kinds serving different purposes, which we’re not going to look into here. See the dreadful OpenSSL documentation if you want to learn more about them.

For our purposes we’re going to be using a pair of memory BIO’s working together as a BIO pair. From the OpenSSL documentation:

A memory BIO is a source/sink BIO which uses memory for its I/O. Data written to a memory BIO is stored in a BUF_MEM structure which is extended as appropriate to accommodate the stored data. […] Any data written to a memory BIO can be recalled by reading from it. Unless the memory BIO is read only any data read from it is deleted from the BIO. [1]


A BIO pair is a pair of source/sink BIOs where data written to either half of the pair is buffered and can be read from the other half. Both halves must usually by handled by the same application thread since no locking is done on the internal data structures. […] One typical use of BIO pairs is to place TLS/SSL I/O under application control, this can be used when the application wishes to use a non standard transport for TLS/SSL or the normal socket routines are inappropriate. [2]

The Steps

These are the necessary steps you’re going to have to implement for a data flow like the one in fig. 2:

  1. Create two memory BIO’s. These are going to be your interface to the OpenSSL engine on the encrypted side of the communication, i.e. the side that contains encrypted data which has to be sent or has just been received over the network (right side of fig. 2).
  2. Create an SSL struct. The SSL struct embodies the TLS session that is going to be established between the two hosts. It is also going to be your interface to the OpenSSL engine on the plaintext side of the communication, which contains the application data that needs to be encrypted and sent, as well as the received data that has been decrypted and needs to be processed by your application (right side of fig. 2)..
  3. Connect the memory BIO’s to the SSL struct. This is the step that builds the essential connection between the plaintext and the encrypted side of the communication: what is written to OpenSSL with one interface, will be able to be retrieved from the other interface after the appropriate processing by OpenSSL.
  4. Set up special_send and special_recv functions to transport the TLS data between the network and the memory BIO’s. Once encrypted data is available for sending in the appropriate memory BIO, it needs to be sent over the network with some special_send function. If encrypted data has been received over the network, special_recv needs to pass it on to OpenSSL for decryption via the appropriate memory BIO. These functions are where the TLS data gets wrapped in or unwrapped from the additional protocol, meaning that any additional, non-TLS headers must be prepended to or removed from the TLS data in these functions.

Let’s have a look at a sketch of each of these steps with some code. I’m only going to provide a basic logical sequence of function calls, since a proper implementation obviously requires you to make your own code structure decisions.

1. Creating the memory BIO’s

Remember that the memory BIO’s represent your application’s interface to the OpenSSL engine on the encrypted side of the communication. Specifically, rbio will be read from to collect the TLS data coming from OpenSSL, whereas wbio will be written to to pass the received TLS data over to OpenSSL. We also set both BIO’s up to return -1 as opposed to EOF in case they’re empty.

#include <openssl/ssl.h>
#include <openssl/err.h>
BIO *rbio = BIO_new(BIO_s_mem()); /* For reading from with BIO_read() */
BIO *wbio = BIO_new(BIO_s_mem()); /* For writing to with BIO_write() */
BIO_set_mem_eof_return(rbio, -1);
BIO_set_mem_eof_return(wbio, -1);

2. Creating the SSL struct

Creating the SSL struct is a matter of a single function call, provided your application has already set up an appropriate SSL_CTX struct containing the TLS parameters. I’m assuming here that you already know how to do this, or that you will find out with some more Googling.

SSL_CTX *ctx = setup_ctx();
if (ctx == NULL) {
SSL *ssl = SSL_new(ctx);
if (ssl == NULL) {

3. Connecting the memory BIO’s to the SSL struct

At this point we link both sides, the “plaintext” and the “encrypted”, by calling SSL_set_bio. As mentioned above, ssl will be the interface to OpenSSL on the plaintext side, whereas wbio and rbio will be the interface on the encrypted side.

SSL_set_bio(ssl, wbio, rbio);
SSL_set_connect_state(ssl); /* If server: SSL_set_accept_state(ssl) */

An important note. Here I’ve defined “write BIO” as the BIO that is written to by BIO_write, “read BIO” accordingly as the BIO that is read from by BIO_read. This is the way I’ve come to look at it while trying to make sense of the whole construction. This, however, causes the order in my function call SSL_set_bio(ssl, wbio, rbio) to differ from the signature of SSL_set_bio, which is SSL_set_bio(SSL *ssl, BIO *rbio, BIO *wbio). I’m sure you can turn my logic around and find a way in which my “read BIO” can be conceived of as a “write BIO”, and vice versa – possibly the “read BIO” is the one a call to SSL_read will read from… I’ll be consistent here and stick to my logic, but bear that in mind when you consult the OpenSSL documentation.

4. Transporting data to and from the BIO’s

Outgoing data. Your application needs to:

  1. Write the application data placed in buffer buf1 to OpenSSL for encryption using SSL_write.
  2. When the corresponding TLS data is available, read it from the read memory BIO rbio into buffer buf2 by calling BIO_read.
  3. From there, in the function special_send add to the TLS data whatever headers you may need to add before sending, then send it over to the other side over the network (here represented by file descriptor fd).
int fd; /* File descriptor for the network, e.g. socket */
int flags; /* Flags controlling the behaviour of special_send */
int numwritten, numread;
int to_write_len, max_read_len, data_len;
unsigned char buf1[BUF_SIZE];
unsigned char buf2[BUF_SIZE];
/* 1 */
numwritten = SSL_write(ssl, buf1, to_write_len);
/* 2 */
if (BIO_ctrl_pending(rbio) > 0) {
	numread = BIO_read(rbio, buf2, max_read_len);
	/* 3 */
	numwritten = special_send(fd, buf2, data_len, flags);

Incoming data. Similarly, your application needs to:

  1. Read the data from the network into buffer buf3 with special_read, stripping it of any additional headers belonging to a different protocol, so that the “clean” TLS data is placed in buf3.
  2. Write the TLS data to the write memory BIO wbio for decryption (and possibly processing by OpenSSL) with BIO_write.
  3. Call SSL_read to read the decrypted data from OpenSSL into buffer buf4, which now contains the decrypted data ready for use by your application.
unsigned char buf3[BUF_SIZE];
unsigned char buf4[BUF_SIZE];
/* 1 */
numread = special_recv(fd, buf3, max_read_len, flags);
/* 2 */
numwritten = BIO_write(wbio, buf3, to_write_len);
/* 3 */
if (BIO_ctrl_pending(wbio) < 0) {
	numread = SSL_read(ssl, buf4, max_read_len);
	/* The decrypted data is now in buf4 */

Note that to_write_len, max_read_len and data_len are mere placeholders for variables actually specifying how many bytes of data are to be read/written by each of the functions used above. What their values need to be in each function call and how they are calculated is up to your implementation to decide. Also, I’m using a dummy signature for special_send and special_recv along the lines of send and recv.

Note the use of BIO_ctrl_pending(BIO *bio) to ensure that data is available for reading.

At this point you’ve seen the logical sequence of function calls that allow you to write plaintext data to OpenSSL and retrieve the corresponding TLS data from a memory buffer, instead of it being sent straight over a TCP/IP socket. And you’ve seen the logical sequence of function calls allowing you to retrieve plaintext data from the encapsulated TLS data you’ve received over the network. It’s up to you now to decide how these logical sequences are to be expanded in your code as well as add all other necessary steps in order for your code to work! :)

Prepending headers to the TLS data

Before I go, let me add a few words on how to prepend additional headers to your TLS data before sending it over the network. The keyword here is flat data structure. More specifically, you’re going to have to build one buffer containing all bytes to be sent, from the very first of your extra header to the very last of the TLS data, all adjacent.

When you call special_send(fd, buf2, data_len, flags), you have data_len bytes of TLS data in buffer buf2 that you need to pass to fd for sending, but before you do that, you need to add a certain header to the TLS data according to the protocol you’re implementing. For simplicity’s sake, assume you have a function unsigned char *create_header() that returns a pointer to a buffer of constant size HEADER_SIZE containing your header. In your special_send function, then, you could do something like this:

int special_send(int fd, unsigned char *buf, size_t data_len, int flags) {
	unsigned char *header = create_header();
	/* Create flat buffer of appropriate size */
	unsigned char *tmp = malloc(HEADER_SIZE + data_len);

	/* Write the header at the beginning of the tmp buffer */
	memcpy((void *)tmp, (void *)header, HEADER_SIZE);

	/* Now write the TLS data right after it */
	memcpy((void *)tmp + HEADER_SIZE, (void *)buf, data_len);
	/* Now do the actual sending */
	int numsent = send(fd, tmp, HEADER_SIZE + data_len, flags);
	return numsent;

Whatever data you need to send, whatever the buffers you take it from, you always do the same: copy the right amount of bytes from whatever buffer to the next location in your flat buffer tmpin the right order. Once you’re done, send it.