Secure Programming Cookbook for C and C++

5.20. Performing Additional Cipher Setup in OpenSSL

Problem

Using OpenSSL, you want to adjust a configurable parameter of a cipher other than the key length.

Solution

OpenSSL provides an obtuse, ioctl()-style API for setting uncommon cipher parameters on a context object:

int EVP_CIPHER_CTX_ctrl(EVP_CIPHER_CTX *ctx, int type, int arg, void *ptr);

Discussion

OpenSSL doesn’t provide much flexibility in adjusting cipher characteristics. For example, the three AES configurations are three specific instantiations of a cipher called Rijndael, which has nine different configurations. However, OpenSSL supports only the three standard ones.

Nevertheless, there are two cases in which OpenSSL does allow for configurability. In the first case, it allows for setting the “effective key bits” in RC2. As a result, the RC2 key is crippled so that it is only as strong as the effective size set. We feel that this functionality is completely useless.

In the second case, OpenSSL allows you to set the number of rounds used internally by the RC5 algorithm. By default, RC5 uses 12 rounds. And while the algorithm should take absolutely variable-length rounds, OpenSSL allows you to set the number only to 8, 12, or 16.

The function EVP_CIPHER_CTX_ctrl( ) can be used to set or query either of these values, given a cipher of the appropriate type. This function has the following arguments:

ctx

Pointer to the cipher context to be modified.

type

Value indicating which operation to perform (more on this a little later). ...