OpenSSL error 0200B009 Bad file descriptor

Question

When I run the OpenSSL CMS encrypt and decrypt demo I receive the following error:

Error Decrypting Data
2900476676:error:0200B009:system library:fread:Bad file descriptor:bss_file.c:245:
2900476676:error:20082002:BIO routines:FILE_READ:system lib:bss_file.c:246:
2900476676:error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length:evp_enc.c:460:

It happens in the CMS_decrypt() method.

Does anyone know what's wrong?

Update #1:

I'm using the library in objective-c (and have also tried it in C++). It happens in this section:

    int error = CMS_decrypt(cms, rkey, rcert, /*out*/ bout, NULL, 0);
    if (!error) {
        fprintf(stderr, "Error Decrypting Data\n");
        ERR_print_errors_fp(stderr);
        printf("error code: %d\n", ERR_get_error());
        assert(false);
    }

Update #2:

Added full decrypt source.

- (void) decryptOrig {
    BIO *in = NULL, *out = NULL, *tbio = NULL;
    X509 *rcert = NULL;
    EVP_PKEY *rkey = NULL;
    CMS_ContentInfo *cms = NULL;
    int ret = 1;

    NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString *documentsDirectory = [paths objectAtIndex:0];


    NSString *iosPathToFile = [NSString stringWithFormat:@"%@/encrypted.enc", documentsDirectory]; //[[NSBundle mainBundle] pathForResource:@"encrypted" ofType:@"enc"];
    NSString *iosPathToCertificate = [[NSBundle mainBundle] pathForResource:@"signer" ofType:@"pem"];
    NSString *iosPathToKey = [[NSBundle mainBundle] pathForResource:@"christof" ofType:@"key"];


    NSString *iosPathToOrigFinal = [NSString stringWithFormat:@"%@/original.txt", documentsDirectory];




    OpenSSL_add_all_algorithms();
    ERR_load_crypto_strings();

    /* Read in recipient certificate and private key */
    tbio = BIO_new_file([iosPathToCertificate cStringUsingEncoding:NSUTF8StringEncoding], "r");



    if (!tbio)
        goto err;

    rcert = PEM_read_bio_X509(tbio, NULL, 0, NULL);

    /*BIO *output = BIO_new(BIO_s_mem());
    X509_print(output, rcert);
    char *temp = malloc(50000);
    BIO_read(output, temp, 50000);

    printf("cert: %s", temp);*/

    //temp for output
    BIO *bout = BIO_new_fp (stdout, BIO_NOCLOSE);


    BIO_reset(tbio);

    rkey = PEM_read_bio_PrivateKey(tbio, NULL, 0, NULL);

    //EVP_PKEY_print_private(bout, rkey, 0, NULL);

    if (!rcert || !rkey)
        goto err;

    /* Open S/MIME message to decrypt */

    in = BIO_new_file([iosPathToFile cStringUsingEncoding:NSUTF8StringEncoding], "r");



    if (!in)
        goto err;

    /* Parse message */
    cms = SMIME_read_CMS(in, NULL);

    //CMS_ContentInfo_print_ctx(bout, cms, 0, NULL);


    if (!cms)
        goto err;

    out = BIO_new_file([iosPathToOrigFinal cStringUsingEncoding:NSUTF8StringEncoding], "w");
    NSLog(iosPathToOrigFinal);
    /*char *mytestoutput = malloc(50000);
    memset(mytestoutput, 0, 50000);
    out = BIO_new_mem_buf(mytestoutput, 50000);*/

    if (!out)
        assert(false);

    /* Decrypt S/MIME message */
    int error = CMS_decrypt(cms, rkey, rcert, out, NULL, 0);
    if (!error) {
        fprintf(stderr, "Error Decrypting Data\n");
        ERR_print_errors_fp(stderr);
        printf("error code: %d\n", ERR_get_error());
        assert(false);
    }


    ret = 0;

err:

    if (ret)
    {
        fprintf(stderr, "Error Decrypting Data\n");
        ERR_print_errors_fp(stderr);
    }

    if (cms)
        CMS_ContentInfo_free(cms);
    if (rcert)
        X509_free(rcert);
    if (rkey)
        EVP_PKEY_free(rkey);

    if (in)
        BIO_free(in);
    if (out)
        BIO_free(out);
    if (tbio)
        BIO_free(tbio);

    return ret;

}

I've removed the bout and used the out in the encrypt method

Update #3:

Is it possible that there's a problem with the symmetric encryption type? CBC etc...?

@sarnold: Oh sorry, I forgot to mention that I'm using the library in C++ (and objective-c). I'm going to update this. — Chris, Feb 28 '12 at 09:19
Incidentally, can you add the code that initializes the parameters? I was surprised to learn [`CMS_decrypt()`](http://www.openssl.org/docs/crypto/CMS_decrypt.html) returns `1` on success, `0` on failure; `if (!error)` is correct but awkward. I think you'd be happier if that variable were named `success`, instead. — sarnold, Feb 28 '12 at 09:25
@sarnold: I've corrected the typo and added the source. The error must be in the decrypt but when I encrypt a file/msg using the terminal it happens the exactly same error (the aforementioned) as when I use my self implemented encrypt method. — Chris, Feb 28 '12 at 09:33
@sarnold: btw. yep you're right. `error` is not really the best name for this but it's just for debug purposes. (I had the method call in the `if`) — Chris, Feb 28 '12 at 09:35
I wonder if `BIO *bout = BIO_new_fp (stdout, BIO_NOCLOSE);` succeeded or failed? What would `stdout` be on an iOS application? — sarnold, Feb 28 '12 at 09:41
@sarnold: That comes from the c++ project. Normally I use the `out` BIO there. You're right, stout doesn't work there but in the iOS simulator it works and prints the output directly on the debug console. But also with the `out` BIO I get the same error. — Chris, Feb 28 '12 at 10:00

oknox · Answer 1 · 2012-10-04T12:36:34.433

Part of the problem may be that the method call you make for decryption is slightly troublesome - which is a tricky one.

You have your output file where it expects to find an input contents file - this also leaves it with a null bio where it expects to put the plain text and so it kicks back this error.

here's how it should look i think

int error = CMS_decrypt(cms, rkey, rcert, NULL, out, 0);

not

int error = CMS_decrypt(cms, rkey, rcert, out, NULL, 0); // won't decrypt

i think what happened was actually that the method signature itself changed in the library at some point, but this old method signature is still in some older demo code of the cms capability. I assume they changed it to better fit their conventions.

good luck

OpenSSL error 0200B009 Bad file descriptor

1 Answers1