0

I’m working with multidocument mongodb transactions, and I’m getting an unexpected error.

In my c++ app, using mongocxx driver, the steps I’m performing are the following:

  1. create session
  2. start transaction
  3. create a bulk_write 1 for this session, where I have multiple updates (~1000) to colletion 1
  4. create another bulk_write 2 for this session, where I have multiple inserts (~1000) to collection 2
  5. execute bulk_write 1
  6. execute bulk write 2
  7. commit transaction

This is the code snippet for the algorithm:

#include <iostream>
#include <vector>

#include <bsoncxx/builder/stream/document.hpp>
#include <bsoncxx/json.hpp>
#include <bsoncxx/exception/exception.hpp>
#include <mongocxx/client.hpp>
#include <mongocxx/instance.hpp>
#include <mongocxx/exception/exception.hpp>
#include <mongocxx/exception/logic_error.hpp>
#include <mongocxx/exception/operation_exception.hpp>



int main(int argc, char** argv)
{
    /* Parameters */

    std::string db_uri = "<PROVIDE URI TO CONNECT WITH A MONGO DB WITH REPLICA SETS>";
    std::string db_name = "db_0";
    std::string collection0_name = "coll_0";
    std::string collection1_name = "coll_1";
    int N_UPDATES = 100000;
    int N_INSERTS = 100000;

    /* Init connection */

    static mongocxx::instance inst{};
    mongocxx::uri client_uri = mongocxx::uri(db_uri);
    mongocxx::options::client client_options;
    mongocxx::options::ssl ssl_options;
    ssl_options.allow_invalid_certificates(true);
    client_options.ssl_opts(ssl_options);
    mongocxx::client client = mongocxx::client(client_uri, client_options);

    /* Reinit collections */

    mongocxx::database db = client[db_name];        
    auto builder = bsoncxx::builder::stream::document{};
    bsoncxx::document::value doc_value = builder
    << "Hello" << "MongoDB"
    << bsoncxx::builder::stream::finalize;    
    db[collection0_name].insert_one(doc_value.view()); /* insert a dummy doc */    
    db[collection0_name].delete_many({}); /* delete all docs */
    db[collection1_name].insert_one(doc_value.view()); /* insert a dummy doc */    
    db[collection1_name].delete_many({}); /* delete all docs */

    /* Create session */

    mongocxx::client_session session = client.start_session();

    /* Start transaction */

    session.start_transaction();

    /* Create bulk operations */

    mongocxx::bulk_write op0 = db[collection0_name].create_bulk_write(session);
    mongocxx::bulk_write op1 = db[collection1_name].create_bulk_write(session);
    std::vector<mongocxx::bulk_write*> bulk_operations;
    bulk_operations.push_back(&op0);
    bulk_operations.push_back(&op1);

    /* Fill update bulk operations */

    for (int i = 0; i < N_UPDATES; i++){

        mongocxx::model::update_one upsert_one{
            bsoncxx::builder::basic::make_document(
                bsoncxx::builder::basic::kvp("field0", i),
                bsoncxx::builder::basic::kvp("field1", i)
            ),
            bsoncxx::builder::basic::make_document(
                bsoncxx::builder::basic::kvp("$set", bsoncxx::builder::basic::make_document(
                    bsoncxx::builder::basic::kvp("field0", i),
                    bsoncxx::builder::basic::kvp("field1", i),
                    bsoncxx::builder::basic::kvp("field2", i),
                    bsoncxx::builder::basic::kvp("field3", i))
                )
            )
        };

        upsert_one.upsert(true); // Set upsert to true: if no document matches {"a": 1}, insert {"a": 2}.
        op0.append(upsert_one);
    }

    /* Fill insert bulk operations */

    for (int i = 0; i < N_INSERTS; i++){

        mongocxx::model::insert_one insert_one{
            bsoncxx::builder::basic::make_document(
                bsoncxx::builder::basic::kvp("field0", i),
                bsoncxx::builder::basic::kvp("field1", i),
                bsoncxx::builder::basic::kvp("field2", i)
            )
        };

        op1.append(insert_one);
    }

    /* Execute transaction */

    for( auto bulk_op : bulk_operations){
        try {
            bulk_op->execute();
        }
        catch (std::exception& e){     
            std::cerr << "Bulk write exception: " << e.what() << std::endl;
            session.abort_transaction();
        }
    }

    session.commit_transaction();   

    return 0;
}

which you can compile using the following command in a linux system with mongocxx installed:

c++ --std=c++11 test.cpp -o test -I/usr/local/include/mongocxx/v_noabi -I/usr/local/include/bsoncxx/v_noabi -L/usr/local/lib -lmongocxx -lbsoncxx

While executing I get the following error:

Bulk write exception: Exec error resulting in state DEAD :: caused by :: operation was interrupted: generic server error
terminate called after throwing an instance of 'mongocxx::v_noabi::operation_exception'
  what():  No transaction started: generic server error
Aborted (core dumped)

Also, I got different errors changing the parameters N_INSERTS and N_UPDATES:

https://docs.google.com/spreadsheets/d/1xZZs5Vb8FCXpvjL2o2fEyujWrDzCTAWCvF49IerZqHw/edit?usp=sharing

Thanks!

Federico Caccia
  • 1,817
  • 1
  • 13
  • 33
  • 2
    Have you tried starting your transaction _before_ creating your bulk writes? You may be running into an error with your order of operations. I would rule this out first before continuing with further debugging efforts. – B. Fleming Aug 06 '19 at 22:38
  • @B.Fleming I tried what you suggested. I get the same error. – Federico Caccia Aug 07 '19 at 00:46
  • @B.Fleming Anyway, do you think it's a bad design creating the bulk operations before starting the transaction? In my code, I have to execute several steps to create the bulk writes, so I have separated the algorithm in two functions: one to create the bulk write for a given session, and the other, that receives the session and the bulk operations, and start the transaction, and lso execute and commit. I think its better organized this way, but let me know if you think there is a reson to start the transaction before. – Federico Caccia Aug 07 '19 at 00:54
  • 1
    Can you please show your code (or an approximation)? There may be something missing that we would be able to identify given a proper code snippet. Regarding design, that's a question best suited for [CodeReview](https://codereview.stackexchange.com/). That being said, I find that it's better to keep directly-related operations closer together as it's easier to make any changes that need to be synchronized across those lines of code. The way you've architected your code may be fine, however, depending on the details. It's hard to make any judgment call on that without seeing any actual code. – B. Fleming Aug 07 '19 at 01:35
  • @B.Fleming I will update my question with a code snippet, thanks – Federico Caccia Aug 07 '19 at 13:27
  • 1
    I'm not familiar with the C++ driver, so unfortunately I won't be able to do too much to help, but your code doesn't look like it should be wrong at first glance. I would encourage you to reduce `N_UPDATES` and `N_INSERTS` to a small value like `10` for initial testing, and to limit yourself to just one of the bulk operations at the start. Begin with minimal functionality to get something that works, and see if you can determine what works and what doesn't. Those steps will help narrow down the point of failure. – B. Fleming Aug 07 '19 at 18:44
  • Thanks for your advice. I have tested changing the amount of inserts and updates. Results on this link: https://docs.google.com/spreadsheets/d/1xZZs5Vb8FCXpvjL2o2fEyujWrDzCTAWCvF49IerZqHw/edit?usp=sharing – Federico Caccia Aug 07 '19 at 21:47
  • 1
    You're still working on tens of thousands of documents. I said to work with a small value like `10`--not `10000`--because you need to know if the issue is with your code or if the issue is with the scale of the problem your code is trying to solve. The more complicated you make the test problem, the more difficult it's going to be to isolate the issues that arise. Start small, then build up gradually. – B. Fleming Aug 07 '19 at 22:18
  • Sure, working with simple cases like you mention, there is no error. Please check at row 2 for example, up to 20k docs both in updates and inserts I found no error. – Federico Caccia Aug 08 '19 at 00:37

0 Answers0