0

I'm testing Sqlite C Api for multiprocess concurrency and I'm hitting a bug that I don't understand.

The test is simple: 1 database, 1 table of 1000 rows with 2 columns. Column 1 are uuids, column 2 is a status set to 0. I'm launching then 2 workers that concurrently select 2 uuids at status 0 and update them to 1 until no uuids are left to be updated.

Here the code in the main loop of the worker:

std::vector<std::string> uuids;
res = sqlite3_exec(database,
    "BEGIN IMMEDIATE TRANSACTION;", nullptr, nullptr, nullptr);
if (res != SQLITE_OK)
{
    Log("Worker: Failed to begin transaction, wait for other worker to finish");
    std::this_thread::sleep_for(std::chrono::milliseconds(5));
    continue;
}
Log("Worker: Begin Transaction");

// SELECT UUIDS
std::string selectString = "SELECT " + m_uuidStr + " FROM " + m_tableStr + " WHERE " + m_statusStr + "=" + std::to_string(0) + " LIMIT " + std::to_string(2);
// Log("Worker: SelectString = " + selectString);
auto selectStatement = selectString.c_str();

res = sqlite3_exec(database,
    selectStatement,
    fill_uuids,
    &uuids, nullptr);
if (res != SQLITE_OK)
{
    Log("Worker: Failed to select uuids");
    return 3;
}

if (uuids.empty())
{
    sqlite3_exec(database, "ROLLBACK;", nullptr, nullptr, nullptr);
    Log("Worker: nothing more to update");
    break;
}

// UPDATE UUIDS
std::stringstream ss;
ss << '\"' << uuids[0] << '\"';
for (int i = 1; i < uuids.size(); ++i)
    ss << ", \"" << uuids[i] << '\"';
Log("Worker: UUIDs selected = " + ss.str());

std::string updateString = "UPDATE " + m_tableStr
    + " SET " + m_statusStr + "=" + std::to_string(1)
    + " WHERE (" + m_uuidStr + " in (" + ss.str() + ")) AND (" + m_statusStr + "=" + std::to_string(0) + ");";
auto updateStatement = updateString.c_str();

// BIG CRASH HERE WITH IMMEDIATE TRANSACTION ///
res = sqlite3_exec(database,
    updateStatement,
    nullptr, nullptr, nullptr);

if (res != SQLITE_OK)
{
    Log("Worker: Failed to update uuids, err = " + std::string(err));
    return 4;
}

// COMMIT CHANGES
res = sqlite3_exec(database,
    "COMMIT;", nullptr, nullptr, nullptr);
if (res != SQLITE_OK)
{
    Log("Worker: Failed to commit, err = " + std::string(err));
    return 5;
}

Log("Worker: Transaction ended");

When using immediate transactions, one of the worker crashes with a negative error code (-1073741819 but I doubt it helps) when executing the UPDATE statement. When using exclusive transactions, I don't have any crash. The version with immediate transactions works with a unique worker. The database is open using the flags SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX, Sqlite is compiled with option SQLITE_THREADSAFE=1.

There are no other process accessing the database while the workers are running. In that respect I don't understand why immediate transactions causes a crash and exclusive transactions do not, as the behavior should be similar (since no thread/process is creating reading statement during the worker execution).

I'm very new to Sqlite so maybe there's an obvious mistake lying in my code. Any help is appreciated.

Cyril
  • 559
  • 5
  • 17
  • Are you doing _"...When any process wants to write, it must lock the entire database file for the duration of its update...."_ ? source: https://www.sqlite.org/faq.html ie SQLite is not a true multi-process database. – Richard Critten Apr 24 '20 at 15:50
  • Note: `SQLITE_THREADSAFE=1` does *not* mean "multi process safe". This seems relevant: https://stackoverflow.com/q/1063438/5910058 as does this: https://sqlite.org/faq.html#q5 – Jesper Juhl Apr 24 '20 at 17:14

0 Answers0