Amazon QLDB increase document transaction limit or counter it

Question

I created an application that is in production now and it has a function that does a lot of calculations and stores the data in QLDB. This function was working fine but our user increased and now whenever we call this function it says document transaction limit: More than 40 documents were modified. Apparently, I didn't know about this limit and I created the function without keeping that in mind and now I am stuck because that function is handling a lot of stuff, and it's in production.

So I want to know if there is a way to increase this limit or how can I counter it.

score 1 · Answer 1 · answered Apr 06 '22 at 15:11

You can't currently achieve what you want.

I don't know what your application looks like, but there's probably something intrinsically wrong with the design, if you're manipulating more than 40 documents at once in a single transaction.

Workaround

For a fast workaround, you'd need, at least, to change your application to do a 40 sized batch inside an atomic transaction. You can use threads for that, but beware of the maxConcurrentTransactions limit and also the OccConflictException that may occur in some scenarios. You can learn more about that here.

Best approach

To really solve your problem, all transactions should be atomic. Meaning you should open your driver and perform only 1 operation of (INSERT, DELETE, UPDATE) along with eventual SELECTs.

Example in Java:

private final QldbDriver qldbDriver;

public void updateSomething() {
    try {
        qldbDriver.execute(txn -> { 
            String select = "SELECT id FROM YourTable WHERE something = 1 AND createdAt = `1649257121113`";
            final Result selectResult = txn.execute(select);
    
            if (selectResult.isEmpty()) {
                throw new NotFoundException("Something not found");
            }
           
            final Integer id = // convert your result into the id
    
            String update = "UPDATE YourTable set something = 2 where id = " + id;
            final Result result = txn.execute(query);
        });
    } catch (final OccConflictException ex) {
        throw new Exception("OCC exception updating!");
    }
}

Short take on QLDB

Always keep in mind that this is not a relational database. All sorting and calculations should be done at the application side. The benefit of working with this database relies mainly on its history (journal) and its exports.

Yeah eventually we leaned towards this solution of opening one driver to do single transaction and to proof check this i first create all the queries, data and everything by category and store them inside another database. e.g in our case its firestore. Then when all data is created with no errors then i start processing it using a function and that function take a single operation checks for the query and data then moves to another operation. Anyway thank you for this informative answer :d — Sulman Azhar, Apr 08 '22 at 09:59

score 0 · Answer 2 · answered Feb 05 '22 at 06:50

0

From Quotas and limits in Amazon QLDB - Amazon Quantum Ledger Database (Amazon QLDB):

In addition to default quotas, QLDB has the following fixed quotas per ledger. These quotas cannot be increased by using Service Quotas:

Number of documents in a transaction: 40

I'm not a QLDB user so I don't understand what that means, but perhaps you can modify your transaction to use fewer documents?

answered Feb 05 '22 at 06:50

John Rotenstein

241,921
22
380
470

Like I said if this cannot be increased then I want a solution that counters it. Because my function is in production and it needs to be ASAP if I have to refactor it with new logic then how will I be able to do that because that function will require a lot QLDB writes – Sulman Azhar Feb 05 '22 at 06:53