Setting the transaction timeout is done where the transaction is begun. This is in your code written using the Moqui tools such as a service or screen, or Java/Groovy/etc code that uses the Moqui TransactionFacade or the JTA interfaces directly.
By default Moqui screens are not run in transactions, unless you set the screen.@begin-transaction attribute to do so. Chances are your problem is in a long-running service, and by default Moqui services ARE run in transactions. Set the timeout using the service.@transaction-timeout attribute on the service that beings the transaction. By default services use a transaction already in place if there is one, so this needs to be on the outer most service where the transaction is actual begun.
For more details about services and transaction management see the Making Apps with Moqui book, available for download from moqui.org.
You may have another issue in your code and that is the socket timing out for the browser request (I see that as well in the log in your screenshot). There are some ways around this, but also some things you can't control so easily like when the browser times out. For a good UI it is also best not have your user sit and wait more than the typical 30-60s where such timeouts start hitting. Change your code to run in the background and if needed add something to your screen to monitor status and/or progress of the job.