- Disable the CICS transaction before starting the batch job, re-enable it when the batch job ends.
- Modify the batch job to use commit intervals, similar to this answer.
Checking to see if the CICS transaction is active is unlikely to behave as you wish. It may be inactive when you check, then you start your batch job, then the CICS transaction becomes active.
Update #1
Though you don't specify, I'm getting the impression this is a long-running CICS transaction and not the normal OLTP-style transaction that finishes in less than 0.10 seconds of clock time.
If this is the case, then creating a batch program that uses the EXCI to execute a CICS program that uses the CICS SPI INQUIRE TASKLIST to locate your transaction may be the way to proceed. If you've got CA-DADs PLUS then you might be able to do this with that product instead of writing programs.