The timeout you are receiving is not a "transaction" timeout. It is a general query timeout. This query could be a Select, Update, Delete etc. The timeout value is generally controlled in the connection string.
I don't believe you can specifically lock rows, pages or tables in SQL Server, which is probably a good thing. Frankly, this isn't the crux of the problem. Your transactions are taking way too long to execute. They should never take so long as to cause other queries to timeout (probably at least 30 seconds). You are going to bring your server to its knees with transactions taking this long. A transaction should be kept to a couple seconds, at the very most.
I would suggest you first take a look at the statements requiring the transaction (Update, Delete or Insert), run them through Query Analyzer in SSMS, and make sure no table or index scans are being done. My guess is this is happening, and the tables involved in the transaction are on the large side (hundred of thousands of rows or more).