It sounds like you're being deluged in spam, either incoming (bad) or being relayed through your server (worse). First, take a look at the mail queue with either sudo postqueue -p or Server Admin -> Mail service -> Maintenance (tab at the bottom of the window) -> Mail Queue (tab at the top). If it shows lots of junk mail addressed to other domains, you're being used as a spam relay, and need to find the hole in your relay settings and plug it (and then flush the mail queue). If it shows lots of incoming and/or bounce messages, you need to tighten your defenses against incoming spam. If it's empty... then it's something else and this answer won't help much (other than the logging parts). In any case, the first thing to do is make sure you have enough logging running: in Server Admin -> Mail -> Settings -> Logging, set the SMTP log detail level to either Information or Debug (Info is my preference); the SMTP log (/var/log/mail.log) is the best place to go for details about just what's going on inside postfix's tiny little mind.
Relay lockdown: your server will relay messages if they either come from a trusted IP address (listed in Server Admin -> Mail -> Settings -> Relay -> Accept SMTP relays only from these hosts and networks) or the sender authenticates with a valid username & password. To find out which it is, look in the SMTP log for a connection & incoming message; it should look something like this (only with a timestamp and servername in front of each line):
postfix/smtpd[17145]: connect from unknown[10.0.0.201]
postfix/smtpd[17145]: 069E852BC63: client=unknown[10.0.0.201], sasl_method=CRAM-MD5, sasl_username=sucker
postfix/cleanup[17153]: 069E852BC63: message-id=<C45A4B44-A531-4977-AAB3-DAE90CBA5CF1@spamguys.com>
postfix/qmgr[72]: 069E852BC63: from=<spammer@spamguys.com>, size=63282, nrcpt=1 (queue active)
Note that these lines may be spread out, as there are other things going on and other messages mixed into the log. What I do is look for a qmgr message with a suspicious from=<>, and then search for the message number (069E852BC63 in this example) to find the smtpd line, then if necessary use the smtpd process ID (17145 above) to find the initial connection. Once you've found the relevant log lines, the things to look for are the source IP address (10.0.0.201 above), and see if it's trusted; if so, track down the computer/device at that address and find out why it's sending spam. The other thing to check is the smtpd line that includes "sasl_method=CRAM-MD5, sasl_username=sucker" -- that means that the client authenticated as a user named "sucker", and you need to change that user's password immediately (and then try to find out how the password was leaked/guessed).
Incoming spam lockdown: on a 10.4 server, you have two lines of spam defense on the server: blacklists and filtering. You should do as much as possible with the blacklist, because it blocks messages before your server even receives them, thus cutting load on your server (it doesn't have to run the spam and virus filters, handle bounce messages, etc). You can subscribe to a DNS blacklist under Server Admin -> Mail -> Settings -> Relay -> Use these junk mail rejection servers (real-time blacklist). I haven't done any good surveys in a few years, but zen.spamhaus.org has a very good reputation, and I always got good results from bl.spamcop.net back when I was running a more significant server. With a blacklist or two in place, the load should be cut down to the point where qmgr and the virus filter can cope (with a caveat I'll get to in a minute). If you want to get a bit fancier, topicdesk.com has a PDF with some additional tricks.
Finally, that caveat about ClamAV: the version of ClamAV included in 10.4 server is no longer supported by current virus definitions; again, topicdesk.com has a PDF with instructions to update the ClamAV engine.
