One of the downsides with qmail is that as default it doesn't check the localpart of the recipient address before it accepts the message. All it does is check that the recipient domain is in its list of allowed recipient domains. (This was a not unreasonable design choice in the early 1990's. It is less so nowadays.) This means that e.g. spammers doing a dictionary spray can fill your server up by throwing a gazillion of mails to non-existent recipients. When qmail tries to deliver the messages, it will find that the localpart doesn't exist and it will generate a bounce message. And since spammers will forge the envelope sender, your server won't be able to deliver those messages, and it will fill the queue up.
The best solution is to not accept those emails in the first place. If your server can reject emails immediately on RCPT TO:nonexistent@example.com, then it will be the sending mailserver's job to issue the bounce and you won't get flooded with bounces.
Another solution would be to reduce the amount of time that qmail will keep trying to deliver bounce messages, so that they will be deleted earlier and thus also not fill up the queue.
And, of course, you could filter incoming spam and discard it instead of allowing it to generate bounces.
Rejecting unknown localparts
The only way to achieve this is to patch qmail, so that qmail-smtpd will check localparts. You can't do this just with configuration options, you need to actually patch and recompile and install the patched qmail-smtpd.
There are a few different patches that can achieve this. I can't recommend any one of them over the other - back when I was managing qmail servers we had another system acting as a proxy which did that filtering, so I never needed to use this. But you can find several patches at netdevice.com/wmail/rcptck and code.dogmap.org/qmail/.
Reducing amount of time qmail tries to deliver bounces
If this can be done for bounces only, I don't know how. But you can do it for all mails, by changing the value of queuelifetime. man qmail-control should tell you how to do it.
Spam filtering
There is way too much to say about spam filtering to fit into one single answer here. I'd start with considering RBL-style blackhole lists and reject connections from IP addresses listed in those. Of course, you need to choose your blackhole lists carefully, and you do risk blocking "real" mail. There's some info about spam filtering specifically for qmail at Chris Hardie's qmail anti-spam howto, and there's info about various blackhole lists at spamhaus.org.
I hope this is enough for a start. I've used qmail for a couple of decades now, including running the mailsystem for an ISP with several million users - but I also have to add that if I were to set such a system up now, qmail would no longer be my first choice. It was a very good mail server when it was created, and it did a lot of things really well, but nowadays I'd probably go with postfix instead. Time and the spammers have moved on, and qmail hasn't.
