I'm loading up Windows Server Backup on a test server to confirm, but the following post should help:
http://social.technet.microsoft.com/Forums/en-US/windowsbackup/thread/d5302218-91ce-4f02-9fba-9c91e9027f69
For Volume Shadow copy based backups,
there is no direct way to set the
maximum size for backup with wbadmin.
Since Windows Server Backup relies on
shadow copies for versioning, you can
use the MAX DIFF AREA Size to control
how much space is allocated for
storing versions.
Moreover, the old backup version will
keep growing until all free space is
used up, Windows Server Backup will
automatically delete the old version
backup to make space for newer ones.
You would adjust this threshold here: Volume ->settings -> use limit -> XXX MB
Or, you may also use "vssadmin resize" command line to increase the amount of storage for shadow copies limit.
vssadmin resize shadowstorage /for=<ForVolumeSpec> /on=<OnVolumeSpec> /maxsize=<MaxSizeSpec>
ex. Vssadmin resize shadowstorage /for=F: /on=C: /Maxsize=900MB