you can call an external executable from your ant file - although that means that it will need to have access to that executable. This may do the trick:
<target name="update-htpasswd">
<exec executable="/usr/bin/htpasswd">
<arg value="-bm"/>
<arg value="${passwordfile}"/>
<arg value="${username}"/>
<arg value="${plain-password}"/>
</exec>
</target>
Guaranteed correct format. Otherwise, you can write your own ant task and call it. Java has built-in MD5 libraries, so you would only have to generate the correct password line in java and append it to the password file. That would be more portable.