See this part here:
MessageDigest algorithm = null;
try{
algorithm = MessageDigest.getInstance("MD5");
}catch(NoSuchAlgorithmException e){}
? That's where that stuff is accessing the MD5 code that's built into the Java runtime. You'll have to come up with your own implementation of MD5 there, which (to put it mildly) will be the tricky part.
All that the posted Java code really does (on top of calling the runtime to do the actual hashing) is to map the resulting hash (part of it, anyway) through a character lookup table.
edit — the lookup table built by that Java code is an array with "$", the digits, the upper-case letters, the lower-case letters, and then (surprisingly) "£". (The last character is surprising because it's not an old-school 7-bit ASCII character code, but whatever.) In JavaScript, that's:
var codes = "$0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz£";
The Java code then takes each 8-bit byte produced by the hash algorithm and looks up a code character by adding 128 to the byte and then dividing by 4. Java bytes are treated as signed values, so that has the effect of mapping every byte into the range 0 ... 63. That value is then used as a lookup into the code array.
Thus if you have a JavaScript MD5 facility that can give you back an array of numbers in the range -128 ... 127 (that is, signed 8-bit values), you could translate the result through the code array like this:
var digest = MagicJavaScriptMD5(source);
var result = [];
for (var i = 0; i < digest.length; ++i)
result.push(codes.charAt(~~((digest[i] + 128) / 4)));
var resultString = result.join('');
EDIT by the OP:
I take the liberty of posting here the right solution, that is highly derived from @Pointy's one. It requires md5.js from http://pajhome.org.uk/crypt/md5/.
/* MD5 in byte[] format */
function byteArray_md5(s) {
var output = [];
var input = rstr_md5(str2rstr_utf8(s)); //here it uses md5.js
for(var i = 0; i < input.length; i++)
output[i] = input.charCodeAt(i);
return output;
}
/* MD5 with custom mapping.
* It's a normal MD5 with a final char mapping of the hash.
*/
function md5WithCustomMapping(source) {
var codes = "$0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz£";
var digest = byteArray_md5(source);
var result = [];
for (var i = 0; i < digest.length; ++i)
result.push(
codes.charAt(
~~( ( digest[i] + 128 * (digest[i]<128 ? 1 : -1) )/4 )
)
);
return result.join('');
}