To deselect an area, as if the end user had clicked it again, there are several ways:
Deselect by key. changing "false" to "true" below would select instead
$('img').mapster('set',false,'key');
Deselect using the area itself:
$('area[mapkey=key]').mapster('set',false);
Also using the area. There's a complementary "select" method as well. The "deselect" and "select" methods only work on areas.
$('area[mapkey=key]').mapster('deselect');
Example: http://jsfiddle.net/jamietre/MZ9aH/
Docs: http://www.outsharked.com/imagemapster/default.aspx?docs.html#select
The code you're using sets initial options - they won't change the current state state of an area. So while the "selected: false" option would make it initially deselected if the map hadn't been bound yet, it won't do anything after the map has been created. The "highlight: false" option determines whether or not an area will be highlighted on mouseover.