I've set the setgid-bit on folders within my samba-share. It works like a charm for windows and linux clients.
Accessing such a folder with a mac-client the setgid is ignored and the group permission of the newly created files/directories is set to the current users default group.
I've tried the the setting unix extensions = no mentioned on this post but this doesn't work.
One thing you can do to force the created files in a share to always belong to a certain group is to use the force group = groupname option in the share configuration in smb.conf. If you want to make sure that only users who already belong to that group can create files use a + sign in front of the group name: force group = +groupname.
More info on the samba manual for smb.conf
