There's a few ways I usually do hit testing.
1) The easiest way is to use a an already made class that you can find online. Some people much smarter than me have created complex classes that allow for much better pixel to pixel interaction. The ones listed by Paras are all good. The problem with these is, for newer users, it can be hard to understand all the code and how to implement them. Usually it is simple once you understand what is going on though. You just replace your hit test with the class file and then enter in the correct arguments.
2) Another method is to actually go into the symbol, create a new layer, and then draw a rectangle(just turn the alpha down to 0%) where you want the hit test to work. This may seem like a stupid method, after all we are just confined to a square once again. BUT, it will actually work MUCH better than you'd expect. Just draw the square maybe slightly smaller than the height and width of your character you're detecting the hit test on, and you should be good to go. Give it an instance name (the hit square that is) and then just perform the hitTest with that square instead of the actual sprite. It works wonderfully and is a very simple solution. For what you're explaining though, this sounds like it might not work. This method is more from a gamer standpoint. It looks good when attacking and getting hit by enemies, but isn't necessarily exact. Also, if you want to do this with two characters (maybe a large attack hitting an enemy) simply draw a hit box for both sprites. This is probably a little more basic than using a pre-made pixel perfect hit detection test, but it works extremely well and takes only a few minutes.