One crazy method would be to analyze each contour separately. Since the contours are rotated at different angles, we have to orient them properly with any common followable rule:
Say, the widest part horizontally.
This can be done using minAreaRect function which bounds the contour with a rotated rectangle. You can find the slope of one of the longest sides and then do a rotation transform to orient according to our rule.
Now you have a collection of images of contours following our rule ,ie. widest part lies horizontally.
To compare the similarity of two contour images say A and B,
compute,
differences1=(A-B)
differences2=(B-A)
differences=differences1 + differences2
or you may do it in one line. This works because OpenCV changes all negative values to 0 and all values greater than 255 will be 255.
Then find the sum of each pixels the image named differences. The smaller the value, the similar the contours are. Or you have use any more efficient parameters to compare the images.
PS: You said using area as a feature is not giving acceptable results. Did you try using both Area and Perimeter simultaneously ? If this is doing good, forget about the previous block of text.
Good Luck!