Annotate on reverse many-to-many

Question

I'm trying to work out why this doesn't work:-

class A(models.Model):
    contacts = models.ManyToManyField(Contact)

class Contact(models.Model):
    name = models.CharField()

If I try and get a count of how many A there are with multiple contacts:-

A.objects.annotate(num_contacts=Count('contacts')).filter(num_contacts__gt=1).count()

there are 10.

but if I have a particular contact and I want to get a count of how many A's they are connected to that have more than 1 contact on them:-

B.a_set.annotate(num_contacts=Count('contacts')).filter(num_contacts__gt=1).count()

I get 0. The num_contacts count always comes out as 1, even when the A has more than 1 contact.

I must have missed something silly but I can't see it. Any ideas?