1

I want to do feature selection between 512 feature maps (3X3 each) from convolutional layers of a neural network. I want to calculate a 512X512 Mutual Information matrix between every two vectors and choose 256 feature maps with the lowest Mutual Information values (excluding rows/columns with all zeros).

For simplicity, let's say I have the following tensors (you don't need pytorch, you can just use numpy):

t2 = torch.Tensor([0.25, 2.1.5. 3.2, 3.1, 2.5])
t3 = torch.Tensor([0.14, 0.9, 6.2 ,3.1  ,1.7])
v2 = t2.data.numpy()
v3 = t3.data.numpy()

I used code I found to calculate the Mutual Information between two vectors (I just changed xrange to range): Python's implementation of Mutual Information

mi_compmi = computeMI(v2,v3)
print('Result: ', mi_compmi)

Things is, I always get the same values of px,py,pxy and same result of 2.32 for all the vectors I assign in v2 and v3, why is that? I get that even when I calculate the MI between the vector and itself, while it should be 0.

Niminim
  • 668
  • 1
  • 7
  • 16

1 Answers1

0

You should use sklearn's mutual_info_score , see here. Seems to work fine and should resolve your issue.

BTW. You should try to go for well-tested functionalities written in libraries instead of relying on random implementation, unless you are forced to do so.

Szymon Maszke
  • 22,747
  • 4
  • 43
  • 83
  • 1
    I tried that before I looked for another implementation, that was my go to option. There was about the same problem - I got 1.609 for different vectors of length 5 (even between the vector and itself, while it should be 0). When I calculated that for a vector of length 10 with itself I got 2.32. – Niminim Jun 25 '19 at 18:05
  • Note: they use natural logarithm, while the implementation above uses log2. The log base doesn't matter to me, though, I just want to get the MI between two vectors. – Niminim Jun 25 '19 at 18:13