As to correctly transform the Normal (which is a direction not a position) from Object space to World space, we multiply it with inverse of Model matrix/_World2Object matrix (in Unity).
Then why Tangent (which is tangent to normal and is direction as well - its not a position) is multiplied with Model Matrix /_Object2World (in Unity) to transform it to World space?
eg code : Code for reading Tangent space normal in shader :
o.normalWorld = normalize( mul(v.normal, _World2Object) );
o.tangentWorld = normalize( mul(_Object2World, v.tangent) );
o.binormalWorld = normalize( cross(o.normalWorld, o.tangentWorld) * v.tangent.w );