MTProto Telegram API PEER_ID_INVALID for User B, but not A

Question

I have the following problem:

When successfully connected to User A, i can send messages to another User X like that:

  await this.api.call('messages.sendMessage', {
    peer: {
      _: 'inputPeerUser',
      user_id: *****123*****,               // user_id of User X
      access_hash: "*****111******"         // access_hash of User X
    },
    message: current.message + ' ' + current.emoji,
    random_id: current.user.userId + Math.floor(Math.random())
  });

Everything works as expected. Now I also have User B, which also likes to send messages to User X.

But, if I use the standard authentication flow like:

1. call sendCode()
2. received the SMS Code, pass it to the MTPROTO Object
3. keep the current MTPROTO Object and authenticate, it works for both Accounts
4. Trying to fetch channels from each user works as well

If i try to send a message to User X from User B i receive:

{
  _: 'mt_rpc_error',
  error_code: 400,
  error_message: 'PEER_ID_INVALID'
}

Are there any restrictions on who can write to whom?

Any help is very appreciated!