How to make nginx to proxy to the same instance multiple ports using load balancing upstream

Question

I have an app that uses 2 backend ports. I use nginx to load balance clients to diferent server instances. Each instance has 2 apps running that listens on diferent ports and intercommunicate between itself. Is there an easy way to tell nginx to pass same client to same instance?

That is a standard way:

stream {
  upstream appA {
    hash   $remote_addr consistent;
    server A:5001
    server B:5001
  }
  upstream appB {
    hash   $remote_addr consistent;
    server A:6001
    server B:6001
  }
  server {
    listen 5000
    proxy_pass appA
  }
  server {
    listen 6000
    proxy_pass appB
  }
}

That is what I want:

client 1, request 5000 -> nginx -> server A: 5001
client 1, request 6000 -> nginx -> server A: 6001

client 2, request 5000 -> nginx -> server B: 5001
client 2, request 6000 -> nginx -> server B: 6001

That is what happens:

client 1, request 5000 -> nginx -> server A: 5001
client 1, request 6000 -> nginx -> server B: 6001

client 2, request 5000 -> nginx -> server B: 5001
client 2, request 6000 -> nginx -> server A: 6001

Since nginx navigates same client to diferent servers, it breaks my app.

I made a workaround with creating main server to listen on both 5000 and 6000 ports. Upstream it to diferent virtual servers, then each server checks url and proxies again to required port. Simplified configs looks something like:

upstream servers {
    hash   $remote_addr consistent;
    server 127.0.0.1:7001
    server 127.0.0.1:7002
}

server {
  listen 7001;
  location / {
    proxy_pass A:5001
  }
  location /workaround-path {
    proxy_pass A:6002
  }
}
server {
  listen 7002;
  location / {
    proxy_pass B:5001
  }
  location /workaround-path {
    proxy_pass B:6002
  }
}
server {
  listen 5000;
  listen 6000;
  proxy_pass servers;
}

The problem is that: First, it does double proxying and runs extra servers. Second, in app I had to specify second connection to domain:6000/workaround-path. I am sure more elegant and simple way exists with nginx, please, help me to find it. Thanks.