Sometimes, the clients' local caching name server always queries the fixed endpoint in a specific time since there is DNS cache locally. You can run ipconfig/flushdns
in a CMD as an administrator account to clear the cache. Then verify if the faulty endpoint continues.
You could also change the DNS time to live (TTL) to a value which best balances your need. Shorter values result in faster cache expiry and thus more round-trips to the Traffic Manager name servers. Longer values mean that it can take longer to direct traffic away from a failed endpoint. Refer to How Traffic Manager Works.
Moreover, the Traffic Manager does not receive DNS queries directly from clients. Rather, DNS queries come from the recursive DNS service that the clients are configured to use. For each DNS query received, Traffic Manager randomly chooses an ONLINE endpoint.
It is important to understand that DNS responses are cached by clients
and by the recursive DNS servers that the clients use to resolve DNS
names. This caching can have an impact on weighted traffic
distributions. When the number of clients and recursive DNS servers is
large, traffic distribution works as expected. However, when the
number of clients or recursive DNS servers is small, caching can
significantly skew the traffic distribution.
Ref: Weighted traffic-routing method
Update
Another thing might be that the TM health probe time. The number of Traffic Manager health checks reaching your endpoint depends on the monitoring interval and the number of locations from where the health checks originate. But this usually happens in a short time nearly few seconds.
Cloud Service 'staging' slots can be configured in Traffic Manager as
External endpoints. Because the External endpoint type is in use,
changes to the underlying service are not picked up automatically.
With external endpoints, Traffic Manager cannot detect when the Cloud
Service is stopped or deleted.
You can go through FAQs to get more details.