TempData is only valid for the current and next request which is why it is stored in the session state. Storing it with the request would not work.
So any limitations that would break saving items in session state (e.g. load balances) will break TempData as well.
See the documentation for TempDataDictionary
You can use a TempDataDictionary object to pass data in the same way
that you use a ViewDataDictionary object. However, the data in a
TempDataDictionary object persists only from one request to the next,
unless you mark one or more keys for retention by using the Keep
method. If a key is marked for retention, the key is retained for the
next request.
A typical use for a TempDataDictionary object is to pass data from an
action method when it redirects to another action method. For example,
an action method might store information about an error in the
controller's TempData property (which returns a TempDataDictionary
object) before it calls the RedirectToAction method. The next action
method can then handle the error and render a view that displays an
error message.