Not able to access InheritableThreadLocal object from parent thread in fall back method

Question

I have InheritableThreadLocal<ConcurrentHashMap<String, Object>> thread that initializes when a request comes via the filter and set some transaction_id in it.

Now at the service layer, I'm calling 10 different API calls via CompletableFuture. All API service class have one execute method that is using RestTempate to make an API call. I put @HystrixCommand on execute method.

execute method is void type but it put the API response in InheritableThreadLocal object.

Problem is when an API call fails Hystrix call FallBackMethod and when I put error response in InheritableThreadLocal, I'm not able to send that error response to the client.

ThreadLocalUtil.class

public class ThreadLocalUtil {

    private static InheritableThreadLocal<ConcurrentHashMap<String, Object>> transmittableThreadLocal = new InheritableThreadLocal<>();

    public static void addDataToThreadLocalMap(String key, Object value) {
        Map<String, Object> existingDataMap = transmittableThreadLocal.get();
        if (value != null) {
            existingDataMap.put(key, value);
        }
    }

    public static Object getDataFromThreadLocalMap(String key) {
        Map<String, Object> existingDataMap = transmittableThreadLocal.get();
        return existingDataMap.get(key);
    }

    public static void clearThreadLocalDataMap() {
        if (transmittableThreadLocal != null) 
            transmittableThreadLocal.remove();
    }

    public static Object getRequestData(String key) {
        Map<String, Object> existingDataMap = transmittableThreadLocal.get();
        if (existingDataMap != null) {
            return existingDataMap.get(key);
        }
        return "-1";
    }

    public static void initThreadLocals() {
        ConcurrentHashMap<String, Object> dataForDataMap = new ConcurrentHashMap<String, Object>();
        String requestId = "REQUEST_ID_" + System.currentTimeMillis();
        dataForDataMap.put("REQUEST_ID", requestId);
        transmittableThreadLocal.set(dataForDataMap);
    }
}

CommonFilter.class

@Component
@Order(1)
public class CommonFilter extends OncePerRequestFilter {

  @Override
  protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
          throws ServletException, IOException {
      try {
          ThreadLocalUtil.initThreadLocals();
          filterChain.doFilter(request, response);
      } catch (Exception e) {
          if (e instanceof ServletException) {
              throw (ServletException) e;
          }
      } finally {
          ThreadLocalUtil.clearThreadLocalDataMap();
      }

  }

EmployeeService.class

@Component
public abstract class EmployeeService {

    @Autowired
    private ThreadLocalUtil threadLocalUtil;

    public abstract void getEmployee(int employeeId);

    public void fallbackMethod(int employeeid) {
        threadLocalUtil.addDataToThreadLocalMap("ErrorResponse", "Fallback response:: No employee details available temporarily");
    }
}

EmployeeServiceImpl.class

@Service
public class EmployeeServiceImpl extends EmployeeService {

    @HystrixCommand(fallbackMethod = "fallbackMethod", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "900"),
            @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "10") })
    public void getEmployee(int employeeId) {
        System.out.println("Getting Employee details for " + employeeId + ", threadLocalUtil : " + threadLocalUtil.getDataFromThreadLocalMap("EMPLOYE_ID"));
        String response = restTemplate.exchange("http://localhost:8011/findEmployeeDetails/{employeeid}",
                HttpMethod.GET, null, new ParameterizedTypeReference<String>() {
                }, employeeId).getBody();

        threadLocalUtil.addDataToThreadLocalMap("Response", response);
    }

    @Autowired
    RestTemplate restTemplate;

    @Autowired
    private ThreadLocalUtil threadLocalUtil;
}

*"I'm not able to send that error response to the client."* I don't see any code attempting to do that, so how can we help with that? --- You do know that `InheritableThreadLocal` is fairly useless in a web server, right? I mean, I can't see where you starts a child thread, but it's very likely that any worker thread is coming from a thread pool, for better performance, so worker threads don't have a parent/child relationship with the request processing thread. — Andreas, Apr 26 '19 at 08:06
*FYI:* `System.currentTimeMillis()` is not granular enough to provide unique Request ID values. Multiple requests may arrive at the same time. Suggest you use an `AtomicLong` instead. — Andreas, Apr 26 '19 at 08:08
@Andreas im trying to sending error response from EmployeeService class. — Devratna, Apr 26 '19 at 09:23
@Andreas child thread is created when HystrixCommand is executed, for example, when we call EmployeeServiceImpl.getEmployee method, due to using HystrixCommand this method execution is happening in a child thread — Devratna, Apr 26 '19 at 09:25
@Andreas in the acual code im appending IP address and System.currentTimeMillis() together so its generating unique key for me — Devratna, Apr 26 '19 at 09:26
EmployeeService class isn't sending anything. It's storing an error message in the ThreadLocal, but that doesn't get used in any code we can see in the question, so, *as I said*, I don't see any code attempting to send that error response to the client. — Andreas, Apr 26 '19 at 13:06
@andreas actually it's just a demo code that I shared, actual code is different but core problem is around in shared code — Devratna, Apr 26 '19 at 20:50
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/192454/discussion-between-devratna-and-andreas). — Devratna, Apr 26 '19 at 20:50

FlapPy · Accepted Answer · 2019-05-02T18:02:48.677

So, first of all since internally Hystrix uses ThreadPoolExecutor (Threads created once and reused), so it is wrong to use InheritableThreadLocal.

From the above question and what you asked in my blog, I understand that you problem is

InheritableThreadLocal becomes null in hystrix fallback method

Further adding to this (you may verify this)

InheritableThreadLocal becomes null in hystrix fallback method only in case of timeouts and not in case of any other exception

I would recommend others to refer to my blog. Hystrix fallback in case of timeout, takes place in hystrix-timer thread. Hystrix fallback execution thread You can verify this by logging Thread.currentThread().getName()

Since the parent of hystrix-timer thread is not your calling thread, and so your transmittableThreadLocal.get() becomes null.

To solve this I would recommend using HystrixCommandExecutionHook and HystrixRequestVariableDefault. Using this you can implement hooks like onStart, onExecutionStart, onFallbackStart etc., in which you need to get/set the threadLocal variables. For more details you can refer to the last section in the blog.

Update: For your use-case you can modify your code as follows:

ThreadLocalUtil.java

public class ThreadLocalUtil {

    private static ThreadLocal<ConcurrentHashMap<String, Object>> transmittableThreadLocal = new ThreadLocal<>();

    public static ConcurrentHashMap<String, Object> getThreadLocalData() {
        return transmittableThreadLocal.get();
    }

    public static void setThreadLocalData(ConcurrentHashMap<String, Object> data) {
        transmittableThreadLocal.set(data);
    }

    public static void addDataToThreadLocalMap(String key, Object value) {
        Map<String, Object> existingDataMap = transmittableThreadLocal.get();
        if (value != null) {
            existingDataMap.put(key, value);
        }
    }

    public static Object getDataFromThreadLocalMap(String key) {
        Map<String, Object> existingDataMap = transmittableThreadLocal.get();
        return existingDataMap.get(key);
    }

    public static void clearThreadLocalDataMap() {
        if (transmittableThreadLocal != null) 
            transmittableThreadLocal.remove();
    }

    public static Object getRequestData(String key) {
        Map<String, Object> existingDataMap = transmittableThreadLocal.get();
        if (existingDataMap != null) {
            return existingDataMap.get(key);
        }
        return "-1";
    }



    public static void initThreadLocals() {
        transmittableThreadLocal.set(new ConcurrentHashMap<>());
        String requestId = "REQUEST_ID_" + System.currentTimeMillis();
        addDataToThreadLocalMap("REQUEST_ID", requestId);
    }
}

EmployeeService.java

@Component
public abstract class EmployeeService {
    public abstract void getEmployee(int employeeId);

    public void fallbackMethod(int employeeid) {
        threadLocalUtil.addDataToThreadLocalMap("ErrorResponse", "Fallback response:: No employee details available temporarily");
    }
}

EmployeeServiceImpl.java

@Service
public class EmployeeServiceImpl extends EmployeeService {

    @HystrixCommand(fallbackMethod = "fallbackMethod", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "900"),
            @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "10") })
    public void getEmployee(int employeeId) {
        System.out.println("Getting Employee details for " + employeeId + ", threadLocalUtil : " + threadLocalUtil.getDataFromThreadLocalMap("EMPLOYEE_ID"));
        String response = restTemplate.exchange("http://localhost:8011/findEmployeeDetails/{employeeid}",
                HttpMethod.GET, null, new ParameterizedTypeReference<String>() {
                }, employeeId).getBody();

        threadLocalUtil.addDataToThreadLocalMap("Response", response);
    }

    @Autowired
    RestTemplate restTemplate;
}

HystrixHook.java

public class HystrixHook extends HystrixCommandExecutionHook {

    private HystrixRequestVariableDefault<ConcurrentHashMap<String, Object>> hrv = new HystrixRequestVariableDefault<>();

    @Override
    public <T> void onStart(HystrixInvokable<T> commandInstance) {
        HystrixRequestContext.initializeContext();
        getThreadLocals();
    }

    @Override
    public <T> void onExecutionStart(HystrixInvokable<T> commandInstance) {
        setThreadLocals();
    }


    @Override
    public <T> void onFallbackStart(HystrixInvokable<T> commandInstance) {
        setThreadLocals();
    }


    @Override
    public <T> void onSuccess(HystrixInvokable<T> commandInstance) {
        HystrixRequestContext.getContextForCurrentThread().shutdown();
        super.onSuccess(commandInstance);
    }

    @Override
    public <T> Exception onError(HystrixInvokable<T> commandInstance, HystrixRuntimeException.FailureType failureType, Exception e) {
        HystrixRequestContext.getContextForCurrentThread().shutdown();
        return super.onError(commandInstance, failureType, e);
    }

    private void getThreadLocals() {
        hrv.set(ThreadLocalUtil.getThreadLocalData());
    }

    private void setThreadLocals() {
        ThreadLocalUtil.setThreadLocalData(hrv.get());
    }
}

AbcApplication.java

public class AbcApplication {
    public static void main(String[] args) {
        HystrixPlugins.getInstance().registerCommandExecutionHook(new HystrixHook());
        SpringApplication.run(Abc.class, args);
    }
}

Hope this helps

if I set ThreadLocal variable in Hook then how will get that variable in the fallback method that exists in another class — Devratna, May 01 '19 at 10:33
As I see, your threadLocal variable is **Mutable** (`ConcurrentHashMap`), so you just have to make sure that **same reference is present in primary and fallback method**. Once you have same reference then whatever you add in primary, will be accessible in fallback. And this can be achieved through hook. Just set the `ConcurrentHashMap` from the **calling thread into HystrixRequestVariableDefault** (`onStart`), and **set it to threadLocal of primary thread (`onExecutionStart`) and fallback thread (`onFallbackStart`)** — FlapPy, May 01 '19 at 16:58
ok @FlapPy, let say i will set threadLocal object in `onExecutionStart` and `onFallbackStart` method, now i want that same object reference in my `EmployeeService.fallbackMethod` method so that i can update some value in same reference of threadLocal object, so how will i do ? — Devratna, May 02 '19 at 05:14
where i use this line of code `HystrixPlugins.getInstance().registerCommandExecutionHook(new HystrixHook());` — Devratna, May 02 '19 at 12:01
I will update my answer for how to do part. And for registerCommandExecutionHook(), you can put this anywhere. **Keeping in mind that this line should execute only once in whole application**. For example you can put this in springboot main function inside `AbcApplication.java` file. — FlapPy, May 02 '19 at 17:37
only one thing that is not clear `how you able to access the threadLocalUtil object in EmployeeService.java without initializing it` — Devratna, May 06 '19 at 07:42
`onStart()` hook is called on **calling thread**, in which initialized `threadLocal` object is copied to `hrv`. Now just before hystrix primary method (`EmployeeServiceImpl.getEmployee()`) execution `onExecutionStart()` hook and just before fallback method (`EmployeeService.fallbackMethod()`) execution `onFallbackStart()` hook is called, which copies initialized `threadlocal` from `hrv` to the `threadLocal` of `hystrix-thread` or `hystrix-timer-thread`, or whatever thread is used from execution. — FlapPy, May 07 '19 at 19:12
when hystrix start reusing thread from hystrix thread pool, then I didn't get the current object of threadlocal, what happens is when i start the application 10 request are serve successfully but 11th request and next all request are throwing null pointer exception because hystrix thread pool size is 10 and when it start re-using thread i didnt get current state of threadLocal object — Devratna, May 08 '19 at 07:56
is there any way so that hystrix start create every time new thread for every new request? (in short, I don't need hystrix pool, can we avoid it) — Devratna, May 08 '19 at 12:13
@SumanthVarada I have already answered your query [here](https://stackoverflow.com/a/59406001/9877669) — FlapPy, Dec 20 '19 at 10:33

Not able to access InheritableThreadLocal object from parent thread in fall back method

ThreadLocalUtil.class

CommonFilter.class

EmployeeService.class

EmployeeServiceImpl.class

1 Answers1

Linked