3

We have a Spring Boot Application (v2.0.7) using REST Controller and JPA with EclipseLink. In one of the tests we made the DB not accessible, which in a few minutes led to the app not responding to REST requests at all.

A Thread Dump was taken which reveals the following:

1) 4 Threads are stuck on socket.read() while executing OraclePreparedStatement.executeQuery(). They will get unstuck after a few minutes (presumably DB timeout). This is understandable.

2) ~180 Threads are in WAITING (parking) state. They execute a RestController method via Spring instrumented proxy, which lead to them being parked waiting on a lock. Here is an example:

"qtp68159840-600" #600 prio=5 os_prio=0 tid=0x00007fe54400c800 nid=0x371d waiting on condition [0x00007fe5246c8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000070727d808> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(GenericScope.java:489)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at com.xxx.XXXApiController$$EnhancerBySpringCGLIB$$f7601a28.getSomeValue(<generated>)
...
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
...
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:873)
...
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
...

Looking at this line in the stack trace:

com.xxx.XXXApiController$$EnhancerBySpringCGLIB$$f7601a28.getSomeValue(<generated>)

where com.xxx.XXXApiController is our REST Controller, and getSomeValue() is one of its methods, we conclude that REST requests actually do get to the Controller (the cglib-instrumented version of it), but then this instrumentation decides that it should make it park and wait on a lock. Actually, the reference to the lock from this particular thread (e.g. 0x000000070727d808) is mentioned only once in the trace and not found anywhere else in the thread dump.

Can anyone help understand why does it make it wait on the lock and what is this lock?

The Controller has an @Autowired field that is a JPA Repository. Possibly, because of this @Autowired, instrumented code of the Controller somehow "knows" there is a problem with the DB connection and locks the Jetty threads? Can anyone clarify how this mechanism works?

UPDATE

Providing Controller code as requested in the comments (tried to show only the relevant parts).

@javax.annotation.Generated(value = "io.swagger.codegen.v3.generators.java.SpringCodegen", date = "2019-04-17T09:25:33.309Z[GMT]")
@RestController
@RefreshScope
public class XXXApiController implements XXXApi {

    private final ObjectMapper objectMapper;

    private final HttpServletRequest request;

    @Autowired
    private DaoService daoService;

    @org.springframework.beans.factory.annotation.Autowired
    public XXXApiController(ObjectMapper objectMapper, HttpServletRequest request) {
        this.objectMapper = objectMapper;
        this.request = request;
    }

    @PreAuthorize("hasAuthority('db-read')")
    @Override
    public ResponseEntity<Something> getSomething(
            @ApiParam(value = "", required=true, allowableValues = "a,b,c,d") @PathVariable("keyType") 
            String keyType, 

            @ApiParam(value = "keyValue for a given keyType", required=true) 
            @PathVariable("keyValue") 
            String keyValue,

            @ApiParam(value = "The troubleshooting optional parameter", allowableValues = "x,y,z") 
            @Valid @RequestParam(value = "from", required = false) 
            String from) {

        request.setAttribute(METHOD, "getSomething");
        if (keyValue == null|| keyValue.length() == 0) {
            return new ResponseEntity<>(HttpStatus.BAD_REQUEST);
        }
        if (from != null && from.equals("cache")) {
            result = daoService.getSomethingFromCache(keyValue);
        } else {
            result = daoService.getSomethingFromDb(keyType, keyValue);
        } 
        ResponseEntity responseEntity;
        if (routingInfo != null) responseEntity = new ResponseEntity<>(result, HttpStatus.OK);
        else responseEntity = new ResponseEntity(HttpStatus.NOT_FOUND);
        return responseEntity;
    }

   ...
}


@Component
@Controller
public class DaoServiceImpl implements DaoService {

    @Autowired
    private SomethingRepository somethingRepository;

    @Override
    public Something getSomethingFromCache(String name) {
        String somethingStr = getFromCache(name);
        if (somethingStr != null) {
            return convertSomethingFromCache(somethingStr);
        }
        return null;
    }    

    @Override
    public Something getSomethingFromDb(String keyType, String value) {
        return createSomething(somethingRepository.findByKey(value));
    }

    ...
}


@Repository
@Transactional
public interface SomethingRepository extends JpaRepository<Something, Long>, SomethingService {
    @Query("select t from Something t where " +
            "(:x is not null and t.x = :x) or " +
            "(:y is not null and t.y= :y)")
    public List<Something> findByFilter(
            @Param("x") String x,
            @Param("y") String y,
            Pageable request);
}
Gena L
  • 420
  • 1
  • 5
  • 16
  • Show the controller code. The lock seems to be related to something in Spring Cloud (`org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean`) that you haven't mentioned in your question. – rainerfrey Aug 14 '19 at 18:26
  • Thanks @rainerfrey, you got me going in the right direction. The only reference to Spring Cloud comes from RefreshScope annotation, which I didn't notice Controller was annotated with. It got occasionally copy-pasted from another class, and shouldn't be here. You can add an answer and I'll accept it. – Gena L Aug 15 '19 at 10:41

0 Answers0