Gradle test fails with OutOfMemory when run together BUT passes when run individually

Question

I have tried everything to try at least understand why this is happening even if I can't solve it.

I have the following integration test which is part of a larger test suite of 893 tests:

@SpringBootTest
@ExtendWith(SpringExtension.class)
@AutoConfigureMockMvc
@ActiveProfiles("test")
@Import(OracleTestDbConfig.class)
public class StyleCCIntegrationTest {

    @Autowired
    MockMvc mockMvc;

    @MockBean
    CustomerChoiceRepository customerChoiceRepository;
    private StyleCCTestUtility styleCCTestUtility;
    @Autowired
    ObjectMapper objectMapper;
    @MockBean
    PriceStrategyClient priceStrategyClient;
    @MockBean
    MerchandiseHierarchyService merchandiseHierarchyService;
    @MockBean
    StyleRepository styleRepository;
    @MockBean
    RegionRepository regionRepository;
    @MockBean
    AsyncService asyncService;

    @Nested
    class StyleCCRequest {

        private static final String STYLE_CC_BASE_URL = "/items/styleCCDetails";

        @BeforeEach
        void setup() {
            styleCCTestUtility = new StyleCCTestUtility();
        }

        @Test
        void shouldReturn200WhenValidStyleCCRequestPassed() throws Exception {
            mockDbAndApiInteractionForStyleCC();
            mockMvc.perform(MockMvcRequestBuilders.post(STYLE_CC_BASE_URL)
                    .contentType(MediaType.APPLICATION_JSON)
                    .content(objectMapper.writeValueAsString(styleCCTestUtility
                            .createStyleCCRequestForCCIdForIT())))
                    .andExpect(status().isOk());
        }

        @Test
        void shouldReturnErrorWhenNoCustomerChoiceStyleFound() throws Exception {
            mockDbAndApiInteractionForStyleCCForNoData();
            mockMvc.perform(MockMvcRequestBuilders.post(STYLE_CC_BASE_URL)
                    .contentType(MediaType.APPLICATION_JSON)
                    .content(objectMapper.writeValueAsString(styleCCTestUtility.createStyleCCRequestForCCId())))
                    .andExpect(status().isNotFound()).andReturn();
        }

        @Test
        void shouldReturnErrorWhenApiNotAvailableForStyleCC() throws Exception {
            when(priceStrategyClient.getPriceStrategyIdsByBMC(anyString(), anyString(), anyString()))
                    .thenAnswer(invocation -> {
                        throw new ItemRuntimeException(ItemExceptions.PRICE_STRATEGY_EXCEPTION);
                    });
            mockMvc.perform(MockMvcRequestBuilders.post(STYLE_CC_BASE_URL)
                    .contentType(MediaType.APPLICATION_JSON)
                    .content(objectMapper.writeValueAsString(styleCCTestUtility.createStyleCCRequestForCCId())))
                    .andExpect(status().isNotFound());
        }

        @Test
        void shouldReturnErrorWhenDbNotReachableForStyleCC() throws Exception {
            when(priceStrategyClient.getPriceStrategyIdsByBMC(anyString(), anyString(), anyString()))
                    .thenReturn(styleCCTestUtility.createPriceStrategyIds());
            when(customerChoiceRepository.findByPriceStrategyIdInAndId(any(), any()))
                    .thenAnswer(invocation -> {
                        throw new ItemRuntimeException(ItemExceptions.GENERAL_ERROR);
                    });
            mockMvc.perform(MockMvcRequestBuilders.post(STYLE_CC_BASE_URL)
                    .contentType(MediaType.APPLICATION_JSON)
                    .content(objectMapper.writeValueAsString(styleCCTestUtility.createStyleCCRequestForCCId())))
                    .andExpect(status().isInternalServerError());
        }

        private void mockDbAndApiInteractionForStyleCC() {
            when(customerChoiceRepository
                    .findByPriceStrategyIdInAndId(any(), any()))
                    .thenReturn(styleCCTestUtility.getCustomerChoices());
            when(merchandiseHierarchyService.getHierarchyDesc(any()))
                    .thenReturn(styleCCTestUtility.createHierarchyIds());
            when(styleRepository.findByIdInAndPriceStrategyIdIn(any(), (List<Long>) any()))
                    .thenReturn(styleCCTestUtility.getStyles());
            when(regionRepository.findByPriceStrategyIdAndStyleId(any(), anyString()))
                    .thenReturn(styleCCTestUtility.createRegionList());
            when(priceStrategyClient.getPriceStrategyIdsByBMC(anyString(), anyString(), anyString()))
                    .thenReturn(singletonList(1L));
        }

        private void mockDbAndApiInteractionForStyleCCForNoData() {
            when(priceStrategyClient.getPriceStrategyIdsByBMC(anyString(), anyString(), anyString()))
                    .thenReturn(styleCCTestUtility.createPriceStrategyIds());
            when(customerChoiceRepository
                    .findByPriceStrategyIdInAndId(any(), any())).thenReturn(emptyList());
            when(styleRepository.findByIdInAndPriceStrategyIdIn(any(), (List<Long>) any()))
                    .thenReturn(emptyList());
            when(regionRepository.findByPriceStrategyIdAndStyleId(any(), anyString()))
                    .thenReturn(emptyList());
            when(merchandiseHierarchyService.getHierarchyDesc(any()))
                    .thenReturn(MerchandiseHierarchy.builder().build());
        }
    }
}

When I try to run this test with :

./gradlew clean test

It fails when running test number 193 which is the above test, and it always fails at the same point in the same way saying that:

<==========---> 83% EXECUTING [9m 16s]
> :test > 193 tests completed, 19 failed
> :test > Executing test com.pem.item.integration.StyleCCIntegrationTest
> :test > Executing test com...integration.StyleCCIntegrationTest$StyleCCRequest
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pem.item.service.exchange.allocation-79"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pem.item.service.exchange.order-78"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pem.item.service.exchange.planning-88"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pem.item.service.exchange.allocation-89"

But when I run the test individually everything works, as it should, because there is nothing in the test that is really fancy.

I don't know what to do, what else I can change, or even where to look to pinpoint this cause. What can I try next?

Also when this test is hit then my CPU utilisation goes up to 90% for some reason.

The OutOfMemoryError is thrown by the TestEventLogger

2021-12-03T11:44:27.897+0530 [DEBUG] [TestEventLogger]     Exception in thread "pem.item.service.exchange.planning-101" java.lang.OutOfMemoryError: Java heap space

"CPU utilisation goes up to 90% for some reason" this is likely the garbage collector which is in overdrive to find some more memory before succumbing to the inevitable. — Thomas, Dec 06 '21 at 12:37
Note that OOM is just a symptom in most cases, i.e. the code listed in the stack trace is just the last straw that broke the camel's neck. The real source of the problem might be elsewhere, ideally try to get a heap dump and analyze it, e.g. using Eclipse MAT or some other suitable tool. — Thomas, Dec 06 '21 at 12:39
@Thomas I am on IntelliJ can you tell me how to get the heap dump from there ? I have never analysed a heap dump. Any pointers on how to get started. — ng.newbie, Dec 06 '21 at 13:13
Have a look here, this should get you started: https://docs.oracle.com/en/java/javase/17/troubleshoot/troubleshooting-memory-leaks.html#GUID-442CF6A1-3DF8-496C-A910-3A239326A161 (it's for Java 17, if you're on another JVM look for the appropriate version). — Thomas, Dec 06 '21 at 13:19
In general, I'd try the following: 1) enable heap dump generation when OOM is detected (Hotspot JVM used to have an option for that), 2) load the heap dump into an analysis tool (Eclipse MAT is standalone and I used it a lot), 3) look for large and unexpected portions of "dominated" heap (i.e. heap that can't be freed because something still has a reference), 4) follow the dominator tree upwards until I find something suspicious or a hint of what might need to be fixed (dominators are the objects that have live references to heap elements, the top should always be a thread). — Thomas, Dec 06 '21 at 13:23

Gradle test fails with OutOfMemory when run together BUT passes when run individually

0 Answers0