Spring boot JPA Hibernate - Old generation memory not being freed up

Question

We are using Spring boot 2.6.6 (Java8) which internally pulls in hibernate version 5.6.7.Final. Database used is Oracle 19c.

We are using "saveAll" method of JPA repository to do batch insert/updates for entities but observing that the old generation memory keeps increasing without getting freed up. We are using "ParNew" for Young generation GC and "CMS" for old generation GC.

Below is what we saw in one of the recent heap dumps generated on OOM -

Problem Suspect 1   One instance of "org.hibernate.engine.internal.StatefulPersistenceContext" loaded by  
 "sun.misc.Launcher$AppClassLoader @ 0x700000000" occupies 3,845,753,568 (82.42%) bytes.  
 The memory is accumulated in one instance of "org.hibernate.engine.internal.EntityEntryContext" loaded by  
 "sun.misc.Launcher$AppClassLoader @ 0x700000000".

Below is the configuration class

@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(entityManagerFactoryRef = "entityManagerFactory", basePackages = { "com.xxx.yyy.persistence.repository" })
public class BootDatasourceConfig {

    
    @ConfigurationProperties(prefix = "yyy.persistence.datasource")
    @Bean(name = "yyyDS")
    @Primary
    public DataSource primaryDataSource() {
        return DataSourceBuilder.create().build();
    }

    @Primary
    @Bean(name = "entityManagerFactory")
    public LocalContainerEntityManagerFactoryBean entityManagerFactory(EntityManagerFactoryBuilder builder, @Qualifier("yyyDS") DataSource dataSource) {
        LocalContainerEntityManagerFactoryBean em = builder.dataSource(dataSource).packages("com.xxx.yyy.persistence.entity").persistenceUnit("persistence").build();
        em.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
        return em;
    }


    @Primary
    @Bean(name = "transactionManager")
    public PlatformTransactionManager transactionManager(@Qualifier("entityManagerFactory") EntityManagerFactory entityManagerFactory) {
        return new JpaTransactionManager(entityManagerFactory);
    }

}

Below is the service class

@Service
@Transactional
public class PersistenceServiceImpl implements PersistenceService {

@Autowired
private Entity1Repository entity1repo;

@Autowired
private Entity2Repository entity2repo;

@Override
public PersistenceResponse process (PersistenceReqMsg reqMsg) {
    List<PersistenceRequest> reqLst = reqMsg.getReqList();

    List<Entity2> entity2Lst = new ArrayList<>();
    List<Entity1> entity1Lst = new ArrayList<>();

    for (PersistenceRequest req : reqLst) {
         for (PersistenceEvt evt : req.getBdy().getEvents()) {
             entity1Lst(...);
         }
         entity2Lst(...)
    }

    if (CollectionUtils.isNotEmpty(entity1Lst)) {
         entity1repo.saveAll(entity1Lst);
    }
    if (CollectionUtils.isNotEmpty(entity2Lst)) {
         entity2repo.saveAll(entity2Lst);
    }
  
    return AAAPersistenceResponse;
}

}

Below are the repo definitions -

@Repository
public interface Entity1Repository extends JpaRepository<Entity1, Long>, JpaSpecificationExecutor<Entity1>, CustomRepository  {}

@Repository
public interface Entity2Repository extends JpaRepository<Entity2, Long>, JpaSpecificationExecutor<Entity2>, CustomRepository  {}

Below are the application prop entries -

yyy.persistence.datasource.type=com.zaxxer.hikari.HikariDataSource
yyy.persistence.datasource.driverClassName=oracle.jdbc.OracleDriver
yyy.persistence.datasource.jdbcUrl=jdbc:oracle:thin:@ddd:1521:ccc
yyy.persistence.datasource.maximumPoolSize=20
yyy.persistence.datasource.idleTimeout=5
yyy.persistence.datasource.minimumIdle=5
yyy.persistence.datasource.connectionTimeout=60000

spring.jpa.database-platform= org.hibernate.dialect.Oracle12cDialect
spring.jpa.show-sql=false
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.Oracle12cDialect
spring.jpa.properties.hibernate.id.new_generator_mappings = false
spring.jpa.properties.hibernate.format_sql = true
spring.jpa.properties.hibernate.current_session_context_class=org.springframework.orm.hibernate5.SpringSessionContext
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
spring.jpa.properties.hibernate.jdbc.batch_size=50
spring.jpa.properties.hibernate.order_inserts=true
spring.jpa.properties.hibernate.order_updates=true

Below is minimal entity definition -

@Entity
@Table(name = "entity1table", uniqueConstraints = @UniqueConstraint(columnNames = { "ref1", "ref2" })) {

@Id
@GeneratedValue(strategy = IDENTITY)
@Column(name = "entity1_id", unique = true, nullable = false)
public Long getEntity1Id() {
    return this.entity1Id;
}

}

From the heap dump it looks like entities are not being cleared from memory and building up over time. The incoming request object "PersistenceReqMsg" will not have an appreciably high number of records (max 250). Since the @Service method is executed in a @Transaction, my understanding is that JPA internally will issue "commit"/"rollback" based on batch size but not sure if JPA is also clearing the entities from persistence context. Could this be the reason for memory not being released?

Besides above JPA repo based code, we have few other service classes where "EntityManager" is being injected via @PersistenceContext and used to execute look up queries and some inserts/updates. Below is the sample -

@PersistenceContext
    private EntityManager em;

public getDetails () {
   em.createNativeQuery(....);
}

Not sure if these could also contribute to the memory building up. Is it better to inject "EntityManagerFactory" instead of "EntityManager" and create a new instance of EM for every execution?