2nd level cache collection state is sometimes not synchronized with database

Question

We're using EhCache as 2nd level cache (4.3.8.Final) in Hibernate (4.3.8.Final) for Queries (only on read-mostly tables), Entities and Collections between entities.

So you could say that 2nd level cache is used intensively for optimisation purposes.

We never faced any big issues up till now...

Our persistence model concerning the relevant Entities basically looks like below:

• First we have 2 entities below which have a bi-directional mapping in a natural parent-child relationship:

  @Entity
  @Table(...)
  @Cacheable
  @Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
  public class Parent extends BaseEntity {
      @Id
      private long id;
  
      @OneToMany(mappedBy="parent", fetch=FetchType.LAZY, cascade = CascadeType.REMOVE)
      @Cache(usage=CacheConcurrencyStrategy.READ_WRITE)
      private Set<Child> children;
  
      // getters and setters
  }
  
  @Entity
  @Table(...)
  @Cacheable
  @Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
  public class Child extends BaseEntity {
      @Id
      private long id;
  
      @ManyToOne(fetch = FetchType.LAZY, optional = false)
      @JoinColumn(name = "parent_id", nullable = false)
      private Parent parent;
  
      // getters and setters
  }
  

• Next we have some InvokingEntity which has a one-directional mapping to Parent entity

       @Entity
       @Table(...)
       @Cacheable
       @Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
       public class InvokingEntity extends BaseEntity {
           @Id
           private long id;
  
           @Fetch(FetchMode.SUBSELECT)
           @OneToMany(fetch = FetchType.LAZY, mappedBy = "invokingEntity", cascade = CascadeType.ALL, orphanRemoval = true)
           @OrderBy("sequence asc")
           @SortComparator(ParentBySequenceComparator.class)
           @Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
           private Set<Parent> parents = new TreeSet<>();
  
           // getters and setters
       }
  

All Entitiesextend from BaseEntity which contains common properties like id and version for optimistic locking purposes.

So what happens next:

A ChildRepository application all Child entities are deleted and afterwards new ones are created; this happens in the same transaction (trx1):

Extract of delete method:

 public void deleteChildsById(List<Long> childIds) {
    if (!childIds.isEmpty()) {
        getSession().createQuery("delete Child as child where child.id in (:childIds)")
                    .setParameterList("childIds", childIds)
                    .setReadOnly(false)
                    .executeUpdate();
    }
} 

Extract of create method:

public void createChilds(List<Child> childs) {
    for (Child child : childs) {
        getSession().save(child);
    }
}

Afterwards, in a 2nd transaction (trx2), the Child entities are retrieved by a natural entity graph traversal:

InvokingEntity invokingEntity = getSession.get(InvokingEntity.class, someId);
Collection<Child> childs = invokingEntity.getFirstParent().getChilds();

Now, the issue is that the childscollection is sometimes empty (in trx2) while we can clearly see multiple Childentities in our database as a result of trx1.

So there clearly is a mismatch between 2nd level cache and DB. Again, this issue only occurs in roughly 10% of all cases. Also not that trx2 is always executed 4-5 seconds AFTER trx1.

What could cause this situation? A mapping, config or query problem?

There existed an issue in Hibernate where 2nd level cache was not invalidated but this seems to be resolved automatic L2 collection cache eviction when an element is added/updated/removed on an earlier version than we're working on.

For completeness, I add an extract of my ehcache config below:

    <defaultCache maxElementsInMemory="10000"
              eternal="false"
              timeToIdleSeconds="1800"
              timeToLiveSeconds="3600"
              overflowToDisk="true"
              diskPersistent="false"
              diskSpoolBufferSizeMB="10"
              diskExpiryThreadIntervalSeconds="120"
              memoryStoreEvictionPolicy="LRU">
    </defaultCache>

Feel free to ask when something is not explained clearly! We're open for all suggestions!