Performance issue in IMap.unlock() when used with along with EntryProcessor and MapListener in Hazelcast

Question

HZ version : 3.5.3

I'm facing a performance issue in IMap.unlock(key) which takes approx 4-5 seconds to finish execution. The scenario is as follows :

I have an employeeList IMap which stores companyId against a list of employees (ArrayList<Employee>). Each value (Arraylist) may contain 1500000 employees.

IMap<Integer, ArrayList<Employee>> employeeListMap = hz.getMap("empList");

// adding MapListener for eviction.
employeeListMap.addEntryListener(new SimpleEvictionListener<Integer,
                                               ArrayList<Employee>>(), false);

int companyId = 1;
ArrayList<Employee> empList = new ArrayList<>();
for(int index = 0; index < 1500000; index++)
{
    empList.add(new Employee(index));
}
employeeListMap.set(companyId, empList);

// lock() takes approx 2ms.
employeeListMap.lock(key);

// EDIT: do some business logic associated with this key.

// executeOnKey() takes approx 3ms.
employeeListMap.executeOnKey(companyId, new ListEntryProcessor<Integer, 
                          ArrayList<Employee>>());

// unlock() takes 4-5sec 
employeeListMap.unlock(companyId);
employeeListMap.destroy();

Employee is a POJO defined as follows.

public class Employee implements Serializable
{
    private static final long serialVersionUID = 1L;
    protected int employeeId;
    protected String name;

    public Employee(int id)
    {
        this.employeeId = id;
        this.name = "name-" + id;
    }

    public int getEmployeeId() 
    {
        return employeeId;
    }

    public void setEmployeeId(int employeeId) 
    {
        this.employeeId = employeeId;
    }

To add a new employee, I have written an entry processor SimpleEntryProcessor which will add a new employee to the list and return true.

public class ListEntryProcessor<K, V> extends AbstractEntryProcessor<K, V> 
{

    private static final long serialVersionUID = 129712L;
    public ListEntryProcessor()
    {
        // We need to modify the backup entries as well.
        super(true);
    }

    @Override
    public Object process(Entry<K, V> entry) 
    {
        ArrayList<Employee> empList = (ArrayList) entry.getValue();
        empList.add(new Employee(-123));
        entry.setValue((V)empList);
        return true;
    }
}

To print the keys on eviction, I have added the following MapListener to employeeMap.

public class SimpleEvictionListener<K, V>  implements  
             EntryEvictedListener<K, V>, MapEvictedListener
{
    public void mapEvicted(MapEvent arg0) 
    {
        syso("map got evicted");
    }

    public void entryEvicted(EntryEvent<K, V> arg0) 
    {
        syso("entry got evicted");
    }
}

IMap configuration is as follows.

<map name="empList">
    <in-memory-format>OBJECT</in-memory-format>
    <backup-count>0</backup-count>
    <max-idle-seconds>1800</max-idle-seconds>
    <eviction-policy>LRU</eviction-policy>
    <time-to-live-seconds>0</time-to-live-seconds>
    <max-size>51000</max-size>
    <eviction-percentage>30</eviction-percentage>
    <merge-policy>com.hazelcast.map.merge.PutIfAbsentMapMergePolicy</merge-policy>
</map>

In this scenario, IMap.unlock() take 4-5 seconds to complete execution.

When I commented out the code employeeListMap.addEntryListener(...) (i.e. without MapListener), IMap.unlock() method took only 1ms.

Is this an open issue with hazelcast ? Any pointers will be of great help.

Note: I understand that I should've stored <employeeId, Employee> in a separate employee IMap and <companyId, <list of emp ids> in a different companyEmps IMap for better results. However it is not possible due to legacy nature of the code.

score 1 · Answer 1 · answered Oct 30 '15 at 14:29

1

The lock time is indeed strange. But when you use an EntryProcessor, you don't need to apply a lock. An EntryProcessor blocks entry, so no concurrent updates can happen.

I'll create a ticket for this issue. It looks like a bug.

Which HZ version are you using?

answered Oct 30 '15 at 14:29

pveentjer

10,545
3
23
40

I understand that I don't need to apply a lock while using EP. However, there are other business operations that I need to perform along with EP in a critical section. Hence I need to take a lock against that key. I have modified the code above (EDIT tag). Please let me know if we are not on the same page. – Dinesh Oct 30 '15 at 14:34

score 1 · Accepted Answer · answered Jan 08 '16 at 20:29

I've put your code snippets into a single class to be able to try it easily: https://gist.github.com/gurbuzali/af8422339bfa81af9750

There was a bug in Hazelcast which serialize the value even if you pass false to employeeListMap.addEntryListener() for includeValue param. The problem becomes more visible in your case because your value size is too big.

Below is the reported issue and fixing PR. The fix will be in 3.5.5 which is not released yet but you can try with the snapshot 3.5.5-SNAPSHOT

https://github.com/hazelcast/hazelcast/issues/6866

https://github.com/hazelcast/hazelcast/pull/6949

Performance issue in IMap.unlock() when used with along with EntryProcessor and MapListener in Hazelcast

2 Answers2