How to avoid creating new instance of byteable data when using ChronicleMap.put

Question

I am learning how to use ChronicleMap 3.12 by using Byteable key and value classes. When I run my code with a loop of ChronicleMap.put operations based on the call stack it seems to create a value object everytime ChronicleMap.put is called. I would assume using Byteable value class will prevent object creation. Could someone tell me if I have potentially done something incorrectly?

Code to create the ChronicleMap (both TestDataKeyForChronicleMap and TestDataForChronicleMap are Byteable):

    map = ChronicleMap.of( TestDataKeyForChronicleMap.class, TestDataForChronicleMap.class)
            .name( "TestDataMapForChronicleMap")
            .entries(aMaxNoOfRecords).
            .create();

The call stack when I am running the ChronicleMap.put

at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at net.openhft.chronicle.core.util.ObjectUtils.lambda$null$0(ObjectUtils.java:71)
    at net.openhft.chronicle.core.util.ThrowingSupplier.lambda$asSupplier$0(ThrowingSupplier.java:42)
    at net.openhft.chronicle.core.util.ObjectUtils.newInstance(ObjectUtils.java:297)
    at net.openhft.chronicle.hash.serialization.impl.InstanceCreatingMarshaller.createInstance(InstanceCreatingMarshaller.java:67)
    at net.openhft.chronicle.hash.serialization.impl.ByteableSizedReader.read(ByteableSizedReader.java:42)
    at net.openhft.chronicle.hash.serialization.impl.ByteableSizedReader.read(ByteableSizedReader.java:31)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext$EntryValueBytesData.innerGetUsing(CompiledMapQueryContext.java:585)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext$EntryValueBytesData.getUsing(CompiledMapQueryContext.java:595)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext$DefaultReturnValue.initDefaultReturnedValue(CompiledMapQueryContext.java:411)
    at net.openhft.chronicle.map.impl.CompiledMapQueryContext$DefaultReturnValue.returnValue(CompiledMapQueryContext.java:400)
    at net.openhft.chronicle.map.MapMethods.put(MapMethods.java:86)
    at net.openhft.chronicle.map.VanillaChronicleMap.put(VanillaChronicleMap.java:716)

Thanks.

Update to include TestDataKeyForChronicleMap:

import java.nio.BufferOverflowException;
import java.nio.BufferUnderflowException;

import net.openhft.chronicle.bytes.Byteable;
import net.openhft.chronicle.bytes.BytesStore;

public class TestDataKeyForChronicleMap implements Byteable
{
    private final static long   MAX_SIZE = 16;
    private BytesStore bytesStore;
    private long offset;

    @Override
    public BytesStore bytesStore() 
    {
        return bytesStore;
    }

    @Override
    public void bytesStore(BytesStore aBytesStore, long anOffset, long aSize)
            throws IllegalStateException, IllegalArgumentException,
            BufferOverflowException, BufferUnderflowException 
    {
        if ( aSize != MAX_SIZE )
        {
            throw new IllegalArgumentException();
        }
        bytesStore = aBytesStore;
        offset = anOffset;
    }

    @Override
    public long maxSize() 
    {
        return MAX_SIZE;
    }

    @Override
    public long offset() 
    {
        return offset;
    }

    /**
     * set key
     * @param aKey1 key 1
     * @param aKey2 key 2
     */
    public void setKey( long aKey1, long aKey2 )
    {
        bytesStore.writeLong( offset, aKey1 );
        bytesStore.writeLong( offset + 8, aKey2 );
    }

    /**
     * get key 1
     * @return key 1
     */
    public long getKey1()
    {
        return bytesStore.readLong( offset );
    }

    /**
     * get key 2
     * @return key 2
     */
    public long getKey2()
    {
        return bytesStore.readLong( offset + 8 );
    }
}

Code for TestDataForChronicleMap:

import java.nio.BufferOverflowException;
import java.nio.BufferUnderflowException;

import net.openhft.chronicle.bytes.Byteable;
import net.openhft.chronicle.bytes.BytesStore;

public final class TestDataForChronicleMap implements TestData, Byteable
{
    private final static long   MAX_SIZE = 48;
    private BytesStore bytesStore;
    private long offset;

    public TestDataForChronicleMap()
    {
        Thread.currentThread().dumpStack();
    }

    @Override
    public BytesStore bytesStore() 
    {
        return bytesStore;
    }

    @Override
    public void bytesStore(BytesStore aBytesStore, long anOffset, long aSize)
            throws IllegalStateException, IllegalArgumentException,
            BufferOverflowException, BufferUnderflowException 
    {
        if ( aSize != MAX_SIZE )
        {
            throw new IllegalArgumentException();
        }
        bytesStore = aBytesStore;
        offset = anOffset;
    }

    @Override
    public long maxSize() 
    {
        return MAX_SIZE;
    }

    @Override
    public long offset() 
    {
        return offset;
    }

    @Override
    public void setKey(long aKey1, long aKey2) 
    {
        bytesStore.writeLong(offset+0, aKey1);
        bytesStore.writeLong(offset+8, aKey2);
        bytesStore.writeLong(offset+16, -1L);
        bytesStore.writeLong(offset+24, -1L);
        bytesStore.writeLong(offset+32, -1L);
        bytesStore.writeLong(offset+40, -1L);
    }

    @Override
    public void setData(long aKey1, long aKey2) 
    {
        bytesStore.writeLong(offset+0, aKey1);
        bytesStore.writeLong(offset+8, aKey2);
        bytesStore.writeLong(offset+16, aKey1);
        bytesStore.writeLong(offset+24, aKey2);
        bytesStore.writeLong(offset+32, aKey2);
        bytesStore.writeLong(offset+40, aKey1);
    }

    @Override
    public boolean isCorrect() 
    {
        return ( bytesStore.readLong(offset+0) ==  bytesStore.readLong(offset+16) &&  bytesStore.readLong(offset+0) ==  bytesStore.readLong(offset+40) ) 
                &&
                (  bytesStore.readLong(offset+8) ==  bytesStore.readLong(offset+24) &&  bytesStore.readLong(offset+8) ==  bytesStore.readLong(offset+32) );
    }

    @Override
    public long getKey1() 
    {
        return bytesStore.readLong(offset+0 );
    }

    @Override
    public long getKey2() 
    {
        return bytesStore.readLong(offset+8 );
    }

    @Override
    public String getPrintableText() 
    {
        StringBuilder builder = new StringBuilder();
        builder.append( bytesStore.readLong(offset+0) );
        builder.append( ' ' );
        builder.append( bytesStore.readLong(offset+8) );
        builder.append( ' ' );
        builder.append( bytesStore.readLong(offset+16) );
        builder.append( ' ' );
        builder.append( bytesStore.readLong(offset+24) );
        builder.append( ' ' );
        builder.append( bytesStore.readLong(offset+32) );
        builder.append( ' ' );
        builder.append( bytesStore.readLong(offset+40) );
        return builder.toString();
    }
}

Code of the put loop:

/**
 * test add
 * @param aMap map to be used
 * @param aNoOfData no of data to be used for testing
 * @return time taken in ms
 */
public final static long TestAdd( final TestDataMap aMap, final long aNoOfData )
{
    long time = System.currentTimeMillis();
    TestData data = aMap.createTestData();
    for( long count=0; count<aNoOfData; count++ )
    {
        data.setData( count, count+aNoOfData);
        aMap.put(data);
    }
    return System.currentTimeMillis() - time;
}

Interface for TestDataMap

public interface TestDataMap 
{
    /**
     * create test data
     */
    public TestData createTestData();

    /**
     * create test data for zero copy
     */
    public TestData createTestDataForZeroCopy();

    /**
     * check if map requires new data in each entry
     * @return true if new data is needed for each entry or false if data can be reused
     */
    public boolean needNewData();

    /**
     * put test data into the map
     * @param aData
     */
    public void put( TestData aData );

    /**
     * get test data from the map
     */
    public TestData get( TestData aData );

    /**
     * get test data from the map with zero copy
     */
    public boolean getZeroCopy( TestData aData );

    /**
     * remove the test data from the map
     */
    public boolean remove( TestData aData );

    /**
     * get if the map contains the test data
     */
    public boolean contains( TestData aData );

    /**
     * get size
     */
    public long getSize();

    /**
     * clear the map
     */
    public void clear();

    /**
     * dispose
     */
    public void dispose();
}

And the TestDataMapForChronicleMap

import java.io.IOException;

import net.openhft.chronicle.bytes.BytesStore;
import net.openhft.chronicle.map.ChronicleMap;

public final class TestDataMapForChronicleMap implements TestDataMap
{
    private ChronicleMap<TestDataKeyForChronicleMap,TestDataForChronicleMap> map;
    private TestDataKeyForChronicleMap key = new TestDataKeyForChronicleMap();

    public TestDataMapForChronicleMap( long aMaxNoOfRecords ) throws IOException
    {
        map = ChronicleMap.of( TestDataKeyForChronicleMap.class, TestDataForChronicleMap.class)
                .name( "TestDataMapForChronicleMap")
                .entries(aMaxNoOfRecords)
                .putReturnsNull( true )
                .create();
        BytesStore bytesStore = BytesStore.wrap( new byte[16] );
        key.bytesStore( bytesStore, 0, bytesStore.capacity() );
    }

    /**
     * put test data into the map
     * @param aData
     */
    public void put( final TestData aData )
    {
        key.setKey( aData.getKey1(), aData.getKey2() );
        map.put( key, (TestDataForChronicleMap)aData );
    }

    /**
     * get test data from the map
     */
    public TestData get( final TestData aData )
    {
        key.setKey( aData.getKey1(), aData.getKey2() );
        return map.getUsing( key, (TestDataForChronicleMap)aData ); 
    }



/**
     * remove the test data from the map
     */
    public boolean remove( final TestData aData )
    {
        key.setKey( aData.getKey1(), aData.getKey2() );
        return map.remove( key ) != null;
    }

    /**
     * get if the map contains the test data
     */
    public boolean contains( final TestData aData )
    {
        key.setKey( aData.getKey1(), aData.getKey2() );
        return map.containsKey( key );
    }

    /**
     * dispose the map and releases all the resources
     * @param shouldRemoveFiles true will remove all the existing memory mapped files from the system
     */
    public void dispose( final boolean shouldRemoveFiles )
    {
        map.close();
    }

    /**
     * get size
     * @return size of the map
     */
    public long getSize()
    {
        return map.size();
    }

    /**
     * clear all the values from the map
     */
    public void clear()
    {
        map.clear();
    }

    @Override
    public boolean getZeroCopy(final TestData aData ) 
    {
        key.setKey( aData.getKey1(), aData.getKey2() );
        return map.getUsing( key, (TestDataForChronicleMap)aData ) != null; 
    }

    @Override
    public TestData createTestData() 
    {
        TestDataForChronicleMap data = new TestDataForChronicleMap();
        BytesStore bytesStore = BytesStore.wrap( new byte[48] );
        data.bytesStore( bytesStore, 0, bytesStore.capacity() );
        return data;
    }

    @Override
    public TestData createTestDataForZeroCopy()
    {
        //return createTestData();
        return null;
    }

    @Override
    public boolean needNewData() 
    {
        return false;
    }

    @Override
    public void dispose() 
    {
        map.close();
    }

    /**
     * get heap memory
     */
    public long getOffHeapMemory()
    {
        return map.offHeapMemoryUsed();
    }
}

TestData code

/**
 * TestData has the following layout
 * long key1
 * long key2
 * long value1 (value = key1)
 * long value2 (value = key2)
 * long value3 (value = key2)
 * long value4 (value = key1)
 */
public interface TestData 
{
    /**
     * set key
     * @param aKey1 key 1
     * @param aKey2 key 2
     */
    public void setKey( long aKey1, long aKey2 );

    /**
     * set data
     * @param aKey1 key 1
     * @param aKey2 key 2
     */
    public void setData( long aKey1, long aKey2 );

    /**
     * check if the data is correct
     */
    public boolean isCorrect();

    /**
     * get key 1
     * @return key 1
     */
    public long getKey1();

    /**
     * get key 2
     * @return key 2
     */
    public long getKey2();

    /**
     * to string
     */
    public String getPrintableText();
}

leventov · Accepted Answer · 2017-01-12T03:43:26.053

1

Apparently you put a value for the same key in a loop, or for repeated keys. On each Map.put() the previous value is returned (so an object should be created) according to Map contract.

The simplest way to avoid this is to add putReturnsNull(true) to your ChronicleMap config, however this makes the ChronicleMap to violate Map contract.

Another way is to use contexts:

static <K, V> void justPut(ChronicleMap<K, V> map, K key, V value) {
    try (ExternalMapQueryContext<K, V, ?> c = map.queryContext(key)) {
        c.updateLock().lock();
        MapEntry<K, V> entry = c.entry();
        if (entry != null) {
            c.replaceValue(entry, value);
        } else {
            c.insert(c.absentEntry(), value);
        }
    }
}

edited Jan 12 '17 at 03:43

answered Jan 12 '17 at 03:25

leventov

14,760
11
69
98

For the value of keys are different in the loop but the instance of the key is the same, but maybe the fact that I only implement Byteable but didn't provide the corresponding equals method to compare using the Byteable content. I will use putReturnsNull(true) since I don't need the return value from put. Thanks. – Thomas Lo Jan 12 '17 at 14:55
This is strange. Could you publish the whole source of TestDataKeyForChronicleMap, and the loop with put()? – leventov Jan 12 '17 at 15:17
I can confirm that putting equal and hashCode doesn't seem to help the situation but using putReturnsNull( true ). I assume in order to satisfy the Map contract ChronicleMap has to create a new instance of the data everytime put is called. I will update the original post to include the source code for TestDataKeyForChronicleMap and the loop with put – Thomas Lo Jan 13 '17 at 15:46
equals() and hashCode() couldn't help, Chronicle Map doesn't use them. I suspect there is a problem with your Byteable implementation – leventov Jan 13 '17 at 15:59
very likely. I added the code for the TestDataKeyForChronicleMap and TestDataForChronicleMap to the original posting. Super thanks for your help – Thomas Lo Jan 13 '17 at 16:02
Could you please also include TestData, I don't see createTestData() code – leventov Jan 13 '17 at 16:17
Could you please also include `TestDataMap`? It seems that it has default implementation of `createTestData()`. – leventov Jan 14 '17 at 16:47
sorry for the late reply really appreciate your help on this. I have added the source code for the interface TestDataMap and the complete source code for the TestDataMapForChronicleMap (reason for the interface is to also use the same test harness for other maps like HashMap and ConcurrentHashMap) – Thomas Lo Jan 16 '17 at 14:18
Thank you. I run your full test without `ChronicleMapBuilder.putReturnsNull(true)` with `aNoOfData` 100, 1 mln, 100 mln and didn't see the stack trace that you have posted, specifically `net.openhft.chronicle.map.MapMethods.put(MapMethods.java:86)`. – leventov Jan 21 '17 at 22:41
that is strange. I went back and verify as much as possible and still seem to get the same stack trace without putReturnsNull(true). According to the source code is there a situation the methods.put(q, valueData, returnValue) (line 716 of VanillaChronicleMap) wouldn't be called? – Thomas Lo Jan 24 '17 at 20:59

How to avoid creating new instance of byteable data when using ChronicleMap.put

1 Answers1