10

I have a large hashmap containing millions of entries, and I want to persist it to disk, so that when it is read from the disk again, I don't have the overhead of inserting the key-value pairs back into the map again.

I am trying to use the cereal library to do this, but it appears that the HashMap datatype needs to derive Generic. Is there a way to do this?

donatello
  • 5,727
  • 6
  • 32
  • 56
  • What's the problem with deriving Generic? – Cubic Jun 23 '12 at 13:06
  • 1
    To derive Generic, for a custom type, we need to write something like: `data Something = Something Int Int deriving Generic` How can this be done if the datatype is in a library on Hackage (other than submitting a patch to the maintainer of the library)? – donatello Jun 23 '12 at 13:11
  • Hmm... Well, I personally suspect that Serializing HashMaps like this isn't going to work, and that you'll end up having to use another implementation that supports the kind of serialization you want, but lets see what the others say. – Cubic Jun 23 '12 at 13:29
  • 1
    Before you give up on the toList/fromList approach, how long are you estimating that'll take? 25 seconds to insert 10million (Int,String) key/value pairs on my 4 year old laptop. – ja. Jun 23 '12 at 14:28
  • @ja. It is fast, but it just seems intuitive that serializing the structure as it is would be easy (and more efficient). It appears that fromList/toList is the only way to go, as the `Generic` way didn't work (see my comment on mergeconflict's answer). – donatello Jun 25 '12 at 20:58
  • You've talked me out of trying to do this. I will just use a normal `Map` which has good-enough performance for me. – Michael Fox Oct 24 '14 at 21:34

5 Answers5

5

You might be able to use stand-alone deriving to generate your own Generic instance for HashMap. You'll probably get a warning about orphan instances, but you also probably don't care :) Anyway, I haven't tried this, but it's probably worth a shot...

stites
  • 4,903
  • 5
  • 32
  • 43
mergeconflict
  • 8,156
  • 34
  • 63
  • 2
    I think this is a good answer. One more point perhaps is a trick on how to avoid orphan instances: simply define a newtype wrapping a HashMap and define the instance for this type. When you need to serialize the HashMap simply wrap it in your type and serialize. – Tener Jun 23 '12 at 20:47
  • 1
    @mergeconflict, @Tener I added a line: `deriving instance (Generic k, Generic v) => Generic (H.HashMap k v)` But GHC complains that all the data constructors of HashMap are not in scope (i.e. [they are not all exported](http://hackage.haskell.org/packages/archive/unordered-containers/0.2.1.0/doc/html/src/Data-HashMap-Base.html#HashMap)). Looks like this approach won't work after all. – donatello Jun 25 '12 at 20:51
1

I am not sure if using Generics is a best shot at achieving high performance. My best bet would actually be writing your own instance for Serializable like this:

instance (Serializable a) => Serializable (HashMap a) where
  ...

To avoid creating orphan instances you can use newtype trick:

newtype SerializableHashMap a = SerializableHashMap { toHashMap :: HashMap a }
instance (Serializable a) => SerializableHashMap a where
  ...

The question is how to define ...?

There is no definite answer before you actually try and implement and benchmark possible solutions.

One possible solution is to use toList/fromList functions and store/read the size of the HashMap.

The other (which would be similar to using Generics) would be to write direct serialization based on internal HashMap structure. Given the fact that you dont really have the internals exported that would be a job for Generics only.

Tener
  • 5,280
  • 4
  • 25
  • 44
1

Currently, there is no way to make HashMap serializable without modifying the HashMap library itself.

It is not possible to make Data.HashMap an instance of Generic (for use with cereal) using stand-alone deriving as described by @mergeconflict's answer, because Data.HashMap does not export all its constructors (this is a requirement for GHC).

So, the only solution left to serialize the HashMap seems to be to use the toList/fromList interface.

donatello
  • 5,727
  • 6
  • 32
  • 56
0

If you can use binary, there's binary-orphans which provides instances for unordered-containers. I couldn't install binary-orphans due to some cabal conflict, but just snatched the parts I needed, e.g.:

{-# LANGUAGE CPP           #-}
{-# LANGUAGE DeriveGeneric #-}

module Bin where

import           Data.Binary
import           Data.ByteString.Lazy.Internal
import           Data.Hashable                 (Hashable)
import qualified Data.HashMap.Strict           as M
import qualified Data.Text                     as T

#if !(MIN_VERSION_text(1,2,1))
import           Data.Text.Binary              ()
#endif

instance  (Hashable k, Eq k, Binary k, Binary v) => Binary (M.HashMap k v) where
  get = fmap M.fromList get
  put = put . M.toList

-- Note: plain `encode M.fromList []` without type annotations won't work
encodeModel :: M.HashMap T.Text Int -> ByteString
encodeModel m =
  encode m
unhammer
  • 4,306
  • 2
  • 39
  • 52
-1

The CerealPlus package provides a definition of Serialize for strict HashMaps.

http://hackage.haskell.org/package/cereal-plus

Carlos Reyes
  • 111
  • 1
  • 3