Memory failure when transposing [(K,[V])] to [(V,[K])]

Question

I've got a 279MB file that contains ~10 million key/value pairs, with ~500,000 unique keys. It's grouped by key (each key only needs to be written once), so all the values for a given key are together.

What I want to do is transpose the association, create a file where the pairs are grouped by value, and all the keys for a given value are stored together.

Currently, I'm using Parsec to read in the pairs as a list of tuples (K,[V]) (using lazy IO so I can process it as a stream while Parsec is processing the input file), where:

newtype K = K Text deriving (Show, Eq, Ord, Hashable)
newtype V = V Text deriving (Show, Eq, Ord, Hashable)

tupleParser :: Parser (K,[V])
tupleParser = ...

data ErrList e a = Cons a (ErrList e a) | End | Err e                

parseAllFromFile :: Parser a -> FilePath-> IO (ErrList ParseError a) 
parseAllFromFile parser inputFile = do                               
  contents <- readFile inputFile                                     
  let Right initialState = parse getParserState inputFile contents   
  return $ loop initialState                                         
  where loop state = case unconsume $ runParsecT parser' state of    
                        Error err             -> Err err             
                        Ok Nothing _ _        -> End                 
                        Ok (Just a) state' _  -> a `Cons` loop state'
        unconsume v = runIdentity $ case runIdentity v of            
                                      Consumed ma -> ma              
                                      Empty ma -> ma                 
        parser' = (Just <$> parser) <|> (const Nothing <$> eof)      

I've tried to insert the tuples into a Data.HashMap.Map V [K] to transpose the association:

transpose :: ErrList ParseError (K,[V]) -> Either ParseError [(V,[K])]          
transpose = transpose' M.empty                                                   
  where transpose' _ (Err e)          = Left e                                
        transpose' m End              = Right $ assocs m                      
        transpose' m (Cons (k,vs) xs) = transpose' (L.foldl' (include k) m vs) xs
        include k m v = M.insertWith (const (k:)) v [k] m                  

But when I tried it, I got the error:

memory allocation failed (requested 2097152 bytes)

I could think of a couple things I'm doing wrong:

1. 2MB seems a bit low (considerably less than the 2GB RAM my machine has installed), so maybe I need to tell GHC it's ok to use more?
2. My problems could be algorithmic/data structure related. Maybe I'm using the wrong tools for the job?
3. My attempt to use lazy IO could be coming back to bite me.

I'm leaning toward (1) for now, but I'm not sure by any means.

Answer 1

Is there the possibility that the data will increase? If yes then I'd suggest not to read the while file into memory and process the data in another way.

One simple possibility is to use a relational database for that. This'd be fairly easy - just load your data in, create a proper index and get it sorted as you need. The database will do all the work for you. I'd definitely recommend this.

---

Another option would be to create your own file-based mechanism. For example:

1. Choose some limit l that is reasonable to load into memory.
2. Create n = d `div` l files, where d is the total amount of your data. (Hopefully this will not exceed your file descriptor limit. You could also close and reopen files after each operation, but this will make the process very slow.)
3. Process the input sequentially and place each pair (k,v) into file number hash v `mod` l. This ensures that the pairs with the same value v will end up in the same file.
4. Process each file separately.
5. Merge them together.

It is essentially a hash table with file buckets. This solution assumes that each value has roughly the same number of keys (otherwise some files could get exceptionally large).

---

You could also implement an external sort which would allow you to sort basically any amount of data.

Answer 2

To allow for files that are larger than available memory, it's a good idea to process them in bite-sized chunks at a time.

Here is a solid algorithm to copy file A to a new file B:

• Create file B and lock it to your machine
• Begin loop
  • If there isn't a next line in file A then exit loop
  • Read in the next line of file A
  • Apply processing to the line
  • Check if File B contains the line already
  • If File B does not contain the line already then append the line to file B
  • Goto beginning of loop
• Unlock file B

It can also be worthwhile making a copy of file A into a temp folder and locking it while you work with it so that other people on the network aren't prevented from changing the original, but you have a snapshot of the file as it was at the point the procedure was begun.

I intend to revisit this answer in the future and add code.