0

I'm developping a little utlity tool which parse some files, extracts a specific information and then store it into a mongodb collection.

It's working right now, but it's doing something like :

#!/bin/bash

find myFolder | while read f
do
    # some things
    ...
    # and then

    echo "db.test.insert({'val': BinData(0, '$data')});" | mongo -quiet --host 127.0.0.1:36427 collectionName
done

But the "problem" here is that I'm opening / closing the mongodb connection everytime.
So here is the question:

Is there a way to start the mongodb connection, and then happen some data into it.

I KNOW that the following code is totally wrong, but it's just to give you an example of the kind of feature I'm looking for :

#!/bin/bash

mongoConn=`mongo 127.0.0.1:36427 collectionName`

find myFolder | while read f
    # ...

    echo "db.test.insert(..);" > mongoConn
done

echo "exit;" > mongoConn
Tiller
  • 436
  • 1
  • 4
  • 22
  • A named pipe could work, that is a special file in your folder which works like a pipe. You probably want to prevent mongodb from closing after reading a single files, so http://stackoverflow.com/questions/5957386/prevent-fifo-from-closing-reuse-closed-fifo might help. – unique2 Nov 03 '13 at 18:12

2 Answers2

2

The following assumes, that you can write more than one command to your mongo tool.

Open a pipe to the mongo command as stdout. After that you can write your commands.

exec > >(mongo ...)
find ... | while read ... ; do
  echo "db.test.insert(...);"
done
ceving
  • 21,900
  • 13
  • 104
  • 178
  • Perfect! Exactly what I was looking for. I didn't know about the `>(...)`. I just changed your code to use `exec 3> >(mongo ...)` and then `echo "db..." >&3` :) – Tiller Nov 03 '13 at 21:49
  • You should be aware that this is bashish --- that is, an extension specific to bash --- and won't work in standard Posix shells. This isn't a problem, but you need to be sure that the magic at the top of your script invokes bash directly (`#!/bin/bash`) rather than the default shell (`#!/bin/sh`), because the default shell might not be bash. – David Given Nov 03 '13 at 23:12
  • You don't need process substitution here. Just pipe the output of the while loop to `mongo`: `find ...| while read ...; do ...; done | mongo ...`. – chepner Nov 04 '13 at 13:50
  • @chepner Yes that it right but it does not make much difference. – ceving Nov 04 '13 at 19:20
0

You could try something like :

#!/bin/bash

deleteTmpFile(){
    [[ -f insert.tmp ]] && rm -rf $1
}

tmpFile="insert.tmp"
deleteTmpFile $tmpFile

find ... | while read f
do
    #...
    echo "db.test.insert({'val': BinData(0, '$data')});" >> $tmpFile
done

cat insert.tmp|mongo -quiet --host 127.0.0.1:36427 collectionName
deleteTmpFile $tmpFile
Idriss Neumann
  • 3,760
  • 2
  • 23
  • 32
  • The problem is that it's a VERY long process. It parses gigas of data, so I can't store the command list. I need to do it live. – Tiller Nov 03 '13 at 17:51
  • You can swap the contents of the file: drain the tmp file and run the pipe on mongodb every 50 queries for example. Otherwise use another language which provide tools which allow us to use a persistent connection. – Idriss Neumann Nov 03 '13 at 17:53