How to Import Multiple JSON Files to Mongodb?

Question

I am newbie on Mongodb. Actually I have thousands of files in different folders. All of files include json data. There are more than 30 millions files. So I think the best way to store this data is document based db.

I am aware of Import more than 1 json file using mongoimport this SO post. However, accepted answer require a collection which has file names in it. I cannot put 30 millions file name in a collection...

How can I import multiple json files to Mongodb on Windows env?

score 3 · Answer 1 · edited Aug 15 '19 at 07:50

3

I was looking for the solution for 2 days and here is the solution which works for me:

C:\MongoDB\Server\3.0\bin>
  for %i in (C:\test\*) do 
    mongoimport --file %i --type json --db mydb --collection mycollection

You just copy and paste this code in the cmd and change the file directories C:\MongoDB\Server\3.0\binand C:\test\.

edited Aug 15 '19 at 07:50

B--rian

5,578
10
38
89

answered Aug 15 '19 at 07:39

Majdoub Wided

31
4

2

Welcome to SO! Please edit your question and explain your solution a bit more. For more guidance, see https://stackoverflow.com/help/how-to-answer – B--rian Aug 15 '19 at 07:42

score 1 · Accepted Answer · answered Jul 26 '13 at 10:10

You will need to write a script in your favourite language that reads each file, JSON-decodes it and then inserts them one by one into MongoDB. In PHP, such a script would be akin to:

<?php
$f = glob("*.json");
$m = new MongoClient;
$c = $m->myDb->myCollection;

foreach ( $f as $fileName )
{
    $contents = json_decode( file_get_contents( $fileName ) );
    $c->insert( $contents );
}
?>

user2321465 · Answer 3 · 2016-03-21T13:31:28.693

1

You can create a batch script which fetches all json file in a given folder and then imports it to the db:

@echo off
for %%f in (*.json) do (
"mongoimport.exe" --jsonArray --db databasename --collection collectioname --file %%~nf.json )

Hope this helps

edited Mar 21 '16 at 13:31

answered Mar 20 '16 at 19:05

user2321465

139
6

score 1 · Answer 4 · answered Jun 27 '17 at 02:38

For anyone searching for a cross-platform solution to this, I created a little perl script that will do this. It takes a database and directory argument and will import any .json files it finds in the directory to mongodb. If you dont give it a directory, it just uses the one you're currently in. I need to refine the regex that checks for the .json files a bit and I'm sure this can be done with less code, (I'm a newbie Perl monk) but this works and I like Perl.. so, to anyone that finds this - enjoy.

#!/usr/bin/perl
use strict;
use warnings;

#this is a script for enumerating over every json file in a folder and importing it into mongodb

my ($database, $directoryPath) = @ARGV;

if(! $database) { #check for required database argument
    die "A database argument must be provided to the script. Ex: perl mongorestore.pl wasp";
}

#if a directory path is not given in arguments, operate in the current directory.
if(!$directoryPath) {
    $directoryPath = '.';
}

#open directory and import json files to mongo
opendir my $dir, $directoryPath or die "Cannot open directory at path $directoryPath.";
my @files = readdir $dir;
importJSONToMongo(@files);
closedir $dir;

#subroutine that takes an array of json files and imports them to the given mongodb database
sub importJSONToMongo {
    foreach my $file (@_) {
        if($file =~ /.json/) { #only import json files - need to make this regex better (it would match *.metadata.json and other extraneous files)

        $file =~ /(^.+?)(?=\.)/; #capture the filename before the '.json' extension
        system("mongoimport -d $database -c $1 --jsonArray --file $directoryPath/$1.json");
        }
    }
}

How to Import Multiple JSON Files to Mongodb?

4 Answers4