0

I have around 6 million rows in my mongodb collection and importing into meilisearch using php artisan scout:import 'model' takes forever to finish.
Importing data with limit option php artisan scout:import 'model' -c 10000 gives me the following error.

MongoDB\Exception\InvalidArgumentException Expected "limit" option to have type "integer" but found "string" at vendor/mongodb/mongodb/src/Exception/InvalidArgumentException.php:59 55▕ 56▕ $expectedType = $typeString; 57▕ } 58▕ ➜ 59▕ return new static(sprintf('Expected %s to have type "%s" but found >"%s"', $name, $expectedType, get_debug_type($value))); 60▕ } 61▕ } 62▕

+27 vendor frames 28 artisan:37 Illuminate\Foundation\Console\Kernel::handle()

I also tried exporting the collection as json from mongodb and manual importing into meilisearch using curl -X POST 'http://127.0.0.1:7700/indexs/posts/documents' / --data @/data/posts.json gives the following error.

{"message":"Invalid JSON: invalid type: map, expected a sequence at line 1 column 0","errorCode":"bad_request","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#bad_request"}curl: (3) URL using bad/illegal format or missing URL Posts.json is the exported json file of mongodb collection using mongoexport command.

How can I import data fast into meilisearch?

Versions

  • "laravel/scout":"^9.1"
  • "laravel/framework": "^8.12",
  • "meilisearch/meilisearch-php": "^0.18.2",
  • mongodb version : "3.6"

OS

  • Ubuntu 20.04
Rhys
  • 67
  • 1
  • 7
  • Did you do you `mongoexport` with the --jsonArray` flag? If not that might help. I don't know lavavel scout so there very well may be a better option that lets you do it directly without curl, but hopefully this helps at lets get started with the curl option. – paul41 Jun 25 '21 at 13:23
  • @paul41 I did `mongoexport` with `--jsonArray` flag but when I inspect the json file, there is "_id" field and the field has another attribute `$oid` assigned to value like `{"_id":{"$oid":'1234568'}}`. There must be primary field which is **default** `id` field to import into meilisearch. – Rhys Jun 26 '21 at 06:35
  • @paul41 I can't set nested attribute "$oid" as primary key in meilisearch. I run the following command `curl -X POST "http://127.0.0.1:7700/indexes" --data '{"uid":"posts","primaryKey":"_id.$oid"}'` but it didn't work. There is no document inside posts index in meilisearch. – Rhys Jun 26 '21 at 06:53
  • 1
    The `id` field can't be nested in MeiliSearch and I don't think there is a way to have `mongoexport` not include the `oid` nesting since that is specifying it is an object id. The two options I can think of are to use a program to un-nest the `_id` field after export, or if there is another unique field in the document you could tell MeiliSearch to use that field instead. – paul41 Jun 26 '21 at 12:55
  • @paul41 using another unique field as primary key works! Thank you :) – Rhys Jun 26 '21 at 14:32

0 Answers0