0

I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my mongo.

I found the s3cmd tool that can transfer my files to the S3 instance (DigitalOcean space), but I can't find a way to actually access those files. I know they are stored in the fs.files and fs.chunks collections, but I can't figure out how to actually use them as input to s3cmd!

Any help will be much appreciated!!!

EDIT: This is the format the images are being stored with:

fs.files

{
  "_id" : ObjectId("5ae97922c1dabec8d2d0bdb0"),
  "filename" : "2b57455f3878d11dabc9c984da7de314_postImage.jpeg",
  "contentType" : "binary/octet-stream",
  "length" : 2291623,
  "chunkSize" : 261120,
  "uploadDate" : ISODate("2018-05-02T08:38:58.549Z"),
  "aliases" : null,
  "metadata" : null,
  "md5" : "9ad420eaa7c28a73e449199430627802"
}

fs.chunks.findOne()

{
  "_id" : ObjectId("5ae2d77f6616b4a9d93cb4b1"),
  "files_id" : ObjectId("5ae2d77f6616b4a9d93cb4b0"),
  "n" : 0,
  "data" : BinData(0,"iVBORw0KGgoAAAANSUhEUgAAAuAAAAJvCAYAAAA6OGQEAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAABWWlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRher38tcAAAAASUVORK5CYII=") }
Sotiris Kaniras
  • 198
  • 2
  • 10

1 Answers1

1

You will have to write a script or program to retrieve files from MongoDB and copy them to S3.

Most probably files in MongoDB are stored in BSON format, so they require to be converted first.

As you have noted that you are using Parse - try these utils to extract images.

https://github.com/parse-server-modules/parse-files-utils

Martynas Saint
  • 1,221
  • 7
  • 15
  • Exactly what I'm looking for! How do I convert them? – Sotiris Kaniras Dec 12 '19 at 20:53
  • Write a program to export from MongoDB to a file system - exactly how depends how you stored things in Mongo, but I doubt you'll find a pre-written script that can do this for you as few people store binary resources in a database. From there use S3 command line tools (eg aws s3 sync) to upload them to S3, or drag and drop. – Tim Dec 12 '19 at 20:56
  • That's what I don't know how to do! I mean it's not as simple as exporting a collection! I found this script, but it doesn't seem to work http://blog.vladimirm.com/2011/06/export-files-from-mongodb-gridfs-with-directory-paths/ Should I be looking for something like this though? – Sotiris Kaniras Dec 12 '19 at 21:11
  • Talk to the developer (of the app) – Martynas Saint Dec 12 '19 at 21:12
  • I'm the developer :P We are a small startup and I work the frontend and the backend! I'm lost – Sotiris Kaniras Dec 12 '19 at 21:30
  • So how did you program the app to put files to the MongoDB in the first place¿ – Martynas Saint Dec 12 '19 at 21:33
  • I'm using `Parse Server` for the app and it handles the uploading! With some digging, I found that this is the format the images are stored... check the edit of the question – Sotiris Kaniras Dec 12 '19 at 21:57
  • 1
    Try this - https://github.com/parse-server-modules/parse-files-utils – Martynas Saint Dec 12 '19 at 22:05
  • There's also a `mongofiles` command-line tool (https://docs.mongodb.com/manual/reference/program/mongofiles/) included with MongoDB server tools. However, `parse-files-utils` looks like a better fit given the context of Parse (which wasn't in the original question). – Stennie Dec 13 '19 at 00:13