2

I've seen some code read large data from mat files instead of doing queries on a database. What are the benefits of doing this as oppose to using a database? Is it possible to easily move the mat file contents into a database and vice versa?

Amro
  • 123,847
  • 25
  • 243
  • 454
mugetsu
  • 4,228
  • 9
  • 50
  • 79

2 Answers2

5

Reading data from mat file, is also a "database" in which you read your data from file.
Eventually, you will have to implement queries by yourself, and take care of many other issues.
Also, it is not a scalable solution, which means that for a large amount of data, it won't work well.
Of course, if you have small amount of data, and only basic queries, the fuss of setting up a database, using SQL isn't worth it.

Regarding your second question, it really depends on the data you have there.

Andrey Rubshtein
  • 20,795
  • 11
  • 69
  • 104
  • @mugetsu: I would argue that the overhead is much lower using mat files for sufficiently small or well-ordered data sets that are trivial to query. – Jonas Dec 14 '11 at 19:39
  • 1
    @Jonas I'm dealing with large mat files that contain images, if you have heard of VOC image or Caltech101 images, thats what I'm using. The table structure is pretty straightforward, but there can be hundreds of thousands of rows. So would a mat be better? – mugetsu Dec 14 '11 at 19:54
  • @mugetsu: If I recall correctly, the Caltech101 set only has a few thousand images, and a handful of annotations. Here, you could just load the annotations into memory, and use them to identify which image to load. Especially with the new matIO interface, it might be a lot easier to not do a database. Databases are, however, very useful if you also need to add more entries, if the annotations cannot easily be stored in a varable in memory because there are so many entries or so many annotations. – Jonas Dec 14 '11 at 20:17
4

I agree with Andrey. It depends on the data and what you want to do with it. I created a small program in Matlab that queries a relatively small .mat database but as the database and users grew performance has been going down.

In the light of this we decided to use a MySQL database. I created a small java application that talks to the database and imported that into Matlab to move data between Matlab and MySQL. But I had to create specific queries for my data. If someone can bring me a better solution I would be grateful.

Perhaps it wouldn't be such a bad idea to generate a general script that moves data between .mat data between Matlab and a SQL database. Store the data in a structure and use that to create the tables.

If you want to discuss something like this further via email I would be happy to. Maybe we can learn a thing or two from each other.

Stefan Gretar
  • 178
  • 1
  • 10
  • 2
    Just a wild guess, try to convert mat to XML, and then to import it using some standard database tool. For example, Microsoft Access. – Andrey Rubshtein Dec 14 '11 at 19:37
  • I am going to have to try this! I have some large .MAT files that I am converting to a SQLite DB and its been a pain as I'm not a real MATLAB guy so I had to build a parser in Python to handle the many MANY nested structs (damn you MATLAB!). Any way I parsed it with Python and then saved out via pandas to the DB. – Matt Camp Jun 04 '17 at 13:02