-2

I thinking about centralized storage of text files with different metadata and content settings(unique lines, key:value lines) but still don't know which technology to use - sql db like PostgreSQL or NoSQL solutions.

Large files: 100 - 600 mb each, small queries to read/write 100 - 500 lines

Any hint ?

Jhon
  • 433
  • 4
  • 19
  • 1
    I voted to close this question because it is way too open-ended. @Jhon, perhaps you should delete this question and write a more detailed description of what you are trying to do. – Gordon Linoff Dec 27 '12 at 17:05
  • Please, provide more details. What kind of system is that? Is it something large? What kind of queries will be used? What information will be stored? Maybe MongoDB would be good for your needs. – shark555 Dec 27 '12 at 17:06
  • @GordonLinoff I wouldn't recommend deleting the question, but rather improving this one. Deleting questions is the road to getting question banned. – Conrad Frix Dec 28 '12 at 21:21
  • @ConradFrix . . . Once someone has answered the question, it becomes rather unfair to change the question. All of a sudden, the answer might seem inappropriate, because the question changed. That's why I recommend asking new questions rather than editing the existing one, when the changes are so significant they would invalidate any answers. – Gordon Linoff Dec 28 '12 at 21:24
  • @GordonLinoff I don't disagree but that having an answer invalidated is bad thing but doesn't having your answer deleted worse? – Conrad Frix Dec 28 '12 at 21:28

1 Answers1

2

Really, the choice between SQL and NoSQL systems depends on what kind of system you're running. SQL is relatively expensive compared to most NoSQL systems because it provides all the ACID guarantees -- Atomicity, Consistency, Integrity, and Durability. These are important guarantees to maintain consistency of your data if you actually require consistent data. If you don't require consistent data (e.g. you are a caching solution or you are Twitter) then NoSQL systems' efficiency becomes much more attractive.

For your specific use case; it doesn't sound like there are many solutions that are going to help you. Modifying the middle of a text file is inherently going to require (at least) rewriting the entire portion of the text file after the edit point out to disk (assuming you actually want the files to be plain text on disk).

You might be able to build a system on top of SQL or NoSQL which represents text files as lines or chunks of lines, and be able to operate on them in a row-oriented manner. But even that type of system is likely to be inefficient for files as large as 100-600MB. Consider storing the files themselves as some kind of structured data in SQL; and then regenerating the files on demand when a user requests the full text file.

Billy ONeal
  • 104,103
  • 58
  • 317
  • 552