0

I have been looking for information in Google and stackoverflow, but I dind't find a good solution.

I need to handle a list, add elements, delete elements... but saved in a file. This is in order to avoid losing the list when the execution finish, because I need to execute my python script periodically. Here are alternatives I found, but they have some problems

  • Shelve module: can't find how to delete a element in the list (such as list.pop() ) instead of deleting all the list.
  • pprint.pformat() : to modify information, I need to delete all the document and save the modifed information, very inefficient.
  • json: tediuos for just a list and doesn't seems to solve my problem

So, which is the best way to handle a list, doing things as easy as mylist.pop() keeping the changes in a file in an efficient way?

southernKid33
  • 351
  • 1
  • 2
  • 14
  • 1
    You may try `pickle`, but if the modifications are frequent, perhaps you should use a database, not a file. – DYZ Apr 05 '19 at 20:35
  • That's the problem, modifications as frequent as 10 per hour. I dind't think about using a database, nice point – southernKid33 Apr 05 '19 at 20:38
  • open file, read in list, manipulate list, save list in file. depending on size of file takes couple ms or longer. See https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files - you can store your list using the csv-module if you like - that handles excaping textstrings as well. – Patrick Artner Apr 05 '19 at 20:59
  • ps: - keeping things in a file and performat are contradictionary. Files are streams, you need to seek, IO is slow to begin with. 10 times a minute an hour for what size of lists are you talking about? 1k items? 100k items? millions? – Patrick Artner Apr 05 '19 at 21:07
  • Not to much, 30k items. I will read list, modify it and save it again, chaging my code to do that only once per day. Using a DB will be complicate the project more thant it's necessary. Thanks everybody , I think I can close the question, as all alternatives have been exposed, with pros and cons – southernKid33 Apr 06 '19 at 09:12

1 Answers1

0

Since this has never been answered before, here is an efficient way. The package pysos can handle disk backed lists with inserts/deletes in constant time.

pip install pysos

Code example:

import pysos

db = pysos.List('somefile')
db.append('saved in the file 0')
db.append('saved in the file 1')
db.append('saved in the file 2')

print(db[1]) # => 'saved in the file 1'

del db[1]

print(db[1]) # => 'saved in the file 2'
dagnelies
  • 5,203
  • 5
  • 38
  • 56