How to append new data in an existing .data file using pickle

Question

I have a .data file which I want to update with new data

Here is my python file named pic.py

import pickle
dataset = [['Foam', 'cake', 430], ['other', 'icecream', 50]]

# creating .data file for first time
outfile = open('database.data', 'wb')
pickle.dump(dataset, outfile)
outfile.close()

newdata = ['type','name','price']

with open('database.data','rb') as temp_data:
    temp_database = pickle.load(temp_data)

temp_database += newdata

with open('database.data','wb') as temp_data:
    pickle.dump(temp_database,temp_data)

now every time when I need to append new data in my .data first I'm loading my .data file using pickle to a temp_database variable and then append my newdata to that variable. After that I have used pickle.dump() method to export my all database that is "temp_database" to database.data file

my problem :>>> "Is there any short method to append new data in .data file using pickle without reading existing data -> append new data -> write and dump new data to existing file? "

https://stackoverflow.com/questions/12761991/how-to-use-append-with-pickle-in-python — Ajay Tom George, Aug 03 '20 at 05:28

score 0 · Answer 1 · answered Aug 03 '20 at 06:29

As long as you keep your temp_database in memory, there is no need to reload the contents of database.data each time. After applying some changes to temp_database you only need to write it back to database.data and hold a reference to temp_database somewhere to be able to modify it again.

For example something like:

import pickle

class DataAccess:
        
    def __init__(self, db_file_name, initial_data=None):
        self.db_file_name = db_file_name
        if initial_data:
            self.temp_database = initial_data
            self.safe_db()
        else:
            with open(self.db_file_name, "rb") as f:
                self.temp_database = pickle.load(f)
    
    def safe_db(self):
        with open(self.db_file_name, "w+b") as f:
            pickle.dump(self.temp_database, f)


# For initialisation with existing data:
# da = DataAccess("database.data", [['Foam', 'cake', 430], ['other', 'icecream', 50]])
da = DataAccess("database.data")

...

# And whenever you modify the database:
da.temp_database += ['some', 'changes']
da.safe_db()

...

da.temp_database += ['another', 'change']
da.safe_db()

Of course, you should add proper error handling.

You may also want to look into pickle.Pickler

If you are concerned about memory issues than you may want to use a real database like SQLite or MongoDB instead.

thanks for your valuable answer but this approch is also doing same things first we are initializing copy of our database as f (like your code > `self.temp_database = pickle.load(f)`) and after some changes in temp_database " f " we "!! overwrite !! " our existing database with pickle.dump that becomes our final self_db — VIJAY, Aug 03 '20 at 09:58
is there any method like append as in lists to extend our .data file without always copy-edit-rewriting whole file — VIJAY, Aug 03 '20 at 10:07

How to append new data in an existing .data file using pickle

1 Answers1