Creating new files on heroku while app is working?

Question

I have a python telegram bot and I have deployed it on Heroku. But problem is that my program actually creates pickled files while working. I have one database which is saving the required data and pickles to save some nested classes which I have to use later at some stage. So these pickle files are one of the important parts of the program. I am using the dill module for pickling. I was able to save these files locally, but can't do when I am doing it in Heroku. I'll share the logs below. It is not even reaching the pickling part, but giving an error in opening the file itself.

import dill


def saving_test(test_path, test_obj):
    try:
        save_test_logger.info('Saving test...')
        try:
            save_test_logger.info('opening file')
            test_file = open(test_path, 'wb')
        except Exception as exc:
            save_test_logger.exception('Error opening file')
            return 0

        dill.dump(test_obj, test_file)
        save_test_logger.debug(f'file saved in {test_path}')
        test_file.close()
        return 1
    except Exception as exc:
        save_test_logger.exception('saving error')
        test_file.close()
        return exc

saving 859ab1303bcd4a65805e364a989ac8ca
2020-10-07T20:53:18.064670+00:00 app[web.1]: Could not open file ./test_objs/859ab1303bcd4a65805e364a989ac8ca.pkl

And I have added logging also to my program, but now I am confused about where can I see the original logs which are supposed to catch the exceptions also. This is my first time using Heroku and I am comparatively new to programming also. So please help me out here to identify the route cause of the problem.

score 1 · Accepted Answer · answered Oct 08 '20 at 05:56

1

I found the problem. Even though I have pushed everything, directory test_objs wasn't there on Heroku. I just added 2 more lines of code to make a directory using os module if the directory does not exist. That solved the problem. I am not deleting the question so that just in case someone gets stuck or confused in a similar kind of situation, this question might be able to help them.

answered Oct 08 '20 at 05:56

Jihjohn

398
4
19

Another thing, heroku doesn't allow for creating of permenant files within heroku itself. If you need that, you'll have to save them in Amazon S3 or something. No issue if you create the file in development commit then push it to heroku. – Nuclearman Oct 08 '20 at 06:34
@Nuclearman I am facing this issue rn. My pickles are getting deleted each time the dyno is getting restarted. Could you please tell me a workaround on that? I am pickling nested class objects for future use. I was actually going to add a new question about the same – Jihjohn Oct 08 '20 at 08:43
@Nuclearman https://stackoverflow.com/questions/64258955/how-to-create-permanent-files-on-heroku checkout this question please – Jihjohn Oct 08 '20 at 08:50

Creating new files on heroku while app is working?

1 Answers1