How to create log file and save all the running logs from Python script

Question

I have below part of Python code extraction.py where i have lot of print statements and there are other logs generated from REST API. I want to create logfile with date and time prefix attached with the log file for example log_yyyy-MM-dd-hhminss to track all the logs from my Python code. This will help to track daily logs from my Python code.

import requests
import json
import shutil
import time
import gzip
import os

extraction request:
if status_code == 202:
    requestUrl = r2.headers["location"]
    print('Extraction is not complete, we shall poll the location URL:')
    print(str(requestUrl))

    requestHeaders = {
        "Prefer": "respond-async",
        "Content-Type": "application/json",
        "Authorization": "token " + token
    }

while (status_code == 202):
    print('As we received a 202, we wait 30 seconds, then poll again (until we receive a 200)')
    time.sleep(30)
    r3 = requests.get(requestUrl, headers=requestHeaders)
    status_code = r3.status_code
    print('HTTP status of the response: ' + str(status_code))

Did you check e.g. https://docs.python.org/3/howto/logging.html? — buran, Sep 02 '21 at 08:42
thanks i just checked but how do we create logfile and track all the logs in this logfile at the end its not mentioned in the document ? — Symonds, Sep 02 '21 at 08:44
Does this link help better? https://docs.python.org/3/howto/logging.html#logging-to-a-file You need to learn how to work with the docs instead of asking such generic question on SO. — buran, Sep 02 '21 at 08:49

Shugla · Accepted Answer · 2021-09-03T14:26:59.000

2

You could create a function that does that for you and replace prints with that. Something along the lines of:

def log(msg):
    t = datetime.datetime.now()
    time = t.strftime("[%d.%m.%y] Time - %H_%M_%S")
    log_msg = str(msg)
    # or any other format (could even use time.ctime() or similar)
    print(log_msg)
    with open("path/" + time + ".log",'a+') as file:
        file.write(log_msg + "\n")

So your code would look like somthing like this:

import requests
import json
import shutil
import time
import gzip
import os
import datetime
def log(msg):
    t = datetime.datetime.now()
    time = t.strftime("[%d.%m.%y] Time - %H_%M_%S")
    log_msg = str(msg)
    print(log_msg)
    with open("path/" + time + ".log",'a+') as file:
        file.write(log_msg + "\n")


extraction request:
if status_code == 202:
    requestUrl = r2.headers["location"]
    log('Extraction is not complete, we shall poll the location URL:')
    log(str(requestUrl))

    requestHeaders = {
        "Prefer": "respond-async",
        "Content-Type": "application/json",
        "Authorization": "token " + token
    }

while (status_code == 202):
    log('As we received a 202, we wait 30 seconds, then poll again (until we receive a 200)')
    time.sleep(30)
    r3 = requests.get(requestUrl, headers=requestHeaders)
    status_code = r3.status_code
    log('HTTP status of the response: ' + str(status_code))

Also as buran pointed out, one should use some logging module that does the writing to file.

edited Sep 03 '21 at 14:26

answered Sep 02 '21 at 08:53

Shugla

36
1
5

i am getting error : log_msg = t.strftime("[%d.%m.%y] Time: %H:%M:%S - ") + msg TypeError: can only concatenate str (not "int") to str – Symonds Sep 02 '21 at 08:59
try casting t.strftime to str `log_msg = str(t.strftime(...)) – Shugla Sep 02 '21 at 09:01
Updated the code, had some bracket mismatches – Shugla Sep 02 '21 at 09:04
This is very naive solution, one should use the built-in logging module or number of third party logging packages. – buran Sep 02 '21 at 09:08
still getting error : log_msg = str(t.strftime("[%d.%m.%y] Time: %H:%M:%S - ")) + msg TypeError: can only concatenate str (not "int") to str – Symonds Sep 02 '21 at 09:09
Ideally you should have this function in some Logger class with some other functionality or as you've said, use some module to do the job. – Shugla Sep 02 '21 at 09:10
Maybe msg is int, try str(msg) – Shugla Sep 02 '21 at 09:11
yes now it working but the logfile which is created its not attached with datetime...actually the datetime is inside the log file which is not needed – Symonds Sep 02 '21 at 09:14
You could change "path/log" to "path/" + time + "log". "a+" mode will create file if it doesn't exist, and will append to file if it exists. – Shugla Sep 02 '21 at 09:17
i tried and getting this error : TypeError: can only concatenate str (not "builtin_function_or_method") to str – Symonds Sep 02 '21 at 09:23
it says : OSError: [Errno 22] Invalid argument: 'C:/Users/[02.09.21] Time: 11:24:58: log'...i think we need for example log_20210902_112731 so that it will not overwrite the file – Symonds Sep 02 '21 at 09:26
I'm not sure if name is problem, but you could try that. Also try adding r"path" instead of "path" for full path. ("path" currently is relative, r"path" should make it absolute) – Shugla Sep 02 '21 at 09:29
It should append to file (add new line) if file with name exists – Shugla Sep 02 '21 at 09:30
can you please update the code ..because i am getting same error when i tried..thanks – Symonds Sep 02 '21 at 09:32
Apparently windows sees ":" as problem in filename, replacing them should do the job. – Shugla Sep 02 '21 at 09:33

How to create log file and save all the running logs from Python script

1 Answers1