3

I want to write kind of a log file back to azure adls gen1 I can write (not append) using

dbutils.fs.put(filename,"random text")

but i cant append it using

with open("/dbfs/mnt/filename.txt","a"):
f.write("random text")

it give me error

1 with  open("/dbfs/mnt/filename.txt", "a") as f:
----> 2   f.write("append values")

OSError: [Errno 95] Operation not supported

alternatively, i tried using logger.basicconfig(logging.basicConfig(filename='dbfs:/mnt/filename.txt', filemode='w')

but looks like its not writing into the path. can anyone help please

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42

3 Answers3

2

Append Only (‘a’) : Open the file for writing. The file is created if it does not exist. The handle is positioned at the end of the file. The data being written will be inserted at the end, after the existing data.

file = open("myfile.txt","a")#append mode 
file.write("Today \n") 

enter image description here

Output of append file:

enter image description here

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
  • If it is useful for you, could you please [accept it as an answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work)? It may help more people who have similar issue. – CHEEKATLAPRADEEP Nov 06 '20 at 04:21
  • Can you show a screenshot of the file's contents? I think that would confirm if it has gotten appended or not? – iratelilkid May 31 '21 at 15:23
  • 1
    @iratelilkid Please find the screenshot of the file content. – CHEEKATLAPRADEEP Jun 01 '21 at 03:06
  • This is not helping, in Databricks you will still have the error OSError: [Errno 95] Operation not supported if trying to append a new line to an existing file. – Carlo Dec 08 '22 at 11:25
1

You can do that to a DBFS file. https://kb.databricks.com/en_US/dbfs/errno95-operation-not-supported

You may need to figure out a logic to read a file from datalake using python CLI and write to it.

nullptr
  • 3,701
  • 2
  • 16
  • 40
Pradeep
  • 11
  • 2
0

I had the same issue. Turns out you can't use standard python file operations on databricks because of the file system (DBFS). You end up getting the error [Errno 95] Operation not supported. You have to use dbutils to do it and I've found that you can't append. The below code is how I've gotten around it. Note that you need to use dbfs:/ file path when using dbutils.fs operations.

dbfs_path_home = "dbfs:/mnt/directory
dbfs_path_log = f"{dbfs_path_home}/logfile.log"

Then when you want to log something, you will need to read in the entire contents of the log file and add the new message. E.g.

def logging(message):
    logmsg = dbutils.fs.head(dbfs_path_log)
    logmsg += message + '\n'
    dbutils.fs.put(dbfs_path_log, logmsg, overwrite=True)
Nemps
  • 21
  • 4