9

Following is sample code, aim is just to merges text files from give folder and it's sub folder. i am getting Traceback occasionally so not sure where to look. also need some help to enhance the code to prevent blank line being merge & to display no lines in merged/master file. Probably it's good idea to before merging file, some cleanup should performed or just to ignores blank line during merging process.

Text file in folder is not more then 1000 lines but aggregate master file could cross 10000+ lines very easily.

import os
root = 'C:\\Dropbox\\ans7i\\'
files = [(path,f) for path,_,file_list in os.walk(root) for f in file_list]
out_file = open('C:\\Dropbox\\Python\\master.txt','w')
for path,f_name in files:
    in_file = open('%s/%s'%(path,f_name), 'r')

    # write out root/path/to/file (space) file_contents
    for line in in_file:
        out_file.write('%s/%s %s'%(path,f_name,line))
    in_file.close()

    # enter new line after each file
    out_file.write('\n')

with open('master.txt', 'r') as f:
  lines = f.readlines()
with open('master.txt', 'w') as f:
  f.write("".join(L for L in lines if L.strip())) 



Traceback (most recent call last):
  File "C:\Dropbox\Python\master.py", line 9, in <module> for line in in_file:
  File "C:\PYTHON32\LIB\encodings\cp1252.py", line  23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0]  
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 972: character maps to <undefined>
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
user1582596
  • 503
  • 2
  • 5
  • 16

3 Answers3

20

The error is thrown because Python 3 opens your files with a default encoding that doesn't match the contents.

If all you are doing is copying file contents, you'd be better off using the shutil.copyfileobj() function together with opening the files in binary mode. That way you avoid encoding issues altogether (as long as all your source files are the same encoding of course, so you don't end up with a target file with mixed encodings):

import shutil
import os.path

with open('C:\\Dropbox\\Python\\master.txt','wb') as output:
    for path, f_name in files:
        with open(os.path.join(path, f_name), 'rb') as input:
            shutil.copyfileobj(input, output)
        output.write(b'\n') # insert extra newline between files

I've cleaned up the code a little to use context managers (so your files get closed automatically when done) and to use os.path to create the full path for your files.

If you do need to process your input line by line you'll need to tell Python what encoding to expect, so it can decode the file contents to python string objects:

open(path, mode, encoding='UTF8')

Note that this requires you to know up front what encoding the files use.

Read up on the Python Unicode HOWTO if you have further questions about python 3, files and encodings.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • My test files very basic in nature, but sometimes there are pasted content and looks like it's creating problem. – user1582596 Aug 31 '12 at 10:28
  • Just tired with above 1st code, and now getting some precise error, it says TypeError: invalid file: and tried to point out to file. but strange thing is that, this is just a blank file. – user1582596 Aug 31 '12 at 10:40
  • 1
    @user1582596: can you use pastie.org to show me the traceback? – Martijn Pieters Aug 31 '12 at 10:42
  • Also, the `copyfileobj` function implementation is exceedingly simple; all it does is read a buffer-size amount from the input, and writes that to the output, until all of the input file has been read. – Martijn Pieters Aug 31 '12 at 10:45
  • 10x, here is the latest code with traceback, as a test, deleted .txt file and but next time same error returns for another text file. http://pastie.org/4630939 – user1582596 Aug 31 '12 at 10:51
  • @user1582596: Ah, my mistake, updated the example. The `os.path.join()` function takes multiple args, not a tuple. – Martijn Pieters Aug 31 '12 at 10:55
  • Sorry, use `.write(b'\n')` instead; we need to write bytes, not python (unicode) strings as the file is open in binary mode. Answer adjusted. – Martijn Pieters Aug 31 '12 at 11:04
  • 1
    @user1582596: that's the code below my changes, where you read all lines in the master file to strip them. You'll still will have to figure out an encoding for those, or devise a way to read and write in binary mode where you remove all whitespace around newlines. Also see the second half of my answer for that part. Dealing with stripping lines in binary mode would be an interesting *new* question on Stack Overflow perhaps. – Martijn Pieters Aug 31 '12 at 11:13
  • is there any way to determine, which text file is creating problem. may be i will try to rectify the issue manually and re-run the program. – user1582596 Aug 31 '12 at 15:46
  • 1
    @user1582596: you can do that by looping over all the filenames, printing the filename, then opening the file in text mode (`open(filename, 'r')`, looping over the file to read all the lines, until you get the exception. The last filename printed is your problem file. – Martijn Pieters Aug 31 '12 at 15:49
  • "print (in_file)" did the tweak, found issue with two text files and rectified manually. many 10x for your time on recent helps. Hats off for your knowledge and sharing knowledge spirit. – user1582596 Aug 31 '12 at 18:08
5

I faced the similar issue while removing the file using os module remove function.

The required changes i performed is:

file = open(filename)

to

file = open(filename, encoding="utf8")

Add an encoding=“utf-8”

UTF-8 is one of the most commonly used encodings, and Python often defaults to using it. UTF stands for “Unicode Transformation Format”, and the '8' means that 8-bit values are used in the encoding. ... UTF-8 uses the following rules: If the code point is < 128, it's represented by the corresponding byte value.

Flair
  • 2,609
  • 1
  • 29
  • 41
Mani Singh
  • 51
  • 1
  • 2
-2

Handling import and decode error with file handling

  1. Open file with full absolute path

(source - absolute path for directory of file folder, getting all files inside file_folder)

import os
file_list = os.listdir(source)
for file in file_list:
    absolute_file_path = os.path.join(source,file)    
    file = open(absolute_file_path)
  1. Encoding the file as we open

file = open(absolute_file_path, mode, encoding, errors=ignore)

dataninsight
  • 1,069
  • 6
  • 13