Why does ?? appear when I copy one text file to another?

Question

When I copy one text file to another the new file has the two characters: (??) at the end which I do not want.

I am using Python3.6.0 on Windows7

This is my script:

from sys import argv
script, from_file, to_file = argv

#Open from_file and get the text from it
indata = open(from_file).read()

#Write the from_file text to to_file
open(to_file, 'w').write(indata)

I run the following in PowerShell:

>echo "This is a test file." > TestSource.txt
>type TestSource.txt
This is a test file.
>python CopyFile.py TestSource.txt TestDestination.txt
>type TestDestination.txt
This is a test file.??

Why do two question marks (??) appear in the file I have created?

Edit: This Related Question was suggested as duplicate. My question is about how Python behaves when I copy one text file to another. Where as this related question is about how Windows PowerShell creates a text file.

Welcome to SO. i have a suspicion that that last two characters aren't question marks, but something with encoding. can you check what their values are? also try saving the file with ascii encoding , `open(to_file, 'w', encoding='ascii').write(indata)` — Nullman, Dec 30 '18 at 08:55
How do I check what their values are? In notepad I see two boxes ਍ഀ. — FergM, Dec 30 '18 at 09:04
Thank you @Nullman. I tried `open(to_file, 'w', encoding='ascii').write(indata)`. It gives the error: *UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range (128).* — FergM, Dec 30 '18 at 09:12
Possible duplicate of [echo "string" > file in Windows PowerShell appends non-printable character to the file](https://stackoverflow.com/questions/8163922/echo-string-file-in-windows-powershell-appends-non-printable-character-to-th) — TheNavigat, Dec 30 '18 at 09:25
Apparently PowerShell is creating an UTF-16 file. Try to use `encoding='utf-16'`. Or change your powershell script to write an actual ASCII/utf-8 file. — Bakuriu, Dec 30 '18 at 09:59

Mark Tolonen · Accepted Answer · 2018-12-30T10:29:20.173

Powershell is creating the file using UTF-16. You have opened the file in text mode (default) without specifying an encoding, so python calls locale.getpreferredencoding(False) and uses that encoding (cp1252 on my US Windows system).

Text mode translates line endings and using the wrong encoding creates problems. To fix this, use binary mode to get a byte-for-byte copy regardless of encoding. I also suggest using with to ensure the files are closed properly:

from sys import argv
script, from_file, to_file = argv

#Open from_file and get the text from it
with open(from_file,'rb') as f:
    data = f.read()

#Write the from_file text to to_file
with open(to_file,'wb') as f:
    f.write(data)

Why does ?? appear when I copy one text file to another?

1 Answers1