-1

I have .txt file which was generated by my application , which we send to Bank every day as part of Payment Processing , but recently Bank observed that in Non prod environment we are sending special Character in 1st row and 1st column itself (file contains three characters in the beginning of the file) , like below

enter image description here Newley added Rows enter image description here

But when we open the file and verify we don't see any as such special character in the file, so could you please suggest me, do we have any tool to check the special character in the .txt file ? your help will be much appreciated

I am in windows OS

mjwills
  • 23,389
  • 6
  • 40
  • 63
Manas
  • 35
  • 6
  • Ask them to send you the file, and compare it to the file you _think_ you sent them. – mjwills Jul 25 '21 at 22:41
  • 2
    Sounds like a BOM. Open it with Notepad++ and it should tell you what type. You could also use hex editor like HxD or write some code to dump out the bytes. I'd guess utf-8, it's the only common one that is 3 bytes. https://en.wikipedia.org/wiki/Byte_order_mark – Retired Ninja Jul 25 '21 at 22:41
  • Please share a [mcve]. – mjwills Jul 25 '21 at 22:42
  • Thank you mjwills - Yes they are also not able to see the special character when they open the File, but their File Parser software is detecting that their is a special character in our file – Manas Jul 25 '21 at 22:59
  • Almost certainly a BOM then. – mjwills Jul 25 '21 at 23:02
  • So are you suggesting me to go for Encoding.UTF32.GetBytes() ? Currently I have Encoding.UTF8.GetBytes() in my code – Manas Jul 25 '21 at 23:04
  • Even in NotePad ++ there is no clue - I am not able to see the special character – Manas Jul 25 '21 at 23:05
  • https://www.google.com/search?q=how+to+see+BOM+in+file – mjwills Jul 25 '21 at 23:08
  • @Manas Notepad++ doesn't show you the characters, but in the lower right it will show the encoding, like UTF-8-BOM. You can also see the encoding in the Encoding menu. – Retired Ninja Jul 25 '21 at 23:28
  • mjwills - I followed the same steps which is mentioned here https://www.ibm.com/support/pages/how-remove-bom-any-textxml-file , Still I don't see the special characters (o;?) in my file – Manas Jul 26 '21 at 00:12
  • Retired Ninja - I can see that in my NotePad ++ in the lower right corner, it is - UTF-8-BOM , but question how I can see the special characters (o;?) in the file – Manas Jul 26 '21 at 00:14
  • To see the leading bytes, use a Hex viewer/editor, or use `File.ReadAllBytes()` and inspect the first 3 bytes. But you can just save the file as UTF-8 NO-BOM. This *version* can be saved using standard .Net methods (e.g., `File.WriteAllText([Path], [Content])` or `new StreamWriter([Path]` etc.) . – Jimi Jul 26 '21 at 02:42
  • It's kind of weird that a procedure that imports UTF-8 encoded files cannot handle the BOM, though. – Jimi Jul 26 '21 at 02:49
  • Jimi - Thanks for the response - I have opened the File in NotePad ++ and then in Encoding Menu I have changed to UTF-8 instead of UTF-8 BOM, But I still not able to see the special Characters and even I have saved the file – Manas Jul 26 '21 at 12:07
  • UTF-8 without BOM doesn't have any *special bytes* anymore, so you cannot *see* anything after you change format. As mentioned, you need a hex/byte viewer (or an advanced editor as UltraEdit ,which lets you switch text/structure/Hex view on the fly). Normal editors don't show the BOM (those bytes are not something you're supposed to *edit*). – Jimi Jul 26 '21 at 13:47

1 Answers1

0

Sounds like this could be because of a mismatch in text encoding that you are using when you create the file vs. what is being expected by the receiver of this file.

SwaroSh
  • 131
  • 4
  • Hi SwaroSh, Thanks for your response, Basically its an existing .txt file and application since 10 years. We only added new row to that .txt File and now Bank Processer (3rd Party) its saying we are sending special character at the beginning of the File, But we are sure we are not sending, so How do I verify from my end in the .txt file that I have special character ? – Manas Jul 25 '21 at 22:36
  • `We only added new row to that .txt File` Show us the file. Particularly that first row. – mjwills Jul 25 '21 at 22:40
  • When you added the new row, are you using one of .NET's built-in encoding functions (e.g. shown below)? Instead, if the data that you are writing is from a source whose encoding is unknown, it could lead to the issue you are describing. `var file = File.Create("test.txt");` `var bytes = Encoding.UTF8.GetBytes("Hello world");` `file.Write(bytes, 0, bytes.Length);` – SwaroSh Jul 25 '21 at 22:56
  • I have added new image which has - newly added rows information – Manas Jul 25 '21 at 22:57
  • Newly added rows used the same way how it was handled earlier - Encoding.UTF8.GetBytes() – Manas Jul 25 '21 at 22:58