1

I'm having a strange issue which causes my terminal's encoding go nuts, and kinda requires me to restart my IDE every time it happens (I'm using PhpStorm 2017.1.2 on Mac OS X 10.12.4). This project has a lot of dependency on this little parsing function of mine and up until now it worked like a charm, but when I tried to parse a big csv file (around 150k records), and tried to console.log every row, it starts to 'glitch' around and make my whole terminal (within the ide) totally unreadable.

The CSV file is formatted like this -

Token , Ad_Group
N000000000089076 , BCZ MY - Keyword 1
N000000000090445 , BCZ SG - Keyword 3
N000000000089102 , BCZ MY - Keyword 47
N000000000090115 , BCZ SG - Keyword 33 [Exact]
N000000000087801 , BCZ AU - Keyword 12 - [Exact]
N000000000088111 , BCZ CA - Keyword 1 - Phrase
N000000000090795 , BCZ UK - Kyeword 89 - Phrase
..and so on

My parsing function looks like this -

exports.csv_tokens_parse = function(file_name) {
 console.log('starting to parse '+file_name+'...');

 Papa.parse(fs.readFileSync(`../dashboard/reports/${file_name}`, {encoding: 'binary'}), 
 {
    step: function (row) {
      console.log(row.data[0][1]); //displaying the ad_groups column
    }
 });
 return file_arr;
};

After running it and going back to the phpstorm's console, i can see that it prints out the first few thousands lines properly, but then something weird happens, and it changes its' printing encoding to something weird and changes my whole shell into it (a screenshot is attached for a better clarification)

messed up encoding in terminal

The screenshot is after i hit ctrl+c to stop the console.log loop and get back to the terminal.. but my typing encoding has changed as well, and you can see me trying to type a simple 'ls' in it.

Gonras Karols
  • 1,150
  • 10
  • 30

1 Answers1

0

Maybe I didn't understand your problem very well, but maybe your problem comes from the encoding :

Papa.parse(fs.readFileSync(../dashboard/reports/${file_name}, {encoding: 'utf8'}),

If this answer doesn't work, can you put more information about the content of the file?

Amine
  • 349
  • 2
  • 16
  • I tried switching to utf8 as well, but unfortunately it didn't help.. There's no more information about the contents than what i've already posted here. All the lines look the same as above, formatted in two columns with a token on the first one and the name of the ad group on the second one - Maybe there are some ad group names that contain special characters like spanish letters, but it didn't have any effect on my previous parsings, with the same contents. – Gonras Karols Apr 24 '17 at 15:20
  • Can you add the line where things start to go crazy? – Amine Apr 24 '17 at 15:22
  • That's the problem.. I can't really track it down, since it's changing the already logged text to gibberish as well..Is there an option to log it all into a text file using nodemon? – Gonras Karols Apr 24 '17 at 15:33
  • Can you try to make the bug happen again on a simple file? (you would have to register whatever data you are trying to parse on a file though) – Amine Apr 24 '17 at 15:39
  • Well, the problem was a piece of text in thai (ปิดบัญชี). Still don't know exactly how to handle these with PapaParse. – Gonras Karols Apr 24 '17 at 21:38