Rails Rake Task How to parse CSV with commas in fields

Question

I have a csv, that contains float numbers with commas except of dots like this "34,21", and I need to parse it in my rake task, I have already tried some solutions like this: Ruby on Rails - Import Data from a CSV file

But none of them doesn't seem to be working properly, they just parse it like 2 fields (32 and 21). Is there a way to fix it with using built-in CSV?

I have already tried this:

task :drugimport, [:filename, :model] => :environment do |task,args|
    CSV.foreach(args[:filename], { encoding: "UTF-8", headers: true, header_converters: :symbol,
        converters: :all}) do |row|
            Moulding.create!(row.to_hash)
        end
end

And this one:

require 'smarter_csv'
options = {}
SmarterCSV.process('input_file.csv', options} do |chunk|
   chunk.each do |data_hash|
       Moulding.create!( data_hash )
   end
end

They both look nice and elegant, except of wrong parsing of fields containing commas.

here is my rows, sorry there is russian, but whatever: http://pastebin.com/RbC4SVzz I didn't changed anything in it, so I pasted to pastebin, will be more useful then here, I guess

here is my import log: http://pastebin.com/rzC0h9rS

Answer 1

In my opinion, you have three possible roads you could go:

1) work with the "bad" input and try to find a workaround

You could try and work line by line and try

line.split (" ,")

which would assume that there is a blank space before the comma. Another approach would be to identify the numerical values via regex and replacing the comma character (this might be easier to fix on the source data!)

2) try to export the CSV with another separator

This depends on where the data comes from. If you can re-export the data, maybe that's the most easy solution. In this case of course, your data would technically not be CSV anymore, but for example SSV (semi-colon-separated values).

3) try other CSV parsers

I can definitely suggest you take a look at other CSV parsers, such as fasterCSV and others (see a list of CSV parsers at ruby-toolbox)

I hope this is helpful advice - sample CSV data would definitely help to help you.

Answer 2

Right, so from what I am seeing you are, as you understand yourself, not passing any options to the parser. When not indicating row_sep or any other form of option, smarter_csv will use the system new line separator which is "\r\n" for windows machines, and "\r" for unix machines.

That being said, try the following...

require 'smarter_csv'
SmarterCSV.process('input_file.csv', :row_sep => :auto, :row_sep => ","} do |chunk|
  chunk.each do |data_hash|
    Moulding.create!( data_hash )
  end
end

I agree with Swards. What I have done assumes quite a lot of things. A glance at some CSV data could be useful.