1

We just found out that CSV export cut off text in one field. Here is the original text for CSV export (from a text field of postgres 9.3):

@payable_approved_unpaid, @payable_paid, @payable_po_unpaid = {}, {}, {}
models.each do |m|
     @payable_po_unpaid[m.id.to_s] = PurchaseOrderx::Order.where(project_id: m.id).sum('po_total') - PaymentRequestx::PaymentRequest.where(project_id: m.id).where(resource_string: 'purchase_orderx/orders').sum('amount')
     @payable_paid[m.id.to_s] = PaymentRequestx::PaymentRequest.where(project_id: m.id).where(wf_state: :paid).sum('amount')
     @payable_approved_unpaid[m.id.to_s] = PaymentRequestx::PaymentRequest.where(project_id: m.id).where('approved = ? AND wf_state != ?', true, :paid).sum('amount')
end

Here is what we get from CSV:

@payable_approved_unpaid, @payable_paid, @payable_po_unpaid = {}, {}, {}

models.each do |m|

     @payable_po_unpaid[m.id.to_s] = PurchaseOrderx::Order.where(project_id: m.id).sum('po_total') - PaymentRequestx::PaymentRequest.where(project_id: m.id).wher

In the same file, there are fields which is much long than this and there is no problem. We have been using the export for a long time and this is first time we are losing text. What could cause the cut=off of text in CSV export?

Here is the method for CSV export, argument_value is the field with cut-off text. In debug, we verified that full context of the column has been assigned to CSV before exporting:

def self.to_csv
      CSV.generate do |csv|
        header = ['id', 'engine_name', 'engine_version', 'argument_name', 'argument_value', 'last_updated_by_id', 'created_at', 'updated_at', 'brief_note', 'global']        
        csv << header
        i = 1
        all.each do |config|
          base = OnboardDatax.engine_config_class.find_by_id(config.engine_config_id)
          row = Array.new
          row << i
          row << (base.global ? nil : base.engine.name)
          row << base.engine_version
          row << base.argument_name
          row << (config.custom_argument_value.present? ? config.custom_argument_value : base.argument_value)
          row << config.last_updated_by_id
          row << config.created_at
          row << config.updated_at
          row << base.brief_note
          row << (base.global ? 't' : 'f')
          csv << row
          i += 1
        end
      end

If wrapping the whole text with quotation mark, then the column can be exported to CSV in its entirety.

user938363
  • 9,990
  • 38
  • 137
  • 303
  • How are you checking the file - could it be an issue with the program you are using to open the csv? – Frederick Cheung Oct 24 '15 at 16:35
  • 2
    By the way, the `i += 1` fiddling can be accomplished more idiomatically using `all.each.with_index(1) do |config, i|`. – 7stud Oct 24 '15 at 18:04
  • `Frederick Cheung`, MS excel is being used to open the csv file, as we always do. – user938363 Oct 25 '15 at 03:57
  • 1
    I'd check with something lower level (eg a text editor first). Excel sometimes does weird things – Frederick Cheung Oct 25 '15 at 04:26
  • `Federick Cheung`, you are right that excel does wired things. When opening csv with notepad++, the full context appears. If you post an answer, will mark it. BTW what's the editor you recommend for opening CSV? Thanks. – user938363 Oct 25 '15 at 15:59

1 Answers1

0

What could cause the cut=off of text in CSV export?

puts File.read('your_cutoff_text.txt').size

--output:--
256

Rails column types:

String:
Limited to 255 characters (depending on DBMS)

My OS automatically adds a newline to the end of a file, so your cutoff text contains exactly 255 characters.

In the same file, there are fields which is much long than this and there is no problem.

Rails column types:

Text: 
Unlimited length (depending on DBMS)
Community
  • 1
  • 1
7stud
  • 46,922
  • 14
  • 101
  • 127
  • `7stud`, not sure why cut off at 256th for this row only. The db field is `text`. Also the next 2nd row after cut-off one has way more texts (more than 700 at least) in the same column. – user938363 Oct 24 '15 at 13:03
  • @user938363, For rows where `config.custom_argument_value` is present, I suggest you print out both `config.custom_argument_value` and `base.argument_value`, or print out a flag indicating which one is being used to produce the value. – 7stud Oct 24 '15 at 18:03