CSV file has random new lines / splits in it. Why is this happening?

Question

I am using Rails and generating a weekly CSV email.

module CSVData
  class Report
    HEADER = ["url", "description", "title", "logo_url", "created_at"]

    attr_reader :start_date, :end_date, :file_name, :posts

    def initialize(time)
      @start_date = time - 7.days
      @end_date = time
      @file_name = "post_data_#{@end_date.strftime("%-m_%-d_%Y")}"
      @posts = Post.where(created_at: @start_date..@end_date)
    end

    def create_csv
      file = File.new(File.join(Dir.pwd, "/tmp/#{@file_name}.csv"), "w+")
      CSV.open(file, "wb") { |csv| write_data(csv) }
      file
    end

    def write_data(csv)
      csv << HEADER
      @posts.each do |post|
        csv << data_row(post)
      end
      csv
    end

    def data_row(post)
      description = post.description || ""
      description = "\"" + description.gsub("\"", "\"\"") + "\""
      description = description.squish

      ["http://#{ENV['HOST']}/#{post.slug}",
      description,
      post.title,
      "http://#{ENV['HOST']}#{post.author.logo.url(:thumb)}",
      post.created_at.strftime("%m/%d/%Y %H:%M")]
    end
  end
end

Mailer

  def post_data_email
    time = Time.now
    @file = CSVData::Report.new(time).create_csv
    attachments["Post_Data_#{time.strftime("%m_%d_%Y")}.csv"] = {
      :content => @file.read
    }

    ## Mail details
  end

When I run it in development it works great and the CSV file is fine. However the automated (with 'whenever gem') email in production has some strange errors where data is split onto a newline. Usually in the "url", "description", or "logo_url"

sometimes in the middle of the description it will just jump to a new line

http://...,"""Here is a description that will be fine and then just 

 jump to a new line.""",Post Title,http://...thumb.jpg?1,08/14/2015 16:27

This one jumped in the middle of the logo_url

"Everything fine until here",http://example.com/logos/1441/thumb.png?143

 9262835,08/11/2015 10:17

one row was missing "ht" in the beginning.

tp://example.com/post,## the rest of the line is fine

It's really baffling me. Any idea what might cause something like this?

EDIT: I have made changes based on comments, but still the same error occurs.

Mailer

  def post_data_email
    time = Time.now
    @file = CSVData::Report.new(time).create_csv
    attachments["Post_Data_#{time.strftime("%m_%d_%Y")}.csv"] = {
      :content => File.read(@file.path)
    }

    ## Mail details
  end

create_csv and write_data methods changed

def create_csv
  csv_string = CSV.generate(write_headers: true, headers: HEADER) { |csv| write_data(csv) }
  file = File.new(File.join(Dir.pwd, "/tmp/#{@file_name}.csv"), "w+")
  file.write(csv_string)
  file.close
  file
end

def write_data(csv)
  @posts.each do |post|
    csv << data_row(post)
  end
  csv
end

Don't use `CSV.open` with an open `File`. Use it with a string `filename`. And refactor everything. — pguardiario, Aug 18 '15 at 08:23
@pguardiario I tried changing it so it wasn't opening a opened file (see edits), but still it's not working. — Peter R, Aug 18 '15 at 10:12
Can you give more details about the automation? Is it a cronjob? If so, what kind of server are you using? — Eric N, Aug 18 '15 at 12:07

score 0 · Answer 1 · edited Oct 07 '21 at 13:31

According to RFC 821: SMTP, email attachments are be cut off at 1000 characters.

text line

The maximum total length of a text line including the is 1000 characters (but not counting the leading dot duplicated for transparency).

The solution is to encode the attachment as base64:

attachments["Post_Data_#{time.strftime("%m_%d_%Y")}.csv"] = {
  :data=> ActiveSupport::Base64.encode64(File.read(@file.path)),
  :encoding => 'base64'
}

Shamelessly copied from How to send a csv attachment with lines longer than 990 characters?

CSV file has random new lines / splits in it. Why is this happening?

1 Answers1