1

I am using scalding 0.12 version with TypedPipe. I want to write the output to csv with headers. How can I add headers with this, I see the option for `writeHeader=true/false" but how do I provide the headers.

nitishagar
  • 9,038
  • 3
  • 28
  • 40
  • Do you have any code to share? At least the part which writes the output to `csv`? – Nader Ghanbari Mar 04 '15 at 04:15
  • The code is not much to share `TypedCsvWithHeader[String](outputString)`, this `TypedCsvWithHeader` just has `writeHeader=true` – nitishagar Mar 04 '15 at 04:26
  • 1
    What is the actual and expected result? Sharing (even a small chunk of code, along with the actual and expected results) makes it a lot easier for others to answer without making a lot of assumptions. – Nader Ghanbari Mar 04 '15 at 04:30
  • Actual result - just the output data `(val1, val2 ...)` Expected result - `header1, header2 ...` `val1, val2, ...` – nitishagar Mar 04 '15 at 04:41

2 Answers2

1

Using the type safe api, I write TSV files with the following code:

val data: TypedPipe[String, String] = ...
data.write(TypedTsv[(String, String)](outputPath))

TypedPipe is defined in the TypedSeperatedFile class. Take a look at the source code:

trait TypedSeperatedFile extends Serializable {
  def separator: String
  def skipHeader: Boolean = false
  def writeHeader: Boolean = false

  def apply[T: Manifest: TupleConverter: TupleSetter](path: String): FixedPathTypedDelimited[T] =
    apply(Seq(path))

  def apply[T: Manifest: TupleConverter: TupleSetter](path: String, f: Fields): FixedPathTypedDelimited[T] =
    apply(Seq(path), f)

  // others apply method
}

object TypedTsv extends TypedSeperatedFile {
  val separator = "\t"
}

One way to write a TSV with header is to implement your own object that extends the TypedSeperatedFile class:

object TypedTsvWithHeader extends TypedSeperatedFile {
  val separator = "\t"
  override val writeHeader = true
}

You can then use it as following:

data.write(TypedTsvWithHeader[(String, String)](outputPath,
  List("header1", "header2")))
Marsellus Wallace
  • 17,991
  • 25
  • 90
  • 154
  • I just realized that you were asking about csv... Luckily it's the same thing! TypedSeperatedFile contains a TypedCsv (and others) as well. – Marsellus Wallace Aug 02 '17 at 14:46
0

.write(Tsv("bcd.tsv", schema , writeHeader = true))

are you looking for something like this ?

Sanchit Grover
  • 998
  • 1
  • 6
  • 9