2

I'm using Apache Spark to build an application. To make the RDDs available from other applications I'm trying two approaches:

  1. Using tachyon
  2. Using a spark-jobserver

I'm new to Tachyon. I completed the following tasks given in the a Running Tachyon on a Cluster

I'm able to access the UI from master:19999 URL.

From the tachyon directory I successfully created a directory./bin/tachyon tfs mkdir /Test But while trying to do the copyFromLocal command I'm getting the following errors:

FailedToCheckpointException(message:Failed to rename hdfs://master:54310/tmp/tachyon/workers/1421840000001/8/93 to hdfs://master:54310/tmp/tachyon/data/93)
dtolnay
  • 9,621
  • 5
  • 41
  • 62
Anju
  • 631
  • 2
  • 9
  • 25
  • I have the exact same problem. After the failed copyFromLocal, the files are shown "In Memory" using `tfs ls`, but they are "incomplete" as reported when I try to `tfs cat` one of the files: _The file /data/test.txt is not complete._ – taranaki Jul 09 '15 at 15:38

1 Answers1

0

You are most likely running tachyon and spark-jobserver under different users, and have HDFS as your underFS.

Check out https://tachyon.atlassian.net/browse/TACHYON-1339 and the related patch.

The easy way out is running tachyon and your spark job server as the same user.

The (slightly) harder way is to port the patch and recompile spark, and then sjs with the patched client.

Rares Mirica
  • 53
  • 5