0

I use the follow codes to write some data into a SequenceFile Format file . when the program run a while , I interrupte the program via red button on eclipse console . however , when I check the data file on hdfs , the sequence file's size is zero. and also can not use 'hadoop fs -text filename' command view the file. when I use SequenceFile.Reader read the file previously created , I meet the 'Exception in thread "main" java.io.EOFException' exception. In this case , how to do? my development environment is eclipse3.7(on windows 7 ) and the hadoop cluster (hadoop version 1.0.3 )on CentOS 6 .

class Sequence extends Thread {

private String uri = "hdfs://172.20.11.60:9000";
private String filePath = "/user/hadoop/input/";
private String fileName = "Sequence-01.seq";
public SequenceFile.Writer writer;
private static int cnt = 0;

private void init() {
    Configuration conf = new Configuration();
    try {
        FileSystem fs = FileSystem.get(URI.create(uri), conf);
        writer = SequenceFile.createWriter(fs, conf, new Path(filePath
                + fileName), LongWritable.class, Text.class);
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

public Sequence() {
    init();
}

@Override
public void run(){
    while(true){
        try {
            writer.append(new LongWritable(100), new Text("hello,world"));
            cnt++;
            if(cnt%100 == 0){
                System.out.println("flush current data to file system");
                writer.syncFs();
            }
        } catch (IOException e) {
            // TODO Auto-generated catch block
            System.out.println("append data error");
            e.printStackTrace();
        }

        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            System.out.println("thread interupted");
            e.printStackTrace();
        }
    }
}

}

public class TestSequenceFile {

/**
 * @param args
 */
public static void main(String[] args) {
    // TODO Auto-generated method stub

    new Sequence().start();
}

}

user2193944
  • 11
  • 1
  • 1

2 Answers2

0

A general advice : not to interrupt the process.

Solution : For me the following code was working fine.

    import java.io.IOException;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;

import org.apache.hadoop.io.Text;


public class SequenceFileWriteDemo {
private static final String[] DATA = {
"One, two, buckle my shoe",
"Three, four, shut the door",
"Five, six, pick up sticks",
"Seven, eight, lay them straight",
"Nine, ten, a big fat hen"};

public static void main(String[] args) throws IOException {
//String uri = "/home/Desktop/inputSort.txt";
String uri = "hdfs://localhost:9900/out1.seq";

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
Path path = new Path(uri);
IntWritable key = new IntWritable();
Text value = new Text();
SequenceFile.Writer writer = null;



try {
writer = SequenceFile.createWriter(fs, conf, path,
    key.getClass(), value.getClass());


    for (int i = 0; i < 130; i++) {
    key.set(100 - i);
    value.set(DATA[i % DATA.length]);


    System.out.printf("[%s]\t%s\t%s\n", writer.getLength(), key, value, key.getClass(), value.getClass());

    writer.append(key, value);
    }
    } finally {
    IOUtils.closeStream(writer);
    }
}}

Please refer to Book Hadoop-The Definitive Guide (O'Reilly publications) for details on Writing to sequence file.

Gargi
  • 113
  • 1
  • 9
0

Yeah Hadoop Definitive guide is best for it here is that example for reading and writing the sequence files.

Actually sequence file forms the sequence of bytes or hadoop Writables mainly used to combine various small files to combine and feed to the map function.

http://javatute.com/javatute/faces/post/hadoop/2014/creating-sequence-file-using-hadoop.xhtml

Ashish
  • 51
  • 5