Map reduce job for continue stream

Question

I have a huge text file which contains text data . Files' each line contains 12 character of data. I need to find a substring of 5 character from that file using map reduce job.

Input file.

abcdefghijkl
kahfdjshjsdh
sdfkjsdjkjks

value to search

cdefg

The 'cdefg' can occurs anywhere in the file. It can be in in two lines. So I don't know how to create a map of last two character of current line and next 3 character of next line.

@bouteillebleu I wrote a simple program which is giving me line by of input string and split that line in 5 character and create a map for that but don't know how to handle last 2 characters of input string. — Ibrar Ahmed, May 06 '17 at 14:13
I have created map in mapper and later will match with "cdefg" string, but my question how I can create maps of 5 characters. — Ibrar Ahmed, May 06 '17 at 14:19
i really don't understand your question also, can you explain more? — Youcef LAIDANI, May 06 '17 at 14:39
I have a file containing lines of 12 characters and I want to find 5 character of string from that file. In Mapper I am getting 12 character of line and can create two maps of 5 character and left 2 character and want to get next 3 character from next line and want to create map of it. So in reducer I can compare that maps with my string. — Ibrar Ahmed, May 06 '17 at 14:47
so your inputs should look like this `[abcde, fghij, klkah, fdjsh, jsdhs, dfkjs, djkjk, s]` ? — Youcef LAIDANI, May 06 '17 at 15:12

score 0 · Answer 1 · edited May 23 '17 at 12:18

0

I have a file containing lines of 12 characters and I want to find 5 character of string from that file. In Mapper I am getting 12 character of line and can create two maps of 5 character and left 2 character and want to get next 3 character from next line and want to create map of it. So in reducer I can compare that maps with my string.

You can concatenate your line all together then you can split the result with 5 character check this Splitting a string at every n-th character :

abcdefghijklkahfdjshjsdhsdfkjsdjkjks
[abcde, fghij, klkah, fdjsh, jsdhs, dfkjs, djkjk, s]

You can inspire the solution from this piece of code :

File file = new File("myFile.txt");
try {
    Scanner scanner = new Scanner(file);
    String result = "";
    while (scanner.hasNextLine()) {
        String line = scanner.nextLine();
        result += line;
    }
    System.out.println(result);
    //here you can use this array
    String spl[] = result.split("(?<=\\G.....)");

    System.out.println(Arrays.toString(spl));
} catch (FileNotFoundException e) {
    e.printStackTrace();
}

Output

abcdefghijklkahfdjshjsdhsdfkjsdjkjks
[abcde, fghij, klkah, fdjsh, jsdhs, dfkjs, djkjk, s]

EDIT

I Want to create map like this abcdefghijklkahfdjshjsdhsdfkjsdjkjks [abcde, bcdef, cdefg, defgh... ]

You can solve this problem like so :

String str = "abcdefghijklkahfdjshjsdhsdfkjsdjkjks";
List<String> list = new ArrayList<>();

for (int i = 0; i < str.length()-4; i++) {
    String s = "";
    for (int j = i; j < i+5; j++) {
        s+=result.charAt(j);
    }
    list.add(s);
}

Output

[abcde, bcdef, cdefg, defgh, efghi, fghij, ghijk, ...., djkjk, jkjks]

edited May 23 '17 at 12:18

Community

1
1

answered May 06 '17 at 14:14

Youcef LAIDANI

55,661
15
90
140

I Want to create map like this abcdefghijklkahfdjshjsdhsdfkjsdjkjks [abcde, bcdef, cdefg, defgh... ] – Ibrar Ahmed May 06 '17 at 15:36
2ndly need to concatenate last 2 characters with next line start character – Ibrar Ahmed May 06 '17 at 15:38
so the last value of your array should be `jkjks` right @IbrarAhmed ? – Youcef LAIDANI May 06 '17 at 15:50
My next line is "hasjfdkjksjkdfjsalkdjf" so next maps should be [jkjks,kjksh, jksha, kshas, shasj .... – Ibrar Ahmed May 06 '17 at 16:33
yes my solution do what you want did you try it @IbrarAhmed – Youcef LAIDANI May 06 '17 at 16:34
Yes try it, it start again for next line, I want to resume scanning from first line, like last 4 character of first line and one character from second line and the last 3 character from first file and first two character from second line. – Ibrar Ahmed May 06 '17 at 16:56
1

I cannot concatenate all lines because they are too many, need a solution which is getting just next line, [I need it in mapper ] – Ibrar Ahmed May 06 '17 at 17:08
mmm, ok i will try to find a solution when i will be free, it need more concentration @IbrarAhmed – Youcef LAIDANI May 06 '17 at 17:10

Map reduce job for continue stream

1 Answers1