1

I export data use latest MLCP version 8.0.8. What is limitation file size to export? I have a lot of files can't save due to file too large.

ERROR contentpump.SingleDocumentWriter: Error saving: /citation/2072452.xml
java.io.IOException: File too large
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1012)
    at com.marklogic.contentpump.SingleDocumentWriter.write(SingleDocumentWriter.java:92)
    at com.marklogic.contentpump.SingleDocumentWriter.write(SingleDocumentWriter.java:51)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
    at com.marklogic.contentpump.DocumentMapper.map(DocumentMapper.java:55)
    at com.marklogic.contentpump.DocumentMapper.map(DocumentMapper.java:34)
    at com.marklogic.contentpump.BaseMapper.run(BaseMapper.java:79)
    at com.marklogic.contentpump.LocalJobRunner$LocalMapTask.call(LocalJobRunner.java:411)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
18/08/01 13:57:48 INFO contentpump.LocalJobRunner:   com.marklogic.mapreduce.MarkLogicCounter:
thichxai
  • 1,073
  • 1
  • 7
  • 16
  • Are you sure that the path hasn't run out of space and is unable to write? Do you know if any of the documents are extremely large? If so, any idea how big they are? What OS are you running this on? – Mads Hansen Aug 01 '18 at 23:46
  • I have checked partition diskspace 56% free available – thichxai Aug 02 '18 at 00:15
  • 1
    Given the exception was thrown from java.io.UnixFileSystem.createFileExclusively, you might be bumping into an OS limit. How big are the files? – kcoleman Aug 02 '18 at 21:17

1 Answers1

0

It’s probably unrelated to MarkLogic and one of the things discussed here:

FileNotFoundException (File too large)

hunterhacker
  • 6,378
  • 1
  • 14
  • 11