FileBackedOutputStream on Appengine

Question

My application on Appengine create a csv file with more 65535 rows

But, I have an error of type OutOfMemoryError when writing :

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)

White this code :

public static byte[] joinLines(Collection<String> lines) {
    final ByteArrayOutputStream stream = new ByteArrayOutputStream();

    boolean firstElement = true;

    for (final String part : lines) {
        String value = part + LINE_SEPARATOR;
        if (firstElement) {
            value = addExcelPrefix(value);
            firstElement = false;
        }

        final int currentSize = value.length();
        try {
            stream.write(value.getBytes(ENCODING), 0, currentSize); // OutOfMemoryError HERE
        } catch (UnsupportedEncodingException e) {
            LOGGER.info(e.getMessage());
        }
    }
    return stream.toByteArray();
}

So I used FileBackedOutputStream of Guava for solve the problem of OutOfMemoryError :

public static byte[] joinLines(Collection<String> lines) throws IOException {
    final FileBackedOutputStream stream = new FileBackedOutputStream(THRESHOLD, true);

    boolean firstElement = true;

    for (final String part : lines) {
        String value = part + LINE_SEPARATOR;
        if (firstElement) {
            value = addExcelPrefix(value);
            firstElement = false;
        }

        final int currentSize = value.length();
        try {
            stream.write(value.getBytes(ENCODING), 0, currentSize);
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
        }
    }

    return stream.asByteSource().read();
}

But, on appengine, I now an error of type SecurityException when creating of temporary file :

java.lang.SecurityException: Unable to create temporary file
    at java.io.File.checkAndCreate(File.java:2083)
    at java.io.File.createTempFile(File.java:2198)
    at java.io.File.createTempFile(File.java:2244)
    at com.google.common.io.FileBackedOutputStream.update(FileBackedOutputStream.java:196)
    at com.google.common.io.FileBackedOutputStream.write(FileBackedOutputStream.java:178)

How to allow create temporary file on Appengine with FileBackedOutputStream ?
In a bucket, how ?

Thanks

What are you going to do with the CSV file after you have generated it? If you are intending to return it as a response to an HTTP request, you can of course write it directly to the HTTP resonse instead of using a ByteArrayOutputStream or FileBackedOutputStream to cache the data. — jarnbjo, Apr 30 '15 at 15:05
The csv file will be saved in a bucket and this code is executed in a task. FileBackedOutputStream does not work on app engine because it can not create its cache file — ctesniere, Apr 30 '15 at 15:21
While you have provided good info on the issue i see no attempt to store it in a bucket (cloud or blobstore). See that api and give it a try. Update your question if still having coding issues with that api. — Zig Mandel, Apr 30 '15 at 16:15

score 0 · Accepted Answer · answered May 07 '15 at 15:42

I used GcsService that solves my problem :

protected String uploadBytesForCsv(Map<Integer, Map<Integer, Object>> rows) throws IOException {
    LOGGER.info("Get Bytes For Csv");

    final Collection<String> lines = cellsToCsv(rows);
    LOGGER.info("number line : " + lines.size());

    boolean firstElement = true;

    final String fileName = getFileName();

    final GcsFilename gcsFilename = new GcsFilename(config.getBucketName(), fileName);
    final GcsService gcsService = GcsServiceFactory.createGcsService();
    final GcsOutputChannel outputChannel = gcsService.createOrReplace(gcsFilename, GcsFileOptions.getDefaultInstance());

    for (final String part : lines) {
        final ByteArrayOutputStream stream = new ByteArrayOutputStream();
        String value = part + LINE_SEPARATOR;
        if (firstElement) {
            value = addExcelPrefix(value);
            firstElement = false;
        }

        final int currentSize = value.length();
        try {
            stream.write(value.getBytes(ENCODING), 0, currentSize);
            outputChannel.write(ByteBuffer.wrap(stream.toByteArray()));
        } catch (UnsupportedEncodingException e) {
            LOGGER.info(e.getMessage());
        }

        stream.flush();
        stream.close();
    }

    outputChannel.close();

    return new UrlBuilder(config.getStorageUrlForExport())
            .setBucketName(config.getBucketName())
            .setFilename(fileName).build();
}

FileBackedOutputStream on Appengine

1 Answers1