Google Appengine JAVA - Zip lots of images saving in Blobstore

Question

I developed a simple media library where you can choose a set of images and download them. When a client request a download, a servlet receives the blob keys to use for create a zip file and then a Task is launched for the procedure

The task iterate through the received blob keys and zip the images into the archive. When the task has finished a mail with the download link is sent to the user.

Here is my problem:

FileWriteChannel writeChannel = fileService.openWriteChannel(file, lock);
OutputStream blobOutputStream = Channels.newOutputStream(writeChannel);
ZipOutputStream zip = new ZipOutputStream(blobOutputStream);

A single channel can handle only this amount of bytes

BlobstoreService.MAX_BLOB_FETCH_SIZE

Because of that, i must open and close the channel every 1mb of data i have to write (same issue for the read, but for the read i used this code and it works). or the write() method throws a null exception

Opening and closing the channel with a normal outputStream, does not presents issue, like this code

But handling a Zip file i also have to manage

ZipOutputStream zip = new ZipOutputStream(blobOutputStream);   
ZipEntry zipEntry = new ZipEntry(image_file_name);
zipOut.putNextEntry(zipEntry);
// while the image has bytes to write
   zipOut.write(bytesToWrite);

After i wrote 1MB of data in the ZipEntry i have to close the channel and open it again.

So here the problem: where i open a new channel i can't access to the previouse zipEntry i was writing and then i cannot continue to write the next 1MB of the image i'm processing.

And, after a open a new channel, if i try to write on the zipEntry object (w/o re-initializing it) i get a ClosedChannel exception

Here is the SAMPLE code i wrote, i know is not working, but explains what i am trying to do.

My question then: How (if is possible, off course) can i create a zip file writing 1MB per time?

I'm also available to other approaches, the thing i need is to zip some images into one zip and save it into the blobstore, if you have other ideas to make this, please tell me

Somehow you should use a buffered reading, like http://stackoverflow.com/questions/4308276/how-to-zip-a-file-while-writing-to-it suggests. That example is of another type of streams but the main principle may apply. I suppose the close between writes is something to avoid and having that buffered reader at mixture would make this rock. — mico, May 30 '13 at 19:27
I have no problems at all with the reading of the Blob. My problem is when i write the zip file. Because i can only write 1MB per channel, i need a way to persist the "ZipEntry" when i close and open a new channel — Deviling Master, Jun 02 '13 at 13:56

Jarandinor · Answer 1 · 2013-06-04T18:49:27.367

You should create you own stream that can manipulate channels. When the blob size limit is reached your stream closes current channel and opens a new one.

Example for local files:

public class ZipChannels {
public static void main(String[] args) throws IOException {
    File dirToZip = new File("target\\dependency");

    //create zip-files
    ChannelOutput out = new ChannelOutput();
    ZipOutputStream zip = new ZipOutputStream(out);
    int b = 0;
    for(File file: dirToZip.listFiles()) {
        ZipEntry zipEntry = new ZipEntry(file.getName());
        zip.putNextEntry(zipEntry);
        BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
        while((b = bis.read()) != -1) {
            zip.write(b);
        }
        bis.close();
        zip.closeEntry();
    }
    zip.close();

    //merge all into one file for check it
    BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("package_all.zip"));
    for (int i = 0; i < out.getChannelCount(); i++) {
        BufferedInputStream bis = new BufferedInputStream(new FileInputStream("package_" + i + ".zip"));
        while((b = bis.read()) != -1) {
            bos.write(b);
        }
        bis.close();
    }
    bos.close();
}

public static class ChannelOutput extends OutputStream {
    private OutputStream channel;
    private int count = 0;
    final private int MAX = 1000000;

    @Override
    public void write(int b) throws IOException {
        if(count++ % MAX == 0) {
            openNewChannel();
        }
        channel.write(b);
    }

    protected void openNewChannel() throws IOException {
        if(channel != null) {
            channel.close();
        }
        channel = new BufferedOutputStream(new FileOutputStream("package_" + (count / MAX) + ".zip")); 
    }
    public int getChannelCount() {
        return count / MAX + 1;
    }

    @Override
    public void close() throws IOException {
        channel.close();
    }
    @Override
    public void flush() throws IOException {
        channel.flush();
    }
}
}

If you have any questions, please feel free to ask.

Google Appengine JAVA - Zip lots of images saving in Blobstore

1 Answers1