File Streamed well without buffering but gets streamed partially while buffering

Question

I have written a code which streams bytes from another server to my server and then I write those contents into my local file. It works fine when I use the read() method which doesn't buffer data. But when I use buffering (intention is that I believe streaming would be faster for big files) I use read(byte[]) method and it gets only partial data while streaming. I am posting the code. Can any one pls point out the mistake or concepts what I am missing.

The following code works fine. (no streaming)

    private void doViewDocument(HttpServletRequest request,
            HttpServletResponse response, DocumentServletService servletService) throws GEMException, MalformedURLException, ProtocolException, IOException {

        final String objectID = request.getParameter(PARAM_OBJECT_ID);

        LOGGER.info("For Viewing Document objectID received from Request == " + objectID);

        if (GEMSharedUtil.isValidObjectId(objectID)) {

            String ebesDocDownloadURL = servletService.getDocumentDownloadUrl(objectID);

            if (!GEMSharedUtil.isValidString(ebesDocDownloadURL)) {             
                //response.setStatus(HttpServletResponse.SC_NOT_ACCEPTABLE);
                response.setHeader("ResponseStatus", "Not_OK");
                throw new GEMException();
            } else {
                HttpURLConnection con = null;
                BufferedInputStream bin = null;
                BufferedOutputStream bout = null;

                try {
                    con = (HttpURLConnection) new URL(ebesDocDownloadURL).openConnection();

                    WASSecurity.preauthenticateWithLTPACookie(con);
                    con.setRequestMethod(REQUEST_METHOD_GET);
                    con.setDoOutput(true); // Triggers POST but since we have set request method so it will override it
                    con.setDoInput(true);
                    con.setUseCaches(false);
                    con.setRequestProperty("Connection", "Keep-Alive");
                    con.setAllowUserInteraction(false);
                //  con.setRequestProperty("Content-Type",
                //  "application/octet-stream");

                    response.setBufferSize(1024);
                    response.setContentType(con.getContentType());
                    response.setContentLength(con.getContentLength());
                    response.setHeader("ResponseStatus", "OK");
                    response.setHeader("Content-Disposition", con
                            .getHeaderField("Content-Disposition"));

                    bin = new BufferedInputStream((InputStream)
                            con.getInputStream(), 1024);

                    bout = new BufferedOutputStream(
                            response.getOutputStream(), 1024);

                    byte[] byteRead = new byte[1024];


                    File file = new File("C:\\Documents and Settings\\weakStudent\\Desktop\\streamed\\testStream.pdf");

                    FileOutputStream fos = new FileOutputStream(file);

                    if(file.length() > 0) {
                        file.delete();
                    }
                    file.createNewFile();

                    BufferedOutputStream fbout = new BufferedOutputStream((OutputStream) fos);
                    int c;
                    while((c= bin.read()) != -1) {
                        bout.write(c);
                        fbout.write(c);
                    }
fos.close();
                    bout.flush();
                    fbout.flush();
                    fbout.close();
                    LOGGER.info("con.getResponseCode():" + con.getResponseCode());

                } finally {
                    try {
                        if (bout != null) {
                            bout.close();
                        }
                    } catch (IOException e) {
                        LOGGER.log(Level.SEVERE, e.getMessage(), e);
                        throw new RuntimeException(e);
                    } finally {
                        try {
                            if (bin != null) {
                                bin.close();
                            }
                        } catch (IOException e) {
                            LOGGER.log(Level.SEVERE, e.getMessage(), e);
                            throw new RuntimeException(e);
                        } finally {
                            if (con != null) {
                                con.disconnect();
                            }
                        }
                    }
                }
            }

        }   //if ends

    }

Now if I have the following while loop it doesn't work correctly.

                while(bin.read(byteRead) != -1) {
                    bout.write(byteRead);
                    fbout.write(byteRead);
                }

Q2) Also would like to know is it mandatory to use BufferedInputStream/BufferedOutputStream for streaming. For eg if I use following code snippet it works

    BufferedInputStream bin = null;

    try {
        //in = request.getInputStream();        

        bin = new BufferedInputStream((InputStream) request
                .getInputStream(), 1024);

        int respcode = HttpURLConnection.HTTP_OK;
        con = createConnection(uploadURL, REQUEST_METHOD_POST);
        con.setRequestProperty("X-File-Name",fileName);

        conOut = con.getOutputStream();
        bout = new BufferedOutputStream(conOut);
        byte[] byteRead = new byte[1024];       

        while (bin.read(byteRead) != -1) {
            bout.write(byteRead);
        }
        bout.flush(); 
        respcode = con.getResponseCode();

But the following again streams partially (no BufferedInputStream used here)

    ServletInputStream in = null;

    try {
        in = request.getInputStream();      

        int respcode = HttpURLConnection.HTTP_OK;
        con = createConnection(uploadURL, REQUEST_METHOD_POST);
        con.setRequestProperty("X-File-Name",fileName);

        conOut = con.getOutputStream();
        bout = new BufferedOutputStream(conOut);
        byte[] byteRead = new byte[1024];       

        while (in.read(byteRead) != -1) {
            bout.write(byteRead);
        }
        bout.flush(); 
        respcode = con.getResponseCode();

close() every stream that you are opening. You ar not closing "fos" — rizzz86, Jul 12 '12 at 09:31

score 1 · Accepted Answer · answered Jul 12 '12 at 09:36

1

A1. Your discarding the number of bytes that were read, telling the output stream to write the entire contents of byteRead buffer, which may contain data from the previous read

int bytesIn = -1;
while((bytesIn = bin.read(byteRead)) != -1) {
    bout.write(byteRead, 0, bytesIn);
    fbout.write(byteRead, 0, bytesIn);
}

UPDATE Basically, all your examples are suffering from the same problem. Your buffer is n bytes long, but the read can return 0 to n bytes in the buffer, you need to take note of the number of bytes the read method returns in order to know how much to write

answered Jul 12 '12 at 09:36

MadProgrammer

343,457
22
230
366

Hi @user99, what you are saying is absoloutely correct. I missed that. So that means if I stream a document of 2048 bytes for instance it will work correctly even my buggy code. Since whole of the byte array would be written and no previous bytes would be there which would make the file corrupt.. Am I right?? – abhihello123 Jul 12 '12 at 09:57
You don't need to initialize `bytesIn`. – user207421 Jul 12 '12 at 10:07
@weakstudent This is the canonical way to copy a stream in Java. Memorize it. It always works. – user207421 Jul 12 '12 at 10:08
Also why does part 1 of Q2) works? Do you have any idea. It also has the buggy code.. – abhihello123 Jul 12 '12 at 10:09
@EJP, thanks. Used streams after a lot many days so forgot :P.. Can you please tell me why part 1 of Q2) where when its not done properly. – abhihello123 Jul 12 '12 at 10:11
@weakstudent Part 1 of Q2 *doesn't* work, unless the file size is a multiple of 1024. – user207421 Jul 12 '12 at 10:14
@EJP, I was experimenting on the same doc. And its not a multiple of 1024. Anyways better to use the proper code only. Thankx. – abhihello123 Jul 12 '12 at 11:49

File Streamed well without buffering but gets streamed partially while buffering

1 Answers1

Linked