HttpURLConnection executing two times

Question

I am using HttpURLConnection to make a request to server and save some data in the Database, but when I make the request it executes two times which adds two identical rows in the database.

Note: I am making the same request to the server from iOS as well and it works perfectly this only happens on Android

This is the code where I make the request:

URL url = new URL(url);
HttpURLConnection httpURLConnection = (HttpURLConnection)url.openConnection();

httpURLConnection.setRequestMethod("POST");
httpURLConnection.setDoOutput(true);
httpURLConnection.setDoInput(true);

OutputStream outputStream = (OutputStream)httpURLConnection.getOutputStream();
BufferedWriter bufferedWriter = new BufferedWriter(new OutputStreamWriter(outputStream, "UTF-8"));

String post_data = URLEncoder.encode("user_name" ,"UTF-8") + "=" + URLEncoder.encode(user,"UTF-8")+"&"
        +URLEncoder.encode("user_id" ,"UTF-8") + "=" + user_id+"&"
        +URLEncoder.encode("manager_id" ,"UTF-8") + "=" + manager+"&"
        +URLEncoder.encode("company_id" ,"UTF-8") + "=" + company+"&"
        +URLEncoder.encode("user_role" ,"UTF-8") + "=" + URLEncoder.encode(user_role ,"UTF-8");

bufferedWriter.write(post_data);
bufferedWriter.close();
outputStream.close();

InputStream inputStream = httpURLConnection.getInputStream();
Reader reader = new InputStreamReader(inputStream);

final char[] buf = new char[256];

final StringBuffer sb = new StringBuffer();

while (true) {
    int length = reader.read(buf);
    if (length == -1) break;
    sb.append(buf, 0, length);
}

reader.close();
inputStream.close();

Answer 1

HttpURLConnection automatic retry mechanism causes the request to be repeated twice.

Problem Analysis And Positioning

HttpURLConnection uses a Sun private HTTP protocol implementation class: HttpClient.java The key is the following method of sending a request and parsing the response header:

569       /** Parse the first line of the HTTP request.  It usually looks
570           something like: "HTTP/1.0 <number> comment\r\n". */
571   
572       public boolean parseHTTP(MessageHeader responses, ProgressSource pi, HttpURLConnection httpuc)
573       throws IOException {
574           /* If "HTTP/*" is found in the beginning, return true.  Let
575            * HttpURLConnection parse the mime header itself.
576            *
577            * If this isn't valid HTTP, then we don't try to parse a header
578            * out of the beginning of the response into the responses,
579            * and instead just queue up the output stream to it's very beginning.
580            * This seems most reasonable, and is what the NN browser does.
581            */
582   
583           try {
584               serverInput = serverSocket.getInputStream();
585               if (capture != null) {
586                   serverInput = new HttpCaptureInputStream(serverInput, capture);
587               }
588               serverInput = new BufferedInputStream(serverInput);
589               return (parseHTTPHeader(responses, pi, httpuc));
590           } catch (SocketTimeoutException stex) {
591               // We don't want to retry the request when the app. sets a timeout
592               // but don't close the server if timeout while waiting for 100-continue
593               if (ignoreContinue) {
594                   closeServer();
595               }
596               throw stex;
597           } catch (IOException e) {
598               closeServer();
599               cachedHttpClient = false;
600               if (!failedOnce && requests != null) {
601                   failedOnce = true;
602                   if (httpuc.getRequestMethod().equals("POST") && (!retryPostProp || streaming)) {
603                       // do not retry the request
604                   }  else {
605                       // try once more
606                       openServer();
607                       if (needsTunneling()) {
608                           httpuc.doTunneling();
609                       }
610                       afterConnect();
611                       writeRequests(requests, poster);
612                       return parseHTTP(responses, pi, httpuc);
613                   }
614               }
615               throw e;
616           }
617   
618       }

In the code on lines 600 - 614:

failedOnce is false by default, indicating whether it has failed once. This also limits sending a maximum of 2 requests. httpuc is request related information. The default value of retryPostProp is true, and the value can be specified by command line parameter (-Dsun.net.http.retryPost=false). streaming: Default false. true if we are in streaming mode (fixed length or chunked).

Use the Linux command socat tcp4-listen:8080,fork,reuseaddr system:“sleep 1”!!stdout to establish an HTTP server that only receives requests and does not return responses. For a POST request, after the first request is sent, the parsing response will encounter an early end of the stream, which is a SocketException: Unexpected end of file from server. After parseHTTP captures it and finds that the above conditions are met, it will retry. The server will receive the second request.

Solution

1. Disable the retry mechanism of HttpURLConnection. Via launcher command parameter -Dsun.net.http.retryPost=false or code to set System.setProperty("sun.net.http.retryPost", "false")

2. Use the Apache HttpComponents library. By default, HttpClient attempts to automatically recover from I/O exceptions. This automatic recovery mechanism is limited to a few exceptions that are considered safe.

HttpClient will not attempt to recover from any logical or HTTP protocol errors; HttpClient will automatically retry methods that are considered idempotent; HttpClient will automatically retry methods that still fail to send HTTP requests to the target server. (For example, the request has not been fully transmitted to the server).

Answer 2

Android uses OkHttp to implement the functionality of HttpURLConnection. OkHttp silently performs retries under certain circumstances:

... OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems ...

The library had several issues in the past with unexpected retries including POST requests (for a good summary see here). Even latest development branch of Android seems to use a more than 5!!! years old version of OkHttp (see here). So, it is very likely that Android's version does not have any of the relevant bug fixes. There is also no simple way to access the internal OkHttp client instance and explicitely disable retries.

So, you have two options:

• A) Use a newer version of OkHttp directly in your app instead of relying on HttpURLConnection
• B) Try to workaround the problem:

It seems that some of the retries are not performed when the content of a POST request is not buffered internally by OkHttp. You can avoid internal buffering by either setting an explicit content lenght through setFixedLengthStreamingMode(long) or by enabling chunked streaming through setChunkedStreamingMode(0).These workarounds seem to avoid at least some of the retries (see comments on this ticket), but there is no guarantee that this will avoid all kinds of retries.

From a security perspective, I heavily recommend to follow approach A) and not use Android's Http(s)URLConnection. As mentioned before, the Android open source project seems to use a completely outdated OkHttp version, that even does not receive any security fixes (at least not from the official OkHttp project).

Be aware, that most smartphone manufacturer use a closed-source fork of the Android open source project. They may use different versions of third-party libraries or even use a different HTTP client library under the hood.