String content sent as byte[] gets its content escaped as query string

Question

I need to send a String over the network, and I'm getting it on the other side in a REST Controller in Spring Boot.

I'm sending the data with the following lines:

byte[] content = sw.toString().getBytes(StandardCharsets.UTF_8); //sw is StringWriter
httpUrlConnection = (HttpURLConnection)new URL("http://127.0.0.1:8080/upload").openConnection();
httpUrlConnection.setDoInput(false);
httpUrlConnection.setDoOutput(true);
httpUrlConnection.setRequestMethod("POST");
httpUrlConnection.setFixedLengthStreamingMode(content.length);
httpUrlConnection.connect();
httpUrlConnection.getOutputStream().write(content);
httpUrlConnection.getOutputStream().flush();

And I'm receiving like this:

@RequestMapping(value="/upload", method = RequestMethod.POST)
public ResponseEntity<String> upload(@RequestBody byte[] uploadedData)
    throws NoSuchAlgorithmException, InvalidKeyException, IOException {
  if(uploadedData == null) {
    log.info("Uploaded data was null.");
    return new ResponseEntity<String>("Data was null after upload.", HttpStatus.INTERNAL_SERVER_ERROR);
  }

  String receivedData = new String(uploadedData, StandardCharsets.UTF_8);
  log.info("" + receivedData);

But while what I'm sending is this...

What I get is this:

-----BEGIN+CERTIFICATE+REQUEST-----%0D%0AMIIB0DCCATkCAQAwgY8xCzAJBgNVBAYTAkFVMSgwJgYDVQQKDB9UaGUgTGVnaW9u%0D%0AIG9mIHRoZSBCb3VuY3kgQ2FzdGxlMRIwEAYDVQQHDAlNZWxib3VybmUxETAPBgNV%0D%0ABAgMCFZpY3RvcmlhMS8wLQYJKoZIhvcNAQkBFiBmZWVkYmFjay1jcnlwdG9AYm91%0D%0AbmN5Y2FzdGxlLm9yZzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqE+pEyKS%0D%0A21JXgeldS83E+SSmQ4s2xYPqV7Yw2ebUKsxnbBz1KcXHdHS1MJ9PYzBJgogdigZW%0D%0A6hEQ83edP%2FAy%2FEQzGqeKUzFqNEsQh3PSdbF9N5k7b81tQHUbfIbNu1ofSBNa%2FEit%0D%0AMkOj1NAmwivpW0AA8aPZhGzYLYWcp0lsC78CAwEAAaAAMA0GCSqGSIb3DQEBBQUA%0D%0AA4GBAB0p0ySmfMkm3z8H4P8WwWJ8bMO2RNXEx0i9fU2ncJfdY0zEPYvM6zpUhwJP%0D%0AT9DsQBPdSy+VLbJ%2FPtYoiKIcupd+vriGYn3mqckXy7RBLqpiVsnw1rGE28oG4I9N%0D%0Au0p2AwDuC+KNuHgtrGxYrRnFTRKZpj2AoGuW1a6eSaNOhPeq%0D%0A-----END+CERTIFICATE+REQUEST-----%0D%0A=

This is not good because it kills the PEM parser.

However, I could overcome this problem by using the following transformations, and the String becomes parseable...

  receivedData = receivedData.replace("-----BEGIN+CERTIFICATE+REQUEST-----", "-----BEGIN CERTIFICATE REQUEST-----");
  receivedData = receivedData.replace("-----END+CERTIFICATE+REQUEST-----", "-----END CERTIFICATE REQUEST-----");
  receivedData = receivedData.replace("%0D%0A", "\r\n");
  receivedData = receivedData.replace("%2F", "/");
  receivedData = receivedData.substring(0, receivedData.length()-1);

In which case parsing was successful...

2014-10-18 16:26:43.864  INFO 2432 --- [nio-8080-exec-1] demo.HelloController                     : PemParser returned: org.spongycastle.pkcs.PKCS10CertificationRequest@f5c81fb9
2014-10-18 16:26:43.870  INFO 2432 --- [nio-8080-exec-1] demo.HelloController                     : SUCCESS

...but I don't think this is a really viable way of doing things as it doesn't look very stable. There has to be a better way. I tried URLEncoder.decode(receivedData, "UTF-8"); but it removes + characters from inside the actual data, which is also not good because it makes the data unparseable.

Does anyone have any useful ideas on how to prevent this URL escaping when I'm sending a String over the network?

What diagnostics have you performed? Have you looked at exactly what's being sent, via (say) Wireshark? Have you logged `sw`? — Jon Skeet, Oct 18 '14 at 14:39
@JonSkeet `sw` is working perfectly fine, that did not have any troubles and logged what was meant to be logged. Aka, the data on the client side works. Only after I receive the String on the other side it is a jarbled mess in that regard. I didn't check in Wireshark, though. — EpicPandaForce, Oct 18 '14 at 14:44
@JonSkeet ...I'll have to check using Wireshark when I'm not testing it only on one computer with localhost where I don't have proper access of the router. This way I'm not using my network interface for this and cannot properly log the packet. I'll look into that later. — EpicPandaForce, Oct 18 '14 at 14:51

score 1 · Accepted Answer · edited May 23 '17 at 12:20

Well, I still don't know the answer for why that happened, but I did make a workaround.

Instead of just sending over the String byte[ ] content, I wrapped the data in a JSON Object with https://jsonp.java.net/index.html according to http://docs.oracle.com/javaee/7/tutorial/doc/jsonp003.htm.

JsonObject model = Json.createObjectBuilder().add("data", sw.toString()).build();

httpsUrlConnection = (HttpsURLConnection)new URL("http://127.0.0.1:8443/upload").openConnection();

httpsUrlConnection.setDoInput(true);
httpsUrlConnection.setDoOutput(true);
httpsUrlConnection.setRequestProperty("Content-Type", "application/json");
httpsUrlConnection.setRequestMethod("POST");
httpsUrlConnection.connect();
try(JsonWriter jsonWriter = Json.createWriter(httpsUrlConnection.getOutputStream()))
{
    jsonWriter.write(model);  
}
httpsUrlConnection.getOutputStream().flush();
System.out.println("Response code: " + httpsUrlConnection.getResponseCode());

Afterwards, on the Spring Boot side, I set it up so that I would receive the JSON object as per https://stackoverflow.com/a/8946142/2413303 :

public class DataObject {
  private String data;

  public String getData() {
    return data;
  }

  public void setData(String data) {
    this.data = data;
  }
}

And in Controller:

@RequestMapping(value="/upload", method = RequestMethod.POST, consumes="application/json")
  public ResponseEntity<String> upload(@RequestBody DataObject uploadedData)
    throws NoSuchAlgorithmException, InvalidKeyException, IOException {
    if(uploadedData == null) {
      log.info("Uploaded data was null.");
      return new ResponseEntity<String>("Data was null after upload.", HttpStatus.INTERNAL_SERVER_ERROR);
    }
    String receivedData = uploadedData.getData();
    log.info("" + receivedData);

This way the characters were not escaped:

2014-10-20 13:03:15.550  INFO 6924 --- [nio-8443-exec-3] demo.HelloController                     : -----BEGIN CERTIFICATE REQUEST-----

EDIT: I have been told that the problem was that with no content type set, Spring defaults to application/x-www-form-urlencoded, because it should have been application/octet-stream.

EDIT2: With octet-stream, sending it over escaped \r and \n to \\r and \\n, and added a " before and after the string for some reason. Not sure why.

String content sent as byte[] gets its content escaped as query string

1 Answers1