Access private MessageHeader field ("Authorization") of HttpURLConnection via Reflection

Question

Currently writing unit tests for an existing library, i am trying to work around the limitation (as explained here) that you cannot retrieve an already set "Authorization"-header, using reflection.

The code i'm using is a very typical snippet that i've used dozens of times to access private fields:

HttpURLConnection conn = (HttpURLConnection) new URL("https://stackoverflow.com").openConnection();
conn.setRequestProperty("Authorization", "Basic Zm9vYmFyOnNlY3JldA==");
try {
    Field requests = conn.getClass().getSuperclass().getSuperclass().getSuperclass().getDeclaredField("requests");
    requests.setAccessible(true);
    MessageHeader headers = (MessageHeader) requests.get(conn); // Problem: returns null
    return headers.getValue(headers.getKey("Authorization"));
} catch (IllegalAccessException | NoSuchFieldException e) {
    e.printStackTrace();
}

However – extraction via Field::get fails and null is returned (see commented line).

Looking at the base class of HttpUrlConnection, which is URLConnection i know i'm looking for the requests field. Debugging through it, i can see the field i want to extract (even showing the "Authorization" values):

In the line of code that fails to return the MessageHeader object, it looks like i have a reference to the field in URLConnection:

But i must be missing something here – can anybody tell, what?

Update

What confuses me is

1. the fact that i'm only importing URLConnection and HttpURLConnection from the java.net-package. However from looking at the first debug screenshot, the conn-object implementation is clearly coming from sun.net.www.protocol.https.
2. the DelegateHttpsURLConnection member (also shown in the first debug screenshot)

Answer 1

When you call getClass() on an object, you get its actual runtime type which doesn’t have to match the compile-time type, as it may be a subclass of it. Therefore, traversing the class hierarchy via getSuperclass() is fragile.

When you know the class declaring the field beforehand, you can just use that class, e.g.

HttpURLConnection conn = (HttpURLConnection)
    new URL("https://stackoverflow.com").openConnection();
try {
  Field f = URLConnection.class.getDeclaredField("requests");
  f.setAccessible(true);
  System.out.println(f.get(conn));
}
catch(ReflectiveOperationException ex) {
  ex.printStackTrace();
}

This, however, will always print null, as this specific connection implementation doesn’t use its superclass state, but delegates to a different implementation. As you’ve found out yourself, it’s stored in a field called delegate.

HttpURLConnection conn = (HttpURLConnection)
    new URL("https://stackoverflow.com").openConnection();
for(Class<?> c = conn.getClass(); c != URLConnection.class; c = c.getSuperclass())
  System.out.print(c.getName() + " > ");
System.out.println(URLConnection.class.getName());

Field delegate = conn.getClass().getDeclaredField("delegate");
delegate.setAccessible(true);
conn = (HttpURLConnection)delegate.get(conn);

for(Class<?> c = conn.getClass(); c != URLConnection.class; c = c.getSuperclass())
  System.out.print(c.getName() + " > ");
System.out.println(URLConnection.class.getName());

Field requests = URLConnection.class.getDeclaredField("requests");
requests.setAccessible(true);
System.out.println(requests.get(conn));

This prints

sun.net.www.protocol.https.HttpsURLConnectionImpl > javax.net.ssl.HttpsURLConnection > java.net.HttpURLConnection > java.net.URLConnection
sun.net.www.protocol.https.DelegateHttpsURLConnection > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection > sun.net.www.protocol.http.HttpURLConnection > java.net.HttpURLConnection > java.net.URLConnection
null

Showing that these two objects have a different class hierarchy and thus, traversing it assuming a particular depth is indeed fragile.

It also prints again null, because even this implementation class doesn’t use the requests field inherited from URLConnection. Instead, the class sun.net.www.protocol.http.HttpURLConnection declares its own field, with the same name, which contains the actual MessageHeader object. That’s why performing getSuperclass().getSuperclass() on the implementation class of the delegate leads to the correct field, which is not URLConnection’s.

When we know this beforehand, we can use

HttpURLConnection conn =
    (HttpURLConnection) new URL("https://stackoverflow.com").openConnection();
conn.setRequestProperty("Authorization", "Basic Zm9vYmFyOnNlY3JldA==");

Field delegate = Class.forName("sun.net.www.protocol.https.HttpsURLConnectionImpl")
    .getDeclaredField("delegate");
Field requests = Class.forName("sun.net.www.protocol.http.HttpURLConnection")
    .getDeclaredField("requests");
AccessibleObject.setAccessible(new Field[] { delegate, requests}, true);
sun.net.www.MessageHeader headers =
    (sun.net.www.MessageHeader)requests.get(delegate.get(conn));
return headers.findValue("Authorization");

Since you’re accessing the sun.net. … package(s) anyway, you could refer to these classes directly instead of using Class.forName, but I considered the latter more robust in this specific case, as all these classes have easy to confuse names and even sometimes the same simple name, just differing by their package.

private String getAuthorizationHeaderValue(HttpURLConnection conn) {
  try {
    Field delegate = sun.net.www.protocol.https.HttpsURLConnectionImpl.class.getDeclaredField("delegate");
    Field requests = sun.net.www.protocol.http.HttpURLConnection.class.getDeclaredField("requests");
    AccessibleObject.setAccessible(new Field[] { delegate, requests}, true);
    sun.net.www.MessageHeader headers = (sun.net.www.MessageHeader)requests.get(delegate.get(conn));
    return headers.findValue("Authorization");
  } catch(ReflectiveOperationException ex) {
    ex.printStackTrace();
    return "";
  }
}

Answer 2

Stepping down the delegate-ladder one step further works and successfully manages to extract the "Authorization"-header value:

private String getAuthorizationHeaderValue(HttpURLConnection conn) {

    try {
        Field delegateField = conn.getClass().getDeclaredField("delegate");
        delegateField.setAccessible(true);
        DelegateHttpsURLConnection delegate = (DelegateHttpsURLConnection) delegateField.get(conn);
        Field requests = delegate.getClass().getSuperclass().getSuperclass().getDeclaredField("requests");
        requests.setAccessible(true);
        MessageHeader headers = (MessageHeader) requests.get(delegate);
        return headers.getValue(headers.getKey("Authorization"));
    } catch (IllegalAccessException | NoSuchFieldException e) {
        e.printStackTrace();
    }

    return "";
}