SOAP, WCF, and message signatures

Question

I have a Java (JAX-WS based) SOAP client that I'm trying to get to talk with a (third-party) WCF-based server. I'm finding the sentiment expressed here to be quite accurate. But the goal still remains.

So long story short, I can coax a valid "security context token" out of the server, but am getting hung up on message-signing issues (I believe).

The server appears to expect the message to be signed using an hmac-sha1 authentication code using client/server secret keys (PSHA1 algorithm). Fair enough. However JAX-WS appears to want to use rsa-sha1 and an X509 certificate to sign the outbound messages (which the server doesn't like), and only seems to use hmac-sha1 if a UsernameToken is provided (which the server also doesn't like).

So I'm trying to manually sign the outbound SOAP messages from within a SOAPHandler implementation. The request that the client sends in order to get a security context token looks like this:

<t:RequestSecurityToken xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">
    <t:TokenType>http://schemas.xmlsoap.org/ws/2005/02/sc/sct</t:TokenType>
    <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>
    <t:Entropy>
        <t:BinarySecret Type="http://schemas.xmlsoap.org/ws/2005/02/trust/Nonce">NzM1MDZjYWVkMTEzNDlkNGEyODY0ZDBlMjlkODEyMTM=</t:BinarySecret>
    </t:Entropy>
    <t:KeySize>256</t:KeySize>
</t:RequestSecurityToken>

And the token that the server sends back look like this:

<t:RequestSecurityTokenResponse xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">
    <t:TokenType>http://schemas.xmlsoap.org/ws/2005/02/sc/sct</t:TokenType>
    <t:RequestedSecurityToken>
        <c:SecurityContextToken xmlns:c="http://schemas.xmlsoap.org/ws/2005/02/sc" u:Id="uuid-106bdbae-76e5-4195-b5d0-cc1c1a7a813e-13">
            <c:Identifier>urn:uuid:c0be4929-da8d-4955-8e13-b25aa7a37217</c:Identifier>
        </c:SecurityContextToken>
    </t:RequestedSecurityToken>
    <t:RequestedAttachedReference>
        <o:SecurityTokenReference xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
            <o:Reference ValueType="http://schemas.xmlsoap.org/ws/2005/02/sc/sct" URI="#uuid-106bdbae-76e5-4195-b5d0-cc1c1a7a813e-13" />
        </o:SecurityTokenReference>
    </t:RequestedAttachedReference>
    <t:RequestedUnattachedReference>
        <o:SecurityTokenReference xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
            <o:Reference URI="urn:uuid:c0be4929-da8d-4955-8e13-b25aa7a37217" ValueType="http://schemas.xmlsoap.org/ws/2005/02/sc/sct" />
        </o:SecurityTokenReference>
    </t:RequestedUnattachedReference>
    <t:RequestedProofToken>
        <t:ComputedKey>http://schemas.xmlsoap.org/ws/2005/02/trust/CK/PSHA1</t:ComputedKey>
    </t:RequestedProofToken>
    <t:Entropy>
        <t:BinarySecret u:Id="uuid-106bdbae-76e5-4195-b5d0-cc1c1a7a813e-14" Type="http://schemas.xmlsoap.org/ws/2005/02/trust/Nonce">dssunihZGy2dnnDHV9PMe3vU3lg/kKKZQkFohvGvCAk=</t:BinarySecret>
    </t:Entropy>
    <t:Lifetime>
        <u:Created>2016-04-08T04:11:54.392Z</u:Created>
        <u:Expires>2016-04-08T19:11:54.392Z</u:Expires>
    </t:Lifetime>
    <t:KeySize>256</t:KeySize>
</t:RequestSecurityTokenResponse>

I'm combining the client and the server BinarySecret keys using PSHA1 as follows:

private byte[] getSharedKey() {
    try {
        //FIXME:  client key first, or server key first?
        P_SHA1 algo = new P_SHA1();
        return algo.createKey(getBinaryClientEntropy(), getBinaryServerEntropy(), 0, getSharedKeySize() / 8);
    }
    catch (Throwable e) {
        LOG.error("Unable to compute shared key!", e);
    }

    return null;

}

I'm then using that key to compute a MAC for the message, like:

Mac mac = Mac.getInstance("HmacSHA1");
SecretKeySpec key = new SecretKeySpec(getSharedKey(), "HmacSHA1");
mac.init(key);

byte[] signatureBytes = mac.doFinal(content);
String signature = Base64.encodeBytes(signatureBytes);

That then goes into the outbound requests (along with a ton of other boilerplate things), as the SignatureValue. Ultimately I end up with something like:

<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope">
    <S:Header xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:scon="http://schemas.xmlsoap.org/ws/2005/02/sc" xmlns:sec="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
        <sec:Security xmlns:env="http://www.w3.org/2003/05/soap-envelope" env:mustUnderstand="true">
            <scon:SecurityContextToken xmlns:util="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" util:Id="uuid-106bdbae-76e5-4195-b5d0-cc1c1a7a813e-55">
                <scon:Identifier>urn:uuid:3ab0f3fb-edd4-4880-af77-d700dda371bb</scon:Identifier>
            </scon:SecurityContextToken>
            <sig:Signature xmlns:sig="http://www.w3.org/2000/09/xmldsig#">
                <sig:SignedInfo>
                    <sig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                    <sig:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#hmac-sha1" />
                </sig:SignedInfo>
                <sig:SignatureValue>ohqViTbUYBG2E3hLldUA1AsPBJM=</sig:SignatureValue>
                <sig:KeyInfo>
                    <sec:SecurityTokenReference>
                        <sec:Reference URI="#uuid-106bdbae-76e5-4195-b5d0-cc1c1a7a813e-55" ValueType="http://schemas.xmlsoap.org/ws/2005/02/sc/sct" />
                    </sec:SecurityTokenReference>
                </sig:KeyInfo>
            </sig:Signature>
        </sec:Security>
    </S:Header>
    <S:Body>
        <ns2:HelloWorld xmlns:ns2="http://tempuri.org/" xmlns:ns3="http://schemas.microsoft.com/2003/10/Serialization/">
            <ns2:name>Test</ns2:name>
        </ns2:HelloWorld>
    </S:Body>
</S:Envelope>

That leads to "An error occurred when verifying security for the message" responses coming back from the server.

Using wcf-storm to fire off requests and Fiddler2 to inspect the outgoing packets, I know that that I should be close. The following request works correctly:

<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope">
    <S:Header xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:scon="http://schemas.xmlsoap.org/ws/2005/02/sc" xmlns:sec="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
        <o:Security xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" s:mustUnderstand="1">
            <u:Timestamp u:Id="_0">
                <u:Created>2016-04-05T23:48:06.110Z</u:Created>
                <u:Expires>2016-04-05T23:53:06.110Z</u:Expires>
            </u:Timestamp>
            <c:SecurityContextToken xmlns:c="http://schemas.xmlsoap.org/ws/2005/02/sc" u:Id="uuid-8085da33-b25c-4f09-b5a9-110635a3ae39-2005">
                <c:Identifier>urn:uuid:91349027-cb32-4c46-9f16-74a6bcb11126</c:Identifier>
            </c:SecurityContextToken>
            <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
                <SignedInfo>
                    <CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                    <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#hmac-sha1" />
                    <Reference URI="#_0">
                        <Transforms>
                            <Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                        </Transforms>
                        <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
                        <DigestValue>AvRXi7pyjulsfdg9afInSFMM+5k=</DigestValue>
                    </Reference>
                </SignedInfo>
                <SignatureValue>TQup7BBN43b8CefrdSRd+X8MBgg=</SignatureValue>
                <KeyInfo>
                    <o:SecurityTokenReference>
                        <o:Reference ValueType="http://schemas.xmlsoap.org/ws/2005/02/sc/sct" URI="#uuid-8085da33-b25c-4f09-b5a9-110635a3ae39-2005" />
                    </o:SecurityTokenReference>
                </KeyInfo>
            </Signature>
        </o:Security>
    </S:Header>
    <S:Body>
        <ns2:HelloWorld xmlns:ns2="http://tempuri.org/" xmlns:ns3="http://schemas.microsoft.com/2003/10/Serialization/">
            <ns2:name>Test</ns2:name>
        </ns2:HelloWorld>
    </S:Body>
</S:Envelope>

The main differences are:

• I've omitted the Timestamp element (though I've tried including it, and didn't seem to make any difference).
• I've omitted the SignedInfo/Reference element, because I'm not sure how its DigestValue is meant to be computed.

So after all of that, I suppose the main question is:

What is the actual algorithm for signing the outbound messages? As in, if I have:

<Envelope>
    <Header>
        HHH...
    </Header>
    <Body>
        BBB...
    </Body>   
</Envelope>

...am I meant to compute the signature value off of <Envelope>...</Envelope> (so the entire thing), or just <Body>...</Body>, or even just the BBB... part? And if I'm meant to use the entire thing, how do I reconcile that against the fact that adding the signature information to the header alters the content that's used as input when computing the signature?

Is there a more straightforward way to get JAX-WS to generate requests using the required signing conventions that I've overlooked?

And then there are some minor bonus questions:

1. Is there an established standard with respect to which order I pass the client and server BinarySecret values when combining them using PSHA1?

2. Are the Timestamp and SignedInfo/Reference entries significant, and if so, what's the correct method for computing the DigestValue?

Answer 1

After some research and a fair bit of trial and error, I managed to get a working solution. I'll start with the bonus questions first:

1. I didn't find any formal documentation, and every reference implementation and code sample that I came across always passed the client key first, and this is also what the server (Microsoft IIS v8.5) expects. So that would appear to be the standard, even if it's not a formal one.

2. Yes, the Timestamp and Reference values are significant in a big way, and very closely relate back to the main question.

So what's the actual algorithm for signing things in an outbound SOAP message, if you have to do so manually, in Java, using JAX-WS?

This reference is a helpful place to start, and should give you a pretty good idea of just how overarchitected things have become in the SOAP world. And some of it is very obtusely described, to boot. For instance:

3.2.2 Signature Validation

1. Obtain the keying information from KeyInfo or from an external source.
2. Obtain the canonical form of the SignatureMethod using the CanonicalizationMethod and use the result (and previously obtained KeyInfo) to confirm the SignatureValue over the SignedInfo element.

If your KeyInfo is a SecurityTokenReference to a SecurityContextToken that doesn't actually contain any key data itself, and your SignatureMethod is Algorithm="http://www.w3.org/2000/09/xmldsig#hmac-sha1", it's clear as mud how any CanonicalizationMethod relates to that or how you're supposed to get from that to knowing that you need to combine the server and client BinarySecret values and take the result to be your key. But I digress.

The algorithm to apply is more-or-less described in the Signature block. For instance, if the server you're talking to expects something like:

<o:Security xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" s:mustUnderstand="1">
    <u:Timestamp u:Id="_0">
        <u:Created>2016-04-11T00:53:44.050Z</u:Created>
        <u:Expires>2016-04-11T00:58:44.050Z</u:Expires>
    </u:Timestamp>
    <c:SecurityContextToken xmlns:c="http://schemas.xmlsoap.org/ws/2005/02/sc" u:Id="uuid-41b0578e-dc47-4467-9b65-b0cebde98309-1">
        <c:Identifier>urn:uuid:9eba64a2-5cf8-4ea9-85e9-359b2edbb13c</c:Identifier>
    </c:SecurityContextToken>
    <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
        <SignedInfo>
            <CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
            <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#hmac-sha1" />
            <Reference URI="#_0">
                <Transforms>
                    <Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                </Transforms>
                <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
                <DigestValue>CwJgqLNOoHJpuiqIOylvVvFli1E=</DigestValue>
            </Reference>
        </SignedInfo>
        <SignatureValue>fJxof0blfd6abX0V4EmPYZ/NGJI=</SignatureValue>
        <KeyInfo>
            <o:SecurityTokenReference>
                <o:Reference ValueType="http://schemas.xmlsoap.org/ws/2005/02/sc/sct" URI="#uuid-41b0578e-dc47-4467-9b65-b0cebde98309-1" />
            </o:SecurityTokenReference>
        </KeyInfo>
    </Signature>
</o:Security>

...you want to start with the Reference element, which points to the element with id "_0" (which in this case is the Timestamp element). You then canonicalize the referenced element according to the nominated Transform algorithm. That's most easily done by using Apache XML Security, roughly like:

SOAPElement timestamp = secHeader.addChildElement(soapFactory.createName("Timestamp", "u", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"));
//[add 'Created' and 'Expires' values, as required]

//once you're done adding stuff, you can canonicalize the element
Canonicalizer canonizer = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
byte[] canonTimestamp = canonizer.canonicalizeSubtree(timestamp);

That will give you something like this (newlines aren't canonical, sorry):

<u:Timestamp xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" u:Id="_0"><u:Created>2016-04-11T00:53:44.050Z</u:Created><u:Expires>2016-04-11T00:58:44.050Z</u:Expires></u:Timestamp>

Now you need to compute the DigestValue of that string. The DigestMethod element in our Reference element tells us that this should be a SHA1 hash (base64-encoded). So simply:

MessageDigest sha1 = MessageDigest.getInstance("SHA-1");
String canonDigestValue = Base64.encodeBytes(sha1.digest(canonTimestamp));

The value that you get goes into the Reference/DigestValue element (assuming you're building an outbound request). Once that is done, the Reference is complete, and since there aren't any additional Reference elements, so is the SignedInfo block.

Now to get the SignatureValue, you canonicalize the SignedInfo element, the same as before:

SOAPElement sigInfo = sigElem.addChildElement(new QName("SignedInfo"));
SOAPElement canon = sigInfo.addChildElement(new QName("CanonicalizationMethod"));
canon.addAttribute(soapFactory.createName("Algorithm"), Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
//[continue adding the other elements...]

//canonicalize the entire, completed 'SignedInfo' block
byte[] bytesToSign = canonizer.canonicalizeSubtree(sigInfo);

Which should net you something like:

<SignedInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></CanonicalizationMethod><SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#hmac-sha1"></SignatureMethod><Reference URI="#_0"><Transforms><Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"></Transform></Transforms><DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"></DigestMethod><DigestValue>CwJgqLNOoHJpuiqIOylvVvFli1E=</DigestValue></Reference></SignedInfo>

...and then you sign the whole thing according to the nominated SignatureMethod algorithm, which in our case is HmacSHA1:

Mac mac = Mac.getInstance("HmacSHA1");
SecretKeySpec key = new SecretKeySpec(getSharedKey(), "HmacSHA1");
mac.init(key);

String signature = Base64.encodeBytes(mac.doFinal(bytesToSign)); 

...where getSharedKey() returns, in this instance, a key derived using the BinarySecret values that the client and server sent during the initial RequestSecurityToken exchange. As in:

private byte[] getSharedKey() {
    try {
        //XXX:  doesn't seem to be formally specified anywhere, but convention appears to be that the client key always goes first
        P_SHA1 algo = new P_SHA1();
        return algo.createKey(getBinaryClientEntropy(),  //the 'BinarySecret' value that the client sent to the server, decoded to raw binary 
                              getBinaryServerEntropy(),  //the 'BinarySecret' value that the server sent to the client, decoded to raw binary
                              0,                         //offset, '0' is what we want here
                              getSharedKeySize() / 8);   //'KeySize' is 256 bits in this case (specified by server), divide by '8' to convert to bytes
    }
    catch (Throwable e) {
        LOG.error("Unable to compute shared key!", e);
    }

    return null;
}

Anyways, at this point you should have a signature value and it can be appended to the Security header in the outbound message, like:

SOAPElement sigValue = sigElem.addChildElement(new QName("SignatureValue"));
sigValue.addTextNode(signature);

And if all goes well, the message is now successfully signed and of acceptable quality for the server.

Though there was one final caveat that I noted, which is that the Timestamp values need to be generated in the server's timezone (which, in this case, is UTC), otherwise it will reject the request due to the timestamps either being from the future or having already expired. A simple problem that could have been solved by standardizing on UNIX epoch timestamps. But for some reason they went with "yyyy-mm-dd'T'hh:mm:ss.msec'Z'" instead. Go figure.

I hope this is helpful to the next unfortunate soul who has to try to get Java talking to .NET using SOAP/XML.

And one final note if you're using Apache XML Security. You need to call org.apache.xml.security.Init.init() before attempting to use the Canonicalizer, for instance from within a static initializer block. If you don't, you'll get an exception (NPE I think it was) when you try to canonicalize.