I'd like to validate the signature on an elliptic curve public key received using ECDHE, but I can't find any reference that clearly explains which bytes are signed.
I set up a TLS v1.2 connection to google.com using cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f). I captured traffic with Wireshark and I can see the client hello, server hello, certificate, and server key exchange messages. My understanding is that the EC public key that Google sent is signed using the private key of certificate Google sent (RSA, in this case).
Here is the Server Key Exchange message:
160303014d0c0001 490300174104f930
e65768e0587ec7e1 b8b537ccd6ae2500
3a364b84a68ed7c0 47d18dd104afb63c
cc72e800495db3cd d629807f0d4501a4
c043c5c7c52aea45 a66692aa11b60201
01007ec0b1ef4994 30f42f3ed9a7a592
92c0f875ad7cd2f8 5b36a7aec804f602
2959549a8a3d0e5c 5825fefa4d69f360
34eaad7138e5da69 61bdfb88ddb5172c
ba64071de0764fc1 c8b895dbc52ec85c
3b7891c53e6d843b 44f80c481a9beb86
c444b32204e9bc6c 6665e6dd26887c5e
fc4e331fbdd66536 686b6b5f16072b52
ee2fee75ca65e28f a0ee0644b91fba30
783798aa83cf28f1 394b1344b43104cb
89aed55030bd7561 d13ae20d4d7bc17e
682e6c6266f04bf6 31665a547e2f15b3
c79fda548a781d39 5d64f4eea75aac96
9374ce60400fdc11 3a3d5a98b62f63b7
6e5324797c938f39 bc1cc5736b612bd7
7a1bc790841d4e25 dae648cab33273e2
588c
Parsing this gives:
160303014d - record header
0c - server key exchange message
000149 - length of message
03 - named curve
0017 - secp256r1 curve
41 - length of public key
04 - not sure what this means
f930e6...04af [32 bytes] - x value of public key
b63ccc...11b6 [32 bytes] - y value of public key
02 - SHA-1 hash
01 - RSA signature
0100 - length of signature
7ec0b1...588c [256 bytes] - signature value
Using the certificate I received in the certificate message, I was able to perform a public key operation on the signature of the EC public key and it looks like a properly padded SHA-1 hash. However, I cannot verify this hash value. I tried many different combinations of the raw EC public key with its headers, but nothing I tried hashes properly.
Here is how I retrieved the hash:
openssl rsautl -inkey ~/googlePubKey.pem -encrypt -in ~/googleECpubkeysig -pubin -raw | hexdump
0000000 0100 ffff ffff ffff ffff ffff ffff ffff
0000010 ffff ffff ffff ffff ffff ffff ffff ffff
*
00000d0 ffff ffff ffff ffff ffff ffff 3000 3021
00000e0 0609 2b05 030e 1a02 0005 1404 3ac5 fb13
00000f0 9ff8 77f1 6a69 09af 472a 90b2 cac6 b4f8
0000100
The last 20 bytes look like a SHA-1 hash, which is the algorithm specified in the signature. Which bytes from the Server Key Exchange should I be hashing to get this value? Or, is there some transformation or other data that I must perform or add prior to hashing?