Filter specific value from Tshark JSON output

Question

I would like to extract the "tls.handshake.certificate_raw" values, but without success so far. The big JSON is killing me.

Here's my JSON file: Download

Thanks in advance!

Answer 1

Since the JSON has objects with duplicate keys, an approach that does not use jq's normal object semantics is warranted.

jq's streaming parser (invoked with the --stream option) allows one to handle objects with duplicate keys:

jq --stream -c '(.[0]|index("tls.handshake.certificate_raw")) as $ix
                | select($ix) | .[0] |= .[$ix+1:]' Tshark.json |
    jq -nc 'fromstream(inputs)'

This produces 503 entities, the first of which is:

["308208df308206c7a00302010202132d000491fc30ee99b24196c0d30000000491fc300d06092a864886f70d01010b050030818b310b3009060355040613025553311330110603550408130a57617368696e67746f6e3110300e060355040713075265646d6f6e64311e301c060355040a13154d6963726f736f667420436f72706f726174696f6e31153013060355040b130c4d6963726f736f6674204954311e301c060355040313154d6963726f736f667420495420544c532043412035301e170d3138313130323139353233375a170d3230313130323139353233375a3021311f301d06035504030c162a2e636c6f2e666f6f747072696e74646e732e636f6d30820122300d06092a864886f70d01010105000382010f003082010a0282010100ab93c55b9e948806364d71adf61ed1ae88c3f3531d8c03df279da3c618391d59636aaeaf9981ac75aaf95e98a97b5bf0fcbc4d819a6c896b29d12652ffa2b603a7edb61eb921a02e2c6730275f3ec23998aab45c5dd4d00cb91ac0cbae0dd8f897cd0b34d81056429c4d854d45b309644df10e7d7fe5a99643eb935263d862a5f4d6750b8202d16c0f89a6b888e8361dcd5314bdb3093b7d6b55106f874824c4a220657f72f8a8ab0c3da27f3f76d343ef85482c88f5423b88cc008c2d9e4ea3b3c373ff22e881491ad2922b6cf5076786b886b1c0e8c603d28d014dd0cd1216c59657c9806b2f1d34795286f729ce712728c12b80197df61790fe629d4732490203010001a38204a33082049f308201f7060a2b06010401d679020402048201e7048201e301e1007600a4b90990b418581487bb13a2cc67700a3c359804f91bdfb8e377cd0ec80ddc1000000166d6053ea50000040300473045022100865c52fbcc0846f7826de85fe28ee5192c706671e208359cc884ef8f862c384a022003dfd681fb2488ccbd8452733eaa7defafe80e1b834fe2a1bb8810cf492883b8007700bbd9dfbc1f8a71b593942397aa927b473857950aab52e81a909664368e1ed18500000166d6053f9a0000040300483046022100d0d6894ad6c1c2ff24a426483e354f21a2172ed4d2c3e544d887d2021f16b6e6022100eb61a67dd9e1b00c244750d43acac412ef9bd47ce77e85db86a6f83ee656c47e0077005ea773f9df56c0e7b536487dd049e0327a919a0c84a11212841875968171455800000166d6053eb20000040300483046022100f81aae0caef665d9071e5ebb949ccfc4033038c9df3227f3e90d46d77ed43574022100f76f1a30eb8b7ff4175cba86e07f46b6f625d1698c91b348a8e8ede7752c2456007500f095a459f200d18240102d2f93888ead4bfe1d47e399e1d034a6b0a8aa8eb27300000166d6053ebd0000040300463044022038393db91d24a1a0121f64446fdb7297406f89ae6d1abe39714073d8dd6bf79202203fec31afceb6a00663f7e3b5e4d9930c0b533dd938bc2ce82117502e97a30863302706092b060104018237150a041a3018300a06082b06010505070302300a06082b06010505070301303e06092b06010401823715070431302f06272b060104018237150887da867583eed90182c9851b81b59e6185f4eb60815d84d2df4282e7937a02016402011d30818506082b0601050507010104793077305106082b060105050730028645687474703a2f2f7777772e6d6963726f736f66742e636f6d2f706b692f6d73636f72702f4d6963726f736f66742532304954253230544c532532304341253230352e637274302206082b060105050730018616687474703a2f2f6f6373702e6d736f6373702e636f6d301d0603551d0e04160414e8e1323d172500a9588138faa2d310619350ac06300b0603551d0f0404030204b030470603551d110440303e82162a2e636c6f2e666f6f747072696e74646e732e636f6d8211632d72696e672e6d73656467652e6e65748211712d72696e672e6d73656467652e6e65743081ac0603551d1f0481a43081a130819ea0819ba08198864b687474703a2f2f6d7363726c2e6d6963726f736f66742e636f6d2f706b692f6d73636f72702f63726c2f4d6963726f736f66742532304954253230544c532532304341253230352e63726c8649687474703a2f2f63726c2e6d6963726f736f66742e636f6d2f706b692f6d73636f72702f63726c2f4d6963726f736f66742532304954253230544c532532304341253230352e63726c304d0603551d2004463044304206092b0601040182372a013035303306082b060105050702011627687474703a2f2f7777772e6d6963726f736f66742e636f6d2f706b692f6d73636f72702f637073301f0603551d2304183016801408fe259f74ea8704c2bcbb8ea8385f33c6d16c65301d0603551d250416301406082b0601050507030206082b06010505070301300d06092a864886f70d01010b050003820201007873266ca28f9b88b653ce3d9eff1d8d61f3cd6513b10228b4c9ea7276f8bca602812c10b45d7055b593880b632bbd8853b945b62033b159c4ad521304f5c2d55edbc027ce3d45aff48fa0c02d4f1b935ae1f2e3633c40bdaa0e618f08ec58996a52b2e0e1a6beb94d481255f5ee70a459ffb5f0dad8b9f739fcc6a7cb97a70229190864a70100e39e0e1c33383413a2e527aab47d1c9930a7f43082478e644ed6e4a3d4625a0fe8bda7fb731e569b38ec1fb75a5b7a51535af0bc96280da67d775fb9131027a13242a39d62e93fc140b7bcf201483d8d6a3ad1326386738dd8c707982bf5003da56a9c096b90d0d3827150e2b1d66393016bbb5b180aba7d98098fbf63ed9b8c85c64cd6414689e684424c68514bbb9f6da3d73165fbd58d972a7fba91fccda255b88c421b30d0961bb431bd2c6bfcb7b5a680acff3777ad911f030af3d552d69a7a6f27ca10ebbcab7195d6a363e769c872da596fd0641889c47f881b155df2ac045a5a016ac769d611e6c0cfcfcbaf9039d13ff6d321fff5abd5f922e8623162192eed42170eb090356549fa0867fa890bdbaf82f2fc41028494a133ccc97e7b6b4a21f6ad1aba032bfb1aa67b9247c97065dcd171270ea387862d92d8acd22994efdf03247d17c8b033f68d88231b655e4d22f30696aaf9e5667cebeb6ed001a3ea0e48a6231d1488a46755828c93a746e4c33a645db0d4",109,2275,0,30]

Explanation

The first invocation of jq uses jq's streaming parser to extract the [path, atomicValue] pairs of interest. The second invocation reconstructs the relevant entities.

Resources

Here are the main stats from a run using /usr/bin/time -lp on a Mac Mini:

real         5.35
user         5.31
sys          0.05
  13279232  maximum resident set size

Answer 2

The following assumes that if a JSON object in the file has a specific key more than once, then only the last one is relevant. If this assumption is not valid, then one option would be to use jq's streaming parser.

The following shows that a simple application of jq to retrieve the named key requires about 26MB RAM. Is that a problem in this day and age?

/usr/bin/time -lp jq '.. | objects | .["tls.handshake.certificate_raw"] // empty'  Tshark.json

Output:

[
  "308204633082034ba00302010202100182f8098ea2e626b91a3b27841fb9af300d06092a864886f70d01010b0500305a310b300906035504061302494531123010060355040a130942616c74696d6f726531133011060355040b130a43796265725472757374312230200603550403131942616c74696d6f7265204379626572547275737420526f6f74301e170d3135313230383132303530375a170d3235303531303132303030305a3064310b300906035504061302555331153013060355040a130c446967694365727420496e6331193017060355040b13107777772e64696769636572742e636f6d312330210603550403131a44696769436572742042616c74696d6f72652043412d3220473230820122300d06092a864886f70d01010105000382010f003082010a0282010100bbe700fe000173ef9b88f05c0875f05c70731fffb57a1f8d0823fdc7c1133fb12106bd4dccdb2db8871931deda6b0dc15dd251d5b05e72d4f066d484f2911841b50926b2039604699553fa074f3ebe863a432bb259b5abd1baf45940725a67b6cde655fc092ba498b2bfcca91fd4bf878f8225fd4580a9ec9d8cf97490cf1cf86550afcdf845abc82f8d8e361ddeea89cdb252d9e509cd36712bdac6d09bc6f6a45b5b39b61f001f19a4fbd3be0ce6fb8fdd0e80e6a4e75ac4c02f72ee91f38367c9cdcace4ab02081fde472af0918e74f42ec4d27424d07a0ddd50d75f73660cc265498a18837f083f7ad295920cfbb25e1d59ee6f3aec101ebb50689e20a8f0203010001a382011930820115301d0603551d0e04160414c012b22874684667e97025741a00455b067d5c44301f0603551d23041830168014e59d5930824758ccacfa085436867b3ab5044df030120603551d130101ff040830060101ff020100300e0603551d0f0101ff040403020186303406082b0601050507010104283026302406082b060105050730018618687474703a2f2f6f6373702e64696769636572742e636f6d303a0603551d1f04333031302fa02da02b8629687474703a2f2f63726c332e64696769636572742e636f6d2f4f6d6e69726f6f74323032352e63726c303d0603551d200436303430320604551d2000302a302806082b06010505070201161c68747470733a2f2f7777772e64696769636572742e636f6d2f435053300d06092a864886f70d01010b050003820101002fe23766c31acf9155ee2914fa5010be998de228f1741607693b44303df5b949f56836feb7308da9e0074942634652b569c69b49f21a0f57c6f6b7beab758d0da2ab33e462854354ab63940ca353a0ea1de4a6a6ac950e6578ec338cbe943ed870abeac7e178ce3395bbfb2c58e08a40a66d41d8d8d9b3f92f02c51b706be23a857b1d18b0d14b9ac16f171029e80ec3e1549654a5919508208d692c9e2f3f50b42ee5571fafa473d2a124bb0d90fe45131da7a129d8660bc3d16cb34fb42b844c4790ec8be2418913b607467742fdbb1e17faff254b8e5057eb8ecc4e153fb125dd410ca09c626d6fba97078bd9f30610c529d4f7ba1690a8d054e14b49c798",
  1778,
  1127,
  0,
  30
]
real         1.42
user         1.34
sys          0.03
  26693632  maximum resident set size

Answer 3

My final command:

jq --stream -c '(.[0]|index("tls.handshake.certificate_raw")) as $ix
            | select($ix) | .[0] |= .[$ix+1:]' Tshark.json |
jq -r -nc 'fromstream(inputs)[0]' | sort -u > Certificates.txt

I grab only the first value from the array and remove the double quotes. After a sort uniq I write the SSL certificates line by line into a file.

Thanks a lot for your great assistance!