How can i parse json file in scala spark 2.0 and can I insert data into hive table?

Question

I want to parse json file in spark 2.0(scala). Next i want to save the data.. in Hive table. How can i parse json file by using scala? json file example) metadata.json:

  {
        "syslog": {
            "month": "Sep",
            "day": "26",
            "time": "23:03:44",
            "host": "cdpcapital.onmicrosoft.com"
        },
        "prefix": {
            "cef_version": "CEF:0",
            "device_vendor": "Microsoft",
            "device_product": "SharePoint Online",
        },
        "extensions": {
            "eventId": "7808891",
            "msg": "ManagedSyncClientAllowed",
            "art": "1506467022378",
            "cat": "SharePoint",
            "act": "ManagedSyncClientAllowed",
            "rt": "1506466717000",
            "requestClientApplication": "Microsoft SkyDriveSync",
            "cs1": "0bdbe027-8f50-4ec3-843f-e27c41a63957",
            "cs1Label": "Organization ID",
            "cs2Label": "Modified Properties",
            "ahost": "cdpdiclog101.cgimss.com",
            "agentZoneURI": "/All Zones",
            "amac": "F0-1F-AF-DA-8F-1B",
            "av": "7.6.0.8009.0",
        }
    },

Thanks

Welcome to StackOverflow please take a look in how to do questions https://stackoverflow.com/help/how-to-ask — Thiago Baldim, Nov 16 '17 at 22:58
I’ve downvotes because no efforts were shown in solving the issue — eliasah, Nov 18 '17 at 18:55

score 0 · Accepted Answer · answered Nov 17 '17 at 05:21

You can use something like:

val jsonDf = sparkSession
      .read
      //.option("wholeFile", true) if its not a Single Line JSON
      .json("resources/json/metadata.json")

    jsonDf.printSchema()

jsonDf.registerTempTable("metadata")

More details about this https://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables

How can i parse json file in scala spark 2.0 and can I insert data into hive table?

1 Answers1