Azure Purview - JSON properties in ADLSGen2 store are getting messed up

Question

I have a ADLS Gen2 lake. In the lake I have a taxonomy.

/environment/source/system/entity/myspecificfilenameconvention/*.json

Purview seems to pick up my paths and conventions just fine, however it is munging a simple JSON file format and giving this in the UI!

The file format is pretty straight forward as below. Any idea why Purview is making a dogs breakfast of it?

[
    {
        "attributes": {
            "type": "MyType",
            "url": "/path"
        },
        "Id": "11111",
        "IsDeleted": false,
        "AccountId": "222222",
        "CreatedById": "333333",
        "CreatedDate": "2019-10-26T08:43:04.000+0000",
        "Field": "created",
        "OldValue": null,
        "NewValue": null
    }, ...

Thanks @HongOoi but I have 100s of GB of the stuff. Tens of thousands of files. — Murray Foxcroft, May 19 '21 at 19:41
Thanks - we are coming up with a plan, the data flows in to the lake hourly on a schedule so we need to think it through a bit. — Murray Foxcroft, May 20 '21 at 05:55
Ideally, change your source to generate UTF-8 output, JSON is defined to be UTF-8 so it's noncompliant right now — Hong Ooi, May 20 '21 at 12:13
The format is indeed the issue - can you add as an answer and I will accept - thanks — Murray Foxcroft, May 21 '21 at 06:54

score 1 · Accepted Answer · answered May 25 '21 at 02:38

1

Thanks for @Hong Ooi comment. Just post an answer to end this question:

"Save your file as UTF-8, not UTF-16" did the trick.

answered May 25 '21 at 02:38

Steve Johnson

8,057
1
6
17

Azure Purview - JSON properties in ADLSGen2 store are getting messed up

1 Answers1