0

Using Azure Data Factory I have created a pipeline to upload any number of JSON files from Azure blob storage. I am loading the JSON data into a stage table with the following fields

  • FileName varchar(200)
  • JSONData nvarchar(max)

Using the SQL Server OPENJSON command, only 5,000 rows are returned, when there should be over 400,000 rows returned.

I used this query to parse the JSON data column into a final table (some info redacted):

 insert into jsonTable
     select * 
     from OPENJSON((select jsondata from rawJSONupload 
                    where filename = 'filename'))
              with (
                    field1 nvarchar(5),
                    field2 real,
                    field3 real,
                    EnteredDate datetime,
                    FilePath nvarchar(500)
                   )

My issue is that I am only getting 5,000 rows returned from the SELECT * FROM OPENJSON(...) query, where I know that I should have 400,000+ rows returned. Is there a max of 5,000 rows returned using OPENJSON, and if so is there a way around that, or do I need to find another way to load JSON data into a table (not one at a time).

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
xxvann
  • 1
  • 1

1 Answers1

0

This is a limitation of Azure Data Factory, not SQL Server (OPENJSON work fine with big json). The upload is limited to 5000 thousand lines. The limit is specified, for example, in this documentation - https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity#limitations-and-workarounds (section "Limitations and workarounds").

You need to look for a solution in the corresponding tags, e.g:

Handle >5000 rows in Lookup @ Azure Data Factory

Handling big JSONs in Azure Data Factory

Arzanis
  • 81
  • 4