Questions tagged [ndjson]

NDJSON is a newline delimited JSON format.

NDJSON is a newline delimited JSON format. It is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It's a great format for log files. It's also a flexible format for passing messages between cooperating processes.

Spec: http://ndjson.org

119 questions
1
vote
2 answers

Read Chunked JSON Stream in Tabulator Grid

I have a Tabulator Grid, and use Ebay's jsonpipe library to load the JSON from my server in a chunked format, so I can begin filling my datagrid immediately even when the results are very large. The method for getting this kind of data into…
Stacey Reiman
  • 666
  • 4
  • 13
1
vote
1 answer

How to retrieve multiple NDJSON objects from the same file using ArduinoJson?

I am trying to use ArduinoJson to parse Google's quickdraw dataset, which contains .ndjson files with multiple objects inside. I figured how to retrieve the first of the objects in the file using the following simple code: DeserializationError…
synchronizer
  • 1,955
  • 1
  • 14
  • 37
1
vote
0 answers

Stream local file to draw map with d3

I found several useful pieces of information to solve my problem, but I can't solve the puzzle by myself. My project has to draw a map with d3. The main problem (as for many others) is that the files to be displayed are huge. After some tests with…
epilurzu
  • 147
  • 8
1
vote
0 answers

How to convert ndjson data into numpy to extract image data?

Following the Google's doodle dataset I would like to know how to get the numpy 28x28 image data out of a .ndjson file (just the image data). I am aware that they also provide the dataset in a numpy version but I am facing a similar issue with…
Mateo Randwolf
  • 2,823
  • 1
  • 6
  • 17
1
vote
2 answers

NewLine error in Elasticsearch bulk API post request

I am trying to use the elasticsearch bulk api to insert multiple records into an index. My JSON looks something like this: request json I am inserting a new line (\\n) at the end of the document but I'm still getting the newline error. Error: { …
1
vote
0 answers

How can I read a json file from a remote server in R?

So I have a collection on json files located on my local machine that I am reading in currently using the command file <- tbl_df(ndjson::stream_in("path/to/file.json") I have copied these files to a linux server (using WinSCP) and I want to stream…
grapestory
  • 183
  • 11
1
vote
1 answer

r How to parse pretty-printed JSON with newlines within character strings

I am struggling to parse a JSON in R which contains newlines both within character strings and between key/value pairs (and whole objects). Here's the sort of format I mean: { "id": 123456, "name": "Try to parse this", "description":…
Tom Wagstaff
  • 1,443
  • 2
  • 13
  • 15
1
vote
2 answers

De-normalize json object into flat objects

I have a json object like { "id": 3590403096656, "title": "Romania Special Zip Hoodie Blue - Version 02 A5", "tags": [ "1ST THE WORLD FOR YOU <3", "apparel", ], "props": [ …
Dakait
  • 2,531
  • 1
  • 25
  • 43
1
vote
1 answer

jq filter to transform jsonlines into a single json map

Input: {"aa":["a1","a2"]} {"bb":["a1","b2"]} Wanted output (no worries about potentially duplicate keys): { "aa":["a1","a2"], "bb":["b1","b2"] } Many thanks!
type
  • 1,137
  • 2
  • 9
  • 16
1
vote
1 answer

How to solve CORS error consuming an API using oboe in Angular2 app?

I have API that returns data like…
char m
  • 7,840
  • 14
  • 68
  • 117
1
vote
1 answer

How to best flatten NDJson data in Python

I have a huge file (>400MB) of NDJson formatted data and like to flatten it into a table format for further analysis. I started iterate through the various objects manually but some are rather deep and might even change over time, so I was hoping…
DirkLX
  • 1,317
  • 1
  • 10
  • 16
1
vote
1 answer

Opening ndJSON format in SQL Server 2016

How do I open ndJSON format in SQL Server 2016? I am able to open with JSON format, but am clueless on how to do it with ndJSON. Is there a particular function in SQL Server that will do this, or is there another approach? Declare @JSON…
Dip
  • 343
  • 7
  • 22
1
vote
1 answer

Left join for D3 mapping returns null that prevents next step of data processing (ndjson-join or jq) command line

I am doing D3 mapping on a state level. Here one problem that i met in data processing. For example, map data are like this, (dat1.ndjson) {state: a, code: aa} {state: b, code: bb} {state: c, code: cc} But usually the information we have are not…
Eleanor
  • 2,647
  • 5
  • 18
  • 30
1
vote
1 answer

How to close a https stream in Node.js

I'm loading a .ndjson file via https. I want to close it after reading 100 lines of the file. const amount = 100; https.get(url, (res) => { var { statusCode } = res; if (statusCode !== 200) { throw new Error(`Request Failed.\n…
Thomas Wagenaar
  • 6,489
  • 5
  • 30
  • 73
1
vote
1 answer

ElasticSearch is unable to recognize Context-Type header with encoding defined

I've spent some time trying to fix the elastic search bulk upload warning: Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header My request is below: POST http://elasticserver/_bulk…
Mando
  • 11,414
  • 17
  • 86
  • 167