Stack Overflow
Stack Overflow

Questions tagged [ndjson]

NDJSON is a newline delimited JSON format.

NDJSON is a newline delimited JSON format. It is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It's a great format for log files. It's also a flexible format for passing messages between cooperating processes.

Spec: http://ndjson.org

119 questions
2
votes
0 answers

Convert json string to ndjson string using java

I'm trying to convert normal json string to new line delimited json format (ndjson). I found the below unix command to perform the same. cat test.json | jq -c '.[]' > testNDJSON.json I need an equivalent java functionality. PFB the json string which…
Jefila
  • 131
  • 1
  • 11
2
votes
2 answers

How do I parse ndjson files?

I performed a GET request from lichess.org, API reference: https://lichess.org/api#operation/apiGamesUser import requests import json import ndjson response = requests.get('https://lichess.org/api/games/user/ardito_bryan', params={'max':10}) I am…
ardito.bryan
  • 429
  • 9
  • 22
2
votes
1 answer

How to write a splittable DoFn in python - convert json to ndjson in apache beam

I have a large dataset in GCS in json format that I need to load into BigQuery. The problem is that the json data is not stored in NdJson but rather in a few large json files, where each key in the JSON should really be a field in json itself. For…
SockworkOrange
  • 355
  • 4
  • 14
2
votes
1 answer

Store invalid JSON columns are STRING or skip them in BigQuery

I have a JSON data file which looks something like below { "key_a": "value_a", "key_b": "value_b", "key_c": { "c_nested/invalid.key.according.to.bigquery": "valid_value_though" } } As we know BigQuery considers…
Amit Yadav
  • 4,422
  • 5
  • 34
  • 79
2
votes
0 answers

How to check ndjson format validation?

So I've been given a NDJSON file to import and then display on a webpage in a table. I keep getting an error inside the data and I feel like it with the format. I will delete the variable with the issue and it will go on to the next one. I've tried…
jenna
  • 113
  • 8
1
vote
2 answers

Need to modify an existing JQ filter of a GitHub CLI GraphQL response

I am creating a GitHub Action workflow which will call a GitHub CLI API request using GraphQL. This gh api graphql response is --paginate and returns JSON Lines (ndjson). I created the GraphQL and jq queries, and I am close to the desired output;…
Christopher Rucinski
  • 4,737
  • 2
  • 27
  • 58
1
vote
0 answers

How to Convert a JSON Stream to NDJSON easily

How can I convert a fetch response stream where the document being downloaded is JSON to a NDJSON stream? For example this code: const response = await fetch(url); ndjsonStreamer = ndjsonStream(response.body).getReader(); works well for ndjson…
Victor Ekpo
  • 11
  • 4
1
vote
0 answers

How to format ndjson files in VS Code

I am trying to format ndjson file in VS Code but it keeps coming up with the message that there is no ndjson file formatter installed. Any idea how to resolve this ? I can't seem to find any extension for formatting ndjson. Shouldn't it just use the…
nabeelfarid
  • 4,156
  • 5
  • 42
  • 60
1
vote
0 answers

How do I deploy a huge NDJSON file into a cluster for distributed computation and storage? (DARPA TC Engagement 5 Theai)

I am a data scientist with the task of anomaly detection on system logs and I want to experiment on the DARPA TC Engagement 5 dataset. I have downloaded the included scripts to import the data and now the data is being parsed and stored in a single…
Elad Cohen
  • 453
  • 3
  • 16
1
vote
0 answers

SpringBoot to Flutter Stream using application/x-ndjson

I am trying to get a Flutter app to process a NDJSON stream properly, but cannot get it to work. I have a SpringBoot server that the Flutter app requests a stream from. SpringBoot side: @GetMapping(value = "/streaming", produces = { // …
jacobw
  • 41
  • 4
1
vote
1 answer

Amazon S3 json files to BigQuery

I want to transfer json files from Amazon S3 to BigQuery, but i've got the problem that BigQuery only support new delimited json files. I don't know how to transform my json files to ndjson. The problem is i dont know how to transform, because the…
databekele
  • 11
  • 2
1
vote
1 answer

convert ndjson to json in python

i need to convert ndjson objects to json in python i see there is a library in pypi.org but i'm not able to use it it is the ndjson…
zanza67
  • 223
  • 4
  • 20
1
vote
0 answers

Reading a JSONstream into an R dataframe

I'm trying to read a (to me) weirdly formatted JSONstream into an R dataframe efficiently. It's for a personal project to learn more R. The json I'm talking about…
Keipi
  • 131
  • 7
1
vote
1 answer

How do you convert an ndjson file to a csv file in Python?

I currently have my file loaded as follows: import ndjson with open('full_simplified_arm.ndjson') as f: data = ndjson.load(f) However, I would like to convert this file into a csv in Python and was wondering how to do this? Thanks
Jane Miller
  • 153
  • 9
1
vote
1 answer

Converting nested JSON streamind data to ndjson

I am currently working with Twitter stream data and I want to convert the nested JSON response to ndjson using python. I saw a few examples using json.normalize but that just seperated it to one level and my output has much deeper levels. I am new…
Hrishikesh Mahajan
  • 43
  • 7
1
2
3 4 5 6 7 8