Stack Overflow
Stack Overflow

Questions tagged [ndjson]

NDJSON is a newline delimited JSON format.

NDJSON is a newline delimited JSON format. It is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It's a great format for log files. It's also a flexible format for passing messages between cooperating processes.

Spec: http://ndjson.org

119 questions
0
votes
2 answers

Replace JSON Key Values and Break up values in python

I have a set of ndJOSN dataset like the below: {'ADDRESS_CITY': 'Whittier', 'ADDRESS_LINE_1': '905 Greenleaf Avenue', 'ADDRESS_STATE': 'CA', 'ADDRESS_ZIP': '90402',}, {'ADDRESS_CITY': 'Cedar Falls', 'ADDRESS_LINE_1': '93323 Maplewood Dr',…
0004
  • 1,156
  • 1
  • 14
  • 49
0
votes
1 answer

How to convert data to desired format and write to a file - Python + Apache Beam

I have a .ndjson file looks like this: {"property_id": "107", "transaction_unique_id": "{C3C3F9B5-FB9E-362B-E053-6B04A8C03ACC}", "price": 80000, "date_of_transfer": "2021-05-07 00:00", "postcode": "BL2 2GY", "property_type": "F", "old_new": "N",…
wawawa
  • 2,835
  • 6
  • 44
  • 105
0
votes
1 answer

Convert types from strings to ints in newline delimited JSON file

According to How to preserve integer data type when exporting to JSON?, it is not currently possible to preserve integer types when exporting from BigQuery to JSON. This minor detail about BigQuery --> GCS JSON exports has been causing us many…
Canovice
  • 9,012
  • 22
  • 93
  • 211
0
votes
1 answer

Concatenate folder of multiple newline-delimited JSON files into single file

We have a directory /our_jsons that has the files: file1.json {"team": 1, "leagueId": 1, "name": "the ballers"} {"team": 2, "leagueId": 1, "name": "the hoopers"} file2.json {"team": 3, "leagueId": 1, "name": "the gamerrs"} {"team": 4, "leagueId":…
Canovice
  • 9,012
  • 22
  • 93
  • 211
0
votes
2 answers

Merging 2 json files

I'm trying to merge both json files but I'm trying to append timestamp from file2 to corresponding frame number in file1.please…
damu
  • 1
  • 2
0
votes
1 answer

Get properties of a JSON object streamed as NDJSON using fetch

I am trying to get NDJSON data from an API using fetch. Since I only need one JSON object, I would like to do this using fetch. The data provided by the API is in the form (formatting by me, actual response is a single line): { "a": "value", …
Wais Kamal
  • 5,858
  • 2
  • 17
  • 36
0
votes
1 answer

How to parse newline delimited JSON with retrofit and moshi?

I'm trying to parse newline delimited json using retrofit and moshi. This is my GET function: suspend fun getDeviceValuesNew(@Path("application-id") applicationId: String, @Path("device-id") deviceId: String) :…
Leo
  • 11
  • 5
0
votes
1 answer

How do you import a ndjson file in jupyter notebook

I have tried the code below but it's not working import json with open("/Users/elton/20210228test2.ndjson") as f: test2data = ndjson.load(f)
Elton Mungani
  • 13
  • 2
0
votes
0 answers

.Net Core POST with x-ndjson header error

I have problem with sending POST request from .NET Core app when I added header Content-Type: application/x-ndjson. Without this header I have 400 error. When I send the same request from Postman, with that header, all good and I have response. When…
Roma Pavliuk
  • 144
  • 8
0
votes
1 answer

Read newline delimited json from Kafka message in Spark Structured Streaming

I developed a Python Kafka producer that sends multiple json records as a nd-json binary string to a Kafka topic. Then I'm trying to read these messages in Spark Structured Streaming with PySpark as follow: events_df =…
user3376554
  • 19
  • 2
  • 5
0
votes
1 answer

Json to new-line delimited json

I'm trying to convert Json file to ndjson. I'm reading the file from GCS(google cloud Storage). sample data: { "Item1" : "INT", "Item2" : "INT", "Item3" : "text", "Item4" : "text", "Item5" : "Date" }{ "Item1" : "INT", "Item2" : "INT", …
Dr.teja
  • 13
  • 3
0
votes
2 answers

Export as JSON using BigQueryToCloudStorageOperator

When I use the BigQuery console manually, I can see that the 3 options when exporting a table to GCS are CSV, JSON (Newline delimited), and Avro. With Airflow, when using the BigQueryToCloudStorageOperator operator, what is the correct value to…
Canovice
  • 9,012
  • 22
  • 93
  • 211
0
votes
2 answers

Convert ndjson to json for HTML table

I would like to know if it is possible to convert the ndjson data from this API: https://lichess.org/api/team/overberg-chess-group/users and turn it into an HTML table. I have found some javascript snippets that will convert normal json into an html…
hpEthan1793
  • 1
0
votes
1 answer

Override method/variable in a gem rails

I am trying to find a way to override a variable in a rails gem actionpack/http/parameters.rb. I have to process an ndjson stream and the rails middleware cannot process ndjson. It uses ActiveSupport::JSON.decode This is the source of the code…
Ndeto
  • 315
  • 6
  • 17
0
votes
2 answers

Lambda - Python - CSV to NDJSON - Fails to dump huge files

I am working on a lambda that converts a CSV file landed in Bucket-A(source) to NDJSON and move it to Bucket-B(destination) The below logic works fine as expected for small files, but my CSV files are expected to be over 200 MB, and some about…
Prashanth kumar
  • 949
  • 3
  • 10
  • 32
1 2 3 4 5
6
7 8