Stack Overflow
Stack Overflow

Questions tagged [ndjson]

NDJSON is a newline delimited JSON format.

NDJSON is a newline delimited JSON format. It is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It's a great format for log files. It's also a flexible format for passing messages between cooperating processes.

Spec: http://ndjson.org

119 questions
0
votes
1 answer

Progress bar for uploading file in Python

I have a Python program in which I want to upload a large .ndjson. Is there a way to add a progress bar so that I know how much of the file is uploaded? My code: import json import pandas as pd df = map(json.loads, open('dump.ndjson')) df =…
taga
  • 3,537
  • 13
  • 53
  • 119
0
votes
1 answer

Extract all (or replace) from ndjson not effective

I am reading a file with one JSON object per line (ndjson) dfjson = pd.read_json(path_or_buf=JsonFicMain,orient='records',lines=True) Here is an example of 2 lines of the content of the dataframe (after dropping columns) nomCommune …
Myrt
  • 34
  • 4
0
votes
1 answer

What's the right way to handle chunked json response with swagger and autogenerated php client?

I'm a bit lost between definitions, generated code and many things that are a bit of a black box, even after debugging. But let's start at the beginning. I have an API written with node using NestJS as framework. NestJS automatically creates the…
nedt
  • 332
  • 3
  • 15
0
votes
1 answer

ND-JSON Split in SFTP

I have a large ND-JSON file in SFTP (~20K lines). Is there a way to generate sub files out of this (~500 lines each) and place in another folder in SFTP? Does Mule 4 has the capability to split a large file and write in SFTP? Or is there a need for…
Triumph Spitfire
  • 663
  • 15
  • 38
0
votes
2 answers

how to assign a javascript variable with ndjson value

I want to assign the data inside a ndjson file into a js variable. I have tried putting it in an array and an object but that is throwing me error. I have tried like this ... var data = [{"attributes":{}} {"attributes":{}} {"attributes":{}}] and…
MaKr
  • 13
  • 5
0
votes
1 answer

When using fetch how to convert response body from application/x-ndjson to application/json?

I'm trying out Lichess API. I'm trying to export games of a user. According to documentation I can receive either PGN or ndjson as response. Right now this don't work: const api_url =…
user11075932
0
votes
0 answers

ndjson request with AWS lambda

I am using AWS lambda as a proxy for my ElasticSearch service. I am facing issues with proxying requests to Multi search API. As this end point accepts payload in the form of ndjson, I am trying to send the following payload as request body and…
user1868744
  • 963
  • 1
  • 13
  • 27
0
votes
1 answer

Save json api results to json file in newline delimited format using php

I am getting a nested json object back from an api call and I am trying to save it to a newline delimited json file so it can be imported into Google Big Query. Here is what I have and it saves it to my file but still not correctly formatted for Big…
user1535268
  • 121
  • 1
  • 2
  • 10
0
votes
1 answer

AWS Lambda Nodejs: Get all objects created in the last 24hours from a S3 bucket

I have a requirement whereby I need to convert all my JSON files in my bucket into one new line delimited JSON for a 3rd party to consume. However, I need to make sure that each newly created new delimited JSON only includes files that were received…
panza
  • 1,341
  • 7
  • 38
  • 68
0
votes
1 answer

How to create an ndjson object from a dictionary in python?

I need help creating a NDJSON object from the following parsed data from on of the leading Advertising Platform. I intend to upload the data to bigquery. I succeeded in creating an NDJSON using pandas but I can't control for datatypes and it…
jarvis
  • 157
  • 1
  • 13
0
votes
1 answer

JSON newline delimited

I have a requirement whereby a third-party system will require to consume JSON files as newline delimited or ndjson from an AWS S3 bucket. I am struggling to find information on how S3 deals with such format. Can I save files as ndsjon? Thank you
panza
  • 1,341
  • 7
  • 38
  • 68
0
votes
0 answers

Indexing a document into elasticsearch with ndjson format

The question is also posted here, https://discuss.elastic.co/t/indexing-jupyter-notebooks/194639?u=venkateshreddypala
Venkateshreddy Pala
  • 873
  • 1
  • 6
  • 11
0
votes
0 answers

How to create a json file with ndjson schema?

How do I create an ndjson file in Visual Studio using the correct schema. At the moment I am getting the following error because my json is not being recognised. I know it is correct because I'm able to issue a Web request via Fiddler with it.
litterbugkid
  • 3,534
  • 7
  • 36
  • 54
0
votes
1 answer

Error with a valid geojson when try to split the file with ndjson-split

I want to remove all the properties from my geojson file so for that I first try to make a ndjson file like that: /root/npm/bin/ndjson-split 'd.features' < test.json > test.ndjson but I get this…
Henri Labarre
  • 53
  • 7
0
votes
1 answer

import ndjson into R skip first n lines

How to read a big ndjson (20GB) file by chunk into R? I have a big data file that I want to read 1M rows at a time. currently, I'm using below code to load data into R. jsonlite::stream_in( file(fileName) ) But I don't need to load all data…
Mahdi Jadaliha
  • 1,947
  • 1
  • 14
  • 22
1 2 3 4 5 6
7
8