Json-Opening Yelp Data Challenge's data set

Question

I am interested in data mining and I am writing my thesis about it. For my thesis I want to use yelp's data challenge's data set, however i can not open it since it is in json format and almost 2 gb. In its website its been said that the dataset can be opened in phyton using mrjob, but I am also not very good with programming. I searched online and looked some of the codes yelp provided in github however I couldn't seem to find an article or something which explains how to open the dataset, clearly. Can you please tell me step by step how to open this file and maybe how to convert it to csv?

https://www.yelp.com.tr/dataset_challenge

https://github.com/Yelp/dataset-examples

Answer 1

data is in .tar format when u extract it again it has another file,rename it to .tar and then extract it.you will get all the json files

Answer 2

yes you can use pandas. Take a look:

import pandas as pd

# read the entire file into a python array
with open('yelp_academic_dataset_review.json', 'rb') as f:
    data = f.readlines()

# remove the trailing "\n" from each line
data = map(lambda x: x.rstrip(), data)


data_json_str = "[" + ','.join(data) + "]"

# now, load it into pandas
data_df = pd.read_json(data_json_str)

Now 'data_df' contains the yelp data ;) Case, you want convert it directly to csv, you can use this script

https://github.com/Yelp/dataset-examples/blob/master/json_to_csv_converter.py

I hope it can help you

Answer 3

To process huge json files, use a streaming parser.

Many of these files aren't a single json, but a stream of jsons (known as "jsons format"). Then a regular json parser will consider everything but the first entry to be junk.

With a streaming parser, you can start reading the file, process parts, and wrote them to the desired output; then continue writing.

There is no single json-to-csv conversion.

Thus, you will not find a general conversion utility, you have to customize the conversion for your needs.

The reason is that a JSON is a tree but a CSV is not. There exists no ultimative and efficient conversion from trees to table rows. I'd stick with JSON unless you are always extracting only the same x attributes from the tree.

Start coding, to become a better programmer. To succeed with such amounts of data, you need to become a better programmer.