0

I have a list of dictionaries:

print(type(train_dataset))
>>> <class 'list'>

print(len(train_dataset))
>>> 4000

train_dataset[0]
>>>
{'id': '7',
 'question': {'stem': 'Who is A',
  'choices': [{'text': 'A is X', 'label': 'A'},
   {'text': 'A is not B', 'label': 'D'}]},
 'answerKey': 'D'}

How can I convert this to a huggingface Dataset object? From their website it seems like you can only convert pandas df (dataset = Dataset.from_pandas(df)) or a dictionary ( dataset = Dataset.from_dict(my_dict)), but it's not clear how to use a list of dictionaries

Progman
  • 16,827
  • 6
  • 33
  • 48
Penguin
  • 1,923
  • 3
  • 21
  • 51
  • I faced the same problem. The only thing I can do is to convert the dataset to a dictionary or a dataframe. – Dammio May 18 '23 at 01:52

1 Answers1

1

From here: https://discuss.huggingface.co/t/convert-a-list-of-dictionaries-to-hugging-face-dataset-object/14670

datasets.Dataset.from_pandas(pd.DataFrame(data=your_data))
Dammio
  • 911
  • 1
  • 7
  • 15