I'm trying to use as less memory as possible when saving json files into HDFS.
Is it possible to save not as json file but as compressed json?
I'm using writeStream() (Java).
I can save as gzip file by using option("compression","gzip"),
but what I really want is compressed json
Thank you!
Asked
Active
Viewed 189 times
1
Kimjungwow
- 65
- 1
- 7
-
It is not possible out of the box. You'd have to write your own custom data source format to do so. You could also start with `foreach*` operators and do the compression and write yourself. – Jacek Laskowski Jul 25 '19 at 10:54
-
I looked for a contribution on GitHub, but found nothing for you. – Jeff Stewart Jul 26 '19 at 02:12