Spark history logs decompress manually

Question

I have some problems with spark history server and therefore tried to uncompress lz4 logs manually. I installed liblz4-tool and downloaded application log application_1510049252849_0303.lz4. But when uncompressing it with lz4 ~/application_1510049252849_0303.lz4 I got the following error:

Error 44 : Unrecognized header : file cannot be decoded

And this I have for every spark history lz4 files. What is the solution? Is there a way to open it manually?

score 6 · Answer 1 · answered Feb 21 '18 at 09:35

6

I had the same error yesterday while trying to decompress the lz4 event logs. Apparently there seems to be some incompatibilities between hadoop lz4 codec and other lz4 codecs: see https://issues.apache.org/jira/browse/HADOOP-12990.

For what it's worth, spark history server can read compressed event logs in lz4, so you don't need to decompress them manually. Hope this helps.

answered Feb 21 '18 at 09:35

Florent Moiny

441
4
7

1

Actually yes, I ended up in writing a simple app using Spark api to decompress it. – St.Antario Feb 21 '18 at 09:36

score 0 · Answer 2 · answered Jul 15 '20 at 05:31

0

lz4 no need to decompress while placing into spark history server. Just you can place your event logs into event log directory. It will automatically pick that application and show it in the Spark History Server UI.

answered Jul 15 '20 at 05:31

Ranga Reddy

2,936
4
29
41

Spark history logs decompress manually

2 Answers2