1

I would like to get data from IceBerg table using AWS Lambda. I was able to create all the code and containers only to discover that AWS Lambda doesn't allow process substitution that spark uses here: https://github.com/apache/spark/blob/121f9338cefbb1c800fabfea5152899a58176b00/bin/spark-class#L92

The error is:

/usr/local/lib/python3.10/dist-packages/pyspark/bin/spark-class: line 92: /dev/fd/63: No such file or directory

Do you maybe have some idea how this can be solved?

Pawel
  • 93
  • 2
  • 7
  • Are you trying to `trigger` from AWS Lambda or run your spark code `in` AWS Lambda? I am assuming later is not the case but if it is then you are going in wrong direction my friend. :) If not, then you should add more details in the question like where did you create containers and hope are you trying to trigger the spark job via Lambda or something of that short. – Frosty Dec 28 '22 at 18:35
  • Did you find a solution? I am facing the same problem – G.M May 11 '23 at 12:30

0 Answers0